Foveation optimization for TV streaming and rendering content assisted by personal devices

Abstract

A mobile device (210), such as a mobile phone, is used to image (406) a nearby user to perform eye tracking at a closer range than would be provided by a camera on a display device (12), such as a TV, and to send (412) the eye tracking data to a server (52) that streams video presented on the display. The server uses (502) the eye tracking data to identify regions of interest (ROI) in the video that are compressed more than regions outside the ROI (504) to facilitate efficient compression of video game images while maintaining the quality of the ROI in the image being viewed by the user.

Description

Technical Field

[0001] The present application involves technically innovative, unconventional solutions that must be rooted in computer technology and produce specific technical improvements. Background Art

[0002] As recognized herein, services for streaming computer simulations (such as computer games) are needed to compress, transmit, and decompress high-resolution images (1080p, 4k, 8K, etc.) from video games originating from remote servers to home console devices within the strict bandwidth limitations typically imposed by the user's Internet Service Provider (ISP) and local network (such as Wi-Fi). Although new video compression codecs are being developed to reduce bandwidth requirements even for high-resolution video, their efficiency is limited.

[0003] There is currently no adequate solution to the aforementioned computer-related technical problems. Summary of the invention

[0004] As understood herein, knowing exactly where a user is looking when playing a streaming game on a TV will help improve the efficiency of the video compression codec by adjusting the compression within the region of interest (ROI) to be relatively low (or uncompressed) and adjusting the compression of the area outside the ROI to be relatively high, thereby highly compressing the image to reduce its data size. In order to achieve this, it is necessary to track the user's gaze. However, as also understood herein, a camera system based on or near a TV is too far away from the user to accurately track the user's gaze. In addition, there is a large variability in the distance between the TV and the user, making it more difficult to track gaze from such a distance.

[0005] The following solution solves this problem for users who stream games or other video content on a display such as a TV.

[0006] Thus, a system includes at least one display such as, but not limited to, a TV display, at least one device having a first camera and a second camera and positionable between the display and a user viewing the display, and at least one server configured to send compressed video for presentation on the display. The device is programmed with instructions executable to generate a first image including an image of the TV display using the first camera. The instructions are also executable to generate a second image including an image of the user using the second camera. The server and / or the device are programmed with instructions executable to use images from the camera to identify a point of gaze (POG) of the user on the TV display, and to identify a region of interest (ROI) in the video based at least in part on the POG. The server is programmed with instructions to compress the video and send the video for presentation on the TV display such that the ROI is presented on the display at a higher resolution and / or higher quality than the video area outside the ROI.

[0007] In an example, the device comprises a mobile communication device, or a tablet computer, or a video game controller, or conventional prescription glasses.

[0008] If desired, the server may be programmed with instructions to identify corresponding speculative ROIs in "N" video frames at least in part using the ROIs identified with the POG, where "N" is an integer greater than zero, and compress the "N" video frames based at least in part on the corresponding speculative ROIs.

[0009] In another aspect, a system includes at least one processor and at least one computer memory, the computer memory not being a transient signal and in turn including instructions executable by the processor to image a user using a first camera on a device. The device is not on a display such as, but not limited to, a TV display. The instructions are executable to image the TV display using a second camera on the device and determine a point of gaze (POG) on the TV display based at least in part on the image. The instructions are also executable to send the POG to at least one remote server to facilitate the server to use a region of interest (ROI) associated with the POG in at least one video frame. The video frame also includes at least one region outside the ROI, such that the server can compress the ROI using a first compression and compress the region outside the ROI using a second compression.

[0010] The history of where the POG exists on a certain game screen can be stored in a database. The information in the database can be fed into a machine learning algorithm to learn correlations with predicted ROI.

[0011] In another aspect, a method includes using a mobile device to generate an image of a user viewing a display such as, but not limited to, a TV display. The method also includes using the mobile device to generate an image of the TV display, and using the image to identify a location on the TV display that the user is viewing. The method includes sending the location to at least one server via a wide area network.

[0012] The details of the present application, both as to its structure and operation, may best be understood with reference to the accompanying drawings, in which like reference numerals refer to like parts, and in which: BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Figure 1 is a block diagram of an example system consistent with the principles of the invention;

[0014] Figure 2 is a schematic diagram illustrating an example system consistent with the principles of the present invention;

[0015] Figure 3 is a flow chart of an exemplary overall logic consistent with the principles of the present invention;

[0016] Figure 4 is a flow chart of example mobile device logic consistent with the principles of the present invention;

[0017] Figure 5 is a flow chart of example server logic consistent with the principles of the present invention;

[0018] Figure 6 is a schematic diagram of an alternative controller-based system consistent with the principles of the present invention;

[0019] Figure 7 is a schematic diagram of an alternative glasses-based system consistent with the principles of the present invention;

[0020] Figure 8 is a schematic diagram illustrating the principle of the present invention;

[0021] Fig. 9 is a flow chart of example logic illustrating further principles; and

[0022] Fig.10 is a schematic diagram of an alternative single-phase implementation. DETAILED DESCRIPTION

[0023] The present disclosure generally relates to a computer ecosystem, which includes various aspects of a computer network that may include consumer electronics (CE) devices. The system herein may include a server and a client component connected via a network so that data can be exchanged between the client and the server component. The client component may include one or more computing devices, including a portable television (e.g., a smart TV, a TV with Internet functionality), a portable computer such as a laptop computer and a tablet computer, and other mobile devices including smart phones and additional examples discussed below. These client devices may operate under a variety of operating environments. For example, some of the client computers may use an operating system such as Microsoft's, or a Unix operating system, or an operating system produced by Apple Computer or Google. These operating environments may be used to execute one or more browsing programs, such as browsers made by Microsoft or Google or Mozilla, or other browser programs that can access websites hosted by Internet servers discussed below.

[0024] The server and / or gateway may include one or more processors that execute instructions that configure the server to receive and transmit data over a network such as the Internet. Alternatively, the client and server may be connected via a local intranet or a virtual private network. The server or controller may be powered by a game console (such as a Sony ), personal computers, etc.

[0025] Information can be exchanged between the client and the server through the network. To this end and for security reasons, the server and / or the client may include firewalls, load balancers, temporary storage devices and agents, and other network infrastructure for reliability and security.

[0026] As used herein, instructions refer to computer-implemented steps for processing information in a system. Instructions may be implemented in software, firmware, or hardware, and include any type of programming steps performed by system components.

[0027] The processor may be any conventional general purpose single-chip or multi-chip processor that can execute logic with the aid of various lines such as address lines, data lines, and control lines, as well as registers and shift registers.

[0028] The software modules described by the flowcharts and user interfaces herein may include various subroutines, programs, etc. Without limiting the present disclosure, the logic stated as being performed by a particular module may be redistributed to other software modules and / or combined together in a single module and / or made available in a shareable library. Although a flowchart format may be used, it should be understood that the software may be implemented as a state machine or other logic method.

[0029] The inventive principles described herein may be implemented as hardware, software, firmware, or a combination thereof; therefore, the illustrative components, blocks, modules, circuits, and steps are described in terms of their functionality.

[0030] In addition to what has been mentioned above, the logic blocks, modules and circuits described below can be implemented or executed with a general purpose processor, digital signal processor (DSP), field programmable gate array (FPGA) or other programmable logic device (such as application specific integrated circuit (ASIC), discrete gate or transistor logic, discrete hardware components or any combination thereof) designed to perform the functions described herein. The processor can be implemented by a controller or a state machine or a combination of computing devices.

[0031] When the functions and methods described below are implemented in software, the functions and methods may be written in an appropriate language such as, but not limited to, C# or C++, and may be stored on or transmitted through a computer-readable storage medium, such as a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a compact disk read-only memory (CD-ROM) or other optical disk storage devices (such as digital versatile disks (DVDs)), magnetic disk storage devices, or other magnetic storage devices including removable thumb drives, etc. A connection may establish a computer-readable medium. Such a connection may include, for example, hard-wired cables, including optical fiber and coaxial cables, as well as digital subscriber lines (DSL) and twisted pairs.

[0032] The components included in one embodiment may be used in other embodiments in any appropriate combination. For example, any of the various components described herein and / or depicted in the drawings may be combined, interchanged, or excluded from other embodiments.

[0033] “A system having at least one of A, B, and C” (similarly, “a system having at least one of A, B, or C” and “a system having at least one of A, B, C”) includes the following systems: having only A; having only B; having only C; having both A and B; having both A and C; having both B and C; and / or having A, B, and C, etc.

[0034] Now specifically refer to Figure 1 , shows an example system 10, which may include one or more of the example devices mentioned above and further described below according to the principles of the present invention. It should be noted that the computerized devices described in all figures herein may include Figure 1 Some or all of the components described in the various devices.

[0035] The first of the example devices included in the system 10 is a consumer electronics (CE) device configured as an example primary display device, and in the illustrated embodiment is an audio video display device (AVDD) 12 of an Internet-enabled TV such as, but not limited to, a TV tuner (equivalently, a set-top box that controls the TV). The AVDD 12 may be based on system. Alternatively, the AVDD 12 may also be a computerized Internet-enabled (“smart”) phone, a tablet computer, a notebook computer, a wearable computerized device (such as a computerized Internet-enabled watch, a computerized Internet-enabled bracelet), other computerized Internet-enabled devices, a computerized Internet-enabled music player, a computerized Internet-enabled headset, a computerized Internet-enabled implantable device (such as an implantable skin device), etc. In any case, it should be understood that the AVDD 12 and / or other computers described herein are configured to employ the principles of the present invention (e.g., communicate with other CE devices to employ the principles of the present invention, execute the logic described herein, and perform any other functions and / or operations described herein).

[0036] Therefore, to adopt this principle, AVDD 12 can be Figure 1Some or all of the components shown are established. For example, the AVDD 12 may include one or more displays 14, which may be implemented by a high-definition or ultra-high-definition "4K" or higher flat screen, and may or may not support touch for receiving user input signals via touch on the display. The AVDD 12 may also include: one or more speakers 16, which are used to output audio according to the principles of the present invention; and at least one additional input device 18 (such as an audio receiver / microphone), which is used to input audible commands to the AVDD 12 to control the AVDD 12, for example. The example AVDD 12 may also include one or more network interfaces 20, which are used to communicate over at least one network 22 (such as the Internet, WAN, LAN, PAN, etc.) under the control of one or more processors 24. Thus, the interface 20 can be, but is not limited to, a Wi-Fi transceiver, which is an example of a wireless computer network interface, such as, but not limited to, a mesh network transceiver. The interface 20 may be, but is not limited to, a Bluetooth transceiver, a Zigbee transceiver, an IrDA transceiver, a wireless USB transceiver, a wired USB, a wired LAN, Powerline, or MoCA. It should be understood that the processor 24 controls the AVDD 12 to employ the principles of the present invention, including other elements of the AVDD 12 described herein, such as controlling the display 14 to present images on the display and receiving input from the display. In addition, it should be noted that the network interface 20 may be, for example, a wired or wireless modem or router or other suitable interface, such as a wireless telephone transceiver, or a Wi-Fi transceiver as mentioned above, etc.

[0037] In addition to the foregoing, the AVDD 12 may also include one or more input ports 26, such as a high-definition multimedia interface (HDMI) port or a USB port for physically connecting (e.g., using a wired connection) to another CE device and / or a headphone port for connecting headphones to the AVDD 12 for presenting audio from the AVDD 12 to the user through the headphones. For example, the input port 26 may be connected to a wired or satellite source 26a of audio-video content via a wire or wirelessly. Thus, the source 26a may be, for example, a separate or integrated set-top box or satellite receiver. Alternatively, the source 26a may be a game console or a disk player.

[0038] The AVDD 12 may also include one or more computer memories 28 that are not transient signals, such as disk-based storage devices or solid-state storage devices, which in some cases are embodied as independent devices in the housing of the AVDD, or as personal video recorders (PVRs) or video disk players for playing back AV programs inside or outside the housing of the AVDD, or as removable storage media. In addition, in some embodiments, the AVDD 12 may include a position or location receiver (such as, but not limited to, a cell phone receiver, a GPS receiver, and / or an altimeter 30) configured to receive geographic location information, such as from at least one satellite or cell phone tower, and provide the information to the processor 24 and / or determine the altitude at which the AVDD 12 is set in conjunction with the processor 24. However, it should be understood that another suitable position receiver other than a cell phone receiver, a GPS receiver, and / or an altimeter may be used in accordance with the principles of the present invention to, for example, determine the position of the AVDD 12 in, for example, all three dimensions.

[0039] Continuing with the description of the AVDD 12, in some embodiments, in accordance with the principles of the present invention, the AVDD 12 may include one or more cameras 32, which may be, for example, thermal imaging cameras, digital cameras (such as webcams), and / or cameras integrated into the AVDD 12 and controllable by the processor 24 to collect pictures / images and / or videos. A Bluetooth transceiver 34 and other near field communication (NFC) elements 36 may also be included on the AVDD 12 for communicating with other devices using Bluetooth and / or NFC technology, respectively. An example NFC element may be a radio frequency identification (RFID) element.

[0040] In addition, the AVDD 12 may include one or more auxiliary sensors 38 that provide input to the processor 24 (e.g., a motion sensor such as an accelerometer, a gyroscope, an odometer, or a magnetic sensor, an infrared (IR) sensor for receiving IR commands from a remote control, an optical sensor, a speed and / or cadence sensor, a gesture sensor (e.g., for sensing gesture commands), etc.). The AVDD 12 may include a wireless TV broadcast port 40 that provides input to the processor 24 for receiving OTATV broadcasts. In addition to the foregoing, it should be noted that the AVDD 12 may also include an infrared (IR) transmitter and / or an IR receiver and / or an IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided for powering the AVDD 12.

[0041] Additionally, in some embodiments, AVDD 12 may include a graphics processing unit (GPU) 44 and / or a field programmable gate array (FPGA) 46. AVDD 12 may use the GPU and / or FPGA for, for example, artificial intelligence processing, such as training a neural network and performing operations (e.g., reasoning) of the neural network in accordance with the principles of the present invention. However, it should be noted that processor 24 may also be used for artificial intelligence processing, such as processor 24 may be a central processing unit (CPU).

[0042] Still refer to Figure 1 In addition to the AVDD 12, the system 10 may also include one or more other computer device types that may include some or all of the components shown for the AVDD 12. In one example, a first device 48 and a second device 50 are shown and may include components similar to some or all of the components of the AVDD 12. Fewer or more devices than shown may be used. The first device 48 may implement, for example, any of the devices discussed below, including a mobile phone that can image a user, glasses that can image a user, etc. The second device 50 may implement, for example, any of the devices discussed below, including a computer simulation controller or console, examples of which include a computer game controller and a console.

[0043] The system 10 may also include one or more servers 52. The server 52 may include at least one server processor 54, at least one computer memory 56 (such as a disk-based storage device or a solid-state storage device), and at least one network interface 58 that allows communication with the server 10 over the network 22 under the control of the server processor 54. Figure 1 The network interface 58 may be a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface, such as a wireless telephone transceiver.

[0044] Thus, in some embodiments, server 52 may be an Internet server and may include and execute "cloud" functions, such that the devices of system 10 may access the "cloud" environment via server 52 in an example embodiment. Alternatively, server 52 may be provided by a server that is associated with Figure 1 The other devices shown are implemented in the same room or nearby a game console or other computer.

[0045] The devices described below may combine some or all of the elements described above.

[0046] The methods described herein may be implemented as software instructions executed by a processor, a suitably configured application specific integrated circuit (ASIC) or field programmable gate array (FPGA) module, or any other convenient manner as will be appreciated by those skilled in the art. Where employed, the software instructions may be embodied in a non-transient device such as a CD ROM or flash drive. The software code instructions may alternatively be embodied in a transient arrangement such as a radio signal or an optical signal, or via a download over the Internet.

[0047] Figure 2 A system is shown where a user 200 views a display 204, such as a TV. More specifically, assume that the user 200, who is unable to focus on the entire display area, focuses on a portion 203 of the display area, referred to herein as a region of interest (ROI).

[0048] A user 200 may employ a computer simulation controller 202, an example of which is a video game controller, to control a computer simulation, such as a video game presented on a display 204. It should be understood that the use of the terms "video game controller" and "video game" is by way of example only and does not imply a loss of generality. The video game may be received from a remote server 206 directly by the display 204 or through a video game console 208 over a wide area network such as the Internet. In addition to or in lieu of the video game, the principles of the present invention are also applicable to video received from a server. For purposes of this disclosure, a video game will be used as an example.

[0049] The mobile device 210, such as a mobile phone or tablet computer, includes at least one camera, and as shown, preferably a front camera 212 and a rear camera 214. The mobile device 210 can be oriented so that the front camera 212 faces the display 204 to capture its image, while the rear camera faces the user 200 to image the user's eyes for eye tracking. Typically, the mobile device 210 also includes a position receiver 216. The mobile device 210 can communicate with the server 206 for purposes disclosed later. It should be noted that the mobile device 210, the server 206, the display 204, the console 208, and the controller 202 may include a plurality of cameras, a plurality of cameras, and a plurality of cameras. Figure 1 Some or all of the electronic components of the various devices shown.

[0050] Now turn to Figure 3, shows the overall logic that can be executed by one or more processors (such as a processor in the mobile device 210 cooperating with a processor in the server 206). Starting at box 300, an image of the user's eyes is received from the rear camera 214 of the mobile device 210 as eye tracking information. Moving to box 302, the position of the ROI 203 is determined based on the eye tracking information. The position is provided to the server 206, and at box 304, the video area to be presented outside the ROI is compressed to a higher degree than the video area to be presented within the ROI. In other words, the resolution and / or image quality of the video in the ROI is higher than the resolution and / or image quality of the video outside the ROI. The differentially compressed video is sent for decoding (including decompression of the ROI and the video area outside the ROI) and presented on the display 204 at box 306.

[0051] Thus, using the facial and gaze tracking features of the mobile device 210, an application running on the mobile device 210 tracks the gaze of the user at a closer range than the display 204, thereby sending the eye tracking data to the server 206. The server uses the eye tracking data to predict the user's gaze on the display 204 and thus locate the ROI of the video compression codec executed by the server, which enables more efficient compression of the video game image to maintain the quality of the image that the user is viewing 203.

[0052] The user's mobile device 210 may be placed near the user 200 , such as on a coffee table or other nearby surface, significantly closer than the display 204 .

[0053] As mentioned above, the mobile device 210 also uses the front-facing camera 212 to image the display 204. Using the image of the display 204, the mobile device 210 can identify the position and orientation of the display 204 relative to the mobile device 210 (via computer vision techniques). Figure 4 As shown at block 400 in FIG. 4 , this operation may be performed during launch of an executable application on the mobile device and, if desired, repeated when one or more motion sensors in the mobile device indicate that the mobile device is being moved.

[0054] Moving to block 402, the mobile device 210 images the display 204. Proceeding to block 404, using image recognition techniques employed by a machine vision algorithm, based on the image of the display 204 and knowing the position of the mobile device 210 based on the position receiver 216 and the position of the mobile device 210 based on one or more motion / orientation sensors in the mobile device (such as Figure 1 The position of display 204 relative to mobile device 210 is determined by measuring the orientation of display 204 using any suitable sensor shown in FIG. 2 .

[0055] Proceeding to block 406, mobile device 210 images user 200 using camera 214 to essentially image the position and orientation (PnO) of the user's face using camera image data, which may include red-green-blue data and / or depth data. Also using the image of user 200, the position of the user's eyes is determined.

[0056] Moving to box 408, the eye image from box 406 is used to determine the line of sight (LOS) of the user's gaze relative to the face. Box 410 indicates that the LOS is projected to the position of the display determined at box 404 to determine the point of gaze (POG) on the display 204 that the user 200 is looking at. The gaze direction of the user's eye can be triangulated to the surface of the display 204 using the previous PnO relative to the display. From this, the user's point of gaze (POG) can be determined within the frame of the display. The POG is sent to the server 206 at box 412, and the server can use the POG as the center of a ROI of a given size. For example, a user sits at an optimal viewing distance of about 1.93 meters from a 50-inch TV with a resolution of 1920x1080 (full HD), where a typical 5-degree high-acuity fovea will project a ROI of approximately 168 mm in diameter to an area on the TV. This will be equivalent to a region of interest of approximately 4% of the entire full HD image. By allowing a certain area to switch compression levels, more than 80% of the image can be highly compressed, saving a lot of bandwidth.

[0057] Although Figure 4 A single user is discussed, but Figure 4 The principle can be extended to multiple users, each with his or her own ROI in a different area of ​​the display.

[0058] Figure 5 Example server 206 logic is shown. Beginning at block 500, a mobile device 210 receives a request such as Figure 4 The POG of the user on the display 204 determined in FIG. 5 is moved to block 502, where the POG is used to determine the ROI of the video. In an example, it can be assumed that the POG is the center of the ROI, and a predetermined area centered on the POG, such as a group of NxM pixels (or a circular area with a constant radius if desired), can be determined as the ROI of the video to be compressed at block 504 to achieve a higher resolution and / or image quality than achieved by a different compression of the video area outside the NxM pixels. Therefore, the video is transmitted to the display 204 at block 506, so that the video area within the ROI is presented on the display 204 with a higher resolution and / or image quality than the video area displayed outside the ROI. It is easy to understand that the ROI is smaller than the entire video frame.

[0059] Understanding the principles of the present invention Figure 2There are delays in the system that should be taken into account. For example, the delay includes the tracking process on the mobile device 210, the communication of POG data from the mobile device 210 to the server 206, the processing and encoding of the video game images by the server 206, the communication of the compressed video game images from the server 206 to the display 204 via the system 208 in the home, and the decoding of the compressed video game images by the home system 208.

[0060] With this in mind, Figure 5 As shown in block 508 in FIG. 5 , the server 206 and / or the mobile device 210 and / or the console 208 predicts “N” future gaze points (and therefore “N” future ROIs, “N” being an integer greater than zero) based on the current and previous eye POGs. As the server 206 and / or the console 208 performs this process, eye tracking data is additionally sent from the mobile device 210 to the server 206 and / or the console 208. At block 510, the positions of the “N” ROIs corresponding to the prediction from block 508 may be used to speculatively compress the next “N” video frames following the current frame.

[0061] A predictive POG may be generated from the current application screen being displayed. A historical dataset of application screen images and observed POGs may be accumulated. The POG from the current display may be predicted using machine learning, an example of which is a deep neural network.

[0062] Additionally, the resulting image at the decoder can be improved by processing the image with a machine learning filter that can dynamically upsample the image to better quality. Such an adaptive filter can be created by creating a historical dataset of application screens and decoding results and training a convolutional deep neural network to restore the image.

[0063] Using the above solution, an improved game streaming service is provided that better adapts to the user's home Internet bandwidth and gaze direction to the display showing the streamed game.

[0064] Figure 6 shows the configuration and operation of Figure 2 The system shown is substantially the same system with the following exceptions. Instead of using a camera on the mobile device to image the user 200 and display 204, cameras 600, 602 on the video game controller 604 are used to track the position and orientation of the display 204 and the face / eyes of the user 200 relative to the controller 604. In this embodiment, one of the cameras 600, 602 may be tilted upward toward the user, and the user may be prompted to hold the controller in a specified manner or orientation via instructions displayed on the display 204 provided by the console 208.

[0065] Figure 7 shows the configuration and operation of Figure 2 Yet another embodiment of a substantially identical system is shown, with the following exceptions: Instead of using the camera on the mobile device to image the user 200 and display 204, the cameras 700, 702 on the glasses 704 are used to track the position and orientation of the display 204 and the face / eyes of the user 200 relative to the glasses 704.

[0066] Instead of streaming video images, local content running on console 208 can incorporate foveated rendering. In such an embodiment, the user's point of gaze (POG) is determined as described herein, and the POG is then used to identify content areas outside of the POG. For content in these areas, the rendering resolution and / or quality and / or other aspects of the content are changed to reduce rendering costs on console 208.

[0067] Reference now Figure 8 , showing additional details of a mobile phone embodiment. A display 800, such as but not limited to a large screen 8K TV display, receives image information, such as but not limited to a video or image, from a video game console 802. The video game console 802 may receive control information from a wireless video game controller 804. A mobile phone 806 may include a first camera 808 that images an ROI 810 on the display 800 and a second camera 812 that images an eye 814 of a user.

[0068] The above application on the phone 806 can prompt the user as to whether the user wants to play the image on the display at the highest resolution or quality, and if so, tilt the phone in a specific orientation, for example, with the camera 808 pointing directly at the display 800. Figure 8 As shown at 816 in FIG. 8 , when the user moves the phone to present the best possible facial image, the display 818 of the mobile phone 806 can present an image of the user's face, such as a profile or direct plan view of the face. Figure 8 As shown at 820 in FIG. 8 , the application can present an indication (such as “OK” and / or a check mark) on the phone display 818 indicating that the phone is properly oriented and the camera 812 is imaging the user's eye 814. The same protocol can be used to ensure that the opposing camera 808 imaging the display 800 is properly oriented.

[0069] It should be noted that in the above relative to Figure 6In the controller embodiments discussed, the display 800 may give feedback regarding correct controller positioning for proper imaging. Alternatively, the controller may include a speaker that gives audible feedback, such as "lower the controller" or "can't see you," to prompt the user to reorient the controller. Alternatively, the controller may include a screen to present Figure 8 The confirmation view shown in .

[0070] The console 802 can communicate wirelessly with the phone 806. The communication can be over a local area network or via WiFi or other suitable protocol. Example use cases include a display 800 that is very large and relatively far from the user, or a mobile tablet or laptop computer that implements a video display relatively close to the user.

[0071] In some embodiments, a visible marker (such as a barcode or a quick response (QR) code indicating the location information of the area) can be inserted into the image on the display 800 to facilitate the application executing on the phone 806 to identify the screen area having the image from the camera 808. Fig. 9 As shown, the phone can image the display 800 at block 900, with the console 802 also sending the image sent to the display to the phone 806 at block 902. The phone can compare the two images from blocks 900 and 902 at block 904 to determine if they match favorably, thereby indicating that the phone is properly oriented.

[0072] The present principles understand that under conditions such as streaming a movie or game over a high traffic network, the image of the user may be degraded. Thus, if the camera cannot see the user's face or send data to the system quickly enough, the system may have to revert to a conventional lower resolution or lower image quality of the rendered image. Alternatively or additionally, the frame rate may be reduced but the resolution and / or image quality remains the same, and then once eye tracking is resumed, the selective compression techniques described herein may be implemented and the frame rate and / or perceived resolution and / or image quality increased.

[0073] Fig.10Shown is an embodiment with only a single, preferably high resolution, wide field of view camera 1000 on a mobile phone 1002 or other device, the single camera may be oriented toward the display 1004 and a mirror 1006, such as a convex mirror, may be placed in front of the user 1008 to present a view of the wide field of view camera 1000 to also image the user via reflection from the mirror 1006. In this embodiment, it is understood that the PnO relative to the display 1004 of the mobile device 1002 is determined via the previously described methods, but the PnO of the user 1008's face / eyes relative to the mobile device 1002 may be determined by approximate size and orientation measurements of the user's 1008 face / eyes reflected in the mirror 1006. Thus, the processing logic for processing images from the camera 1000 attached to the mobile device 1002 via computer vision techniques will assume that the user's 1008 face / eyes are mirrored and adjust the computer vision processing accordingly.

[0074] It will be appreciated that while the principles of the invention have been described with reference to certain exemplary embodiments, these embodiments are not intended to be limiting and that various alternative arrangements may be used to implement the subject matter claimed herein.

Claims

1. A system for gaze point optimization, the system comprising: at least one display; at least one device including a first camera and a second camera and positionable between the display and a user viewing the display; at least one server configured to send compressed video for presentation on the display; The device is programmed with instructions executable to: generating a first image using the first camera, the first image comprising an image of the display, wherein a position of the display relative to the device is determined based on the first image; generating a second image using the second camera, the second image comprising an image of the user, wherein a line of sight (LOS) of the user's gaze relative to the user's face is determined based on the second image; The server and / or the device are programmed with instructions executable to: identifying a gaze point POG of the user at the display by projecting the LOS to the location of the display based at least in part on the first image and the second image from the first camera and the second camera; identifying a region of interest (ROI) in the video based at least in part on the POG; The server is programmed with instructions executable to: The video is compressed and sent for presentation on the display, the compression causing the ROI to be presented on the display at a higher resolution and / or image quality than areas of the video outside of the ROI.

2. The system of claim 1, wherein the device comprises a mobile communication device.

3. The system of claim 1, wherein the device comprises a tablet computer.

4. The system of claim 1, wherein the device comprises a computer simulated controller.

5. The system of claim 1, wherein the device comprises glasses.

6. The system of claim 1, wherein the server is programmed with instructions to: The video frames in the ROI are compressed less than areas of the video frames outside the ROI.

7. The system of claim 1, wherein the server is programmed with instructions to: identifying, in "N" video frames, corresponding speculative ROIs using, at least in part, the ROIs identified using the POG, where "N" is an integer greater than zero; The "N" video frames are compressed based at least in part on the corresponding speculative ROIs.

8. A system for fixation point optimization, the system comprising: at least one processor; at least one computer memory that is not a transient signal and includes instructions executable by the at least one processor to: imaging a user using a first camera on a device that is not on a display to generate a first image, wherein a line of sight (LOS) of a gaze of the user relative to a face of the user is determined based on the first image; imaging the display using a second camera on the device to generate a second image, wherein a position of the display relative to the device is determined based on the second image; determining a point of gaze POG on the display by projecting the LOS to a location of the display based at least in part on the first image and the second image; as well as The POG is sent to at least one remote server to facilitate the server to use a region of interest ROI associated with the POG in at least one video frame, the at least one video frame also including at least one area outside the ROI, so that the server can compress the ROI using a first compression and compress the area outside the ROI using a second compression.

9. The system of claim 8, wherein the device comprises a mobile communication device.

10. The system of claim 8, wherein the device comprises a tablet computer.

11. The system of claim 8, wherein the device comprises a computer simulated controller.

12. The system of claim 8, wherein the device comprises eyeglasses.

13. The system of claim 8, comprising the server, the server being programmed with instructions to: The ROI is compressed using the first compression, and the region outside the ROI is compressed using the second compression.

14. The system of claim 13, wherein the server is programmed with instructions to: identifying, in "N" video frames, corresponding speculative ROIs using, at least in part, the ROIs identified using the POG, where "N" is an integer greater than zero; The "N" video frames are compressed based at least in part on the corresponding speculative ROIs.

15. A method for fixation point optimization, the method comprising: generating, using a mobile device, a first image of a user viewing a display, wherein a line of sight (LOS) of the user's gaze relative to a face of the user is determined based on the first image; generating a second image of the display using the mobile device, wherein a position of the display relative to the mobile device is determined based on the second image; identifying a location on the display at which the user is viewing by projecting the LOS onto the location of the display using the first image and the second image; as well as The position is sent via a network to at least one server or at least one computer simulation console.

16. The method of claim 15, wherein the server or console is programmed with instructions to: Compressing a video and sending the video for presentation on the display, the compression causing the video to be presented in the location on the display that the user is viewing to be presented on the display at a higher resolution and / or image quality than video outside the location on the display that the user is viewing. The method of claim 15 , wherein the mobile device comprises a mobile communication device. The method of claim 15 , wherein the mobile device comprises a tablet computer.

19. The method of claim 15, wherein the mobile device comprises a video game controller.

20. The method of claim 15, wherein the server or console is programmed with instructions to: identifying, in "N" video frames, corresponding speculative regions of interest (ROIs) using, at least in part, the locations on the display that the user is viewing, where "N" is an integer greater than zero; The "N" video frames are compressed based at least in part on the corresponding speculative ROIs.

21. The method of claim 15, wherein the computer simulation console is programmed with instructions to: determining a point of gaze POG towards the display; and Using the POG reduces the rendering resolution / quality of video images presented outside of the POG.

Images