Motion blur correction by eye tracking

Abstract

The user's eyes (402) and, optionally, head (400) are tracked as the user's gaze follows a moving object (202) on the display (200). Motion blur of the moving object is coupled (408) to the eye / head tracking. Motion blur of other objects in the frame may also be coupled to the eye / head tracking.

Description

Technical Field

[0001] This application generally relates to motion blur correction by eye tracking.

Background Art

[0002] Motion blur is typically calculated by the movement of an object across a scene. Specifically, in the case of an object moving rapidly from left to right, the game renderer can choose to "striped" the movement across the screen.

[0003] To achieve motion blur, a full-screen radial blur can be created considering only the movement of the camera, or a software shader can be programmed to create a velocity buffer and mark the strength of the movement to apply a motion blur effect to the object, using a more selective per-object motion blur. Due to the motion blur effect, the object appears "striped".

Summary of the Invention

[0004] As understood herein, current motion blur technology does not consider the movement of the player / viewer's eyes (and sometimes the head). That is, a moving object may be rendered with motion blur even though the viewer is gazing at and following the object, in which case, in the real world, the object would not appear blurred by the movement.

[0005] Accordingly, the system includes at least one computer medium, which is not a transient signal, and at least one processor-executable instruction for performing gaze tracking of a user looking at a display and implementing at least one motion blur function for at least one object presented on the display according to the gaze tracking.

[0006] In some examples, gaze tracking includes one or more of head tracking, eye tracking, and user body motion tracking. In other words, it is possible to implement overall motion accounting that takes into account the movement of the viewer's eyes, head, and virtual body (such as sitting in a car or on a train).

[0007] In some embodiments, the instructions may be executable to identify that an object is moving across the display, identify that gaze tracking is following the object, identify that the object is moving across the display, and in response to identifying that gaze tracking is following the object, implement a motion blur function as if there is no motion blur of the object.

[0008] In an exemplary embodiment, the instructions may be executable to identify that an object is moving across the display, identify that gaze tracking is not following the object, identify that the object is moving across the display, and in response to identifying that gaze tracking is not following the object, implement a motion blur function as if there is motion blur of the object. In such an example, the motion blur of the object may be proportional to the difference between the motion vector of the object and the motion vector of gaze tracking.

[0009] In some implementations, the instructions may be executable to identify that an object is moving across the display, identify that gaze tracking is moving relative to the display, identify that the object is not moving across the display, and in response to identifying that gaze tracking is moving relative to the display, implement a motion blur function as if there is motion blur of the object.

[0010] A hardware implementation may be provided in which a motion blur function is at least partially implemented by establishing a first luminance and a first exposure in a first region of a display that includes an object and establishing a second luminance and / or a second exposure in a second region of the display that does not include the object.

[0011] In another aspect, the method includes implementing a first motion blur of an object presented on a display along a user's line of sight that follows the object as the object moves, and implementing a second motion blur of the first object along a user's line of sight that does not follow the object as the object moves.

[0012] In another aspect, the apparatus includes at least one display and at least one processor configured to identify a first motion vector associated with a viewer of the display, identify a second motion vector associated with an object presented on the display, and implement motion blur of the object based on the first and second motion vectors.

[0013] The details of the present application can be best understood with reference to the accompanying drawings, both as to its structure and operation, in which like reference numerals indicate like parts.

Brief Description of the Drawings

[0014]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

DETAILED DESCRIPTION OF THE INVENTION

[0015] The present disclosure generally relates to a computer ecosystem including characteristics of a consumer electronics (CE) device network not limited to a computer game network. The systems herein may include a server component and a client component connectable via a network so that data can be exchanged between the client component and the server component. The client component may include one or more computing devices including game consoles such as Sony PlayStation (registered trademark), game consoles manufactured by manufacturers such as Microsoft and Nintendo, virtual reality (VR) headsets, augmented reality (AR) headsets, portable televisions (such as smart TVs, Internet-enabled TVs, etc.), portable computers such as laptops and tablet computers, mobile devices such as smartphones, and additional examples described below. These client devices may operate in various operating environments. For example, some client computers employ a Linux (registered trademark) operating system, an operating system of Microsoft, a Unix (registered trademark) operating system, an operating system of Apple or Google. These operating environments can be used to execute one or more browsing programs such as browsers made by Microsoft, Google, Mozilla, and other browser programs that can access websites hosted by Internet servers described below. Also, the operating environment according to this principle can be used to execute one or more computer game programs.

[0016] The server and / or gateway may include one or more processors that execute instructions configuring the server to receive and transmit data via a network such as the Internet. Alternatively, the client and the server can be connected by a local intranet or a virtual private network. The server or controller can be instantiated by a game console such as Sony PlayStation (registered trademark), a personal computer, or the like.

[0017] Information may be exchanged between the client and the server via a network. For this purpose and for security, the server, and / or the client may include a firewall, a load balancer, temporary storage, a proxy, and other network infrastructure for reliability and security. One or more servers can form an apparatus that implements a method for providing a secure community, such as an online social website, to network members.

[0018] The processor may be a single-chip or multi-chip processor capable of executing logic by various lines such as address lines, data lines, control lines, registers, shift registers, etc.

[0019] The components included in one embodiment can be used in other embodiments in suitable combinations. For example, any of the various components described herein and / or shown in the drawings can be integrated, replaced, or excluded from other embodiments.

[0020] A "system having at least one of A, B, and C" (similarly "a system having at least one of A, B, and C", "a system having at least one of A, B, and C") includes a system having only A, only B, only C, A and B, A and C, B and C, and / or A, B, and C.

[0021] Referring specifically to FIG. 1 here, an example of the system 10 is shown. This system 10 may include one or more of the above-described exemplary devices and will be further described below in accordance with the present principle. The first of the exemplary devices included in the system 10 is a consumer electronics (CE) device such as an audio-video device (AVD) 12, such as an Internet-enabled TV with a TV tuner (equivalently, a set-top box that controls the TV). The AVD 12 may alternatively be a computer-controlled Internet-enabled (“smart”) telephone, a tablet computer, a notebook computer, an HMD, a wearable computer device, a computer-controlled Internet-enabled music player, a computer-controlled Internet-enabled headset, a computer-controlled Internet-enabled embedded device such as an embedded skin device. In any case, it should be understood that the AVD 12 is configured to implement the present principle (e.g., communicate with other CE devices to implement the present principle, execute the logic described herein, and perform other functions and / or operations described herein).

[0022] Therefore, in order to implement such a principle, the AVD12 can be established by some or all of the components shown in FIG. 1. For example, the AVD12 can be implemented by a high-definition or ultra-high-definition "4K" or higher flat screen, and may include one or more displays 14 that are touch-enabled to receive user input signals via touches on the display. The AVD12 may also include one or more speakers 16 for outputting sound according to this principle, and at least one additional input device 18 such as a voice receiver / microphone for inputting audible commands to the AVD12 to control the AVD12. The exemplary AVD12 may also include one or more network interfaces 20 for communicating via at least one network 22 such as the Internet, WAN, LAN, etc. under the control of one or more processors 24. Therefore, the interface 20 may be, but is not limited to, a Wi-Fi transceiver, such as a mesh network transceiver, which is an example of a wireless computer network interface. It should be understood that the processor 24 controls the AVD12 to implement this principle, including other elements of the AVD12 described herein, such as controlling the display 14 to present images and receive inputs therefrom. Further, it should be noted that the network interface 20 may be a wired or wireless modem or router, or other suitable interfaces such as a wireless telephone transceiver or the aforementioned Wi-Fi transceiver.

[0023] In addition to the above, the AVD12 may include one or more input ports and / or output ports 26, such as a high-quality multimedia interface (HDMI (registered trademark)) port and a USB port for physically connecting to other CE devices, and / or a headphone port for connecting headphones to the AVD12 to provide audio to the user via the headphones from the AVD12. For example, the input port 26 can be connected, either wired or wirelessly, to a cable or satellite source 26a of audio-video content. Thus, the source 26a may be a separate or integrated set-top box, or a satellite receiver. Alternatively, the source 26a may be a game console or a disc player containing content. When implemented as a game console, the source 26a may include some or all of the components described below in relation to the CE device 48.

[0024] The AVD12 can further include one or more computer memories 28, such as disk-based or solid-state storage that is not a transient signal, and in some cases, can be embodied in the AVD's chassis as a stand-alone device, or as an internal or external personal video recording device (PVR) or video disc player for playing AV programs, or as a removable memory media or the server described below. Also, in some embodiments, the AVD12 can include a position or location receiver, such as a cellular phone receiver, a GPS receiver, and / or an altimeter 30, configured to receive geographical location information from a satellite or a cellular phone base station, provide that information to the processor 24, and / or determine the altitude at which the AVD12 is disposed in conjunction with the processor 24, but is not limited thereto. The component 30 can also be implemented by an inertial measurement unit (IMU), typically a combination of an accelerometer, a gyroscope, and a magnetometer, or by an event-based sensor, to determine the three-dimensional placement and orientation of the AVD12.

[0025] Continuing the description of the AVD12, in some embodiments, the AVD12 may include one or more cameras 32, such as an infrared camera, a digital camera such as a webcam, an event-based sensor, and / or a camera integrated with the AVD12 and controllable by the processor 24 to collect photos / images and / or videos according to this principle. Also, the AVD12 may include a Bluetooth (registered trademark) transceiver 34 and another near-field communication (NFC) element 36 for communicating with other devices that use Bluetooth (registered trademark) and / or NFC technology, respectively. An example of the NFC element may be a radio frequency identification (RFID) element.

[0026] Furthermore, the AVD12 may include one or more auxiliary sensors 38 that provide inputs to the processor 24 (for example, motion sensors such as accelerometers, gyroscopes, cyclometers, or magnetic sensors, infrared (IR) sensors, optical sensors, speed and / or cadence sensors, event-based sensors, gesture sensors (for example, for sensing gesture commands)). The AVD12 may include an OTA television broadcast port 40 for receiving OTA television broadcasts that provide inputs to the processor 24. In addition to the above, it should be noted that the AVD12 may also include an infrared (IR) transmitter and / or IR receiver and / or IR transceiver 42, such as an IR data association (IRDA) device. A battery (not shown) may be provided to supply power to the AVD12, or it may be a kinetic energy harvester that converts kinetic energy into power for charging the battery and / or supplying power to the AVD12. It may include a graphics processing unit (GPU) 44 and a field programmable gate array 46. One or more tactile generators 47 may be provided to generate tactile signals that can be sensed by a person holding or in contact with the device.

[0027] Continuing to refer to FIG. 1, in addition to the AVD 12, the system 10 may include one or more other CE device types. In one example, the first CE device 48 may be a computer game console that can be used to transmit the audio and video of a computer game to the AVD 12 via commands sent directly to the AVD 12 and / or via the server described below, while the second CE device 50 may include components similar to those of the first CE device 48. In the illustrated example, the second CE device 50 may be configured as a computer game controller operated by a player or a head-mounted display (HMD) worn by the player. In the illustrated example, only two CE devices are shown, but it should be understood that fewer or more devices may be used. The devices described herein may implement some or all of the components shown for the AVD 12. Any of the components shown in the following figures may incorporate some or all of the components shown for the AVD 12.

[0028] Referring now to at least one of the servers 52 described above, it includes at least one server processor 54, at least one tangible computer-readable storage medium 56 such as disk-based or solid-state storage, and at least one network interface 58 that enables communication with other devices in FIG. 1 via the network 22 under the control of the server processor 54 and that can actually facilitate communication between the server and the client device in accordance with this principle. Note that the network interface 58 may be, for example, a wired or wireless modem or router, a Wi-Fi transceiver, or other suitable interface such as a wireless phone transceiver.

[0029] Thus, in some embodiments, server 52 may be an Internet server or an entire server "farm", and may include and execute "cloud" functionality such that, for example, in an exemplary embodiment for a network game application, devices of system 10 can access a "cloud" environment via server 52. Alternatively, server 52 may be implemented on one or more game machines or other computers in the same room as or near the other devices shown in FIG. 1.

[0030] The components shown in the following figures may include some or all of the components shown in FIG. 1.

[0031] FIG. 2 shows a display 200 presenting an animated moving object 202 (an airplane in the illustrated example) that is moving left as indicated by motion vector 204. Viewer 206 is viewing a stationary object 210 (a tree in this case) on display 200 as a point of gaze (POG 208). In the example of FIG. 2, the POG is stably fixed to the stationary object 210 and does not follow the moving object 202, so the motion vector in the direction of travel of the moving object 202, i.e., the x-dimension, of POG 208 is zero. Thus, a motion blur indicated by line 212 is applied to the moving object 202 in accordance with the principles described below. The length of line 212 may be proportional to the difference in magnitude between the motion vector 212 of the moving object 202 and the motion vector of POG 208. The direction in which line 212 extends away from the moving object 202 may be the direction towards the location of POG 208.

[0032] Here, consider the case of FIG. 3 where POG 208 is following a moving object 202 as indicated by viewer motion vector 300. In this case, assume that the viewer is accurately tracking the moving object, that is, the difference in magnitude between the viewer's motion vector 300 and the object's motion vector 204 is zero. As a result, a minimal (in this case zero) motion blur is applied to the moving object 202. On the other hand, since the viewer's POG 208 is moving away from the stationary object 210, motion blur 302 can be applied to the stationary object 210 as needed. The length of line 302 can be proportional to the difference in magnitude between the motion vector 300 (which is zero) of the stationary object 210 and the motion vector of POG 208. The direction in which line 302 extends away from the stationary object 210 can be the direction towards the location of POG 208.

[0033] Referring now to FIG. 4. First, at block 400, head tracking and / or body tracking (e.g., when the viewer is on a moving platform) can be received, and at block 402, eye tracking can be received. Thus, if desired, the motion vector of the viewer's point of gaze (POG) may be determined at block 404 as the sum of the motion vectors due to body movement, head movement, and eye movement. However, in some embodiments (e.g., when the display is an HMD and the viewer's eyes are tracked by an internal camera on the HMD), only eye tracking may be used to determine the viewer's POG and any movement thereof with respect to the display. Combining the various motion vectors into the POG's motion vector can be done using vector algebra.

[0034] Proceeding to block 406, the motion vector(s) of the object(s) on the display that the viewer is looking at are identified. This can be done by accessing the metadata of the object(s) associated with the rendering of the computer simulation.

[0035] Next, block 408 indicates whether motion blur is applied (or not) based on the difference between the motion vector of the object and the motion vector of the POG. For example, if the motion vector indicates that the viewer is tracking a moving object, motion blur may not be applied to that moving object, while, if desired, motion blur may be applied to an object that is not moving on the display based on the viewer's POG moving relative to a stationary object. On the other hand, if the viewer is staring at a point in space and not tracking a moving object, motion blur may be applied to the moving object.

[0036] Motion blur can be achieved via software by blurring the edges / representing motion lines as shown in FIGS. 2 and 3 in accordance with both the direction or movement of the POG and the difference between the motion vector of the display object and the viewer's POG. In some cases, the blur may occur implicitly (e.g., when the viewer rapidly moves their eyes across the VR HMD, the image may appear blurred to the viewer's vision). In some cases, focus can be achieved by reducing or eliminating motion blur in the portion of the scene that the eyes are tracking.

[0037] FIG. 5 shows something like a VR HMD 500 that can be implemented using any or all of the components shown in FIG. 1 for CE device 48, including a position sensor 502, a motion sensor 504 for sensing head movement, a camera 506 for tracking the eyes, and a transceiver 508 for sending signals to a motion blur computer 510, such as a computer simulation console or server, from the sensors of the HMD.

[0038] Figures 6 and 7 illustrate a hardware-assisted embodiment in which a display 600, such as any display herein, has the ability to increase the luminance for only a small portion 602 of the display where an object 604 is presented. A luminance control circuit 606 and an exposure control circuit 608 may be provided to selectively change the exposure and luminance of portions of the display 600, such as portion 602. For example, the luminance can be increased or the exposure time can be shortened to reduce motion blur. For example, in the case of a VR HMD, after briefly exposing an image, a second period during which the display goes black can be provided to "strobe" the screen and suppress blur. The display can vary the exposure time across the entire screen and achieve high exposure by distributing power and heat to only a very small part of the screen, but that period is very short. As a specific example, a certain display shows an image at 50% luminance for 20% of the time (and 0% luminance for the remaining time). If the display uses 100% luminance for only 10% of the time, the same number of photons may be sent. In the case of the moving object (airplane) in FIGS. 2 and 3, the luminance of the airplane can be increased and the exposure time can be shortened to suppress blur (FIG. 3), while for areas where blur due to eye movement is desired, the opposite can be done to FIG. 2, i.e., lower the luminance for a longer exposure time.

[0039] Blur can be added or removed along one axis. Relative movement of the eye with respect to the display vs movement by everything else (head, body, etc.) along other axes.

[0040] In the case of hardware, not only the frame buffer but also a blur / persistence mask can be added. This mask is a per-pixel / per-region mask that tells the hardware how long to persist an image. Not only the frame buffer but also the velocity field can be passed. Asynchronous reprojection may occur near the HMD, i.e., the HMD may perform low-latency tracking and warp the field of view just before rendering it, so the viewer can achieve ultra-low-latency reprojection with a blur equivalent to that of combined head tracking and / or eye tracking. The HMD may also be able to perform a similar operation with eye tracking, combining the eye movement with the passed velocity field to create a current snapshot of pixel movement from the eye, and determining whether to adjust the blur by passing it through a software filter executed on the HMD or by dynamically adjusting the pixel exposure.

[0041] In this specification, specific embodiments are shown and described in detail, but it should be understood that the subject matter encompassed by the present invention is limited only by the claims.

Claims

1. Rather than a transient signal, perform gaze tracking of a user looking at a display, include instructions executable by at least one processor to implement at least one motion blur function for at least one object presented on the display according to the gaze tracking, the instructions identify the object as moving across the display, identify the gaze tracking as following the object, be executable to implement the motion blur function as if there is no motion blur of the object in response to the object being identified as moving across the display and the gaze tracking being identified as following the object, a system comprising at least one computer medium.

2. The system according to claim 1, comprising the display and the at least one processor that executes the instructions.

3. The system according to claim 1, wherein the gaze tracking includes head tracking.

4. The system according to claim 1, wherein the gaze tracking includes eye tracking.

5. The system according to claim 3, wherein the gaze tracking includes eye tracking.

6. The instructions identify the object as moving across the display, identify the gaze tracking as not following the object, be executable to implement the motion blur function as if there is motion blur of the object in response to the object being identified as moving across the display and the gaze tracking being identified as not following the object, the system according to claim 1.

7. The system according to claim 6, wherein the motion blur of the object is proportional to the difference between the motion vector of the object and the motion vector of the gaze tracking.

8. The instructions identify the object as not moving across the display, identify the gaze tracking as moving with respect to the display, The system according to claim 1, executable to implement the motion blur function as the motion blur of the object in response to being identified that the object is not moving across the display and that the gaze tracking is identified as moving relative to the display.

9. The instructions are executable to implement at least partially the motion blur function by establishing a first luminance and a first exposure in a first region of the display that includes the object, and establishing a second luminance and / or a second exposure in a second region of the display that does not include the object. The system according to claim 1.

10. implementing a first motion blur of an object presented on a display in response to the user's line of sight following the object as the object moves, and implementing a second motion blur of the object in response to the user's line of sight not following the object as the object moves, including wherein the first motion blur is less than the second motion blur. A method.

11. The object is a first object, and the method includes implementing a motion blur of a second object presented on the display in response to the user's line of sight moving relative to the display. The method according to claim 10.

12. at least one display, and identifying a first motion vector associated with a viewer of the display, identifying a second motion vector associated with an object presented on the display, at least one processor configured with instructions to implement a motion blur of the object based on the first and second motion vectors, and comprising wherein the processor identifies the object as moving across the display, and identifies gaze tracking as following the object based at least in part on the first motion vector. An apparatus comprising instructions for implementing a minimum motion blur of the object in response to the object being identified as moving across the display and gaze tracking being identified as tracking the object. **Claim 13** The processor identifies the object as moving across the display, identifies gaze tracking as not tracking the object based at least in part on the first motion vector, and in response to the object being identified as moving across the display and gaze tracking being identified as not tracking the object, comprises instructions for implementing a motion blur greater than the minimum motion blur of the object, the apparatus of claim 12. **Claim 14** The apparatus of claim 13, wherein the motion blur of the object is proportional to a difference between the first and second motion vectors. **Claim 15** The processor identifies the object as not moving across the display, identifies gaze tracking as moving relative to the display based at least in part on the first motion vector, and in response to the object being identified as not moving across the display and gaze tracking being identified as moving relative to the object, comprises instructions for implementing a motion blur of the object, the apparatus of claim 12. **Claim 16** The processor establishes a first luminance and a first exposure in a first region of the display that includes the object, and establishes a second luminance and / or a second exposure in a second region of the display that does not include the object, thereby at least partially implementing a motion blur, the apparatus of claim 12.

Images