Warping for spatial light modulation displays using eye tracking

Abstract

To improve the image quality of a spatial optical modulation display.SOLUTION: The MR device acquires a first image frame having a set of color fields. The first image frame corresponds to a first position of the pupil of the viewer. The MR device then determines a second position of the viewer's pupil based on, for example, data received from an eye tracking device coupled to the MR device. The MR device generates a second image frame corresponding to the second position of the pupil based on the first image frame. The second image frame is generated by shifting the color field by a shift value based on a second position of the pupil of the viewer. The MR device transmits the second image frame to the display device of the MR device in order to be displayed thereon.SELECTED DRAWING: Figure 6

Description

[Technical Field]

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 028,418, filed May 21, 2020, and entitled "WARPING FOR SPATIAL LIGHT MODULATING DISPLAYS USING EYE TRACKING," the entire contents of which are incorporated herein by reference for all purposes.

[0002] The following regular US patent applications are being filed concurrently, the entire disclosures of which are incorporated herein by reference for all purposes: U.S. Patent Application No. 17 / 326,034, filed May 20, 2021, and entitled "WARPING FOR LASER BEAM SCANNING DISPLAYS USING EYE TRACKING" U.S. Patent Application No. 17 / 326,036, filed May 20, 2021, and entitled "WARPING FOR SPATIAL LIGHT MODULATING DISPLAYS USING EYE TRACKING" [Background technology]

[0003] Modern computing and display technologies have facilitated the development of "mixed reality" (MR) systems for so-called "virtual reality" (VR) or "augmented reality" (AR) experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner in which they appear or can be perceived as real. VR scenarios typically involve the presentation of digital or virtual image information without transparency to actual real-world visual input. AR scenarios typically involve the presentation of digital or virtual image information as an extension to the user's visualization of the real world around them (i.e., transparency to real-world visual input). Thus, AR scenarios involve the presentation of digital or virtual image information with transparency to real-world visual input.

[0004] MR systems typically employ wearable display devices (e.g., head-mounted displays, helmet-mounted displays, or smart glasses) that are at least loosely coupled to the user's head. MR systems typically generate and display color data, which increases the realism of the MR scenario. Various optical systems generate images, including color images, at various depths to display MR (VR and AR) scenarios.

[0005] According to various scenarios, a user may change the position of their head, or simply their eyes (i.e., their gaze). Changes in a viewer's head and / or pupil position present challenges to spatial light modulation display technology, introducing artifacts such as color breakup into the displayed image. Summary of the Invention [Means for solving the problem]

[0006] Described herein are techniques and technologies for improving the image quality of spatial light modulating displays in which a user moves their eyes and thereby modifies their line of sight (e.g., line of sight).

[0007] As an example, when a user wearing a head-mounted display device views a virtual object and moves their eyes to look in different directions (e.g., to follow the virtual object as it moves or to choose to look at a different object), the virtual object can be rendered based on the user's field of view as determined by the position of their pupils. Changes in the user's pupil position require adjustments to the way the color field of the image frame will be displayed on the spatial light modulator display.

[0008] Embodiments may shift one or more color fields of a rendered image frame based on eye tracking data associated with a position of a user's pupil. One or more processors coupled to the MR device may acquire a first image frame having a set of color fields, the first image frame corresponding to a first view associated with a first position of the viewer's pupil. The processor may then determine a second position of the viewer's pupil based on, for example, data received from an eye tracking device coupled to the MR device. The processor may then generate a second image frame based on the first image frame, corresponding to a second view associated with the second position of the viewer's pupil. For example, the processor may shift one or more color fields of the set of color fields by a shift value based on the second position of the viewer's pupil. The processor may then transmit the second image frame to an eyepiece display device of the MR device for display on the eyepiece display device. All six color fields of the second image frame are displayed on the same spot relative to a second position of the viewer's pupil. According to various embodiments, a first color field of the set of color fields of the first image frame may be shifted by a first shift value and a second color field may be shifted by a second shift value. The first and second values ​​are determined based on the first and second positions of the viewer's pupil.

[0009] Some embodiments provide a method for transforming an image frame based on a position of a viewer's pupil. The method may include acquiring, by a computing device, a first image frame having a set of color fields. The first image frame corresponds to a first view associated with a first position of the viewer's pupil. The method may also include determining, by the computing device, a second position of the viewer's pupil. The computing device may first generate, based on the image frame, a second image frame corresponding to a second view associated with the second position of the viewer's pupil. The generating step may include shifting one or more color fields of the set of color fields by a shift value based on the second position of the viewer's pupil. The method may also include transmitting, by the computing device, the second image frame to an eyepiece display device for display on the eyepiece display device. All six color fields of the second image frame are displayed on the same spot relative to the second position of the viewer's pupil. In some embodiments, the first view is also associated with a first head position of the viewer, and the method further includes estimating, by the computing device, a second head position of the viewer. An intermediate warped image frame may be generated from the first image frame using the second head position of the viewer. The second image frame is generated using the intermediate warped image frame.

[0010] In some embodiments, the method may also include receiving, by the computing device, data from the eye tracking device associated with a position of the viewer's pupil, and a second position of the pupil is determined based on the data received from the eye tracking device.

[0011] In some embodiments, the method may also include calculating, by the computing device, an eye velocity as a function of a first position of the pupil, a second position of the pupil, and a time elapsed for the viewer's pupil to travel from the first position to the second position. The eye velocity is calculated relative to the eyepiece display device. The method may also include calculating, by the computing device, a shift value based on the eye velocity.

[0012] Various embodiments provide a system that includes one or more processors for implementing a method for transforming image frames based on the position of a viewer's pupils, as described above.

[0013] An embodiment provides a non-transitory computer-readable medium having a sequence of instructions stored thereon which, when executed by one or more processors, causes the processors to perform a method for transforming image frames based on a position of a viewer's pupils, as described above.

[0014] Additional and other objects, features, and advantages of the present disclosure are set forth in the detailed description, drawings, and claims. [Brief explanation of the drawings]

[0015] The drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale, and that elements of similar structure or function are represented by like reference numerals throughout the drawings. To better understand how the above-listed and other advantages and objects of the various embodiments of the present disclosure are obtained, a detailed description of the present disclosure, briefly described above, will be given by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. It should be understood that these drawings depict only exemplary embodiments of the present disclosure and, therefore, should not be considered limiting of its scope; the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0016] [Figure 1] FIG. 1 diagrammatically depicts an exemplary mixed reality (e.g., VR or AR) system with real-world objects and virtual objects within a user's field of view, according to some embodiments.

[0017] [Figure 2] FIG. 2 diagrammatically depicts an exemplary mixed reality system illustrating a user's eyeball with a real-world or virtual object in the user's line of sight, according to some embodiments.

[0018] [Figure 3] FIG. 3 illustrates an example method for transforming image frames based on the position of a user's (eg, viewer's) pupils, according to some embodiments.

[0019] [Figure 4] FIG. 4 illustrates a series of drawings associated with a first example VR scenario without any visual corrections applied onto the rendered image.

[0020] [Figure 5] FIG. 5 illustrates a series of drawings associated with a first example VR scenario with conventional head pose-based correction applied to the rendered image.

[0021] [Figure 6] FIG. 6 illustrates a series of drawings associated with a first example VR scenario with eye-tracking-based shifting applied to a rendered image, according to some embodiments.

[0022] [Figure 7] FIG. 7 illustrates a series of drawings associated with a second example VR scenario without any visual corrections applied onto the rendered image.

[0023] [Figure 8]FIG. 8 illustrates a series of drawings associated with a second example VR scenario with eye-tracking-based shifting applied to a rendered image, according to some embodiments.

[0024] [Figure 9] FIG. 9 illustrates a series of drawings associated with a third example VR scenario without any visual corrections applied on the rendered image.

[0025] [Figure 10] FIG. 10 illustrates a series of drawings associated with a third example VR scenario with eye-tracking-based shifting applied to a rendered image, according to some embodiments.

[0026] [Figure 11] FIG. 11 illustrates a series of drawings associated with a fourth example VR scenario without any visual corrections applied on the rendered image.

[0027] [Figure 12] FIG. 12 illustrates a series of drawings associated with a fourth example VR scenario with conventional head pose-based correction applied to the rendered image.

[0028] [Figure 13] FIG. 13 illustrates a series of drawings associated with a fourth example VR scenario with eye-tracking-based shifting applied to a rendered image, according to some embodiments.

[0029] [Figure 14] FIG. 14 illustrates a series of drawings associated with a fifth example VR scenario without any visual corrections applied on the rendered image.

[0030] [Figure 15]FIG. 15 illustrates a series of drawings associated with a fifth example VR scenario with eye-tracking-based shifting applied to a rendered image, according to some embodiments.

[0031] [Figure 16] FIG. 16 is a block diagram that schematically depicts an illustrative computing system, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0032] Detailed Description The present disclosure relates to a spatial light modulating display system that projects image frames including a set of color fields that are shifted using shift values ​​calculated based on eye tracking data to eliminate visual artifacts such as color breakup, and a method for using the same to generate mixed reality experience content.

[0033] Various embodiments of the present disclosure are directed to systems, methods, and articles of manufacture for enhancing warping of virtual content for a spatial light modulating display device using data from an eye tracking device. Other objects, features, and advantages of the present disclosure are set forth in the detailed description, drawings, and claims.

[0034] Various embodiments will now be described in detail with reference to the drawings, which are provided as illustrative examples of the present disclosure to enable those skilled in the art to practice the present disclosure. It should be noted that the following figures and examples are not intended to limit the scope of the present disclosure. Where certain elements of the present disclosure can be implemented partially or completely using known components (or methods or processes), only those portions of such known components (or methods or processes) necessary for understanding the present disclosure will be described, and detailed descriptions of other portions of such known components (or methods or processes) will be omitted so as not to obscure the present disclosure. Furthermore, various embodiments encompass present and future known equivalents of the components referenced herein as examples. Illustrative Mixed Reality Scenarios and Systems

[0035] The following description relates to an exemplary augmented reality system with which the eye tracking enhanced warping system may be implemented. However, it should be understood that the embodiments are also suitable for use in other types of display systems (including other types of mixed reality systems), and thus the embodiments are not limited to only the exemplary system disclosed herein.

[0036] Mixed reality (MR) scenarios often involve the presentation of virtual content (e.g., color images and sounds) corresponding to virtual objects in relation to real-world objects. For example, referring to FIG. 1 , a user 100 of an MR device 102 (e.g., a wearable component) including a headset sees a real-world physical object 104. According to various embodiments, a virtual object 114 may be rendered on an eyepiece display device of the MR device 102 relative to the real-world object 104. For example, the real-world object 104 may be in the form of a table, and a virtual fairy (as an exemplary virtual object 114) may be rendered on the display device to appear as if it were placed on the table. The MR device 102 must also consider the user's line of sight (e.g., line of sight) used to generate / render the real-world object 104 and the virtual object 114. For example, the user 100 may move their eyes from a first position providing a first field of view 105 to a second position providing a second field of view 106. Although the position of the real-world object 104 remains the same relative to real-world coordinates, the position of the real-world object 104 shifts within the user's field of view (e.g., the real-world object 104 is closer to the edge of the first field of view 105 and in the center of the second field of view 106). Unless properly handled, the change in the user's pupil position can result in the virtual object 114 appearing blurred or with color breakup (as discussed in more detail below) on the eyepiece display device of the MR device 102.

[0037] According to various embodiments, data from an eye tracking device 108 coupled to the MR device 102 may be used to appropriately render the virtual object 114 on the eyepiece display device. Such eye tracking data may be determined, for example, by projecting light onto the end user's eye and detecting a return or reflection of at least a portion of the projected light. The eye tracking device 108 may output a pixel index of the display device where the user's gaze is directed. For example, the eye tracking device 108 may determine a first position of the user's pupil at time t1 as the center pixel of the display device and a second position of the user's pupil at time t2 as 10 pixels to the right of the center pixel. According to some embodiments, the MR device 102 may be capable of calculating eye velocity for the user as a function of the first position, the second position, and the difference Δt between time t1 and time t2.

[0038] For example, MR device 102 may acquire a first image frame having a set of color fields (e.g., red, green, blue) when the user's pupil is at a first position (illustrated by normal vector 115 to the user's pupil) associated with first field of view 105. MR device 102 may receive data from eye tracking device 108 and use the data from eye tracking device 108 to determine a second position of the user's pupil (illustrated by normal vector 116 to the user's pupil) associated with second field of view 106. MR device 102 may then generate a second image frame corresponding to second field of view 106 by performing a shift of one or more color fields of the first image frame based on at least the first and second positions of the user's pupil, among other steps. Details of the shifts and additional steps that may be performed are described in more detail below.

[0039] The following description provides five example scenarios in which eye-tracking data can be used to correct visual artifacts / anomalies / glitches resulting from a user of an MR device shifting their gaze (e.g., the position of their pupils). The gaze shift can be in addition to a head pose change or can be on its own (e.g., a user changes their gaze without moving their head). According to various embodiments, eye-tracking-based shifting can be performed as a stand-alone correction or in addition to head pose-based warping (e.g., eye-tracking-based shifting can be performed on a head pose-based warped image).

[0040] 2 is a schematic representation of an exemplary AR / VR headset 200 relative to the eyes of a user of the headset 200. The headset 200 includes a display device 202 (e.g., a spatial light modulator display) that is positioned in front of the user's eyes 204. An eye-tracking device coupled to the headset 200 may track the position of the user's eyes 204 (e.g., the user's pupils). The line of sight of the user 208 may be directed at a real or virtual object 206.

[0041] According to various embodiments, data from the eye tracking device may be used to correctly render real or virtual object 206 on display device 202 based on the user's line of sight or any changes in the user's line of sight. Figure 3 illustrates an example method for transforming image frames based on the position of the user's (e.g., viewer's) pupils.

[0042] In step 302, a computing device acquires a first image frame having a set of color fields. The first image frame may have six color fields and may correspond to a first perspective associated with a first position of a viewer's pupil. According to various embodiments, the computing device may be coupled to an eyepiece display device that includes a spatial light modulating display.

[0043] In step 304, the computing device may receive data associated with the viewer's pupil position from an eye tracking device coupled to a headset worn by the viewer.

[0044] In step 306, the computing device may determine a second position of the viewer's pupil based on the data received from the eye tracking device. The second position of the viewer's pupil is associated with the second view. In some embodiments, the position of the viewer's pupil may be determined relative to the viewer's head position. In other embodiments, the position of the viewer's pupil may be determined relative to the headset (e.g., relative to the pixels of the headset's display device).

[0045] In step 308, the computing device may estimate a second head position, also associated with the second view.

[0046] In step 310, the computing device may generate an intermediate warped image frame from the first image frame using the viewer's second head position, whereby the intermediate warped image is generated using head pose-based warping.

[0047] In step 312, the computing device generates a second image frame based on the first image frame (e.g., using an intermediate warped image frame generated from the first image frame) corresponding to a second view associated with a second position of the viewer's pupil. The second image frame may be generated by shifting one or more color fields of the set of color fields by a shift value based on the second position of the viewer's pupil.

[0048] According to various embodiments, the computing device may calculate an eye velocity for the viewer as a function of a first position of the pupil, a second position of the pupil, and the time elapsed for the viewer's pupil to travel from the first position to the second position. In some embodiments, the eye velocity is calculated relative to the headset (e.g., relative to the pixels of the headset's display device). The computing device may then calculate a shift value based on the eye velocity. In some embodiments, all six color fields of the first image frame may be shifted by the same shift value. In other embodiments, the first color field may be shifted by a first shift value and the second color field may be shifted by a second shift value. Six different shift values ​​(e.g., a different shift value for each of the six color fields) may be calculated based on the viewer's various pupil positions.

[0049] In step 314, the computing device may transmit the second image frame to the eyepiece display device for display on the eyepiece display device. All six color fields of the second image frame are displayed on the same spot on the display device relative to the second position of the viewer's pupil. Thus, although the first image frame may include color breakup artifacts, the color breakup is eliminated in the second image frame displayed on the display device. Example Scenario 1

[0050] In a first example VR scenario, a virtual object (e.g., a virtual fairy) appears in the user's peripheral vision. The user looks at the virtual object and fixes their gaze on it, while the user turns their head / neck to face the virtual object. While the user rotates their head, the virtual object remains stationary and the user's gaze is fixed on the virtual object.

[0051] 4 illustrates a series of drawings (Drawings 1-9) illustrating clockwise head movement relative to a virtual object along with a color field displayed on a display device at an exemplary rate of 60 fps. No visual correction (e.g., head pose-based warping or eye-tracking-based correction) is applied in the drawings shown in FIG.

[0052] As the headset 200 and display device 202 move clockwise relative to the user's pupils, the user's 208 line of sight remains fixed on the virtual object 206. In the third drawing, a first color field 400 is formed at a predetermined location (e.g., a pixel) on the display device 202. As the user continues to move their head without shifting their line of sight, a second (e.g., green) color field 402, a third (e.g., blue) color field 404, a fourth (e.g., red) color field 406, a fifth (e.g., green) color field 408, and a sixth (e.g., blue) color field 410 are displayed on the display device, as shown in drawings 4-8, respectively. As shown in these drawings, the color fields 402-410 are formed outside the user's line of sight. Rather, the color fields 402-410 are formed normal to the display device 202, as opposed to the user's 208 line of sight. As a result, color fields 402-410 are mapped to the same location on the display device, and the image displayed on display device 202 exhibits color breakup. The final drawing (i.e., FIG. 9) illustrates the final rendering of color field 412 on the display device as perceived by a user. As shown, too much blur and color breakup appears on color field 412 illustrated in FIG. 9.

[0053] Conventional solutions to this problem may warp image frames based on the user's estimated head pose. The MR system may detect and / or predict head pose (e.g., using inertial measurement units). The MR system may then warp or transform the rendered virtual content from the source frame of reference to the warped virtual content in the output frame of reference.

[0054] A head pose-based solution for the first scenario is illustrated in FIG. 5. The MR system can figure out the head pose in the first figure and then extrapolate the head pose to estimate where the head would be at different times (as illustrated in subsequent figures 2-8). The different times may correspond to times at which color fields 500, 502, 504, 506, 508, and 510 would be displayed on a display device, as illustrated in figures 3-8, respectively. Once the head pose is extrapolated, the system can then shift one or more color fields (e.g., red, green, and blue) so that the color fields 500-510 align on the display device. Head pose-based warping can place the color fields 500-510 in the user's line of sight (e.g., all color fields appear in the user's line of sight 208, illustrated with the normal to the pupil), thereby eliminating color breakup. However, drawing 9 of FIG. 5 illustrates that while head pose-based warping provides an image 512 of a virtual object without color breakup, the virtual object may still appear blurry on the display device due to the rate at which the display is updated. In the example scenario discussed herein, the display is assumed to update at 360 Hz. Thus, the image may change every 2.8 ms, which would result in an image with 2.8 ms worth of blur, even if the colors are aligned. If the display were configured to update at a much faster rate, such as 100,000 Hz (e.g., in the case of a microLED display, a digital light processing (DLP) display, or a ferroelectric liquid crystal display), the resulting image would have only 0.01 ms worth of blur, which would be imperceptible.

[0055] FIG. 6 illustrates a series of drawings (e.g., drawings 1-8) of an eye-tracking-based solution for the first scenario, according to various embodiments. The position of the user's pupil may be tracked using a high-sample-rate, robust, high-accuracy eye-tracking device coupled to the MR system. The eye-tracking device may determine the position of the user's pupil at different times when the color fields will be displayed on the display device. Thus, instead of tying the color fields to the real world (as is done in conventional warping techniques and / or systems), the eye-tracking-based solution "locks" or "ties" them to the user's pupil or retina. The system may then calculate eye velocity (e.g., the velocity of the user's pupil relative to the headset or display device worn by the user) in x and y pixels to shift the color fields relative to the rendered frame. For example, if the user's gaze moves at 10 pixels / ms and the time offset between the color fields is 3 ms, the rendered fields will each be shifted by 30 pixels. As a result, instead of the color fields aligning to the same location on the display, one or more of the color fields are shifted on the display to align them to the same location on the user's retina.

[0056] For example, without any correction, all six color fields may be mapped to an exemplary pixel coordinate (e.g., 10, 20) on a display device. Based on eye tracking, the system may consider two subsequent positions of the user's eye. For example, the user's eye may move 10 pixels / time units to the left, where one time unit is the difference when different color fields are displayed on the display device. If the color fields are mapped to pixel coordinate (10, 20), the centroids of the six color fields should be mapped to (10, 20). Thus, each color field may need to be shifted by a certain offset value, such that the first color field may be mapped to (35,20), the second color field may be mapped to (25,20), the third color field may be mapped to (15,20), the fourth color field may be mapped to (5,20), the fifth color field may be mapped to (-5,20), and the sixth color field may be mapped to (-15,20). Thus, each color field will be shifted (e.g., displaced) by exactly the correct interval to correspond to eye velocity. The center of gravity of the color fields is at (10,20), while each color field is shifted by a certain offset value determined based on eye velocity (or the position of the user's pupil at different times). For example, the red color field is shifted to the right of the (10,20) coordinate to properly align with where the user is looking. The next color field (e.g., a green color field, a blue color field, and a repeating pattern), instead of being co-located with the red color field, is shifted on the display to align the color fields over the same location on the user's retina.

[0057] As illustrated in the series of drawings in Figure 6, the color fields 610 all appear in the user's line of sight and are aligned to the same location on the user's retina. Therefore, color breakup is eliminated. Shifting the color fields on the rendered image frame 612 works well with spatial light modulator displays because the entire rendered frame is not displayed at once, but rather, is displayed at six different times. The embodiment shifts the color fields 610 by a shift value (measured in pixels) calculated as the product of the eye velocity (measured in pixels / millisecond) and the time (measured in milliseconds) that each color field will be displayed on the display device.

[0058] According to various embodiments, eye velocity may be calculated as a function of a first position of the user's pupil, a second position of the user's pupil, and the time elapsed for the user's pupil to travel from the first position to the second position. The eye velocity is calculated relative to the headset. For increased accuracy, it may be desirable to calculate the eye velocity as late as possible. For example, the eye velocity may be calculated just before the first color field is shifted by the shift value, leaving just enough time for the system to use the eye velocity to calculate the shift value.

[0059] In some embodiments, the same eye velocity may be used to calculate the shift values ​​for all six color fields. In yet other embodiments, a first eye velocity may be calculated for the first color field and a second eye velocity may be calculated for the second color field. In some embodiments, different eye velocities may be calculated to determine different shift values ​​for each of the six color fields. For an MR device rendering at 60 Hz, eye-tracking-based shifting may be performed at 360 Hz.

[0060] In some embodiments, the eye velocity may be zero. For example, a user may not change their line of sight, but simply change their focus (e.g., looking from a distant object to a closer object when both objects are in the same line of sight). In such an embodiment, the rendered image may still be warped or shifted; however, because the eye velocity would be zero, the shift value would be calculated as zero (e.g., no shift would be applied to the rendered image or color field). Example Scenario 2

[0061] In a second example VR scenario, a virtual object (e.g., a virtual gnome) appears on a real-world object (e.g., a desk) in front of the user. The user attempts to view the gnome's surroundings by rocking their body from side to side so that the user can see what is behind the gnome. In such a scenario, the user's head undergoes translation without rotation. The user's gaze remains on the gnome while the user's body and head move from side to side.

[0062] 7 illustrates a series of drawings (e.g., drawings 1-9) illustrating head movement along with color fields 700, 702, 704, 706, 708, 710 rendered on a display device using head pose-based warping to shift the rendered image; only head orientation is used, not translation (e.g., the head is moving but not rotating). As the headset 200 and display device 202 move from left to right relative to the virtual object 206, the user's 208 line of sight remains fixed on the virtual object 206. In the third drawing, a first color field 700 is formed over a predetermined location (e.g., a pixel) on the display device 202. As the user continues to move their head without shifting their gaze, a second (e.g., green) color field 702, a third (e.g., blue) color field 704, a fourth (e.g., red) color field 706, a fifth (e.g., green) color field 708 and a sixth (e.g., blue) color field 710 are displayed on the display device, as shown in Figures 4-8, respectively.

[0063] The MR system can determine the head pose in the first figure and then extrapolate the head pose to estimate where the head will be at different times (as shown in Figures 2-8). The different times may correspond to times at which color fields 700-710 will be displayed on the display device, as shown in Figures 3-8. Once the head pose is extrapolated, the system can then shift one or more color fields (e.g., red, green, blue) so that they align on the display device. As shown in these figures, the color fields are formed outside the user's line of sight. Rather, the color fields are formed normal to the display device 202, as opposed to the user's line of sight. As a result, the color fields are mapped to the same location on the display device, and the image 712 displayed on the display device 202 exhibits color breakup. Figure 9 illustrates the final rendering of the color fields on the display device as perceived by the user. As shown, too much blur and color breakup appears on image 712. If the system were to attempt to correct the artifacts using head pose-based warping, which tracks orientation but not translation, the correction would not be sufficient. The system would require head pose (which tracks both orientation and translation) and object depth plane (or vergence-divergence motion depth) in six degrees of freedom.

[0064] FIG. 8 illustrates a series of figures (e.g., figures 1-8) illustrating an eye-tracking-based solution for the second scenario, according to various embodiments. The position of the user's pupil may be tracked using a high-sample-rate, robust, high-accuracy eye-tracking device coupled to the MR system. The eye-tracking device may determine the position of the user's pupil at different times when the color fields will be displayed on the display device. Thus, instead of tying the color fields to the real world (as is done in conventional warping techniques and / or systems), the eye-tracking-based solution "locks" or "ties" them to the user's pupil or retina. The system may then calculate eye velocity (e.g., the velocity of the user's pupil relative to a headset or display device worn by the user) in x and y pixels to shift the color fields relative to the rendered frame. As a result, instead of the color fields aligning to the same location on the display, one or more of the color fields are shifted on the display to align them to the same location on the user's retina.

[0065] As illustrated in the series of drawings in Figure 8, color fields 810 are depicted that all appear in the user's line of sight and align to the same location on the user's retina. Thus, color breakup and blurring are eliminated on the rendered image 812, as shown in drawing 8 of Figure 8. Example Scenario 3

[0066] In a third example VR scenario, a virtual object (e.g., a virtual mouse) appears behind a real-world object (e.g., a coffee cup) in front of the user. For example, the mouse has its head peeking out from behind the coffee cup. The mouse runs from the coffee cup to another object in the scene. The user does not move but follows the mouse using their gaze. In such a scenario, the virtual object moves, but the user's head remains stationary. The user's gaze remains on the moving virtual object.

[0067] 9 illustrates a series of drawings (e.g., drawings 1-9) illustrating the movement of a virtual object 206 along with color fields rendered on a display device. As the virtual object 206 moves from right to left relative to the headset 200 and display device 202, the user's 208 line of sight remains fixed on the virtual object 206. In the third drawing, a first color field 900 is formed at a predetermined location (e.g., pixel) on the display device 202. As the virtual object continues to move relative to the headset without the user shifting their line of sight from the virtual object, a second (e.g., green) color field 902, a third (e.g., blue) color field 904, a fourth (e.g., red) color field 906, a fifth (e.g., green) color field 908, and a sixth (e.g., blue) color field 910 are displayed on the display device, as shown in drawings 4-8, respectively. Because the user's head is not moving, conventional head pose-based warping would not be able to correct the blurring and color breakup on the displayed image 912, as shown in drawing 9 of Figure 9. That is, because the head pose is the same, the applied head pose-based warping would result in a zero pixel shift for each color field. A conventional approach to correct this is to use the head pose plus the motion vectors for the virtual object to perform the warping.

[0068] FIG. 10 illustrates a series of diagrams illustrating an eye-tracking-based solution for the third scenario, according to various embodiments. The position of the user's pupil may be tracked using a high-sample-rate, robust, high-accuracy eye-tracking device coupled to the MR system. The eye-tracking device may determine the position of the user's pupil at different times when the color fields will be displayed on the display device. Thus, instead of tying the color fields to the real world (as is done in conventional warping techniques and / or systems), the eye-tracking-based solution "locks" or "ties" them to the user's pupil or retina. The system may then calculate eye velocity (e.g., the velocity of the user's pupil relative to a headset or display device worn by the user) in x and y pixels to shift the color fields relative to the rendered frame. As a result, instead of the color fields aligning to the same location on the display, one or more of the color fields are shifted on the display to align them to the same location on the user's retina.

[0069] As illustrated in the series of drawings in Figure 10, color fields 1010 are depicted that all appear in the user's line of sight and align to the same location on the user's retina. Thus, color breakup and blurring are eliminated on the rendered image 1012, as shown in drawing 8 of Figure 10. Example Scenario 4

[0070] The third scenario illustrates how rendering and image display become complicated when virtual objects are in motion. Head pose-based warping is not sufficient to correct blurring and color breakup on the displayed image. The next scenario illustrates that in some cases, head pose-based warping is not only sufficient to correct artifacts, but also worsens the quality of the displayed image.

[0071] In a fourth example VR scenario, a virtual object (e.g., a virtual fish) moves around the user. The user turns their entire body to follow the virtual object. Both the headset and the user's pupils rotate relative to the real world but are fixed relative to each other.

[0072] 11 illustrates a series of drawings (drawings 1-8) illustrating headset and pupil movement, with the remainder remaining fixed relative to one another, along with a color field rendered on a display device. Head pose warping is not applied to the drawings illustrated in FIG. 11. As the headset 200 (including the display device 202) and the user's line of sight, along with the virtual object 206, move counterclockwise, the line of sight of the user 208 and the normal vector to the display device 202 remain fixed on the virtual object 206. In the third drawing, a first color field 1100 is formed on a predetermined location (e.g., a pixel) on the display device 202. As the user continues to move their head without shifting their gaze, a second (e.g., green) color field 1102, a third (e.g., blue) color field 1104, a fourth (e.g., red) color field 1106, a fifth (e.g., green) color field 1108, and a sixth (e.g., blue) color field 1110 are displayed on the display device, as shown in Figures 4-8, respectively. Because the user is following the moving object with their eyes and their head, the resulting image 1112 is clear without any blurring or color breakup.

[0073] On the other hand, for an MR system that applies head pose-based warping to every rendered image frame, example scenario 4 results in a low-quality image: head pose-based warping degrades the quality of the displayed image by introducing blur and / or color breakup, as illustrated in FIG.

[0074] The MR system can figure out the head pose in the first figure and then extrapolate the head pose to estimate where the head would be at different times (as illustrated in subsequent figures). The different times may correspond to times at which color fields would be displayed on a display device, as illustrated in Figures 3-8. Once the head pose is extrapolated, the system can then shift one or more color fields (e.g., red, green, blue) so that they align on the display device. Head pose-based warping aims to place color fields in the user's line of sight, but does not consider the movement of virtual objects. Therefore, head pose-based warping attempts to lock virtual objects to the world, introducing blurring and color breakup into the rendered image 1212, as illustrated in the last figure of Figure 12. In fact, the color breakup introduced by head pose-based warping is worse than the artifacts previously illustrated, because the colors do not even blend together.

[0075] FIG. 13 illustrates an eye-tracking-based solution to the fourth scenario, according to various embodiments. Because the system calculates eye velocity relative to the headset, when the headset and pupil move simultaneously, the eye velocity is zero. Therefore, the eye-velocity-based shift value is zero. As illustrated in the drawing of FIG. 13, the color fields 1310 all appear in the user's line of sight and are aligned to the same location on the user's retina. Therefore, both color breakup and blur are eliminated on the displayed image 1312. Therefore, the result is identical to FIG. 11, even when the eye-tracking-based solution is applied by default. Example Scenario 5

[0076] In a fifth exemplary VR scenario, a virtual object (e.g., a virtual robot) is at the center of the user's field of view. However, the user is not looking at the virtual object. Instead, the user may be comparing two other virtual objects or areas next to the virtual object. For example, the user may be looking at area 216 located to the right of the virtual object and area 226 located to the left of the virtual object. That is, the user may be blinking their eyes back and forth between the two areas and across the virtual object. This eye movement may be referred to as a "saccade." This scenario is similar to the third exemplary VR scenario, except that the eyes move at a faster speed.

[0077] 14 illustrates a series of drawings illustrating saccadic eye movement along with color fields rendered on a display device using head pose-based warping for the rendered image. While headset 200 and display device 202 remain stationary, a user moves their eyes from left to right relative to areas 216 and 226, and the user's 208 line of sight moves between the two areas. In the third drawing, a first color field 1400 is formed on a predetermined location (e.g., pixel) on display device 202. As the user continues to move their eyes, a second (e.g., green) color field 1402, a third (e.g., blue) color field 1404, a fourth (e.g., red) color field 1406, a fifth (e.g., green) color field 1408, and a sixth (e.g., blue) color field 1410 are displayed on the display device, as shown in drawings 4-8, respectively. 14, the displayed image 1412 exhibits color breakup. Because the user's head is not moving, traditional head pose-based warping would not be able to correct the artifacts on the displayed image 1412 (e.g., the head pose is the same, and therefore warping applied based on head pose would be zero warping). In addition, motion vector-based warping (which could improve on Scenario 3) cannot be applied here because there is no real object and no virtual object that the user's eyes are following.

[0078] FIG. 15 illustrates an eye-tracking-based solution for the fifth scenario, according to various embodiments. The position of the user's pupil may be tracked using a high-sample-rate, robust, high-accuracy eye-tracking device coupled to the MR system. The eye-tracking device may determine the position of the user's pupil at different times when the color fields will be displayed on the display device. Thus, instead of tying the color fields to the real world (as is done in conventional warping techniques and / or systems), the eye-tracking-based solution "locks" or "ties" them to the user's pupil or retina. The system may then calculate eye velocity (e.g., the velocity of the user's pupil relative to a headset or display device worn by the user) in x and y pixels to shift the color fields relative to the rendered frame. As a result, instead of the color fields aligning to the same location on the display, one or more of the color fields are shifted on the display to align them to the same location on the user's retina. 15, the color fields 1510 all appear in the user's line of sight and are aligned to the same location on the user's retina, thus eliminating color breakup on the displayed image 1512.

[0079] The displayed image 1512 shown in drawing 9 of Figure 15 may contain blurring that may be due to display persistence between color fields. Instead of seeing different colors as the user blinks their eyes around, the user sees a strobe effect. In some embodiments, artificial blurring may be added to the rendered image in addition to the color field shift to correct for strobe artifacts.

[0080] In some embodiments, eye-tracking-based shifting may also be applicable to two independent monocular displays. Tracking for each eye may be used to warp the corresponding display. In some embodiments, eye tracking for one eye may be used for both eyes (e.g., when the device loses tracking in one eye).

[0081] According to various embodiments, an image frame may be rendered using a predicted head pose at the centroid of the six color fields. Single-frame time-delay warping may be performed to correct the rendering. An eye-tracking-based shift may then be applied to the warped rendered image to shift one or more of the six color fields by a shift value (which may be the same shift value for all color fields or a different shift value for one or more of the six color fields). The eye-tracking-based color field shift may be applied on the headset portion of the MR device. System Architecture Overview

[0082] 16 is a block diagram of an exemplary computing system 1600 (e.g., an MR device) according to some embodiments. The computer system 1600 includes a bus 1606 or other communication mechanism for communicating information, interconnecting subsystems and devices such as a processor 1607, system memory 1608 (e.g., RAM), static storage device 1609 (e.g., ROM), disk drive 1610 (e.g., magnetic or optical), communication interface 1614 (e.g., modem or Ethernet card), display 1611 (e.g., CRT or LCD), input device 1612 (e.g., keyboard), and cursor control.

[0083] According to some embodiments, computer system 1600 performs specific operations by processor 1607 executing one or more sequences of one or more instructions contained in system memory 1608. Such instructions may be read into system memory 1608 from another computer-readable / usable medium, such as static storage device 1609 or disk drive 1610. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the present disclosure. Thus, embodiments are not limited to any specific combination of hardware circuitry and / or software. The term "logic" may refer to any combination of software or hardware used to implement all or part of the present disclosure.

[0084] The terms "non-transitory computer-readable medium" or "computer-usable medium," as used herein, refer to any medium that participates in providing instructions to the processor 1607 for execution. Such media may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as the disk drive 1610. Volatile media include dynamic memory, such as the system memory 1608.

[0085] Common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM (e.g., NAND flash, NOR flash), any other memory chip or cartridge, or any other medium from which a computer can read.

[0086] In some embodiments, execution of sequences of instructions for practicing the present disclosure is performed by a single computer system 1600. According to some embodiments, two or more computer systems 1600 coupled by a communications link 1615 (e.g., a LAN, PTSN, or wireless network) may cooperate with each other to perform the sequences of instructions required to practice the present disclosure.

[0087] Computer system 1600 may transmit and receive messages, data, and instructions, including programs, e.g., application code, through communications link 1615 and communications interface 1614. Received program code may be executed by processor 1607 as it is received, and / or stored in disk drive 1610 or other non-volatile storage for later execution. Database 1632 in storage medium 1631 may be used to store data accessible by system 1600 via data interface 1633.

[0088] The present disclosure includes methods that may be implemented using the subject devices. The methods may include the act of providing such a suitable device. Such provisioning may be performed by a user. In other words, the act of "providing" simply requires the user to obtain, access, approach, locate, configure, activate, power on, or otherwise act to provide the device required in the subject methods. The methods recited herein may occur in any order of the recited events and the recited sequence of events that is logically possible.

[0089] Exemplary aspects of the present disclosure, along with details regarding material selection and manufacturing, are described above. As for other details of the present disclosure, these may be understood in connection with the aforementioned referenced patents and publications and are generally known or may be understood by those skilled in the art. The same may be true with respect to the method-based aspects of the present disclosure in terms of additional operations as commonly or logically adopted.

[0090] Additionally, while the present disclosure has been described with reference to several embodiments incorporating various features, the present disclosure is not limited to that described or illustrated, as each variation of the disclosure is discussed. Various modifications may be made to the present disclosure as described, and equivalents (whether recited herein or not included for purposes of brevity to some extent) may be substituted without departing from the spirit and scope of the present disclosure. Additionally, when a range of values ​​is provided, it is understood that all intervening values ​​between the upper and lower limits of that range, and any other stated value or intervening value within the stated range, are encompassed within the present disclosure.

[0091] It is also contemplated that any optional features of the described inventive variations may be set forth and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that plural identical items are present. More specifically, as used in this specification and the claims associated therewith, the singular forms "a," "an," "said," and "the" include plural references unless specifically stated otherwise. In other words, the use of articles allows for "at least one" of the items of the present subject matter in the above description and in the claims associated with this disclosure. Furthermore, it should be noted that such claims may be drafted to exclude any optional element. Accordingly, this language is intended to serve as a predicate for the use of exclusive terminology such as "solely," "only," and the like in connection with the recitation of claim elements, or the use of a "negative" limitation.

[0092] Without the use of such exclusive terminology, the term "comprising" in the claims associated with this disclosure shall be deemed to permit the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claim, or the addition of features may be deemed to change the nature of the elements recited in such claim. Except as specifically defined herein, all technical and scientific terms used herein should be given the broadest possible commonly understood meaning while maintaining claim legitimacy.

[0093] The scope of the present disclosure should not be limited to the examples provided and / or this specification, but rather should be limited only by the scope of the claim language associated with this disclosure.

[0094] In the foregoing specification, the present disclosure has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made therein without departing from the broader spirit and scope of the present disclosure. For example, the foregoing process flows are described with reference to a particular order of process actions. However, the order of many of the described process actions may be changed without affecting the scope or operation of the present disclosure. The specification and drawings are, therefore, to be regarded in an illustrative rather than a restrictive sense.

Claims

1. 1. A method for transforming an image frame based on a position of a viewer's pupil, the method comprising: acquiring, by a computing device, a first image frame having a set of color fields, the first image frame corresponding to a first perspective associated with a first position of a pupil of the viewer; determining, by the computing device, a second position of a pupil of the viewer; and generating, by the computing device, a second image frame corresponding to a second view associated with a second position of a pupil of the viewer based on the first image frame, wherein the generating includes: shifting one or more color fields of the set of color fields by a shift value based on a second position of the viewer's pupil. and transmitting, by the computing device, the second image frame to the eyepiece display device for display on the eyepiece display device, wherein all six color fields of the second image frame are displayed on the same spot relative to a second position of the viewer's pupil; A method comprising:

2. 10. The method of claim 1, further comprising receiving, by the computing device, data associated with a position of a pupil of the viewer from an eye tracking device, wherein a second position of the pupil is determined based on the data received from the eye tracking device.

3. The first perspective is also associated with a first head position of the viewer, and the method further comprises: estimating, by the computing device, a second head position of the viewer; and generating, by the computing device, an intermediate warped image frame from the first image frame using a second head position of the viewer, wherein the second image frame is generated using the intermediate warped image frame; The method of claim 1 , comprising:

4. The method of claim 1 , wherein the eyepiece display device is a spatial light modulating display.

5. The method of claim 1 , wherein the pupil position is determined relative to the viewer's head position.

6. The method of claim 1 , wherein the position of the pupil is determined relative to pixels of the eyepiece display device.

7. calculating an eye velocity as a function of a first position of the pupil, a second position of the pupil, and a time elapsed for the viewer's pupil to travel from the first position to the second position, the eye velocity being calculated relative to the eyepiece display device; calculating, by the computing device, the shift value based on the eye velocity; The method of claim 1 further comprising:

8. 10. The method of claim 1, wherein the first image frame includes six color fields, and all six color fields are shifted based on first and second positions of the viewer's pupil.

9. The method of claim 8 , wherein each color field is shifted at a different shift value determined based on the position of the viewer's pupil.

10. 2. The method of claim 1, wherein a first color field of the set of color fields of the first image frame is shifted by a first shift value and a second color field is shifted by a second shift value, the first shift value and the second shift value being determined based on first and second positions of the viewer's pupil.

11. 1. A system comprising: one or more processors, acquiring a first image frame having a set of color fields, the first image frame corresponding to a first perspective associated with a first position of a viewer's pupil; determining a second position of the viewer's pupil; generating a second image frame corresponding to a second perspective associated with a second position of a pupil of the viewer based on the first image frame, said generating including: shifting one or more color fields of the set of color fields by a shift value based on a second position of the viewer's pupil. and transmitting the second image frame for display; one or more processors configured to execute instructions to perform the 1. A display device, comprising: displaying a second image frame, wherein all six color fields of the second image frame are displayed on the same spot relative to a second position of the viewer's pupil; a display device configured to: A system comprising:

12. 1. An eye tracking device comprising: tracking the viewer's pupils; transmitting data associated with a position of a pupil of the viewer to the one or more processors, wherein a second position of the pupil is determined based on data received from the eye tracking device; an eye tracking device configured to The system of claim 11 further comprising:

13. The first perspective is also associated with a first head position of the viewer, and the one or more processors further: estimating a second head position of the viewer; and generating an intermediate warped image frame from the first image frame using a second head position of the viewer, the second image frame being generated using the intermediate warped image frame; and The system of claim 11 configured to execute instructions to:

14. The system of claim 11 , wherein the display device is a spatial light modulating display.

15. The system of claim 11 , wherein the pupil position is determined relative to the viewer's head position.

16. The system of claim 11 , wherein the position of the pupil is determined relative to pixels of the display device.

17. The one or more processors further comprise: calculating an eye velocity as a function of a first position of the pupil, a second position of the pupil, and a time elapsed for the viewer's pupil to travel from the first position to the second position, wherein the eye velocity is calculated relative to the display device; calculating the shift value based on the eye velocity; The system of claim 11 configured to execute instructions to:

18. 12. The system of claim 11, wherein the first image frame includes six color fields, all six color fields being shifted based on first and second positions of the viewer's pupil, each color field being shifted by a different shift value determined based on the position of the viewer's pupil.

19. 12. The system of claim 11, wherein a first color field of the set of color fields of the first image frame is shifted by a first shift value and a second color field is shifted by a second shift value, the first shift value and the second shift value being determined based on first and second positions of the viewer's pupil.

20. 1. A non-transitory computer-readable medium having a sequence of instructions stored thereon that, when executed by one or more processors, causes the processors to perform a method for transforming image frames based on a position of a viewer's pupils, the method comprising: acquiring a first image frame having a set of color fields, the first image frame corresponding to a first perspective associated with a first position of the viewer's pupil; determining a second position of the viewer's pupil; generating a second image frame corresponding to a second perspective associated with a second position of a pupil of the viewer based on the first image frame, said generating including: shifting one or more color fields of the set of color fields by a shift value based on a second position of the viewer's pupil. and transmitting the second image frame to the eyepiece display device for display on the eyepiece display device, wherein all six color fields of the second image frame are displayed on the same spot relative to a second position of the viewer's pupil; 1. A non-transitory computer-readable medium comprising:

Images