Warping for laser beam scanning displays using eye tracking.

Abstract

An embodiment transforms image frames based on a viewer's pupil position to eliminate visual artifacts that form on image frames displayed on a scanning-type display device. An MR system acquires a first image frame corresponding to a first view associated with a first pupil position. The system receives data from an eye tracking device, determines a second pupil position, and generates a second image frame corresponding to a second view associated with the second pupil position. A first set of pixels in the second image frame are shifted by a first shift value, and a second set of pixels in the second image frame are shifted by a second shift value, the shift value being calculated based on at least the second pupil position. The system transmits the second image frame to an eyepiece display device for display thereon.

Description

[Technical Field]

[0001] This application claims the benefit of priority to U.S. Provisional Patent Application No. 63 / 028,411, filed May 21, 2020, and entitled "WARPING FOR LASER BEAM SCANNING DISPLAYS USING EYE TRACKING," the entire contents of which are incorporated herein by reference for all purposes.

[0002] The following regular US patent applications are being filed concurrently, the entire disclosures of which are incorporated herein by reference for all purposes: U.S. Patent Application No. 17 / 326,034, filed May 20, 2021, and entitled "WARPING FOR LASER BEAM SCANNING DISPLAYS USING EYE TRACKING" U.S. Patent Application No. 17 / 326,036, filed May 20, 2021, and entitled "WARPING FOR SPATIAL LIGHT MODULATING DISPLAYS USING EYE TRACKING" [Background technology]

[0003] Modern computing and display technologies have facilitated the development of "mixed reality" (MR) systems for so-called "virtual reality" (VR) or "augmented reality" (AR) experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner in which they appear or can be perceived as real. VR scenarios typically involve the presentation of digital or virtual image information without transparency to actual real-world visual input. AR scenarios typically involve the presentation of digital or virtual image information as an extension to the user's visualization of the real world around them (i.e., transparency to real-world visual input). Thus, AR scenarios involve the presentation of digital or virtual image information with transparency to real-world visual input.

[0004] MR systems typically employ wearable display devices (e.g., head-mounted displays, helmet-mounted displays, or smart glasses) that are at least loosely coupled to the user's head. Various optical systems generate images, including color images, at various depths to display MR (VR and AR) scenarios.

[0005] According to various scenarios, a user may change the position of their head, or simply their eyes (i.e., their gaze). Changes in the viewer's head and / or pupil position present challenges to laser beam scanning display technology, introducing artifacts such as swirling and mis-stitching onto the displayed image. Summary of the Invention [Means for solving the problem]

[0006] Described herein are techniques and technologies for improving the image quality of laser beam scanning displays in which a user moves their eyes and thereby modifies their line of sight (e.g., line of sight).

[0007] As an example, if a user wearing a head-mounted display device views a virtual object and moves their eyes to look in different directions (e.g., to follow the virtual object as it moves or to choose to look at a different object), the virtual object can be rendered based on the user's field of view as determined by the position of their pupils. Changes in the user's pupil position require adjusting the way image frames will be displayed on the laser beam scanning display.

[0008] Some embodiments provide a method for transforming an image frame based on a position of a viewer's pupil. The method may include acquiring, by a computing device, a first image frame. The first image frame corresponds to a first view associated with a first position of the viewer's pupil. The method may also include determining, by the computing device, a second position of the viewer's pupil. The computing device may generate, based on the first image frame, a second image frame corresponding to a second view associated with the second position of the viewer's pupil. The generating step may include shifting a first set of pixels of the second image frame by at least a first shift value calculated based on the second position of the viewer's pupil, and shifting a second set of pixels of the second image frame by at least a second shift value calculated based on the second position of the viewer's pupil. The second image frame may be transmitted to an eyepiece display device for display on the eyepiece display device. In some embodiments, the first view is also associated with a first head position of the viewer, and the method further includes estimating, by the computing device, a second head position of the viewer, and generating, by the computing device, an intermediate warped image frame from the first image frame using the second head position of the viewer. The second image frame is generated using the intermediate warped image frame.

[0009] In some embodiments, the method may also include receiving, by the computing device, data from the eye tracking device associated with a position of the viewer's pupil, and a second position of the pupil is determined based on the data received from the eye tracking device.

[0010] In some embodiments, the method may include calculating a first eye velocity as a function of a first position of the pupil, a second position of the pupil, and an elapsed time for the viewer's pupil to travel from the first position to the second position. The first eye velocity is calculated relative to the eyepiece display device. The method may also include calculating a second eye velocity and calculating, by a computing device, a first shift value based on the first eye velocity and a second shift value based on the second eye velocity. In some embodiments, the method may include calculating, by the computing device, a second shift value based on the first shift value and a subsequent position of the viewer's pupil.

[0011] Various embodiments provide a system that includes one or more processors for implementing a method for transforming image frames based on the position of a viewer's pupils, as described above.

[0012] An embodiment provides a non-transitory computer-readable medium having a sequence of instructions stored thereon which, when executed by one or more processors, causes the processors to perform a method for transforming image frames based on a position of a viewer's pupils, as described above.

[0013] Additional and other objects, features, and advantages of the present disclosure are set forth in the detailed description, drawings, and claims. [Brief explanation of the drawings]

[0014] The drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale, and that elements of similar structure or function are represented by like reference numerals throughout the drawings. To better understand how the above-listed and other advantages and objects of the various embodiments of the present disclosure are obtained, the detailed description of the present disclosure, briefly described above, will be given by reference to specific embodiments thereof, which are illustrated in the accompanying drawings. It should be understood that these drawings depict only exemplary embodiments of the present disclosure and, therefore, should not be considered limiting of its scope; the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0015] [Figure 1] FIG. 1 diagrammatically depicts an exemplary mixed reality (e.g., VR or AR) system with real-world objects and virtual objects within a user's field of view, according to some embodiments.

[0016] [Figure 2] FIG. 2 diagrammatically depicts an exemplary mixed reality system illustrating a user's eyeball with a real-world or virtual object in the user's line of sight, according to some embodiments.

[0017] [Figure 3] FIG. 3 illustrates an example method for transforming image frames based on the position of a user's (eg, viewer's) pupils, according to some embodiments.

[0018] [Figure 4] FIG. 4 illustrates an exemplary manner for splitting image frames for visualization purposes of a VR scenario illustrated in a set of adjacent numbers, according to various embodiments.

[0019] [Figure 5]FIG. 5 illustrates a series of drawings associated with a first example VR scenario without any visual corrections applied onto the rendered image.

[0020] [Figure 6] FIG. 6 illustrates a series of drawings associated with a first example VR scenario with conventional head pose-based correction applied to the rendered image.

[0021] [Figure 7] FIG. 7 illustrates a series of drawings associated with a first example VR scenario with eye-tracking-based shifting applied to a rendered image, according to some embodiments.

[0022] [Figure 8] FIG. 8 illustrates a series of drawings associated with a second example VR scenario without any visual corrections applied onto the rendered image.

[0023] [Figure 9] FIG. 9 illustrates a series of drawings associated with a second example VR scenario with eye-tracking-based shifting applied to a rendered image, according to some embodiments.

[0024] [Figure 10] FIG. 10 illustrates a series of drawings associated with a third example VR scenario without any visual corrections applied on the rendered image.

[0025] [Figure 11] FIG. 11 illustrates a series of drawings associated with a third example VR scenario with eye-tracking-based shifting applied to a rendered image, according to some embodiments.

[0026] [Figure 12]FIG. 12 illustrates a series of drawings associated with a fourth example VR scenario without any visual corrections applied on the rendered image.

[0027] [Figure 13] FIG. 13 illustrates a series of drawings associated with a fourth example VR scenario with conventional head pose-based correction applied to the rendered image.

[0028] [Figure 14] FIG. 14 illustrates a series of drawings associated with a fourth example VR scenario with eye-tracking-based shifting applied to a rendered image, according to some embodiments.

[0029] [Figure 15-1] FIG. 15 illustrates a series of drawings associated with a fifth example VR scenario without any visual corrections applied on the rendered image. [Figure 15-2] FIG. 15 illustrates a series of drawings associated with a fifth example VR scenario without any visual corrections applied on the rendered image.

[0030] [Figure 16-1] FIG. 16 illustrates a series of drawings associated with a fifth example VR scenario with eye-tracking-based shifting applied to a rendered image, according to some embodiments. [Figure 16-2] FIG. 16 illustrates a series of drawings associated with a fifth example VR scenario with eye-tracking-based shifting applied to a rendered image, according to some embodiments.

[0031] [Figure 17] 17-21 illustrate additional processing that may be used in conjunction with eye-tracking based shifting to improve the quality of the displayed image frame, according to some embodiments. [Figure 18]17-21 illustrate additional processing that may be used in conjunction with eye-tracking based shifting to improve the quality of the displayed image frame, according to some embodiments. [Figure 19] 17-21 illustrate additional processing that may be used in conjunction with eye-tracking based shifting to improve the quality of the displayed image frame, according to some embodiments. [Figure 20] 17-21 illustrate additional processing that may be used in conjunction with eye-tracking based shifting to improve the quality of the displayed image frame, according to some embodiments. [Figure 21] 17-21 illustrate additional processing that may be used in conjunction with eye-tracking based shifting to improve the quality of the displayed image frame, according to some embodiments.

[0032] [Figure 22] FIG. 22 is a block diagram that schematically depicts an illustrative computing system, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0033] Detailed Description The present disclosure relates to an MR system, including a laser beam scanning display system that projects image frames in which visual artifacts such as rotation and mis-stitching are eliminated using eye tracking data, and a method for using the same to generate mixed reality experience content.

[0034] In some embodiments, an MR system may include more than one (e.g., two) scanning sources (e.g., lasers) that scan simultaneously. For example, one laser may scan from the top of the display to the center of the display, while a second laser may start at the center and scan to the bottom of the display. Each laser may have a predetermined scanning pattern (e.g., scanning row by row, or scanning odd rows first and continuing with even rows). Therefore, visual artifacts formed on the displayed image may include a vertical line that splits the displayed object down the middle, with each half (e.g., top and bottom halves) independent, separated in the middle, with a diagonal line, a gap, or a slight diagonal line below it. The displayed object may end up with two diagonals separated. Embodiments provide a solution for eliminating books and other visual artifacts using eye tracking data to warp rendered image frames.

[0035] Various embodiments of the present disclosure are directed to systems, methods, and articles of manufacture for improving warping of virtual content for a laser beam scanning display device using data from an eye tracking device. Other objects, features, and advantages of the present disclosure are set forth in the detailed description, drawings, and claims.

[0036] Various embodiments will now be described in detail with reference to the drawings, which are provided as illustrative examples of the present disclosure to enable those skilled in the art to practice the present disclosure. It should be noted that the following figures and examples are not intended to limit the scope of the present disclosure. Where certain elements of the present disclosure can be implemented partially or completely using known components (or methods or processes), only those portions of such known components (or methods or processes) necessary for understanding the present disclosure will be described, and detailed descriptions of other portions of such known components (or methods or processes) will be omitted so as not to obscure the present disclosure. Furthermore, various embodiments encompass present and future known equivalents of the components referenced herein as examples. Illustrative Mixed Reality Scenarios and Systems

[0037] The following description relates to an exemplary augmented reality system with which the eye tracking enhanced warping system may be implemented. However, it should be understood that the embodiments are also suitable for use in other types of display systems (including other types of mixed reality systems), and thus the embodiments are not limited to only the exemplary system disclosed herein.

[0038] Mixed reality (MR) scenarios often involve the presentation of virtual content (e.g., images and sounds) in relation to real-world objects and corresponding virtual objects. For example, referring to FIG. 1 , a user 100 of an MR device 102 (e.g., a wearable component) including a headset sees a real-world physical object 104. According to various embodiments, a virtual object 114 may be rendered on an eyepiece display device (e.g., a laser beam scanning display device) of the MR device 102 relative to the real-world object 104. For example, the real-world object 104 may be in the form of a table, and a virtual fairy (as an exemplary virtual object 114) may be rendered on the display device to appear as if it were placed on the table. The MR device 102 must also consider the user's line of sight (e.g., line of sight) used to generate / render the real-world object 104 and the virtual object 114. For example, the user 100 may move their eyes from a first position providing a first field of view 105 to a second position providing a second field of view 106. Although the position of the real-world object 104 remains the same relative to real-world coordinates, the position of the real-world object 104 shifts within the user's field of view (e.g., the real-world object 104 is closer to the edge of the first field of view 105 and in the center of the second field of view 106). Unless properly handled, the change in the user's pupil position can result in the virtual object 114 appearing rotated or corrupted (e.g., mis-stitched) on the eyepiece display device of the MR device 102. These artifacts on the displayed image frames are discussed in more detail below.

[0039] According to various embodiments, data from the eye tracking device 108 coupled to the MR device 102 may be used to appropriately render the virtual object 114 on the eyepiece display device. Such eye tracking data may be determined, for example, by projecting light onto the end user's eye and detecting a return or reflection of at least a portion of the projected light. The eye tracking device 108 may output a pixel index of the display device where the user's gaze is directed. For example, the eye tracking device 108 may determine a first position of the user's pupil as the center pixel of the display device at time t1 and a second position of the user's pupil as 10 pixels to the right of the center pixel at time t2. According to some embodiments, the MR device 102 may be capable of calculating eye velocity for the user as a function of the first position, the second position, and the difference Δt between time t1 and time t2. According to various embodiments discussed herein, the system may calculate different eye velocities for each unit of the scan pattern (e.g., different eye velocities for each row when scanning row by row).

[0040] For example, the MR device 102 may acquire a first image frame corresponding to a first position of the user's pupil (illustrated by a normal vector 115 to the user's pupil) associated with the first field of view 105. The MR device 102 may receive data from the eye tracking device 108 and use the data from the eye tracking device 108 to determine a second position of the user's pupil (illustrated by a normal vector 116 to the user's pupil) associated with the second field of view 106. The MR device 102 may then generate a second image frame corresponding to the second field of view 106 by performing a shift of one or more sets (e.g., rows) of pixels of the second image frame based on at least the first and second positions of the user's pupil, among other steps. In some embodiments, the second image frame may be generated using head pose-based warping, and the pixels of the second image frame may be shifted as described herein to correct for visual artifacts. Details of the shifts and additional steps that may be performed are described in more detail below.

[0041] 2 is a schematic representation of an exemplary AR / VR headset 200 relative to the eyes of a user of the headset 200. The headset 200 includes a display device 202 (e.g., a laser beam scanning display) that is positioned in front of the user's eyes 204. An eye tracking device coupled to the headset 200 may track the position of the user's eyes 204 (e.g., the user's pupils). The user's line of sight 208 may be directed at a real or virtual object 206.

[0042] According to various embodiments, data from the eye tracking device may be used to correctly render the real or virtual object 206 on the display device 202 based on the user's line of sight or any changes in the user's line of sight. For example, different sets of pixels (e.g., different rows) of a rendered image frame may be shifted by different shift values ​​to generate an image frame for display on a laser beam scanning display device. According to various embodiments, the shift values ​​may be determined based on data provided by the eye tracking device. For example, the MR system may calculate the eye velocity of the user's pupil. A different eye velocity value may be calculated for each shift (e.g., in the case of row-by-row shifts, a different eye velocity may be used for each row).

[0043] FIG. 3 illustrates an exemplary method for transforming image frames based on the position of a user's (eg, viewer's) pupils.

[0044] In step 302, a computing device acquires a first image frame corresponding to a first view associated with a first position of a viewer's pupil. According to various embodiments, the computing device may be coupled to an eyepiece display device including a laser beam scanning display that displays data on pixels of the eyepiece display device according to a predetermined scanning pattern. The data is displayed on a first set of pixels of the eyepiece display device at a different instant in time than a second set of pixels of the eyepiece display device.

[0045] In step 304, the computing device may receive data associated with the viewer's pupil position from an eye tracking device coupled to a headset worn by the viewer.

[0046] In step 306, the computing device may determine a second position of the viewer's pupil based on the data received from the eye tracking device. The second position of the viewer's pupil is associated with the second view. In some embodiments, the position of the viewer's pupil may be determined relative to the viewer's head position. In other embodiments, the position of the viewer's pupil may be determined relative to the headset (e.g., relative to the pixels of the headset's display device).

[0047] In step 308, the computing device may estimate a second head position, also associated with the second view.

[0048] In step 310, the computing device may generate an intermediate warped image frame from the first image frame using the viewer's second head position, whereby the intermediate warped image is generated using head pose-based warping.

[0049] In step 312, the computing device may generate a second image frame corresponding to a second view associated with a second position of the viewer's pupil based on the first image frame (e.g., in some embodiments, using an intermediate warped image frame generated from the first image frame). The second image frame may be generated by shifting a different set of pixels of the second image frame by a different shift value determined based on the second position of the viewer's pupil. For example, the first set of pixels may be shifted by a first shift value, and the second set of pixels may be shifted by a second shift value. The first shift value and the second shift value may be calculated based on at least the second position of the viewer's pupil. For example, the shift value may also be calculated based on the user's eye velocity.

[0050] According to various embodiments, the computing device may calculate an eye velocity for the viewer as a function of a first position of the pupil, a second position of the pupil, and the time elapsed for the viewer's pupil to travel from the first position to the second position. In some embodiments, the eye velocity is calculated relative to the headset (e.g., relative to the pixels of the headset's display device). The computing device may then calculate shift values ​​based on the eye velocity such that a first set of pixels may be shifted by a first shift value and a second set of pixels may be shifted by a second shift value. The number of shift values ​​may depend on the scanning pattern of the MR system. For example, if the MR system scans row-by-row, it may be possible to calculate as many shift values ​​as there are rows in the display device.

[0051] According to various embodiments, the MR system may select a set of pixels according to a scanning pattern. For example, the set of pixels may correspond to a predetermined number of rows of the display device. In some embodiments, the first set of pixels may correspond to a first row of pixels, and the second set of pixels may correspond to a second row of pixels. Because all pixels of the laser beam scanning display device are not activated simultaneously, the displayed image frame exhibits visual artifacts, such as the separation of the top and bottom portions of the exemplary displayed object. In other embodiments, the MR system may first display the odd rows one by one, and then, after the odd rows are completed, display the even rows one by one. The entire display operation can be completed within 6 milliseconds for one MR system to display one image frame. Those skilled in the art will understand that the scanning patterns discussed herein are for illustrative purposes only, and that an MR system can display image frames on a display device using any scanning pattern.

[0052] In step 314, the computing device may transmit the second image frame to the eyepiece display device for display on the eyepiece display device, the displayed image frame being free of visual artifacts such as rotation and mis-stitching.

[0053] The following description provides five example scenarios in which eye-tracking data can be used to correct visual artifacts / anomalies / glitches resulting from a user of an MR device shifting their gaze (e.g., the position of their pupils). The gaze shift can be in addition to a head pose change or can be on its own (e.g., a user changes their gaze without moving their head). According to various embodiments, eye-tracking-based shifting can be performed as a stand-alone correction or in addition to head pose-based warping (e.g., eye-tracking-based shifting can be performed on a head pose-based warped image).

[0054] FIG. 4 illustrates an exemplary manner for splitting image frames for visualization purposes of the VR scenarios illustrated in the adjacent set of numbers, according to various embodiments. For visualization purposes, each 60 Hz frame may be split into six sub-frames. There is a slight overlap between the top and bottom halves of the image. Rendering occurs at 60 Hz, but each section of the display device is flashed twice per rendered frame. The virtual object 400 should appear as a vertical bar, although sometimes this will not be visible to the user due to visual artifacts, as discussed below in connection with each scenario. Example Scenario 1

[0055] In a first example VR scenario, a virtual object (e.g., a virtual fairy) appears in the user's peripheral vision. The user looks at the virtual object and fixes their gaze on it, while the user turns their head / neck to face the virtual object. While the user rotates their head, the virtual object remains stationary and the user's gaze is fixed on the virtual object.

[0056] 5 illustrates a series of drawings (Drawings 1-9) illustrating clockwise head movement relative to a virtual object along with rendered images displayed on a display device at an exemplary rate of 60 fps. No visual correction (e.g., head pose-based warping or eye-tracking-based correction) is applied in the drawings shown in FIG.

[0057] As the headset 200 and display device 202 move clockwise relative to the user's pupils, the user's 208 line of sight remains fixed on the virtual object 206. In the third drawing, a first set of pixels 500 is displayed at a predetermined location (e.g., pixel 1) on the display device 202. As the user continues to move their head without shifting their line of sight, the next sets of pixels 502, 504, 506, 508, 510 (based on the scanning pattern of the MR system) are displayed on the display device, respectively, as shown in drawings 4-8. As shown in these drawings, the pixels are formed outside the user's line of sight, fixed at the center of a circle representing the virtual object on the drawing. Rather, the pixels are formed normal to the display device 202, as opposed to the user's line of sight 208. As a result, the image of the object 512 displayed on the display device 202 appears tilted (e.g., a rotation artifact) and split at the seams (e.g., a mis-stitching artifact). The angle of swivel and the offset of the mis-stitching may be proportional to the angular velocity of the headset. The final figure (i.e., Figure 9) illustrates the final rendering of the image frames in the duration simulation on the display device. Although the average position is correct, the object 512 (illustrated in the duration simulation) appears tilted (swivel) and torn at the seams (mis-stitching) compared to the virtual object 400 illustrated in Figure 4. The angle of swivel and the offset of the mis-stitching are proportional to the angular velocity of the head rotation. This rendering is unacceptable in an MR system.

[0058] Conventional solutions to this problem may warp image frames based on the user's estimated head pose. The MR system may detect and / or predict head pose (e.g., using an inertial measurement unit). The MR system may then warp or transform the rendered virtual content from the source frame of reference to the warped virtual content in the output frame of reference.

[0059] A head pose-based solution for the first scenario is illustrated in FIG. 6. The MR system can understand the head pose in the first figure and then extrapolate the head pose to estimate where the head will be at different times (as illustrated in subsequent figures 2-8). Figures 1-9 include row-by-row warping of the image. For this scenario, head pose estimation with three degrees of freedom may be sufficient to correct visual artifacts. Once the head pose is extrapolated, the system can then warp the rendered image based on the estimated head pose. Head pose-based warping places pixels on the user's line of sight (e.g., all pixels are formed on the line of sight of user 208, illustrated with a normal to the pupil), thereby eliminating visual artifacts. The final figure (i.e., FIG. 9) illustrates the final rendering of the image frame in the duration simulation on a display device. Object 612 (illustrated in the duration simulation) appears the same as virtual object 400 illustrated in FIG. 4. Even with head pose-based warping, rotation and mis-stitching can occur, but they are proportional to the error in the estimated head pose, as opposed to being proportional to the angular velocity of the headset: virtual objects will appear accurate if the head pose is accurate.

[0060] FIG. 7 illustrates a series of figures (e.g., figures 1-8) illustrating an eye-tracking-based solution (as opposed to head pose, depth plane, or pixel motion tracking) for the first scenario, according to various embodiments. The position of the user's pupil may be tracked using a high-sample-rate, robust, high-accuracy eye-tracking device coupled to the MR system. The eye-tracking device may determine the position of the user's pupil at different times when a set (e.g., a row) of pixels will be displayed on the display device. Thus, instead of tying elements of the virtual object to the real world (as is done in conventional warping techniques and / or systems), the eye-tracking-based solution "locks" or "ties" them to the user's pupil or retina. The system may then calculate eye velocity (e.g., the velocity of the user's pupil relative to a headset or display device worn by the user) in x and y pixels each time the system will shift a set of pixels of the virtual object relative to a rendered frame (as illustrated in subsequent figures 2-7). The final drawing (i.e., drawing 8) illustrates the final rendering of the image frames in the duration simulation on a display device. Object 712 appears identical to virtual object 400 illustrated in FIG.

[0061] According to various embodiments, eye velocity may be calculated as a function of a first position of the user's pupil (e.g., as shown in FIG. 2), a second position of the user's pupil (e.g., as shown in FIG. 3), and the time elapsed for the user's pupil to travel from the first position to the second position (e.g., from the position in FIG. 2 to the position in FIG. 3 of FIG. 7). The eye velocity is calculated relative to the headset. For increased accuracy, it may be desirable to calculate the eye velocity as late as possible. For example, the first eye velocity may be calculated just before the first set of pixels are shifted by the first shift value, leaving just enough time for the system to use the first eye velocity to calculate the first shift value.

[0062] In some embodiments, a different eye velocity may be used for each set of pixels (e.g., each row) to be shifted. A first eye velocity may be calculated for the first set of pixels, and a second eye velocity may be calculated for the second set of pixels. For an MR device rendering at 60 Hz, the eye-tracking-based shifting may be performed at 360 Hz.

[0063] In some embodiments, the eye velocity may be zero. For example, a user may not change their line of sight, but simply change their focus (e.g., looking from a distant object to a closer object when both objects are in the same line of sight). In such an embodiment, the rendered image may still be warped or shifted; however, because the eye velocity would be zero, the shift value would be calculated as zero (e.g., no shift would be applied to the rendered image). Example Scenario 2

[0064] In a second example VR scenario, a virtual object (e.g., a virtual gnome) appears on a real-world object (e.g., a desk) in front of the user. The user attempts to view the gnome's surroundings by rocking their body from side to side so that the user can see what is behind the gnome. In such a scenario, the user's head undergoes translation without rotation. The user's gaze remains on the gnome while the user's body and head move from side to side.

[0065] FIG. 8 illustrates a series of drawings (e.g., drawings 1-9) illustrating head movement along with an image rendered on a display device using head pose-based warping to shift the rendered image; only head orientation, not translation, is used. In this exemplary scenario, there is no rotation. As the headset 200 and display device 202 move from left to right relative to the virtual object 206, the user's 208 line of sight remains fixed on the virtual object 206. In the third drawing, a first set of pixels is formed at a predetermined location (e.g., pixel) on the display device 202. As the user continues to move their head without shifting their line of sight, the next set of pixels (based on the scanning pattern of the MR system) is displayed on the display device, as shown in drawings 4-8, respectively. As shown in these drawings, the pixels are formed outside the user's line of sight, fixed at the center of a circle representing the virtual object on the drawing. Rather, the pixels are formed on a normal to the display device 202, as opposed to the user's line of sight 208. Figure 9 illustrates the final rendering of a virtual object on a display device. As shown, the displayed image 812 (illustrated in the duration simulation) contains motion artifacts such as swirling and mis-stitching due to the use of a three-degree-of-freedom head pose (e.g., orientation only). If the system were to attempt to correct the artifacts using head-pose-based warping, which tracks orientation but not translation, the correction would not be sufficient. The system would require head pose (which tracks both orientation and translation) and object depth planes (or vergence-divergence motion depth) in six degrees of freedom.

[0066] FIG. 9 illustrates a series of figures (e.g., figures 1-8) illustrating an eye-tracking-based solution for the second scenario, according to various embodiments. The position of the user's pupil may be tracked using a high-sample-rate, robust, high-accuracy eye-tracking device coupled to the MR system. The eye-tracking device may determine the position of the user's pupil at different times when a set of pixels (e.g., a row) will be displayed on the display device. Thus, instead of tying elements of the eye-tracking-based solution virtual object to the real world (as is done in conventional warping techniques and / or systems), the system "locks" or "ties" them to the user's pupil or retina. The system may then calculate eye velocity (e.g., the velocity of the user's pupil relative to a headset or display device worn by the user) in x and y pixels for each unit (e.g., each row) of the scan pattern to shift the set of pixels (e.g., a row) relative to the rendered frame. The final figure (i.e., figure 8) illustrates the final rendering of the image frame in the duration simulation on the display device. Object 912 appears identical to virtual object 400 shown in Figure 4. As shown in the drawing of Figure 9, all pixels appear in the user's line of sight, eliminating visual artifacts on the displayed image frame (as shown in drawing 8 of Figure 9). Example Scenario 3

[0067] In a third example VR scenario, a virtual object (e.g., a virtual mouse) appears behind a real-world object (e.g., a coffee cup) in front of the user. For example, the mouse has its head peeking out from behind the coffee cup. The mouse runs from the coffee cup to another object in the scene. The user does not move but follows the mouse using their gaze. In such a scenario, the virtual object moves, but the user's head remains stationary. The user's gaze remains on the moving virtual object.

[0068] 10 illustrates a series of drawings (e.g., drawings 1-9) illustrating the movement of a virtual object along with a rendered image rendered on a display device. As the virtual object 206 moves from right to left relative to the headset 200 and display device 202 (as shown in drawings 3-8), the line of sight of the user 208 remains fixed on the virtual object 206. In the third drawing, a first set of pixels 1002 is formed at a predetermined location (e.g., pixel) on the display device 202. As the virtual object continues to move relative to the headset without the user shifting their line of sight from the virtual object, the next sets of pixels 1003, 1004, 1005, 1006, 1007 (based on the scanning pattern of the MR system) are displayed on the display device, as shown in drawings 4-8, respectively. As shown in each of these figures, pixels 1003, 1004, 1005, 1006, and 1007 are each formed outside the user's line of sight 1000, which is fixed at the center of circle 1001, representing the virtual object on the figure. Rather, pixels 1003, 1004, 1005, 1006, and 1007 are formed normal 1010 to the display device 202, as opposed to the user's 208 line of sight 1000. Figure 9 illustrates the final rendering of the virtual object in a duration simulation on the display device. Because the user's head is not moving, conventional head pose-based warping (even with six degrees of freedom instead of three) would not be able to correct for the rotation and mis-stitching on the displayed image 1012, as shown in Figure 9 of Figure 10. That is, because the head pose is identical, the applied head pose-based warping would result in a zero pixel shift for the image. A conventional approach to correct this is to use the head pose plus the motion vectors for the virtual object to perform the warping. Even if a 6-DOF head pose is used with the correct depth plane, the rendered image will still have rotation and mis-stitching.

[0069] FIG. 11 illustrates a series of figures (e.g., figures 1-9) illustrating an eye-tracking-based solution for the third scenario, according to various embodiments. The position of the user's pupil may be tracked using a high-sample-rate, robust, high-accuracy eye-tracking device coupled to the MR system. The eye-tracking device may determine the position of the user's pupil at different times when sets (e.g., rows) 1102, 1103, 1104, 1105, 1106 of pixels would be displayed on the display device (e.g., as illustrated in figures 3-8). Thus, instead of tying elements of the virtual object to the real world (as is done in conventional warping techniques and / or systems), the eye-tracking-based solution "locks" or "ties" them to the user's pupil or retina. The system may then calculate eye velocity (e.g., the velocity of the user's pupil relative to a headset or display device worn by the user) in x and y pixels for each unit (e.g., each row) of the scan pattern to shift that set of pixels (e.g., a row) relative to the rendered frame. The final drawing (i.e., drawing 8) of FIG. 11 illustrates the final rendering of the image frame in the duration simulation on the display device. Object 1112 appears identical to virtual object 400 illustrated in FIG. 4. As illustrated in the drawing of FIG. 11, all pixels appear in the user's line of sight, eliminating visual artifacts on the displayed image frame (as illustrated in drawing 8 of FIG. 11). Example Scenario 4

[0070] The third scenario illustrates how rendering and image display become complicated when a virtual object is in motion. Head pose-based warping is not sufficient to correct for rotation and mis-stitching on the displayed image. The next scenario illustrates that in some cases, head pose-based warping is not only sufficient to correct the artifacts, but also worsens the quality of the displayed image.

[0071] In a fourth example VR scenario, a virtual object (e.g., a virtual fish) moves around the user. The user turns their entire body to follow the virtual object. Both the headset and the user's pupils rotate relative to the real world but are fixed relative to each other.

[0072] FIG. 12 illustrates a series of drawings (Drawings 1-8) illustrating headset and pupil movement, with the remainder remaining fixed relative to one another, along with a rendered image rendered on a display device. Head pose warping is not applied in the drawings illustrated in FIG. 12. Referring to the reference numerals illustrated in FIG. 2, as the headset 200 (including the display device 202) and the user's gaze illustrated in FIG. 12 move counterclockwise, along with the virtual object 206, the line of sight of the user 208 and the normal vector to the display device 202 remain fixed on the virtual object 206. In the third drawing, a first set of pixels 1202 is formed at a predetermined location (e.g., pixel) on the display device 202. As the user continues to move their head without shifting their gaze, the next sets of pixels 1203, 1204, 1205, 1206, 1207 (based on the scanning pattern of the MR system) are displayed on the display device, respectively, as shown in FIGS. 4-8. As shown in these figures, pixels are formed outside the user's line of sight, fixed at the center of a circle representing the virtual object on the drawing. Rather, pixels are formed normal to the display device 202, as opposed to the user's line of sight 208. Drawing 9 of FIG. 12 illustrates the final rendering of the virtual object in a duration simulation on a display device. Object 1212 appears identical to virtual object 400 illustrated in FIG. 4. Because the user is following the moving object with their eyes and their head, the resulting image is sharp, without any swirling or mis-stitching. This result is obtained without applying head pose-based warping.

[0073] On the other hand, for MR systems that apply head pose-based warping to every rendered image frame, exemplary scenario 4 (virtual objects move around the user, the user turns their entire body to follow the virtual objects, and both the headset and the user's pupils rotate relative to the real world but are fixed relative to each other) results in low-quality images. That is, head pose-based warping degrades the quality of the displayed image by introducing swirling and / or mis-stitching, as illustrated in FIG. 13.

[0074] The MR system can figure out the head pose in a first drawing and then extrapolate the head pose to estimate where the head would be at different times (as illustrated in drawings 3-8 of FIG. 13). Once the head pose is extrapolated, the system can then shift one or more sets (e.g., rows) of pixels. Head pose-based warping aims to place pixels in the user's line of sight, but does not consider the movement of virtual objects. Thus, head pose-based warping attempts to lock the virtual object to the world, introducing swirling and mis-stitching onto the displayed image 1312, as illustrated in drawing 9 of FIG. 13.

[0075] FIG. 14 illustrates a series of figures (figures 1-8) illustrating an eye-tracking-based solution for the fourth scenario, according to various embodiments. Because the system calculates eye velocity relative to the headset, when the headset and pupil move simultaneously, the eye velocity is zero. Therefore, the eye-velocity-based shift value is zero. As illustrated in the figures of FIG. 14, pixels appear in the user's line of sight, eliminating visual artifacts on the displayed image frame. Therefore, the result is identical to FIG. 12, even when the eye-tracking-based solution is applied by default. Figure 8 of FIG. 14 illustrates the final rendering of the virtual object in a duration simulation on a display device. Object 1412 appears identical to virtual object 400 illustrated in FIG. 4. Example Scenario 5

[0076] In a fifth exemplary VR scenario, a virtual object (e.g., a virtual robot) is at the center of the user's field of view. However, the user is not looking at the virtual object. Instead, the user may be comparing two other virtual objects or areas next to the virtual object. For example, the user may be looking at area 216 located to the right of the virtual object and area 226 located to the left of the virtual object. That is, the user may be blinking their eyes back and forth between the two areas and across the virtual object while the virtual object and the user's head remain stationary. This eye movement may be referred to as a "saccade." This scenario is similar to the third exemplary VR scenario, except that the eyes move at a faster speed.

[0077] FIG. 15 illustrates a series of drawings illustrating saccadic eye movement along with a rendered image rendered on a display device using head pose-based warping for the rendered image. While the headset 200 and display device 202 remain stationary, the user moves their eyes from left to right relative to areas 216 and 226, and the user's 208 line of sight moves between the two areas. In the first drawing, a first set of pixels 1502 is formed at a predetermined location (e.g., pixel) on the display device 202. As the user continues to move their eyes, the next set of pixels (based on the scanning pattern of the MR system) is displayed on the display device, as shown in drawings 2-10, respectively. As shown in these drawings, the pixels are formed outside the user's line of sight, fixed at the center of a circle representing a virtual object on the drawing. Rather, the pixels are formed on a normal to the display device 202, as opposed to the user's line of sight 208. The final drawing (i.e., drawing 10 of FIG. 15) illustrates the final rendering of the virtual object in the duration simulation on a display device. As shown in drawing 9 of FIG. 15, the displayed image 1512 exhibits rotation and mis-stitching.

[0078] In this scenario, the artifacts formed are asynchronous and transient instead of synchronous and repeatable. Periodic artifacts are seen. They never return to the same spot on the retina as in the other scenarios. Because the user's head is not moving, traditional head pose-based warping would not be able to correct the artifacts on the displayed image (e.g., the head pose is the same, and therefore warping applied based on head pose would result in zero warping). In addition, motion vector-based warping (which could improve on Scenario 3) cannot be applied here because there is no real object and no virtual object that the user's eyes are following.

[0079] FIG. 16 illustrates an eye-tracking-based solution for the fifth scenario (e.g., a saccadic eye between two areas and across a virtual object while the virtual object and the user's head remain stationary, according to various embodiments). The position of the user's pupil may be tracked using a high-sample-rate, robust, high-accuracy eye-tracking device coupled to the MR system. The eye-tracking device may determine the position of the user's pupil at different times when a set of pixels (e.g., a row) will be displayed on the display device. Thus, instead of tying elements of the eye-tracking-based solution virtual object to the real world (as is done in conventional warping techniques and / or systems), the system "locks" or "ties" them to the user's pupil or retina. The system may then calculate eye velocity (e.g., the velocity of the user's pupil relative to a headset or display device worn by the user) in x and y pixels for each unit (e.g., each row) of the scan pattern to shift the set of pixels (e.g., a row) relative to the rendered frame. As illustrated in drawing 10 of Figure 16, multiple clear copies of a virtual object 1610 are rendered, but each copy is undistorted (e.g., swirling and mis-stitching are eliminated, but stroboscopic effects are present). For improved quality, as shown in rendered image 1612, it may be desirable to add digital blur to the rendered image (e.g., using a horizontal boxcar filter).

[0080] Instead of seeing swirling and mis-stitching as the user blinks their eyes around, the user will see a stroboscopic effect. Adding artificial blur would help with this, but for a full 2D blur function, this can be computationally prohibitive.

[0081] In some embodiments, eye-tracking-based shifting may also be applicable to two independent monocular displays. Tracking for each eye may be used to warp the corresponding display. In some embodiments, eye tracking for one eye may be used for both eyes (e.g., when the device loses tracking in one eye).

[0082] According to various embodiments, single-frame time-delay warping may be performed to modify the rendering. Subsequently, eye-tracking-based shifting may be applied to the warped rendered image to shift different sets of pixels by different shift values. Thus, while eye-tracking-based pixel shifting may be implemented as a stand-alone solution, eye-tracking-based pixel shifting may also be performed on rendered image frames following head pose-based warping. Eye-tracking-based pixel shifting may be applied on the headset portion of an MR device.

[0083] Figures 17-21 illustrate additional processing that can be used in conjunction with eye-tracking-based shifting to improve the quality of the displayed image frame. Figure 17 illustrates visual differences that can be present on a rendered image. If the rendered image is displayed without further processing, when a user looks at a seam, the fovea may simultaneously see pixels from above the seam and below the seam. Thus, differences in the seam are easily perceived, especially because they are temporally close. When eye-tracking-based shifting is applied as illustrated in Figure 18, delaying the data sent to the lower laser by one scan time, the differences are corrected. Even if different colors and spoke angles are projected simultaneously, as shown in Figure 19, the user's fovea cannot see both at the same time because they are spatially separated.

[0084] 20 illustrates another example of eye-tracking-based shifting with frame doubling. Rendering 2000 is improved by frame-doubling rendering 2002. According to various embodiments, different rendered frames may be displayed simultaneously. For example, doubling the frame rate from 60 Hz to 120 Hz, as illustrated in rendered frame 2002, may reduce mis-stitching by a factor of 10, resulting in fewer visible tears, shorter latency from motion to image drawing, and require a smaller frame buffer.

[0085] According to various embodiments, mis-stitching may be further reduced by minimizing the flyback time and / or minimizing the number of overlapping rows. Reducing the flyback time and the number of overlapping rows to zero will eliminate the mis-stitching from the displayed image frame, as shown in FIG. 21 . Using conventional systems, a virtual object 2100 may be displayed on a display device with mis-stitching 2102. Reducing the flyback time and the number of overlapping rows to zero will eliminate the mis-stitching from the displayed image frame 2104. System Architecture Overview

[0086] 22 is a block diagram of an exemplary computing system 2200 (e.g., an MR device) according to some embodiments. The computer system 2200 includes a bus 2206 or other communication mechanism for communicating information, interconnecting subsystems and devices such as a processor 2207, a system memory 2208 (e.g., RAM), a static storage device 2209 (e.g., ROM), a disk drive 2210 (e.g., magnetic or optical), a communication interface 2214 (e.g., a modem or Ethernet card), a display 2211 (e.g., CRT or LCD), an input device 2212 (e.g., a keyboard), and cursor control.

[0087] According to some embodiments, computer system 2200 performs specific operations by processor 2207 executing one or more sequences of one or more instructions contained in system memory 2208. Such instructions may be read into system memory 2208 from another computer-readable / usable medium, such as static storage device 2209 or disk drive 2210. In alternative embodiments, hard-wired circuitry may be used in place of or in combination with software instructions to implement the present disclosure. Thus, embodiments are not limited to any specific combination of hardware circuitry and / or software. The term "logic" may refer to any combination of software or hardware used to implement all or part of the present disclosure.

[0088] The terms "non-transitory computer-readable medium" or "computer-usable medium," as used herein, refer to any medium that participates in providing instructions to processor 2207 for execution. Such media may take many forms, including but not limited to, non-volatile media and volatile media. Non-volatile media include, for example, optical or magnetic disks, such as disk drive 2210. Volatile media include dynamic memory, such as system memory 2208.

[0089] Common forms of computer readable media include, for example, a floppy disk, a flexible disk, a hard disk, magnetic tape, any other magnetic medium, a CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with a pattern of holes, RAM, PROM, EPROM, FLASH-EPROM (e.g., NAND flash, NOR flash), any other memory chip or cartridge, or any other medium from which a computer can read.

[0090] In some embodiments, execution of sequences of instructions for practicing the present disclosure is performed by a single computer system 2200. According to some embodiments, two or more computer systems 2200 coupled by a communications link 2215 (e.g., a LAN, PTSN, or wireless network) may cooperate with each other to perform the sequences of instructions required to practice the present disclosure.

[0091] Computer system 2200 may transmit and receive messages, data, and instructions, including programs, e.g., application code, through communications link 2215 and communications interface 2214. Received program code may be executed by processor 507 as it is received, and / or stored in disk drive 2210 or other non-volatile storage for later execution. Database 2232 in storage medium 2231 may be used to store data accessible by system 2200 via data interface 2233.

[0092] The present disclosure includes methods that may be implemented using the subject devices. The methods may include the act of providing such a suitable device. Such provisioning may be performed by a user. In other words, the act of "providing" simply requires the user to obtain, access, approach, locate, configure, activate, power on, or otherwise act to provide the device required in the subject methods. The methods recited herein may occur in any order of the recited events and the recited sequence of events that is logically possible.

[0093] Exemplary aspects of the present disclosure are described above, along with details regarding material selection and manufacturing. As for other details of the present disclosure, these may be understood in connection with the aforementioned referenced patents and publications and are generally known or may be understood by those skilled in the art. The same may be true with respect to the method-based aspects of the present disclosure in terms of additional operations as commonly or logically adopted.

[0094] Additionally, while the present disclosure has been described with reference to several embodiments incorporating various features, the present disclosure is not limited to that described or illustrated, as each variation of the disclosure is discussed. Various modifications may be made to the present disclosure as described, and equivalents (whether recited herein or not included for purposes of brevity to some extent) may be substituted without departing from the spirit and scope of the present disclosure. Additionally, when a range of values ​​is provided, it is understood that all intervening values ​​between the upper and lower limits of that range, and any other stated value or intervening value within the stated range, are encompassed within the present disclosure.

[0095] It is also contemplated that any optional features of the described inventive variations may be set forth and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that plural identical items are present. More specifically, as used in this specification and the claims associated therewith, the singular forms "a," "an," "said," and "the" include plural references unless specifically stated otherwise. In other words, the use of articles allows for "at least one" of the items of the present subject matter in the above description and in the claims associated with this disclosure. Furthermore, it should be noted that such claims may be drafted to exclude any optional element. Accordingly, this language is intended to serve as a predicate for the use of exclusive terminology such as "solely," "only," and the like in connection with the recitation of claim elements, or the use of a "negative" limitation.

[0096] Without the use of such exclusive terminology, the term "comprising" in the claims associated with this disclosure shall be deemed to permit the inclusion of any additional elements, regardless of whether a given number of elements are recited in such claim, or the addition of features may be deemed to change the nature of the elements recited in such claim. Except as specifically defined herein, all technical and scientific terms used herein should be given the broadest possible commonly understood meaning while maintaining claim legitimacy.

[0097] The scope of the present disclosure should not be limited to the examples provided and / or this specification, but rather should be limited only by the scope of the claim language associated with this disclosure.

[0098] In the foregoing specification, the present disclosure has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes may be made therein without departing from the broader spirit and scope of the present disclosure. For example, the foregoing process flows are described with reference to a particular order of process actions. However, the order of many of the described process actions may be changed without affecting the scope or operation of the present disclosure. The specification and drawings are, therefore, to be regarded in an illustrative rather than a restrictive sense.

Claims

1. 1. A method for transforming an image frame based on a position of a viewer's pupil, the method comprising: acquiring, by a computing device, a first image frame, the first image frame corresponding to a first perspective associated with a first position of a pupil of the viewer; determining, by the computing device, a second position of a pupil of the viewer, wherein the first position of the pupil and the second position of the pupil are determined relative to a head position of the viewer, the first position of the pupil being determined at a first time when a first set of pixels is displayed on an eyepiece display device, and the second position of the pupil being determined at a second time when a second set of pixels is displayed on the eyepiece display device; generating, by the computing device, a second image frame corresponding to a second view associated with a second position of a pupil of the viewer based on the first image frame, wherein the generating includes: shifting the first set of pixels of the second image frame by at least a first shift value calculated based on a second position of the viewer's pupil; shifting the second set of pixels of the second image frame by at least a second shift value calculated based on a second position of the viewer's pupil, the first set of pixels being provided in a first row of the eyepiece display device and the second set of pixels being provided in a second row of the eyepiece display device; and transmitting, by the computing device, the second image frame to the eyepiece display device for display on the eyepiece display device; A method comprising:

2. 10. The method of claim 1, further comprising receiving, by the computing device, data associated with a position of a pupil of the viewer from an eye tracking device, wherein a second position of the pupil is determined based on the data received from the eye tracking device.

3. The first perspective is also associated with a first head position of the viewer, and the method further comprises: estimating, by the computing device, a second head position of the viewer; and generating, by the computing device, an intermediate warped image frame from the first image frame using a second head position of the viewer, wherein the second image frame is generated using the intermediate warped image frame; The method of claim 1 , comprising:

4. 2. The method of claim 1, wherein the eyepiece display device is a laser beam scanning display that displays data on pixels of the eyepiece display device according to a predetermined scanning pattern, and the data is displayed on a first set of pixels of the eyepiece display device at a different instant than a second set of pixels of the eyepiece display device.

5. The method of claim 4 , wherein the laser beam scanning display comprises at least two simultaneous laser scans.

6. The method of claim 1 , wherein the pupil position is determined relative to the viewer's head position.

7. The method of claim 1 , wherein the position of the pupil is determined relative to pixels of the eyepiece display device.

8. calculating a first eye velocity and a second eye velocity as a function of a first position of the pupil, a second position of the pupil, and an elapsed time for the viewer's pupil to travel from the first position to the second position, wherein the first eye velocity and the second eye velocity are calculated with respect to the eyepiece display device; calculating, by the computing device, the first shift value based on the first eye velocity and the second shift value based on the second eye velocity; The method of claim 1 further comprising:

9. The method of claim 1 , further comprising calculating, by the computing device, the second shift value based on the first shift value and the second position of the viewer's pupil.

10. 1. A system comprising: one or more processors, acquiring a first image frame, the first image frame corresponding to a first perspective associated with a first position of a viewer's pupil; determining a second position of a pupil of the viewer, wherein the first position of the pupil and the second position of the pupil are determined relative to a head position of the viewer, the first position of the pupil being determined at a first time when a first set of pixels is displayed on a display device, and the second position of the pupil being determined at a second time when a second set of pixels is displayed on the display device; generating a second image frame corresponding to a second perspective associated with a second position of a pupil of the viewer based on the first image frame, the generating including: shifting the first set of pixels of the second image frame by at least a first shift value calculated based on a second position of the viewer's pupil; shifting the second set of pixels of the second image frame by at least a second shift value calculated based on a second position of the viewer's pupil, the first set of pixels being provided in a first row of the display device and the second set of pixels being provided in a second row of the display device; and transmitting the second image frame to the display device for display on the display device; one or more processors configured to execute instructions to perform the the display device configured to display a second image frame; A system comprising:

11. 1. An eye tracking device comprising: tracking the viewer's pupils; transmitting data associated with a position of a pupil of the viewer to the one or more processors, wherein a second position of the pupil is determined based on data received from the eye tracking device; an eye tracking device configured to The system of claim 10 further comprising:

12. The first perspective is also associated with a first head position of the viewer, and the one or more processors further: estimating a second head position of the viewer; and generating an intermediate warped image frame from the first image frame using a second head position of the viewer, the second image frame being generated using the intermediate warped image frame; and The system of claim 10 configured to execute instructions to:

13. 11. The system of claim 10, wherein the display device is a laser beam scanning display that displays data on pixels of the display device according to a predetermined scanning pattern, and the data is displayed on a first set of pixels of the display device at a different moment than a second set of pixels of the display device.

14. The system of claim 13 , wherein the laser beam scanning display includes at least two simultaneous laser scans.

15. The system of claim 10 , wherein the pupil position is determined relative to the viewer's head position.

16. The system of claim 10 , wherein the position of the pupil is determined relative to pixels of the display device.

17. The first perspective is also associated with a first head position of the viewer, and the one or more processors further: calculating a first eye velocity and a second eye velocity as a function of a first position of the pupil, a second position of the pupil, and an elapsed time for the viewer's pupil to travel from the first position to the second position, wherein the first eye velocity and the second eye velocity are calculated relative to the display device; calculating the first shift value based on the first eye velocity and calculating the second shift value based on the second eye velocity; The system of claim 10 configured to execute instructions to:

18. The first perspective is also associated with a first head position of the viewer, and the one or more processors further: The system of claim 10 , configured to execute instructions for calculating the second shift value based on the first shift value and the second position of the viewer's pupil.

19. A non-transitory computer readable medium having sequences of instructions stored thereon that, when executed by one or more processors, cause the processors to: acquiring a first image frame, the first image frame corresponding to a first perspective associated with a first position of a viewer's pupil; determining a second position of a pupil of the viewer, wherein the first position of the pupil and the second position of the pupil are determined relative to a head position of the viewer, the first position of the pupil being determined at a first time when a first set of pixels is displayed on an eyepiece display device, and the second position of the pupil being determined at a second time when a second set of pixels is displayed on the eyepiece display device; generating a second image frame corresponding to a second perspective associated with a second position of a pupil of the viewer based on the first image frame, said generating including: shifting the first set of pixels of the second image frame by at least a first shift value calculated based on a second position of the viewer's pupil; shifting the second set of pixels of the second image frame by at least a second shift value calculated based on a second position of the viewer's pupil, the first set of pixels being provided in a first row of the eyepiece display device and the second set of pixels being provided in a second row of the eyepiece display device; and transmitting the second image frame to an eyepiece display device for display on the eyepiece display device; A non-transitory computer-readable medium for causing

20. The sequences of instructions, when executed by one or more processors, further cause the processors to: calculating a first eye velocity and a second eye velocity as a function of a first position of the pupil, a second position of the pupil, and an elapsed time for the viewer's pupil to travel from the first position to the second position, wherein the first eye velocity and the second eye velocity are calculated with respect to the eyepiece display device; calculating the first shift value based on the first eye velocity and calculating the second shift value based on the second eye velocity; 20. The non-transitory computer-readable medium of claim 19,

Images