Field Subcode Timing in Field Sequential Display

Abstract

To provide a preferable computer-implemented method for warping multi-field color virtual content for sequential projection.SOLUTION: First and second color fields having different first and second colors are obtained. A first time for projection of a warped first color field is determined. A first pose corresponding to the first time is predicted. For each one of the first colors in the first color field, (a) input representing the one of the first colors in the first color field is identified; (b) the input is reconfigured as a series of pulses creating a plurality of pieces of per-field input; and (c) each one in the series of pulses is warped on the basis of the first pose. The warped first color field is generated on the basis of the warped series of pulses. Pixels on a sequential display display the warped first color field.SELECTED DRAWING: Figure 23

Description

Technical Field

[0001] (Cross - Reference to Related Applications) This application claims the benefit of priority of U.S. Provisional Patent Application No. 62 / 702,181, filed on Jul. 23, 2018, entitled “Intra - Field Sub Code Timing In Field Sequential Displays”, which is incorporated herein by reference in its entirety for all purposes as if fully set forth herein.

[0002] This application is related to U.S. Patent Application No. 15 / 924,078, filed on Mar. 16, 2018, entitled “Mixed Reality System with Color Virtual Content Warping and Method of Generating Virtual Content Using Same”, the content of which is incorporated herein by reference in its entirety. (Field of the Invention)

[0003] The present disclosure relates to a field - sequential display system that projects one or more color codes for virtual content over time to different geometric positions and a method of generating mixed - reality experience content using the same.

Background Art

[0004] Modern computing and display technologies have facilitated the development of "Mixed Reality" (MR) systems for so-called "Virtual Reality" (VR) or "Augmented Reality" (AR) experiences, in which digitally reproduced images or portions thereof are presented to a user in a manner that appears to be, or can be perceived as, real. VR scenarios typically involve the presentation of digital or virtual image information without transparency to actual real-world visual input. AR scenarios typically involve the presentation of digital or virtual image information as an augmentation of the visualization of the actual world around the user (i.e., transparency to real-world visual input). Thus, AR scenarios involve the presentation of digital or virtual image information with transparency to real-world visual input.

[0005] MR systems typically generate and display color data, which increases the realism of the MR scenario. Many of these MR systems display color data by sequentially projecting sub-images in different (e.g., primary) colors or "fields" (e.g., red, green, and blue) corresponding to a color image in rapid succession. Projecting the color sub-images at a sufficiently high rate (e.g., 60 Hz, 120 Hz, etc.) can deliver a smooth color MR scenario to the user's memory.

[0006] Various optical systems generate images, including color images, at various depths for displaying MR (VR and AR) scenarios. Some such optical systems are described in U.S. Utility Patent Application No. 14 / 555,585, filed November 27, 2014 (Attorney Docket No. ML.20011.00), the content of which is hereby incorporated by reference in its entirety as if fully and explicitly set forth herein.

[0007] An MR system typically employs a wearable display device (e.g., a head-mounted display, a helmet-mounted display, or smart glasses) that is at least loosely coupled to the user's head and thus moves when the user's head moves. When the user's head movement is detected by the display device, the displayed data can be updated to account for changes in the head pose (i.e., the orientation and / or location of the user's head). The change in position presents the challenge to field sequential display technology.

SUMMARY OF THE INVENTION

MEANS FOR SOLVING THE PROBLEM

[0008] Techniques and technologies for improving the image quality of a field sequential display that receives motion intended to project a still image are described herein.

[0009] As an example, when a user wearing a head-mounted display device views a virtual representation of a virtual object on a display and walks around the perimeter of the area where the virtual object appears, the virtual object can be rendered for each viewpoint and give the user the perception of walking around the perimeter of an object that shares a relationship with real space, as opposed to the relationship with the display surface. However, changes in the user's head pose require adjustment of the timing of the field sequential projector to maintain a still image projection from the dynamic display system.

[0010] Conventional field sequential displays can project in a time series where colors are specified for a single image frame, and any time difference between fields is not noticed when viewed on a static display. For example, a red pixel displayed at a first time and a blue pixel displayed 10 ms later will appear to overlap since the geometric position of the pixels does not change over a discernible amount of time.

[0011] However, in a movable projector such as a head-mounted display, the movement in the same 10 ms interval may correspond to a significant shift in the red and blue pixels that was intended to overlap.

[0012] In some embodiments, warping the color of individual images within a field sequence can improve the perception of the images because each frame will be based on the appropriate perspective of the field at a given time during a change in head pose. Such methods and systems implementing this solution are described in U.S. Patent Application No. 15 / 924,078.

[0013] In addition to the specific field warping that should occur to correct for general head pose changes in a field sequential display, the subcode of a given field itself should also be adjusted to appropriately convey a rich image representing the intended color.

[0014] In one embodiment, a computer-implemented method of warping multi-field color virtual content for sequential projection includes obtaining first and second color fields having different first and second colors. The method also includes determining a first time for projection of the warped first color field. The method further includes predicting a first pose corresponding to the first time. For each one of the first color within the first color field, the method includes (a) identifying an input representing one of the first color within the first color field, (b) reconstructing the input as a series of pulses creating the input per a plurality of fields, and (c) warping each one of the series of pulses based on the first pose. The method also includes generating the warped first color field based on the warped series of pulses. Additionally, the method includes activating pixels on a sequential display and displaying the warped first color field based on the warped series of pulses.

[0015] In one or more embodiments, a series of pulses includes a center pulse centered at a first time, a second pulse occurring before the center pulse, and a third pulse occurring after the center pulse. The end of the decay phase of the second pulse is temporally aligned with the start of the growth phase of the center pulse, and the start of the growth phase of the third pulse is temporally aligned with the end of the decay phase of the center pulse. The center of gravity of the center pulse occurs at the first time, the center of gravity of the second pulse occurs at a second time before the first time, and the center of gravity of the third pulse occurs at a third time after the first time. In some embodiments, the difference between the first time and the second time is equal to the difference between the first time and the third time. In some embodiments, the center pulse includes a set of first time slots each having a first duration, and the second and third pulses each include a set of second time slots each having a second duration that exceeds the first duration. Pixels on a sequential display are activated within a set of the first time slots or within a set of the second time slots. In some embodiments, pixels on a sequential display are activated during the time slots of the center pulse according to a color code associated with one of the colors within a first color field of a first color. In various embodiments, pixels on a sequential display are activated for the time slots in the second pulse and the corresponding time slots in the third pulse.

[0016] In one or more embodiments, the method may also include determining a second time for projection of a warped second color field. The method may further include predicting a second pose corresponding to the second time. For each one of the second colors within the second color field, the method may include (a) identifying an input representing one of the second colors within the second color field, (b) reconstructing the input as a series of pulses creating the input per multiple fields, and (c) warping each one of the series of pulses based on the second pose. The method may also include generating a warped second color field based on the warped series of pulses. Additionally, the method may include activating pixels on a sequential display based on the warped series of pulses and displaying the warped second color field based on the warped series of pulses.

[0017] In another embodiment, a system for warping multi-field color virtual content for sequential projection includes a warping unit for receiving first and second color fields having different first and second colors for sequential projection. The warping unit includes a pose estimator for determining a first time for projection of a warped first color field and predicting a first pose corresponding to the first time. For each one of the first colors within the first color field, the warping unit includes (a) identifying an input representing one of the first colors within the first color field, (b) reconstructing the input as a series of pulses creating the input per multiple fields, and (c) a conversion unit for warping each one of the series of pulses based on the first pose. The conversion unit is further configured to generate a warped first color field based on the warped series of pulses. The conversion unit is also configured to activate pixels on a sequential display based on the warped series of pulses and display the warped first color field.

[0018] In yet another embodiment, a computer program product is embodied in a non-transitory computer-readable medium, the computer-readable medium storing a sequence of instructions that, when executed by a processor, cause the processor to execute a method for warping multi-field color virtual content for sequential projection. The method includes obtaining first and second color fields having different first and second colors. The method also includes determining a first time for projection of the warped first color field. The method further includes predicting a first pose corresponding to the first time. For each one of the first colors within the first color field, the method includes (a) identifying an input representing one of the first colors within the first color field, (b) reconstructing the input as a series of pulses creating the input per field, and (c) warping each one of the series of pulses based on the first pose. The method also includes generating the warped first color field based on the warped series of pulses. Additionally, the method includes activating pixels on a sequential display and displaying the warped first color field based on the warped series of pulses.

[0019] In one embodiment, a computer-implemented method for warping multi-field color virtual content for sequential projection includes obtaining first and second color fields having different first and second colors. The method also includes determining a first time for projection of the warped first color field. The method further includes determining a second time for projection of the warped second color field. Additionally, the method includes predicting a first pose at the first time and predicting a second pose at the second time. In addition, the method includes generating a warped first color field by warping the first color field based on the first pose. The method also includes generating a warped second color field by warping the second color field based on the second pose.

[0020] In one or more embodiments, the first color field includes first color field information at X, Y locations. The first color field information may include a first lightness at the first color. The second color field may include second color field information at X, Y locations. The second color field information may include a second lightness at the second color.

[0021] In one or more embodiments, the warped first color field includes warped first color field information at first warped X, Y locations. The warped second color field may include warped second color field information at second warped X, Y locations. Warping the first color field based on the first pose may include applying a first transformation to the first color field to generate the warped first color field. Warping the second color field based on the second pose may include applying a second transformation to the second color field to generate the warped second color field.

[0022] In one or more embodiments, the method includes sequentially sending the warped first and second color fields to a projector, and the sequential projector sequentially projecting the warped first color field and the warped second color field. The warped first color field may be projected at a first time, and the warped second color field may be projected at a second time.

[0023] In another embodiment, a system for warping multi-field color virtual content for sequential projection includes a warping unit for receiving first and second color fields having different first and second colors for sequential projection. The warping unit includes a pose estimator for determining first and second times for projection of the respective warped first and second color fields and predicting first and second poses at the respective first and second times. The warping unit also includes a conversion unit for generating the warped first and second color fields by warping the respective first and second color fields based on the respective first and second poses.

[0024] In yet another embodiment, a computer program product is embodied in a non-transitory computer-readable medium, the computer-readable medium storing a sequence of instructions that, when executed by a processor, cause the processor to execute a method for warping multi-field color virtual content for sequential projection. The method includes obtaining first and second color fields having different first and second colors. The method also includes determining a first time for projection of the warped first color field. The method further includes determining a second time for projection of the warped second color field. Additionally, the method includes predicting a first pose at the first time and predicting a second pose at the second time. In addition, the method includes generating a warped first color field by warping the first color field based on the first pose. The method also includes generating a warped second color field by warping the second color field based on the second pose.

[0025] In yet another embodiment, a computer-implemented method for warping multi-field color virtual content for sequential projection includes obtaining an application frame and an application pose. The method also includes estimating a first pose for a first warping of the application frame at a first estimated display time. The method further includes using the application pose and the estimated first pose to perform a first warping of the application frame and generate a first warped frame. Additionally, the method includes estimating a second pose for a second warping of the first warped frame at a second estimated display time. In addition, the method includes using the estimated second pose to perform a second warping of the first warping frame and generate a second warped frame.

[0026] In one or more embodiments, the method includes displaying a second warped frame at approximately a second estimated display time. The method may also include estimating a third pose for a third warping of a first warped frame at a third estimated display time and performing the third warping of the first warping frame using the estimated third pose to generate a third warped frame. The third estimated display time may be after the second estimated display time. The method may also include displaying the third warped frame at approximately the third estimated display time.

[0027] In another embodiment, a computer-implemented method for minimizing color break-up ("CBU") artifacts includes predicting CBU artifacts based on received eye or head tracking information. The method also includes increasing a color field rate based on the predicted CBU artifacts.

[0028] In one or more embodiments, the method includes predicting a second CBU based on received eye or head tracking information and an increased color field rate, and decreasing a bit depth based on the predicted second CBU artifacts. The method may also include displaying an image using the increased color field rate and the decreased bit depth. The method may further include displaying an image using the increased color field rate.

[0029] Additional and other objects, features, and advantages of the present disclosure are set forth in the detailed description, figures, and claims for carrying out the invention. This specification also provides, for example, the following items. (Item 1) A computer-implemented method for warping multi-field color virtual content for sequential projection, the method comprising: obtaining first and second color fields having different first and second colors; Determining a first time for projection of a warped first color field; Predicting a first pose corresponding to the first time; For each one of the first colors within the first color field, Identifying an input representing the one color among the first colors within the first color field; Reconstructing the input as a series of pulses creating an input per plurality of fields; Warping each one of the series of pulses based on the first pose; Performing; Generating the warped first color field based on the warped series of pulses; Activating pixels on a display sequentially based on the warped series of pulses and displaying the warped first color field; A method comprising. (Item 2) The method according to item 1, wherein the series of pulses includes a center pulse centered at the first time, a second pulse occurring before the center pulse, and a third pulse occurring after the center pulse. (Item 3) The end of the decay phase of the second pulse is temporally aligned with the start of the growth phase of the center pulse, The method according to item 2, wherein the start of the growth phase of the third pulse is temporally aligned with the end of the decay phase of the center pulse. (Item 4) The method according to item 2, wherein the center of gravity of the center pulse occurs at the first time, the center of gravity of the second pulse occurs at a second time before the first time, and the center of gravity of the third pulse occurs at a third time after the first time. (Item 5) The method according to item 4, wherein the difference between the first time and the second time is equal to the difference between the first time and the third time. (Item 6) The central pulse includes a set of first time slots each having a first duration, and the second pulse and the third pulse each include a set of second time slots each having a second duration that exceeds the first duration, the method according to item 2. (Item 7) The pixels on the sequential display are activated within a part of the set of first time slots or the set of second time slots, the method according to item 6. (Item 8) The pixels on the sequential display are activated during the time slots of the central pulse according to a color code associated with one of the colors in the first color within the first color field, the method according to item 7. (Item 9) The pixels on the sequential display are activated for the time slots in the second pulse and the corresponding time slots in the third pulse, the method according to item 7. (Item 10) Determining a second time for projection of the warped second color field; Predicting a second pose corresponding to the second time; For each one of the second colors within the second color field, Identifying an input representing the one color among the second colors within the second color field; Reconstructing the input as a series of pulses creating an input per plurality of fields; Warping each one of the series of pulses based on the second pose; And performing; Generating the warped second color field based on the warped series of pulses; Activating pixels on a sequential display based on the warped series of pulses and displaying the warped second color field; Further including the method according to item 1. (Item 11) A system for warping multi-field color virtual content for sequential projection, the system comprising a warping unit for receiving first and second color fields having different first and second colors, The warping unit, a pose estimator for determining a first time for projection of the warped first color field and predicting a first pose corresponding to the first time, a conversion unit and The conversion unit, for each one of the first colors in the first color field, identifying an input representing the one color among the first colors in the first color field, reconstructing the input as a series of pulses creating an input per plurality of fields, warping each one of the series of pulses based on the first pose, and generating the warped first color field based on the warped series of pulses, activating pixels on a sequential display based on the warped series of pulses and displaying the warped first color field A system that performs. (Item 12) The series of pulses includes a center pulse centered at the first time, a second pulse occurring before the center pulse, and a third pulse occurring after the center pulse, the system according to item 11. (Item 13) The end of the decay phase of the second pulse is temporally aligned with the start of the growth phase of the center pulse, The start of the growth phase of the third pulse is temporally aligned with the end of the decay phase of the center pulse, the system according to item 12. (Item 14) The center of gravity of the center pulse occurs at the first time, the center of gravity of the second pulse occurs at a second time before the first time, and the center of gravity of the third pulse occurs at a third time after the first time, the system according to item 12. (Item 15) The center pulse includes a set of first time slots each having a first duration, and the second pulse and the third pulse each include a set of second time slots each having a second duration exceeding the first duration, the system according to item 12. (Item 16) The pixels on the sequential display are activated within a part of the set of the first time slots or the set of the second time slots, the system according to item 15. (Item 17) The pixels on the sequential display are activated during the time slots of the center pulse according to a color code associated with one of the colors in the first color within the first color field, the system according to item 16. (Item 18) The pixels on the sequential display are activated for the time slots in the second pulse and the corresponding time slots in the third pulse, the system according to item 16. (Item 19) The posture estimator is configured to determine a second time for the projection of the warped second color field and predict a second posture corresponding to the second time, The conversion unit, For each one of the second colors within the second color field, Identifying an input representing the one color among the second colors within the second color field, Reconstructing the input as a series of pulses creating an input per a plurality of fields, Warping each one of the series of pulses based on the second posture, And performing Based on the warped series of pulses, generating the warped second color field; Based on the warped series of pulses, sequentially activating pixels on a display and displaying the warped second color field; The system according to item 11, further configured to perform the above. (Item 20) A computer program product embodied in a non-transitory computer-readable medium, the non-transitory computer-readable medium storing a series of instructions which, when executed by a processor, cause the processor to execute a method for warping multi-field color virtual content for sequential projection, The method includes: Obtaining first and second color fields having different first and second colors; Determining a first time for projection of the warped first color field; Predicting a first pose corresponding to the first time; For each one of the first colors in the first color field, Identifying an input representing the one color among the first colors in the first color field; Reconstructing the input as a series of pulses for creating an input per multiple fields; Warping each one of the series of pulses based on the first pose; And performing the above; Based on the warped series of pulses, generating the warped first color field; Based on the warped series of pulses, sequentially activating pixels on a display and displaying the warped first color field; The computer program product including the above.

Brief Description of Drawings

[0030] The drawings illustrate the design and usability of various embodiments of the present disclosure. Note that the figures are not drawn to exact scale and elements of similar structure or function are represented by like reference numerals throughout the figures. To gain a deeper understanding of the advantages and other advantages and objects listed above on various embodiments of the present disclosure and the manner of obtaining them, the forms for carrying out the invention of the present disclosure briefly described above will be given by referring to the specific embodiments illustrated in the accompanying drawings. It should be understood that these drawings depict only typical embodiments of the present disclosure and are, therefore, not to be considered as limiting its scope. The present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0031]

Figure 1

[0032]

Figure 2A

Figure 2B

Figure 2C

[0033]

Figure 3

Figure 4

[0034]

Figure 5

[0035]

Figure 6

[0036]

Figure 7A

Figure 7B

[0037]

Figure 8

[0038]

Figure 9A

Figure 9B

[0039]

Figure 10

[0040]

Figure 11

[0041]

Figure 12

[0042]

Figure 13

[0043]

Figure 14

[0044]

Figure 15

[0045]

Figure 16

[0046]

Figure 17

[0047]

Figure 18A

[0048]

Figure 18B

[0049]

Figure 19

[0050]

Figure 20

[0051]

Figure 21

[0052]

Figure 22

[0053]

Figure 23

[0054]

Figure 24

[0055]

Figure 25

[0056] Various embodiments of the present disclosure are directed to systems, methods, and articles of manufacture for warping virtual content from a source in a single embodiment or multiple embodiments. Other objects, features, and advantages of the present disclosure are set forth in the detailed description, figures, and claims.

[0057] Various embodiments will now be described in detail with reference to the drawings and will be provided as illustrative examples of the present disclosure to enable those skilled in the art to practice the present disclosure. It should be noted that the following figures and examples are not meant to limit the scope of the present disclosure. If an element of the present disclosure can be implemented partially or fully using known components (or methods or processes), only those parts of such known components (or methods or processes) necessary for understanding the present disclosure will be described, and detailed descriptions of other parts of such known components (or methods or processes) will be omitted so as not to obscure the present disclosure. Further, various embodiments include current and future known equivalents of the components referenced herein as examples.

[0058] The virtual content warping system can be implemented independently of the mixed reality system, but some of the following embodiments are described in relation to an AR system for illustrative purposes only. Further, the virtual content warping system described herein can also be used in the same manner in a VR system. (Illustrative Mixed Reality Scenarios and Systems)

[0059] The following description relates to an illustrative augmented reality system in which the warping system can be practiced. However, it should be understood that the embodiments are also suitable for use in other types of display systems (including other types of mixed reality systems) and thus the embodiments are not limited to only the illustrative systems disclosed herein.

[0060] Mixed reality (e.g., VR or AR) scenarios often involve presenting virtual content (e.g., color images and sounds) corresponding to virtual objects in relation to real-world objects. For example, referring to FIG. 1, an augmented reality (AR) scene 100 is depicted, and a user of AR technology is viewing a physical park-like setting 102 in the real world featuring people, trees, buildings in the background, and a physical concrete platform 104 in the real world. In addition to these items, the user of AR technology also "sees" a virtual robot image 106 standing on the physical concrete platform 104 and a flying virtual comic-like avatar character 108 that appears anthropomorphic like a bumblebee, but these virtual objects 106, 108 do not exist in the real world.

[0061] Similar to AR scenarios, VR scenarios must also consider the pose used to generate / render virtual content. Accurately warping virtual content with respect to a reference AR / VR display frame and warping the warped virtual content can improve the AR / VR scenario or at least not degrade the AR / VR scenario.

[0062] The following description relates to an exemplary AR system in which the present disclosure may be practiced. However, it should be understood that the present disclosure is also suitable for use in other types of augmented reality and virtual reality systems in itself, and thus the present disclosure is not limited to only the exemplary systems disclosed herein.

[0063] Referring to FIG. 2A, an embodiment of an AR system 200 according to some embodiments is illustrated. The AR system 200 can be operated with a projection subsystem 208 to provide an image of a virtual object mixed with a physical object within the field of view of a user 250. This approach employs one or more at least partially transparent surfaces through which the surrounding environment including the physical object can be seen and through which the AR system 200 generates an image of the virtual object. The projection subsystem 208 is stored within a control subsystem 201, and the control subsystem 201 is operably coupled to a display system / subsystem 204 through a link 207. The link 207 can be a wired or wireless communication link.

[0064] Regarding AR applications, it may be desirable to spatially position various virtual objects within the field of view of the user 250 relative to respective physical objects. The virtual object can take any of a variety of forms having any of a variety of data, information, concepts, or logical structures that can be represented as an image. Non-limiting examples of virtual objects can include virtual text objects, virtual numerical objects, virtual alphanumeric objects, virtual tag objects, virtual field objects, virtual chart objects, virtual map objects, virtual instrument objects, or virtual visual representations of physical objects.

[0065] The AR system 200 includes a frame structure 202 worn by the user 250, a display system 204 supported by the frame structure 202 such that the display system 204 is positioned in front of the eyes of the user 250, and a speaker 206 incorporated in or connected to the display system 204. In the illustrated embodiment, the speaker 206 is supported by the frame structure 202 (e.g., earbuds or headphones) such that the speaker 206 is positioned adjacent to the ear canal (inside or around) of the user 250.

[0066] The display system 204 is designed to present a light-based radiation pattern that can be comfortably perceived by the user 250's eyes as an augmentation to the surrounding environment that includes both two-dimensional and three-dimensional content. The display system 204 presents a sequence of frames at a high frequency, which provides the perception of a single coherent scene. To achieve this purpose, the display system 204 includes a projection subsystem 208 and a partially transparent display screen, through which the projection subsystem 208 projects an image. The display screen is positioned within the field of view of the user 250 between the user 250's eyes and the surrounding environment.

[0067] In some embodiments, the projection subsystem 208 takes the form of a scanning-based projection device, the display screen takes the form of a waveguide-based display, and within the display, scanned light from the projection subsystem 208 is input, for example, to generate an image at a single optical viewing distance closer than infinity (e.g., arm's length), images at a plurality of separate optical viewing distances or focal planes, and / or image layers stacked at a plurality of viewing distances or focal planes for representing a three-dimensional (3D) object. These layers within the clear aperture can be stacked closely enough together to appear continuous to the human peripheral visual system (e.g., one layer is within the cone of confusion of an adjacent layer). Additionally, or alternatively, the photographic elements can be blended across two or more layers, and even when those layers are stacked more sparsely (e.g., one layer is outside the cone of confusion of an adjacent layer), the perceived continuity of the transition between layers within the clear aperture can be increased. The display system 204 can be monocular or binocular. The scanning assembly includes one or more light sources that generate a light beam (e.g., emit light of different colors in a defined pattern). The light sources can take any of a variety of forms, such as a set of RGB sources (e.g., laser diodes capable of outputting red, green, and blue light), and the set of RGB sources is operable to generate coherent collimated light of red, green, and blue, respectively, according to a defined pixel pattern specified within each frame of pixel information or data. Laser light provides high color saturation and is highly energy efficient. The optical coupling subsystem includes an optical waveguide input device (e.g., one or more reflective surfaces, diffraction gratings, mirrors, dichroic mirrors, or prisms, etc.), and the optical waveguide input device optically couples light into the end of the display screen. The optical coupling subsystem further includes a collimation element that collimates light from an optical fiber. Optionally, the optical coupling subsystem includes an optical modulation device that is configured to converge the light from the collimation element toward a focus at the center of the optical waveguide input device, thereby enabling the size of the optical waveguide input device to be minimized.Accordingly, display subsystem 204 generates a series of composite image frames of pixel information that present to the user an undistorted image of one or more virtual objects. Display subsystem 204 may also generate a series of color composite sub-image frames of pixel information that present to the user an undistorted color image of one or more virtual objects. Further details explaining the display subsystem are provided in U.S. Patent Application No. 14 / 212,961, entitled "Display System and Method" (Attorney Docket No. ML.20006.00) and No. 14 / 331,218, entitled "Planar Waveguide Apparatus With Diffraction Element(s) and Subsystem Employing Same" (Attorney Docket No. ML.20020.00), the contents of which are hereby incorporated by reference in their entirety as if fully and explicitly set forth herein.

[0068] AR system 200 further includes one or more sensors mounted on frame structure 202 to detect the position (including orientation) and movement of user 250's head and / or the position of user 250's eyes and the inter-ocular distance. Such sensors may include image capture devices, microphones, inertial measurement units (IMUs), accelerometers, compasses, GPS units, wireless devices, gyroscopes, and the like. For example, in one embodiment, AR system 200 includes a head-mounted transducer subsystem that includes one or more inertial transducers for capturing inertial measurements indicative of the movement of user 250's head. Such devices may be used to sense, measure, or collect information about the movement of user 250's head. For example, these devices may be used to detect / measure the movement, speed, acceleration, and / or position of user 250's head. The position (including orientation) of user 250's head is also known as the "head pose" of user 250.

[0069] The AR system 200 of FIG. 2A may include one or more forward-facing cameras. The cameras may be employed for any number of purposes, such as recording images / videos from the forward direction of the system 200. Additionally, the cameras may be used to capture information about the environment in which the user 250 is located, such as information indicating the distance, orientation, and / or angular position of the user 250 relative to the environment and specific objects within the environment.

[0070] The AR system 200 may further include a rear-facing camera for tracking the angular position of the user 250's eyes (the direction in which one or both eyes are facing), blinking, and depth of focus (by detecting eye convergence). Such eye tracking information may be determined, for example, by projecting light onto the end user's eyes and detecting at least a portion of the return or reflection of the projected light.

[0071] The augmented reality system 200 further includes a control subsystem 201 that can take any of a variety of forms. The control subsystem 201 includes several controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers such as application-specific integrated circuits (ASICs), programmable gate arrays (PGAs), such as field PGAs (FPGAs), and / or programmable logic controllers (PLUs). The control subsystem 201 may include a digital signal processor (DSP), a central processing unit (CPU) 251, a graphics processing unit (GPU) 252, and one or more frame buffers 254. While the CPU 251 controls the overall operation of the system, the GPU 252 renders the frames (i.e., converts a three-dimensional scene into a two-dimensional image) and stores these frames in the frame buffer 254. Although not shown, one or more additional integrated circuits may control the reading of frames into and / or the reading of frames from the frame buffer 254, and the operation of the display system 204. The reading into and / or from the frame buffer 254 may employ dynamic addressing, for example, the frames are over-rendered. The control subsystem 201 further includes a read-only memory (ROM) and a random access memory (RAM). The control subsystem 201 further includes a three-dimensional database 260, from which the GPU 252 can access the three-dimensional data of one or more scenes for rendering the frames, and the synthetic sound data associated with the virtual sound sources included in the three-dimensional scene.

[0072] The augmented reality system 200 further includes a user orientation detection module 248. The user orientation module 248 can detect the instantaneous position of the head of the user 250 and predict the position of the head of the user 250 based on the position data received from the sensor. The user orientation module 248 also tracks the eyes of the user 250, particularly the direction and / or distance on which the user 250 is focused, based on the tracking data received from the sensor.

[0073] Figure 2B depicts an AR system 200’ according to some embodiments. The AR system 200’ depicted in Figure 2B is similar to the AR system 200 depicted and described above in Figure 2A. For example, the AR system 200’ includes a frame structure 202, a display system 204, a speaker 206, and a control subsystem 201’ operably coupled to the display subsystem 204 through a link 207. The control subsystem 201’ depicted in Figure 2B is similar to the control subsystem 201 depicted and described above in Figure 2A. For example, the control subsystem 201’ includes a projection subsystem 208, an image / video database 271, a user orientation module 248, a CPU 251, a GPU 252, a 3D database 260, a ROM, and a RAM.

[0074] The difference between the control subsystem 201’ and thus the AR system 200’ depicted in Figure 2B and the corresponding system / system component depicted in Figure 2A is the presence of a warping unit 280 within the control subsystem 201’ depicted in Figure 2B. The warping unit 280 is a separate warping block independent of the GPU 252 or the CPU 251. In other embodiments, the warping unit 280 can be a component within a separate warping block. In some embodiments, the warping unit 280 can be inside the GPU 252. In some embodiments, the warping unit 280 can be inside the CPU 251. Figure 2C shows that the warping unit 280 includes a pose estimator 282 and a conversion unit 284.

[0075] The various processing components of the AR systems 200, 200' can be included within a distributed subsystem. For example, the AR systems 200, 200' include a local processing and data module (i.e., control subsystems 201, 201') that is operably coupled to a part of the display system 204 by means of a wired conductor or wireless connectivity 207, etc. The local processing and data module can be mounted in various configurations, such as a configuration fixedly attached to the frame structure 202, a configuration fixedly attached to a helmet or hat, a configuration built into headphones, a configuration removably attached to the user 250's torso, or a configuration removably attached to the user 250's waist in a belt-coupled configuration, etc. The AR systems 200, 200' can further include a remote processing module and a remote data repository that are operably coupled to the local processing and data module by means of a wired conductor or wireless connectivity, etc., whereby the remote modules are operably coupled to each other and are available as resources for the local processing and data module. The local processing and data module can include a power-efficient processor or controller and digital memory such as flash memory, both of which can be used to assist in the processing, caching, and storing of data captured from sensors and / or data obtained and / or processed using the remote processing module and / or the remote data repository, possibly for passage to the display system 204 after processing or reading. The remote processing module can include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. The remote data repository can include a relatively large-scale digital data storage facility, which can be available through other networking configurations in an Internet or "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed within the local processing and data module, enabling fully autonomous use from any remote module.The couplings between the various components described above can include one or more wired interfaces or ports to provide wire or optical communication, or one or more wireless interfaces or ports via RF, microwave, IR, etc. to provide wireless communication. In some implementations, all communication can be wired, while in other implementations, all communication can be wireless except for optical fiber. (Summary of Problems and Solutions)

[0076] When an optical system generates / renders color virtual content, it can use a source reference system, which can be related to the pose of the system when the virtual content is rendered. In an AR system, the virtual content to be rendered can have a predetermined relationship with the actual physical object. For example, FIG. 3 illustrates an AR scenario 300 that includes a virtual flowerpot 310 positioned on top of an actual physical table 312. The AR system renders the virtual flowerpot 310 based on a source reference system in which the location of the actual table 312 is known, such that the virtual flowerpot 310 appears to be placed on top of the actual table 312. The AR system can use the source reference system to render the virtual flowerpot 310 at a first time and display / project the rendered virtual flowerpot 310 in an output reference system at a second time after the first time. If the source reference system and the output reference system are the same, the virtual flowerpot 310 will appear in the intended location (e.g., on top of the actual physical table 312).

[0077] However, if the reference system of the AR system changes in the gap between the first time when the virtual flower pot 310 is rendered and the second time when the rendered virtual flower pot 310 is displayed / projected (e.g., along with rapid user head movement), the misalignment / difference between the source reference system and the output reference system can result in visual artifacts / anomalies / glitches. For example, FIG. 4 shows an AR scenario 400 that includes a virtual flower pot 410 rendered to be positioned on top of an actual physical stand 412. However, since the AR system is rapidly moved to the right after the virtual flower pot 410 is rendered but before it is displayed / projected, the virtual flower pot 410 is displayed to the right of its intended position 410’ (indicated by the imaginary line). Thus, the virtual flower pot 410 appears to float in mid-air to the right of the actual physical stand 412. This artifact will be corrected when the virtual flower pot is re-rendered in the output reference system (assuming the AR system movement stops). However, the artifact will still be visible to some users, and the virtual flower pot 410 will appear as a glitch by temporarily jumping to an unexpected position. This glitch and other such ones can negatively affect the illusion of continuity of the AR scenario.

[0078] Some optical systems may include a warping system that warps or transforms the reference system of the source virtual content from the source reference system in which the virtual content is generated to the output reference system in which the virtual content will be displayed. In the example depicted in FIG. 4, the AR system can detect and / or predict the output reference system and / or orientation (e.g., using an IMU or eye tracking). The AR system can then warp or transform the rendered virtual content from the source reference system into the warped virtual content within the output reference system. (Color Virtual Content Warping System and Method)

[0079] FIG. 5 schematically illustrates the warping of virtual content according to some embodiments. Source virtual content 512 in a source reference system (rendering pose) represented by ray 510 is warped to warped virtual content 512' in an output reference system (estimated pose) represented by ray 510'. The warping depicted in FIG. 5 may represent a head rotation to the right 520. The source virtual content 512 is located at source X, Y locations, while the warped virtual content 512' is transformed to output X', Y' locations.

[0080] FIG. 6 depicts a method of warping virtual content according to some embodiments. In step 612, a warping unit 280 receives virtual content, a base pose (i.e., the current pose of AR systems 200, 200' (current reference system)), a rendering pose (i.e., the pose of AR systems 200, 200' used to render the virtual content (source reference system)), and an estimated illumination time (i.e., the estimated time at which display system 204 will be illuminated (estimated output reference system)). In some embodiments, the base pose may be newer / more recent / most recent than the rendering pose. In step 614, a pose estimator 282 uses the base pose and information about AR systems 200, 200' to estimate the pose at the estimated illumination time. In step 616, a conversion unit 284 uses the estimated pose (from the estimated illumination time) and the rendering pose to generate warped virtual content from the received virtual content.

[0081] When virtual content includes color, some warping systems warp all of the color sub-images or fields corresponding to / forming a color image using a single X', Y' location in a single output reference system (e.g., a single estimated pose from a single estimated illumination time). However, some projection display systems (e.g., sequential projection display systems) do not project all of the color sub-images / fields simultaneously, as in some AR systems. For example, there may be some latency during the projection of each color sub-image / field. This latency during the projection of each color sub-image / field, which is a difference in illumination time, can result in color interference fringe artifacts in the final image during fast head movement.

[0082] For example, FIG. 7A schematically illustrates the warping of color virtual content using some warping systems according to some embodiments. Source virtual content 712 has three color sections, namely, a red section 712R, a green section 712G, and a blue section 712B. In this example, each color section corresponds to a color sub-image / field 712R'', 712G'', 712B''. Some warping systems use a single output reference system (e.g., an estimated pose) represented by a light ray 710'' (e.g., a reference system 710'' corresponding to the green sub-image and its illumination time t1) to warp all three color sub-images 712R'', 712G'', 712B''. However, some projection systems do not project the color sub-images 712R'', 712G'', 712B'' simultaneously. Instead, the color sub-images 712R'', 712G'', 712B'' are projected at three slightly different times (represented by light rays 710', 710'', 710''' at times t0, t1, and t2). The size of the delay during the projection of the sub-images may depend on the frame / refresh rate of the projection system. For example, if the projection system has a frame rate of 60 Hz or less (e.g., 30 Hz), the delay may result in color interference fringe artifacts with fast-moving viewers or objects.

[0083] FIG. 7B illustrates color interference fringe artifacts generated by a virtual content warping system / method similar to that depicted in FIG. 7A according to some embodiments. The red sub-image 712R’’ is warped using the output reference system (e.g., estimated pose) represented by the ray 710’’ in FIG. 7A, but is projected at time t0 represented by the ray 710’, so the red sub-image 712R’’ appears to over-warp the intended warping. This over-warp amount appears as the right interference fringe image 712R’’ in FIG. 7B. The green sub-image 712G’’ is warped using the output reference system (e.g., estimated pose) represented by the ray 710’’ in FIG. 7A and is projected at time t1 represented by the ray 710’’, so the green sub-image 712G’’ is projected using the intended warping. This is represented by the central image 712G’’ in FIG. 7B. The blue sub-image 712B’’ is warped using the output reference system (e.g., estimated pose) represented by the ray 710’’ in FIG. 7A, but is projected at time t2 represented by the ray 710’’’, so the blue sub-image 712B’’ appears to not reach the intended warping. This negative over-warp amount appears as the left interference fringe image 712B’’ in FIG. 7B. FIG. 7B illustrates the reconstruction in the user's perception of warped virtual content including a body having three overlapping R, G, B color fields (i.e., a body rendered in color). FIG. 7B includes a red right interference fringe image color break (``CBU'') artifact 712R'', a central image 712G'', and a blue left interference fringe image CBU artifact 712B''.

[0084] Figure 7B exaggerates the overshoot and negative overshoot effects for illustrative purposes. The size of these effects depends on the frame / field rate of the projection system and the relative speed of the virtual content and the output reference system (e.g., the estimated pose). When these overshoot and negative overshoot effects are smaller, they can appear as color / rainbow interference fringes. For example, at a sufficiently low frame rate, a white virtual object such as a baseball can have color (e.g., red, green, and / or blue) interference fringes. Instead of having interference fringes, a virtual object with a series of solid colors matching the sub-images (e.g., red, green, and / or blue) can cause glitches (i.e., during fast movement, it jumps to an unexpected position and then appears to jump back to the expected position after fast movement). Such a solid color virtual object can also appear to vibrate during fast movement.

[0085] To address these and other limitations, the systems described herein warp color virtual content using several reference systems corresponding to the number of color sub-images / fields. For example, FIG. 8 depicts a method of warping color virtual content according to several embodiments. In step 812, a warping unit 280 receives virtual content, the base poses of the AR systems 200, 200' (i.e., the current pose (current reference system), the rendering pose (i.e., the pose of the AR systems 200, 200' used to render the virtual content (source reference system)), and the estimated illumination time for each sub-image / color field (R, G, B) (i.e., the estimated time that the display system 204 is illuminated for each sub-image related to the display system 204 (the estimated output reference system of each sub-image)). In step 814, the warping unit 280 divides the virtual content into each sub-image / color field (R, G, B).

[0086] In steps 816R, 816G, and 816B, the pose estimator 282 estimates the pose at each respective estimated illumination time for the R, G, B sub-images / fields using the base pose (e.g., the current reference system) and information about the AR systems 200, 200'. In steps 818R, 818G, and 818B, the conversion unit 284 uses the respective estimated R, G, and B poses and the rendering pose (e.g., the source reference system) to generate the warped virtual content for R, G, and B from the received virtual content sub-images / color fields (R, G, B). In step 820, the conversion unit 284 combines the warped R, G, B sub-images / fields for sequential display.

[0087] FIG. 9A schematically illustrates the warping of color virtual content using a warping system according to some embodiments. The source virtual content 912 is the same as the source virtual content 712 in FIG. 7A. The source virtual content 912 has three color sections, namely, a red section 912R, a green section 912G, and a blue section 912B. Each color section corresponds to a color sub-image / field 912R', 912G'', 912B'''. The warping system according to the embodiments herein warps each corresponding color sub-image / field 912R', 912G'', 912B''' using the respective output reference systems (e.g., the estimated poses) represented by the light rays 910', 910'', 910'''. These warping systems consider the timing of the projection (i.e., t0, t1, t2) of the color sub-images 912R', 912G'', 912B''' when warping the color virtual content. The timing of the projection depends on the frame / field rate of the projection system used to calculate the timing of the projection.

[0088] FIG. 9B illustrates warped color sub-images 912R’, 912G’’, 912B’’’ generated by a virtual content warping system / method similar to that depicted in FIG. 9A. The red, green, and blue sub-images 912R’, 912G’’, 912B’’’ are warped using respective output reference systems (e.g., the estimated pose) represented by rays 910’, 910’’, 910’’’ and projected at times t0, t1, t2 represented by the same rays 910’, 910’’, 910’’’, so the sub-images 912R’, 912G’’, 912B’’’ are projected using the intended warping. FIG. 9B illustrates the reconstruction in the perception of a user of warped virtual content according to some embodiments that include a body having three overlapping R, G, B color fields (i.e., a body rendered in color). FIG. 9B is a substantially accurate rendering of the body in color since the three sub-images / fields 912R’, 912G’’, 912B’’’ are projected using the intended warping at the appropriate times.

[0089] The warping system according to embodiments herein uses a corresponding reference system (e.g., the estimated pose) that takes into account the projection timing / illumination time to warp the sub-images / fields 912R’, 912G’’, 912B’’’ instead of using a single reference system. As a result, the warping system according to embodiments herein minimizes warping-related color artifacts such as CBU while warping color virtual content into separate sub-images of different colors / fields. More accurate warping of color virtual content contributes to a more realistic and immersive AR scenario. (Illustrative Graphics Processing Unit)

[0090] FIG. 10 schematically depicts an exemplary Graphics Processing Unit (GPU) 252 for warping color virtual content into an output reference system corresponding to various color sub-images or fields, according to one embodiment. The GPU 252 includes an input memory 1010 for storing the generated color virtual content to be warped. In one embodiment, the color virtual content is stored as primitives (e.g., triangle 1100 in FIG. 11). The GPU 252 also includes a command processor 1012, which (1) receives / reads the color virtual content from the input memory 1010, (2) divides the color virtual content into color sub-images and schedules the color sub-images to a scheduling unit, and (3) transmits the scheduling unit in waves or warps along a rendering pipeline for parallel processing. The GPU 252 further includes a scheduler 1014 that receives the scheduling unit from the command processor 1012. The scheduler 1014 determines whether a "new job" from the command processor 1012 or an "old job" (described below) returning from downstream in the rendering pipeline should be transmitted at any given time along the rendering pipeline. In effect, the scheduler 1014 determines the order in which the GPU 252 processes various input data.

[0091] The GPU 252 includes a GPU core 1016, which has several parallel-executable cores / units ("shader cores") 1018 for processing the scheduling units in parallel. The command processor 1012 divides the color virtual content into a number equal to the number of shader cores 1018 (e.g., 32). The GPU 252 also includes a "First In First Out" ("FIFO") memory 1020, which receives the output from the GPU core 1016. From the FIFO memory 1020, the output can be routed back to the scheduler 1014 as an "old job" for insertion into additional processing of the rendering pipeline by the GPU core 1016.

[0092] The GPU 252 further includes a raster operation unit ("ROP") 1022 that receives the output from the FIFO memory 1020 and rasterizes the output for display. For example, primitives of color virtual content can be stored as the coordinates of the vertices of a triangle. After processing by the GPU core 1016 (during which the three vertices 1110, 1112, 1114 of the triangle 1100 can be warped), the ROP 1022 determines the pixels 1116 inside the triangle 1100 defined by the three vertices 1110, 1112, 1114 and fills those pixels 1116 in the color virtual content. The ROP 1022 can perform a depth test on the virtual content. To process the color virtual content, the GPU 252 can include one or more ROPs 1022R, 1022B, 1022G for parallel processing of sub-images of different primary colors.

[0093] The GPU 252 also includes a buffer memory 1024 for temporarily storing the warped color virtual content from the ROP 1022. The warped color virtual content in the buffer memory 1024 can include luminance / color and depth information at one or more X, Y positions of the field of view in the output reference system. The output from the buffer memory 1024 can be routed back to the scheduler 1014 as "old work" for insertion into additional processing of the rendering pipeline by the GPU core 1016 or for display in the corresponding pixels of the display system. Each fragment of the color virtual content in the input memory 1010 is processed by the GPU core 1016 at least twice. The GPU core 1016 first processes the vertices 1110, 1112, 1114 of the triangle 1100 and then processes the pixels 1116 inside the triangle 1100. When all fragments of the color virtual content in the input memory 1010 have been warped and depth-tested (if necessary), the buffer memory 1024 will contain all of the luminance / color and depth information required to display the field of view in the output reference system. (Color Virtual Content Warping System and Method)

[0094] In standard image processing without head pose changes, the results of the processing by the GPU 252 are color / brightness values and depth values at each X, Y value (e.g., each pixel). However, with head pose changes, the virtual content is warped to follow the head pose change. In the case of color virtual content, each color sub-image is warped separately. In existing methods for warping color virtual content, the color sub-images corresponding to the color image are warped using a single output reference system (e.g., corresponding to the green sub-image). As explained above, this can result in other visual artifacts such as color interference fringes and CBU.

[0095] FIG. 12 depicts a method 1200 for warping color virtual content while minimizing visual artifacts such as CBU. In step 1202, the warping system (e.g., its GPU core 1016 and / or warping unit 280) determines the projection / illumination times for the R, G, and B sub-images. This determination uses the frame rate and other characteristics of the associated projection system. In the example in FIG. 9A, the projection times correspond to t0, t1, and t2 and the light rays 910’, 910’’, 910’’’.

[0096] In step 1204, the warping system (e.g., its GPU core 1016 and / or pose estimator 282) predicts the pose / reference system corresponding to the projection times for the R, G, and B sub-images. This prediction uses various system inputs including the current pose, system IMU velocity, and system IMU acceleration. In the example in FIG. 9A, the pose / reference systems for R, G, B correspond to the light rays t0, t1, and t2 and 910’, 910’’, 910’’’.

[0097] In step 1206, a warping system (e.g., its GPU core 1016, ROP 1022, and / or transformation unit 284) warps the R sub-image using the pose / reference system of R predicted in step 1204. In step 1208, a warping system (e.g., its GPU core 1016, ROP 1022, and / or transformation unit 284) warps the G sub-image using the pose / reference system of G predicted in step 1204. In step 1210, a warping system (e.g., its GPU core 1016, ROP 1022, and / or transformation unit 284) warps the B sub-image using the pose / reference system of B predicted in step 1204. Warping separate sub-images / fields using their respective pose / reference systems differentiates these embodiments from existing methods for warping color virtual content.

[0098] In step 1212, a projection system operably coupled to the warping system projects the R, G, and B sub-images at the projection times for the R, G, and B sub-images determined in step 1202.

[0099] As described above, the method 1000 depicted in FIG. 10 may also be executed on a separate warping block 280 independent of any GPU 252 or CPU 251. In yet another embodiment, the method 1000 depicted in FIG. 10 may be executed on the CPU 251. In still other embodiments, the method 1000 depicted in FIG. 10 may be executed on various combinations / sub-combinations of the GPU 252, CPU 251, and separate warping units 280. The method 1000 depicted in FIG. 10 is an image processing pipeline that may be executed using various execution models at a particular time according to system resource availability.

[0100] Warping the color virtual content using the predicted pose / reference system corresponding to each color sub-image / field reduces color moiré and other visual anomalies. Reducing these anomalies results in a more realistic and immersive mixed reality scenario. (System Architecture Overview)

[0101] FIG. 13 is a block diagram of an illustrative computing system 1300 according to some embodiments. The computer system 1300 includes a bus 1306 or other communication mechanism for communicating information interconnecting subsystems and devices, and the subsystems and devices include a processor 1307, a system memory 1308 (e.g., RAM), a static storage device 1309 (e.g., ROM), a disk drive 1310 (e.g., magnetic or optical), a communication interface 1314 (e.g., a modem or Ethernet® card), a display 1311 (e.g., a CRT or LCD), an input device 1312 (e.g., a keyboard), and a cursor control, etc.

[0102] According to some embodiments, the computer system 1300 performs specific operations by the processor 1307 executing one or more sequences of one or more instructions included within the system memory 1308. Such instructions may be read into the system memory 1308 from another computer-readable / usable medium, such as the static storage device 1309 or the disk drive 1310. In an alternative embodiment, hardwired circuitry may be used in place of, or in combination with, software instructions to implement the present disclosure. Accordingly, embodiments are not limited to any specific combination of hardware circuitry and / or software. In one embodiment, the term “logic” may mean any combination of software or hardware used to implement all or part of the present disclosure.

[0103] As used herein, the term "computer-readable medium" or "computer-usable medium" refers to any medium involved in providing instructions for execution to a processor 1307. Such a medium can take many forms including, but not limited to, non-volatile and volatile media. Non-volatile media includes, for example, optical or magnetic disks such as disk drive 1310. Volatile media includes dynamic memory such as system memory 1308.

[0104] Computer-readable media in general form includes, for example, floppy (registered trademark) disk, flexible disk, hard disk, magnetic tape, any other magnetic medium, CD-ROM, any other optical medium, punch cards, paper tape, any other physical medium with patterns of holes, RAM, PROM, EPROM, FLASH (registered trademark)-EPROM (e.g., NAND flash, NOR flash), any other memory chip or cartridge, or any other medium readable by a computer.

[0105] In some embodiments, the execution of a series of instructions for practicing the present disclosure is performed by a single computer system 1300. According to some embodiments, two or more computer systems 1300 coupled by a communication link 1315 (e.g., LAN, PTSN, or wireless network) can cooperate with each other to perform a series of instructions required to practice the present disclosure.

[0106] Computer system 1300 can transmit and receive programs, such as messages, data, and instructions including application code, through communication link 1315 and communication interface 1314. The received program code, when it is received, can be executed by processor 1307 and / or stored in disk drive 1310 or other non-volatile storage device for later execution. Database 1332 in storage medium 1331 can be used to store data accessible by system 1300 via data interface 1333. (Alternative warping / rendering pipeline)

[0107] FIG. 14 depicts a warping / rendering pipeline 1400 for multi-field (color) virtual content according to some embodiments. Pipeline 1400 embodies two aspects: (1) multi-stage / decompression warping, (2) frequency variation between the application frame and the illumination frame. ((1) Multi-stage / decompression warping)

[0108] Pipeline 1400 includes one or more warping stages. At 1412, an application CPU (“client”) generates virtual content, which is processed by application GPU 252 for one or more (e.g., R, G, B) frames and pose 1414. At 1416, a warping / synthesizer CPU and its GPU 252 perform a first warping using a first estimated pose for each frame. In the second half of pipeline 1400 (i.e., closer to illumination), warping unit 1420 performs a second warping for each frame 1422R, 1422G, 1422B using a second estimated pose for each frame. The second estimated pose can be more accurate than each first estimated pose since the second estimated pose is determined to be closer to illumination. The twice-warped frames 1422R, 1422G, 1422B are displayed at t0, t1, and t2.

[0109] The first warping can be a best guess that can be used to align the frame of virtual content for subsequent warping. This can be computationally intensive warping. The second warping can be a sequential correction warping of each singly warped frame. The second warping is less computationally intensive and can reduce the time between the second estimate of pose and display / illumination, thereby increasing accuracy. ((2) Frequency variation)

[0110] In some embodiments, the frequency (i.e., frame rate) of the client or application and the display or illumination may not match. In some embodiments, the illumination frame rate can be twice the application frame rate. For example, the illumination frame rate can be 60 Hz and the application frame rate can be 30 Hz.

[0111] To address the warping problem with such frequency mismatches, pipeline 1400 generates two sets of doubly warped frames 1422R, 1422G, 1422B (for projection at t0 - t2) and 1424R, 1424G, 1424B (for projection at t3 - t5) per frame 1414 from application CPU 1412 and GPU 252. Using the same frame 1414 and the first warped frame 1418, warping unit 1420 sequentially generates the first and second sets of doubly warped frames 1422R, 1422G, 1422B and 1424R, 1424G, 1424B. This provides twice the number of warped frames 1422, 1424 per application frame 1414. The second warping is less computationally intensive and can further reduce processor / power requirements and heat generation.

[0112] Pipeline 1400 depicts a 2:1 lighting / application ratio, although that ratio can vary in other embodiments. For example, the lighting / application ratio can be 3:1, 4:1, 2.5:1, etc. In embodiments with fractional ratios, the most recently generated application frame 1414 can be used in the pipeline. (Method for minimizing alternative color bleeding)

[0113] FIG. 15 depicts a method 1500 for minimizing color bleeding (CBU) artifacts in warping multi-field (color) virtual content for sequential display according to some embodiments. In step 1512, the CPU receives eye and / or head tracking information (e.g., from an eye tracking camera or IMU). In step 1514, the CPU analyzes the eye and / or head tracking information and predicts CBU artifacts (e.g., based on the characteristics of the display system). In step 1516, if CBU is predicted, method 1500 proceeds to step 1518, and the CPU increases the color field rate (e.g., from 180 Hz to 360 Hz). In step 1516, if CBU is not predicted, method 1500 proceeds to step 1526, and the image (e.g., segmented and warped field information) is displayed using the system default color field rate and bit depth (e.g., 180 Hz and 8 bits).

[0114] After increasing the color field rate in step 1518, the system re-analyzes the eye and / or head tracking information and predicts CBU artifacts in step 1520. In step 1522, if CBU is predicted, method 1500 proceeds to step 1524, and the CPU decreases the bit depth (e.g., from 8 bits to 4 bits). After decreasing the bit depth, the image (e.g., segmented and warped field information) is displayed using the increased color field rate and decreased bit depth (e.g., 360 Hz and 4 bits).

[0115] In step 1522, if no CBU is predicted, method 1500 proceeds to step 1526, and the image (e.g., segmented and warped field information) is displayed using an increased color field rate and the system default bit depth (e.g., 180 Hz and 8 bits).

[0116] After the image (e.g., segmented and warped field information) is displayed using the adjusted or system default color field rate and bit depth, the CPU returns to step 1512 and, prior to repeating method 1500, resets the color field rate and bit depth to the system default values in step 1528.

[0117] In response to the predicted CBU, by adjusting the color field rate and bit depth, method 1500 depicted in FIG. 15 illustrates a way to minimize CBU artifacts. Method 1500 can be combined with other methods described herein (e.g., method 800) to further reduce CBU artifacts. Most of the steps in method 1500 depicted in FIG. 15 are performed by the CPU, but some or all of these steps can instead be performed by the GPU or a dedicated component. (Color Virtual Content Warping Using Intra-Field Subcode Timing in a Field Sequential Display System)

[0118] Referring now to FIG. 16A, an exemplary field sequential illumination sequence, according to some embodiments, is shown with respect to a change in head pose. As discussed in connection with FIG. 9A, the input image 1610 has three color sections, namely, a red section, a green section, and a blue section. Each color section corresponds to a respective color sub-image / field 1620, 1630, 1640 of the input image 1610. In some embodiments, the warping system takes into account the projection timings t0, t1, and t2 of the color fields when warping color virtual content.

[0119] In a red-green-blue (RGB) color system, the various colors can be formed from combinations of red, green, and blue color fields. Each color can be represented using a code that includes integers representing each of the red, green, and blue color fields. The red, green, and blue colors can each use 8 bits, and they have integer values from 0 to 255 corresponding to the sub-codes. For example, the red color can be represented as (R = 255, G = 0, B = 0), the green color can be represented as (0, 255, 0), and the blue color can be represented as (0, 0, 255). The various shades are formed by modifying the integer values representing the amounts of the primary color fields (red, green, blue). This is discussed in more detail below.

[0120] FIG. 16B shows the field bit depth pattern of the sigmoid growth / plateau / decay form for the complete subcodes of each constituent color field. For example, for the red color field, the complete subcode includes all colors with the code (255, X, Y), where each of x and y can take any value from 0 to 255. The sigmoid function (e.g., the field bit depth pattern) 1620' corresponds to the complete subcode of the red color field, the sigmoid function 1630' corresponds to the complete subcode of the green color field, and the sigmoid function 1640' corresponds to the complete subcode of the blue color field. As shown, each of the sigmoid functions 1620', 1630', and 1640' has a sigmoid growth segment 1602, a plateau segment 1604, and a decay segment 1606.

[0121] Assuming the source input image 1610, when the user's head moves, the red, green, and blue of the color field should be displayed using the appropriate warping corresponding to the given time at which each field is located within the sequence. In some embodiments, for a given bit depth of a color field, the timing is positioned at the centroid of the display sequence of that color field allocated for that field. For example, the centroid of the red color field display sigmoid function 1620' is aligned with the head pose position at the first time (t0), the centroid of the green color field display sigmoid function 1630' is aligned with the head pose position at a second time (t1) after the first time, and the centroid of the blue color field display sigmoid function 1640' is aligned with the head pose position at a third time (t2) after the first and second times.

[0122] Figure 17 illustrates the geometric relationships regarding the non-uniform timing sequences of the respective fields when receiving head pose changes. The geometric positions regarding the red, green, and blue color fields are offset from each other, but the degree of change coincides with the degree of change in the head pose, presenting a more uniform image with overlapping fields at a given pixel and generating the desired net color field.

[0123] Each of FIGS. 16 and 17 illustrates the field bit depth patterns of the sigmoid growth / plateau / decay form regarding the complete sub-codes of the constituent color fields.

[0124] However, it will be understood that it is not simply created as a combination of constituent sub-codes with equal colors, and that various colors require different amounts of red, green, and blue sub-codes. For example, looking at the International Commission on Illumination (CIE) 1931 color matching represented in grayscale by 1810 in FIG. 18A, any one color is a combination of multiple field inputs represented by sub-codes. The sigmoid functions 1620', 1630', and 1640' in FIG. 16B represent the maximum potential of each field (e.g., (255, 0, 0) for the color red, (0, 255, 0) for the color green, (0, 0, 255) for the color blue as sub-coded by scheme 1810).

[0125] Specific colors may not share such uniform sub-codes. For example, pink may have a combination of red 255, green 192, and blue 203 represented as (255, 192, 203), while orange may have a combination of red 255, green 165, and blue 0 represented as (255, 165, 0).

[0126] The sub - codes of the constituent color will have a corresponding variable sigmoid form. Using the red of the color field as an exemplary set, the various sub - codes of the red color field are illustrated in FIG. 18B by sigmoid functions 1822, 1824, and 1826, and each sigmoid function corresponds to a different sub - code. For example, the first sub - code of red represented by sigmoid function 1822 (e.g., (255, 10, 15)) can be red throughout the field time in the sequence, while sigmoid functions 1824 and 1826 correspond to different sub - codes of red (i.e., the second sub - code (e.g., (255, 100, 100)) and the third sub - code (e.g., (255, 150, 200)) respectively) that correspond to less time of activation of a given pixel under the pulse of the spatial light modulator within the field time allotted to the sequence. Generally, in display technology, the start of the growth phase is common to any sub - code, but the decay part starts from a non - uniform time. Thus, the centers of gravity resulting from a particular sigmoid pattern and any given sub - code are shifted relative to each other when the sub - code starts at the common start time of the field timing in the sequence.

[0127] In a conventional field - sequential display system, the sub - codes are started at a common time such that the centers of gravity regarding the sigmoids of the sub - codes will be offset from each other. As shown in FIG. 18B, the center of gravity regarding the first sub - code of red represented by sigmoid function 1822 appears at t0, while the centers of gravity regarding the second and third sub - codes of red represented by sigmoid functions 1824 and 1826 appear at t 0-n and t 0-n-m respectively. Grouping 1850 shows the range of possible head - pose positions where each sub - code may need to be warped for effective visual recognition during the head movement of the head - mounted display device.

[0128] The different barycentric times for sub - codes within a single field (i.e., color) appear as different positions when the user's head posture changes, which, since the warping will be applied to the offset position for that sub - code, can result in color separation that would otherwise occur regardless of any warping of that field. In other words, pixels intended to be pink can be geometrically offset from pixels intended to be orange because the timing of the head posture does not match the barycentric pattern timing of the sub - code.

[0129] Figure 19 more specifically illustrates this principle for a single field with various sub - code possibilities where the user's head position is at x, y at t0, where t0 can be correctly aligned with the first sub - code represented by the sigmoid function 1822, but the specific barycenters for the second and third sub - codes represented by the sigmoid functions 1824 and 1826 respectively correspond to x1, y1 and x2, y2 in geometric space. If the spatial light modulator that conveys this image data were to become active at a common time t0, the appearance of the pixels that convey the image data for the second and third sub - codes represented by the sigmoid functions 1824 and 1826 respectively would appear offset from where they should appear. This problem is similarly exacerbated when extended to the green and blue color fields and their respective sub - codes.

[0130] In some embodiments, this is corrected by having successively smaller head - posture samples such that any given color sub - code with its sigmoid barycenter can have its timing adjusted with respect to a given head posture. For example, a specific head posture for t 0-n-m can be calculated and applied with respect to the third sub - code represented by the sigmoid function 1826, for t 0-nA new specific head pose can be calculated and applied with respect to a second sub-code represented by the sigmoid function 1824, and the specific head pose with respect to t0 can be calculated and applied with respect to a first sub-code represented by the sigmoid function 1822. For a realistic augmented reality perception, the projector frequency is ideally faster than 120 Hz. For a field sequential display with three fields, this allows only a few milliseconds for any single head pose calculation. Sampling of additional head poses for each of the hundreds of sub-codes within each field can be prohibitively costly in terms of computing power and the desired form factor.

[0131] According to some embodiments, the sigmoid function shape for a given sub-code can be adjusted. Various display systems and spatial light modulators employ media and components that do not respond instantaneously to inputs. FIG. 20 illustrates an exemplary delay that can occur in some systems. For example, for a liquid crystal on silicon (LCoS) display, when a given pixel can be activated, a given liquid crystal layer can induce a delay t b as described above. This delay can exacerbate any head pose changes already present with the sub-code or can result in the contouring of an image where a single color-coded sub-code presents a band across the image. FIG. 21 illustrates such an exaggerated effect of image contouring in a field sequential display that is susceptible to timing issues of pixel enabling of sub-codes when the display is moving.

[0132] In order to alleviate these timing concerns without sacrificing excessive computing power, in some embodiments, the center of gravity for each sigmoid representing a subcode is temporally modified to correspond to the common head pose time for all subcodes of the common field. As depicted in FIG. 22, rather than starting from the common source time, the subcodes are started at different times and present their respective bit-depth sigmoid centers of gravity at a common time t0. In some embodiments, the start time for a single or all subcodes is further offset further away, such that the sigmoids are calculated such that the pixel / response times are aligned with the common head pose measurement at time t0 - t b are calculated to align at. In other words, the modulation and timing of all field input values (i.e., red, green, blue) to the spatial light modulator are constructed such that the center of gravity of the output light for each subcode is the same within the field channel.

[0133] In some embodiments, rather than creating a single subcode input (such as the second subcode represented by the single sigmoid function 1826 in FIG. 22), a series of pulses create the input per one or more fields. In FIG. 23, the center pulse 2302 is centered at a timing (t0) of a field within the frame of the sequential display. That is, the center pulse is centered at the time for the projection of the warped color field (e.g., the time of the head pose samples used to warp the color field). The center of gravity of the pulse 2302 is at time t0.

[0134] The second pulse 2304 (however, measured relative to the center pulse 2302 which may be referred to as the first pulse and thus occurs before the center pulse 2302 referred to as the second pulse) is measured from the center of gravity of the center pulse 2302 at time t0, and the end of the decay phase of the second pulse 2304 is temporally aligned with the start of the growth phase of the center pulse 2302 at time t 0-p The center of gravity of the second pulse 2304 is at time t c2is present, and it occurs during a predetermined amount of time before time t0 (e.g., t0 - t in FIG. 23 c2 ), i.e., it occurs at a time prior to that time).

[0135] The third pulse 2306 (occurring after the central pulse 2302) is measured from the center of gravity of the central pulse 2302 at time t0, and the start of the growth phase of the third pulse 2306 is temporally aligned with the end of the decay phase of the central pulse 2302 at time t 0+r . The center of gravity of the third pulse 2306 is at time t c3 , and it occurs during a predetermined amount of time after time t0 (e.g., t c3 - t0 in FIG. 23), i.e., it occurs at a time after that time).

[0136] In some embodiments, the difference between time t c3 and time t0 may be equal to the difference between time t0 and time t c2 . That is, the center of gravity of the second pulse 2304 occurs a predetermined amount of time before the center of gravity of the central pulse 2302, and the center of gravity of the third pulse 2306 occurs the same predetermined amount of time after the center of gravity of the central pulse 2302. Such symmetry of the center of gravity creates a selective bit depth with a more uniform distribution around the head pose sample throughout the sequence of fields. For example, a single pulse for a subcode of a desired bit depth requires precise timing for a specific bit depth centered on the head pose time; a bit depth that is spread with lower pulses regarding the cumulative bit depth around the head pose timing is less prone to color aliasing due to a change in direction or variable speed of head pose change because only one of the one or more pulses will be temporally aligned with the head pose sample (e.g., the central pulse 2302).

[0137] As depicted in FIG. 23, the second pulse 2304 is added to the central pulse 2302 at t 0-p , and the third pulse 2306 is added to the central pulse 2302 at t 0+r . As illustrated in FIG. 23, the growth phase of the second pulse 2304 is at time t0-y may start at, and the decay phase of the second pulse 2304 may end at time t 0-p That is, the second pulse 2304 may be defined between time t 0-y and time t 0-p The growth phase of the third pulse 2306 may start at time t 0+r and the decay phase of the third pulse 2306 may end at time t 0+x That is, the third pulse 2306 may be defined between time t 0+r and time t 0+x Those skilled in the art will understand that the decay of the second pulse 2304 may be longer or shorter than the growth phase of the third pulse 2306, and that aligning the centers of gravity may require different timings for each t0, even though equal distributions result in the intended location of the center of gravity in time, so that p and r are not necessarily equal.

[0138] FIG. 23 illustrates three separate pulses 2302, 2304, 2306 that grow from the center of gravity at time t0 of a sigmoid function representing a given color subcode (e.g., the color subcode represented by the single sigmoid function 1826 of FIG. 22) toward the edge of the sigmoid function. The center pulse 2302 is used in combination with the second pulse 2304 and the third pulse 2306 to create 256 modulation steps for each field (i.e., color).

[0139] Pulses 2302, 2304, 2306 illustrated in FIG. 23 can be used in conjunction with a computer-implemented method for warping multi-field color virtual content for sequential projection. For example, when first and second color fields (e.g., one or more of red, blue, or green) having different first and second colors (e.g., red, blue, or green sub-codes) are obtained, a first time for projection of the first color field to be warped can be determined. When predicting a first pose corresponding to the first time (e.g., time t0), for each one of the first plurality of colors within the first color field, an input representing that one of the first plurality of colors within the first color field (e.g., a color sub-code represented by the single sigmoid function 1824 of FIG. 22) can be identified, and the input can be reconstructed as a series of pulses (e.g., a center pulse 2302 centered at the first time t0, a second pulse 2304, and a third pulse 2306) that create the input per one or more fields. Each one of the series of pulses can be warped based on the first pose. And a warped first color field can be generated based on the warped series of pulses; pixels on the sequential display can be activated based on the warped series of pulses to display the warped first color field.

[0140] In some embodiments, the center pulse 2302 can include a series of short time slots (ts 1-1 , ts 1-2 , ts 1-3 , ts 1-4 , ts 1-5 , ts 1-6 ) arranged from the center outward. That is, the time slots ts 1-1 , ts 1-2 are formed adjacent to the center of gravity at time t0. The time slots ts 1-3 , ts 1-4 , ts 1-5 , ts 1-6 progress outward from time t0 such that the time slots ts 1-1 , ts 1-2are arranged with respect to. Pixels on the display device (e.g., LCoS pixels) may or may not be activated in each slot (ts 1-1 ts 1-2 ts 1-3 ts 1-4 ts 1-5 ts 1-6 ). That is, the pixels on the sequential display may be activated during a part of the time slot of the center pulse 2302. The pixels on the sequential display may be activated according to the sub - code associated with the center pulse 2302. In some embodiments, only a part of the time slot may be turned on. For example, for the lowest color code, only the center time slots (e.g., ts 1-1 ts 1-2 ) may be turned on (i.e., only the center time slots may result in active pixels on the display device). The higher the color code, the more time slots may be turned on moving outward from the center.

[0141] According to some embodiments, the second pulse 2304 and the third pulse 2306 may include time slots larger than the time slots of the center pulse 2302 (ts 1-1 ts 1-2 ts 1-3 ts 1-4 ts 1-5 ts 1-6 ). For example, the second pulse 2304 may include a time slot (ts 1-1 ts 1-2 ts 1-3 ts 1-4 ts 1-5 ts 1-6 ) that is longer (i.e., larger) than the time slots of the center pulse 2302 (ts 2-1 ts 2-2 ts 2-3 ts 2-4 ). The time slots of the second pulse 2304 (ts 2-1 ts 2-2 ts 2-3 ts2-4 ) can be arranged from a later time to an earlier time. That is, time slot ts 2-1 is the time slot ts within the second pulse 2304 2-2 , ts 2-3 , ts 2-4 occurs later in time relative to. Similarly, the third pulse 2306 has a time slot (ts 1-1 , ts 1-2 , ts 1-3 , ts 1-4 , ts 1-5 , ts 1-6 ) of the central pulse 2302 and has a longer duration time slot (ts 3-1 , ts 3-2 , ts 3-3 , ts 3-4 ). The time slots (ts 3-1 , ts 3-2 , ts 3-3 , ts 3-4 ) of the third pulse 2306 can be arranged from an earlier time to a later time. That is, time slot ts 3-1 occurs earlier in time relative to the time slots ts 3-2 , ts 3-3 , ts 3-4 within the third pulse 2306. Therefore, the pulses can be arranged to grow outward from the central pulse 2302.

[0142] In some embodiments, the pixels on the display may be activated during a portion of the time slots of the second pulse 2304 and / or the third pulse 2306. When the time slots are turned on within the second pulse 2304 and the third pulse 2306 to create a higher color code, in order to maintain the overall center of gravity within the color code, consideration is given to turning on together the slots of the third pulse 2306 corresponding to the slots of the second pulse 2304. If system constraints require turning on a single slot within the second pulse 2304 or the third pulse 2306 for adjacent codes (which is often the case), consideration is given to keeping additional slots short or using spatial / temporal dithering in order to prevent too large a shift in light energy from the center of gravity. This also avoids additional contouring artifacts associated with head or eye movement.

[0143] The center pulse 2302 may be considered the least significant bit (LSB) of the digital color code, while the second pulse 2304 and the third pulse 2306 are similar to the most significant bits (MSB) of the digital color code. The combination of the center pulse 2302 and the second pulse 2304 and the third pulse 2306 results in many possible combinations that can be used to construct 256 modulation steps.

[0144] For maximum brightness, a single pulse may need to be created for the highest modulation step that merges the center pulse 2302, the second pulse 2304, and the third pulse 2306. In the transition from three pulses to one pulse, smaller time slots may be turned on to keep the step size small. In this case, smaller slots may be added at the start of the second pulse 2304 and arranged from later times to earlier times. For example, as illustrated in FIG. 23, the time slot ts 2-4 (i.e., the time slot at the start of the second pulse 2304) is arranged with smaller time slots (ts 2-4-1 , ts 2-4-2, ts 2-4-3 can be divided into. That is, the time slot ts 2-4-1 is the time slot ts within the second pulse 2304 2-4-2 and ts 2-4-3 that occurs later in time. Similarly, smaller slots are added at the end of the third pulse 2306 and arranged from earlier times to later times. For example, as shown in FIG. 23, the time slot ts 3-4 (i.e., the time slot at the end of the third pulse 2306) can be divided into smaller time slots (ts 3-4-1 , ts 3-4-2 , ts 2-4-3 ) arranged from earlier times to later times. That is, the time slot ts 3-4-1 occurs earlier in time than the time slots ts 3-4-2 and ts 3-4-3 within the third pulse 2306. In both cases, the short time slots (i.e., ts 2-4-1 , ts 2-4-2 , ts 2-4-3 and ts 3-4-1 , ts 3-4-2 , ts 2-4-3 ) are arranged in the same direction as the larger time slots (i.e., ts 2-1 , ts 2-2 , ts 2-3 , ts 2-4 and ts 3-1 , ts 3-2 , ts 3-3 , ts 3-4 ) of their respective pulses (i.e., the second pulse 2304 and the third pulse 2306).

[0145] Many light modulators (e.g., LCoS, lasers in scanning displays, digital light processing (DLP), liquid crystal displays (LCD), and / or other display technologies) have asymmetric on and off times, so the lengths and sequence of the three pulses may need to be asymmetric to keep the center of gravity at a fixed point. If the on time is longer than the off time, for example, the center of gravity will be later than the center time within the field. According to various embodiments, each of the three pulses can be constructed in a manner similar to the asymmetric slot lengths and sequences.

[0146] The combination of the pulse lengths of the center pulse 2302, the second pulse 2304, and the third pulse 2306 can generate more than 256 possible combinations. Some of these combinations are used to create 256 modulation steps. The combinations can be selected based on several factors, including the best fit to a desired brightness response curve (i.e., linear gamma, standard red / green / blue (sRGB) gamma), minimum variation in the center of gravity across all color codes, minimum variation in the center of gravity for adjacent color codes, and smaller brightness variations for that combination across temperature and process.

[0147] Since the on and off times can vary with temperature, voltage, process, and other variables, different sets of the 256 combinations can be selected for different conditions. For example, a first set for cooling the temperature can be selected when the device is first turned on, and a different second set can be selected when the device is heated and reaches a steady-state temperature. Any number of sets can be used to limit contouring and maximize image quality across operating conditions.

[0148] In some embodiments, the symmetry property of the bit depth timing in FIG. 23 reduces the interference between sub - codes (depending on the left - to - right movement direction of the head posture), thus preventing overly bright or overly dark streaks. That is, when the sub - codes are not adjusted in time and the user moves the user's head in a particular direction, the bits of a particular sub - code (presenting color information) may appear in places not intended to appear, simply due to the poor timing of the bit - depth sigmoid form regarding the sub - code. As illustrated in FIG. 24, zone 2250 is set up to present a particular sub - code 2406 with color when the head movement is such that two other sub - codes 2402 and 2404 within the same field are in the decay phase, and depicts the area where, when the color of any sub - code is not intended to be displayed to the user, the display pixels may be inadvertently displayed based on a given head - posture timing sample. Those skilled in the art will understand that additional configurations are possible to construct the desired bit depth of one or more sub - codes.

[0149] FIG. 25 depicts a method of warping color virtual content according to some embodiments. The steps depicted in FIG. 25 may be performed for each color field (R, G, B). In some embodiments, the steps depicted in FIG. 25 may be performed as sub - steps of steps 816R, 816G, and / or 816B.

[0150] Each color field (R, G, B) contains one or more colors represented by sub - codes. For each color (e.g., sub - code) among the one or more colors of the selected color field, in step 2502, the pose estimator identifies an input (e.g., sigmoid) representing the sub - code for the color field. In step 2504, the pose estimator reconstructs the input as a series of pulses (e.g., three pulses) and creates an input per one or more fields. In step 2506, the conversion unit warps each one of the series of pulses based on the first pose. In step 2508, the conversion unit generates a warped first color field based on the warped series of pulses. In step 2510, the conversion unit sequentially activates pixels on the display based on the warped series of pulses and displays the warped first color field. The same steps 2502 - 2510 can be performed for all color fields (R, G, B).

[0151] This disclosure includes methods that can be implemented using the devices of the present subject matter. The methods can include the act of providing such a suitable device. Such providing can be performed by a user. In other words, the act of "providing" simply requires that the user act to obtain, access, approach, position, set up, activate, power on, or otherwise provide the device required in the method of the present subject matter. The methods recited herein can be performed in any order of the recited logical events and in the recited order of events.

[0152] Exemplary aspects of the present disclosure have been described above, along with details regarding material selection and manufacturing. Regarding other details of the present disclosure, these are understood in relation to the aforementioned reference patents and publications and are generally known or understandable by those skilled in the art. The same can apply to the method - based aspects of the present disclosure from the perspective of additional acts as generally or logically employed.

[0153] In addition, although the present disclosure has been described with reference to several examples incorporating various features, the present disclosure is not limited to what is described or illustrated as would be considered with respect to each variation of the disclosure. Various changes may be made to the present disclosure being described, and equivalents (whether listed herein or not, or included for some brevity purposes) may be substituted without departing from the spirit and scope of the present disclosure. In addition, when a range of values is provided, it is to be understood that all intervening values between the upper and lower limits of that range, as well as any other stated value or intervening values within the stated range, are included within the present disclosure.

[0154] It is also contemplated that any optional features of any of the variations of the invention being described may be described and claimed independently, or in combination with any one or more of the features described herein. References to singular items include the possibility that there are a plurality of the same items present. More specifically, as used in this specification and the claims associated herewith, the singular forms "a", "an", "said", and "the" include references to the plural unless specifically stated otherwise. In other words, the use of the articles enables "at least one" of the items of the subject matter in the above description and the claims associated with the present disclosure. Further, it should be noted that such claims may be drafted to exclude any optional elements. Accordingly, the recitation herein is intended to serve as a antecedent for the use of exclusive terminology such as "solely", "only", etc., or the use of "negative" limitations in connection with the recitation of elements of a claim.

[0155] Without using such exclusive technical terms, the term "comprising" in the claims associated with the present disclosure shall be construed to allow the inclusion of any additional elements, whether or not a given number of elements are recited in such claims, or the addition of features may be regarded as transforming the nature of the elements recited in such claims. Except where specifically defined herein, all technical and scientific terms used herein should be given the broadest generally understood meaning possible while maintaining the validity of the claims.

[0156] The scope of the present disclosure should not be limited to the examples provided and / or the specification of the subject matter, but rather should be limited only by the scope of the language of the claims associated with the present disclosure.

[0157] In the foregoing specification, the present disclosure has been described with reference to its specific embodiments. However, it will be apparent that various modifications and changes can be made thereto without departing from the broader spirit and scope of the present disclosure. For example, the foregoing process flow is described with reference to a particular order of process actions. However, many of the orders of the process actions described can be changed without affecting the scope or operation of the present disclosure. The specification and drawings are, therefore, to be regarded in an illustrative rather than a limiting sense.

Claims

1. A computer-implemented method for warping multi-field color virtual content for sequential projection, the method comprising: obtaining a primary color field including a plurality of colors, each of the plurality of colors representing a different shade of the primary color field; determining a first time for projection of the warped primary color field; predicting a pose corresponding to the first time; with respect to a selected one of the plurality of colors within the primary color field, identifying an input representing the selected one of the plurality of colors within the primary color field; reconstructing the input as a series of pulses that create an input per field; warping each one of the series of pulses based on the pose, wherein the selected one of the plurality of colors within the primary color field is warped individually; generating the warped primary color field based on the warped series of pulses; activating pixels on a sequential display based on the warped series of pulses to display the warped primary color field. A method comprising the above.

2. The method of claim 1, wherein the series of pulses includes a center pulse centered at the first time, a second pulse occurring before the center pulse, and a third pulse occurring after the center pulse.

3. The end of the decay phase of the second pulse is time-aligned with the start of the growth phase of the center pulse, The start of the growth phase of the third pulse is time-aligned with the end of the decay phase of the center pulse. The method of claim 2.

4. The center of gravity of the center pulse occurs at the first time, the center of gravity of the second pulse occurs at a second time before the first time, and the center of gravity of the third pulse occurs at a third time after the first time, according to the method of claim 2.

5. The difference between the first time and the second time is equal to the difference between the first time and the third time, according to the method of claim 4.

6. The center pulse includes a set of first time slots each having a first duration, and the second and third pulses each include a set of second time slots each having a second duration that exceeds the first duration, according to the method of claim 2.

7. The pixels on the sequential display are activated within a part of the set of first time slots or the set of second time slots, according to the method of claim 6.

8. The pixels on the sequential display are activated during the time slots of the center pulse according to a color code associated with the selected one of the plurality of colors within the primary color field, according to the method of claim 7.

9. The pixels on the sequential display are activated for the time slots in the second pulse and the corresponding time slots in the third pulse, according to the method of claim 7.

10. The primary color field is one of a red, green, or blue color field, according to the method of claim 1.

11. A system for warping multi-field color virtual content for sequential projection, the system comprising a warping unit for receiving a primary color field including a plurality of colors, each of the plurality of colors representing a different shade of the primary color field, The warping unit is Determine a first time for projection of the warped primary color field, and a pose estimator for predicting a pose corresponding to the first time, a conversion unit, and is provided with, wherein the conversion unit, with respect to one selected color among the plurality of colors in the primary color field, identifies an input representing the one selected color among the plurality of colors in the primary color field, reconfigures the input as a series of pulses that create an input per plurality of fields, based on the pose, warping each one of the series of pulses, wherein the one selected color among the plurality of colors in the primary color field is warped individually, generates the warped primary color field based on the warped series of pulses, activates pixels on the display sequentially based on the warped series of pulses, and displays the warped primary color field to perform, a system.

12. The series of pulses includes a central pulse centered at the first time, a second pulse occurring before the central pulse, and a third pulse occurring after the central pulse, the system according to claim 11.

13. The end of the decay phase of the second pulse is time-aligned with the start of the growth phase of the central pulse, The start of the growth phase of the third pulse is time-aligned with the end of the decay phase of the central pulse, the system according to claim 12.

14. The center of gravity of the central pulse occurs at the first time, the center of gravity of the second pulse occurs at a second time before the first time, and the center of gravity of the third pulse occurs at a third time after the first time, the system according to claim 12.

15. The central pulses each include a set of first time slots each having a first duration, The system according to claim 12, wherein the second pulse and the third pulse each include a set of second time slots each having a second duration that exceeds the first duration.

16. The system according to claim 15, wherein the pixels on the sequential display are activated during a set of the first time slots or a part of the set of the second time slots.

17. The system according to claim 16, wherein the pixels on the sequential display are activated during the time slots of the central pulses according to a color code associated with the selected one of the plurality of colors in the primary color field.

18. The system according to claim 16, wherein the pixels on the sequential display are activated for a time slot in the second pulse and a corresponding time slot in the third pulse.

19. A computer-implemented method for warping multi-field color virtual content for sequential projection, obtaining a first primary color field including a plurality of first colors and a second primary color field including a plurality of second colors different from the plurality of first colors of the first primary color field, wherein each of the plurality of first colors represents a different shade of the first primary color field, and each of the plurality of second colors represents a different shade of the second primary color field, for each one of the first primary color field and the second primary color field, determining a first time for projection of the warped primary color field, predicting a pose corresponding to the first time, for each one of the plurality of colors in the primary color field, Identifying an input representing the one color among the plurality of colors within the primary color field; Reconfiguring the input as a series of pulses that create an input per a plurality of fields; Warping each one of the series of pulses based on the posture, wherein each color among the plurality of colors within the primary color field is warped individually; Performing; Generating the warped primary color field based on the series of warped pulses corresponding to all of the plurality of colors within the primary color field; Activating pixels on a display sequentially based on the series of warped pulses and displaying the warped primary color field; A computer-implemented method including performing.

Images