Intra-field subcode timing in field sequential displays

Abstract

Embodiments provide a computer-implemented method for warping multi-field color virtual content for sequential projection. A first color field and a second color field having different first colors and second colors are acquired. A first time for projecting a warped first color field is determined. A first pose corresponding to the first time is predicted. For each of the first colors in the first color field, (a) an input representing one of the first colors in the first color field is identified; (b) the input is reconfigured to create a series of pulses of per-field input; and (c) each of the series of pulses is warped based on the first pose. A warped first color field is generated based on the warped series of pulses. Pixels on a sequential display are activated based on the warped series of pulses to display the warped first color field.

Description

[0001] This application is a divisional application of the patent application filed on July 23, 2019, with application number 201980048711.7 and entitled "Field Subcode Timing in Field Sequence Display".

[0002] Cross-references to related applications

[0003] This application claims priority to U.S. Provisional Patent Application No. 62 / 702,181, filed July 23, 2018, entitled "Intra-Field Sub Code Timing In Field Sequential Displays," the entire disclosure of which is incorporated herein by reference for all purposes, as if set forth in its entirety.

[0004] This application relates to U.S. Patent Application No. 15 / 924,078, filed March 16, 2018, entitled “Mixed Reality System with ColorVirtual Content Warping and Method of Generating Virtual Content Using Same,” the contents of which are incorporated herein by reference in their entirety. Technical Field

[0005] This disclosure relates to a field-sequential display system that projects one or more color codes onto virtual content at different geometric locations over time, and a method for generating mixed reality experience content using the field-sequential display system. Background Technology

[0006] Modern computing and display technologies have facilitated the development of "mixed reality" (MR) systems for so-called "virtual reality" (VR) or "augmented reality" (AR) experiences, in which digitally reproduced images, or portions thereof, are presented to the user in a way that appears real or could be perceived as real. VR scenes typically involve the presentation of digital or virtual image information without transparency to actual real-world visual input. AR scenes typically involve the presentation of digital or virtual image information as an enhancement of the visualization of the real world around the user (i.e., transparency to real-world visual input). Therefore, AR scenes involve the presentation of digital or virtual image information with transparency to real-world visual input.

[0007] MR systems typically generate and display color data, which enhances the realism of MR scenes. Many of these MR systems display color data by rapidly and continuously projecting sub-images of different colors (e.g., primary colors) or "fields" (e.g., red, green, and blue) corresponding to color images. Projecting color sub-images at sufficiently high rates (e.g., 60Hz, 120Hz, etc.) can deliver a smooth, color MR scene in the user's mind.

[0008] Various optical systems generate images, including color images, at various depths for displaying MR (VR and AR) scenes. Some such optical systems are described in U.S. Utility Model Patent Application No. 14 / 555,585 (Attorney’s File No. ML.20011.00), filed November 27, 2014, the contents of which are incorporated herein by reference in their entirety as if fully described.

[0009] MR systems typically employ wearable displays (e.g., head-mounted displays, helmet-mounted displays, or smart glasses) that are at least loosely coupled to the user's head and thus move as the user's head moves. If the display detects head movement, it can update the data being displayed to account for changes in head posture (i.e., the orientation and / or position of the user's head). Changes in position pose challenges to field-sequential display techniques. Summary of the Invention

[0010] This article describes skills and techniques for improving the image quality of motion-exposed field-sequence displays designed to project still images.

[0011] As an example, if a user wearing a head-mounted display views a virtual representation of a virtual object on the screen and walks around the area where the virtual object appears, the virtual object can be rendered for each viewpoint, giving the user the feeling that they are walking around an object that shares a relationship with real space rather than with the display surface. However, changes in the user's head posture will alter the image, and to maintain static image projection from a dynamic display system, the timing of the field sequence projector needs to be adjusted.

[0012] A conventional field-sequence display projects colors for a single image frame in a specified time sequence, and the time difference between fields is not noticeable when viewed on a static display. For example, a red pixel displayed first and a blue pixel displayed 10 milliseconds later will appear to overlap because the geometric position of the pixels does not change within a perceptible amount of time.

[0013] However, in a motion projector such as a head-mounted display, motion performed at the same 10-millisecond intervals may correspond to a noticeable offset between red and blue pixels intended to overlap.

[0014] In some embodiments, distorting the color of a single image within a field sequence can improve image perception because each frame will be based on the appropriate viewpoint of the field at a given time during head pose changes. Such methods and systems for implementing this solution are described in U.S. Patent Application No. 15 / 924,078.

[0015] In addition to the specific field distortions that should occur to correct for general head posture changes in field sequence displays, the subcodes of a given field should also be adjusted to properly convey a rich image representing the expected colors.

[0016] In one embodiment, a computer-implemented method for distorting multi-field color virtual content for sequential projection includes acquiring a first color field and a second color field having different first and second colors. The method further includes determining a first time for projecting the distorted first color field. The method also includes predicting a first pose corresponding to the first time. For each of the first colors in the first color field, the method includes: (a) identifying an input representing that color in the first color field; (b) reconfiguring the input to create a series of pulses for a plurality of per-field inputs; and (c) distorting each of the series of pulses based on the first pose. The method also includes generating the distorted first color field based on the distorted series of pulses. Additionally, the method includes activating pixels on a sequential display based on the distorted series of pulses to display the distorted first color field.

[0017] In one or more embodiments, a series of pulses includes a central pulse centered at a first time, a second pulse occurring before the central pulse, and a third pulse occurring after the central pulse. The end of the decay phase of the second pulse is time-aligned with the start of the growth phase of the central pulse, and the start of the growth phase of the third pulse is time-aligned with the end of the decay phase of the central pulse. The centroid of the central pulse occurs at the first time, the centroid of the second pulse occurs at a second time before the first time, and the centroid of the third pulse occurs at a third time after the first time. In some embodiments, the difference between the first time and the second time is equal to the difference between the first time and the third time. In some embodiments, the central pulse includes a first time slot group, each having a first duration, and the second and third pulses include a second time slot group, each having a second duration greater than the first duration. Pixels on the sequential display are activated during a subset of the first or second time slot group. In some embodiments, during a time slot of the central pulse, pixels on the sequential display are activated according to a color code associated with one of the first colors in a first color field. In various embodiments, pixels on the sequential display are activated in time slots within the second pulse and corresponding time slots within the third pulse.

[0018] In one or more embodiments, the method may further include determining a second time for projecting the distorted second color field. The method may further include predicting a second pose corresponding to the second time. For each of the second colors in the second color field, the method may include: (a) identifying an input representing one of the second colors in the second color field; (b) reconfiguring the input to create a series of pulses for each field input; and (c) distorting each of the series of pulses based on the second pose. The method may further include generating the distorted second color field based on the distorted series of pulses. Additionally, the method may include activating pixels on a sequential display based on the distorted series of pulses to display the distorted second color field based on the distorted series of pulses.

[0019] In another embodiment, a system for distorting multi-field color virtual content for sequential projection includes: a distortion unit that receives a first color field and a second color field having different first and second colors for sequential projection. The distortion unit includes a pose estimator that determines a first time for projecting the distorted first color field and predicts a first pose corresponding to the first time. The distortion unit also includes a transformation unit that, for each of the first colors in the first color field, (a) identifies an input representing that color; (b) reconfigures the input to create a series of pulses for each field input; and (c) distorts each of the series of pulses based on the first pose. The transformation unit is also configured to generate the distorted first color field based on the distorted series of pulses. The transformation unit is further configured to activate pixels on a sequential display based on the distorted series of pulses to display the distorted first color field.

[0020] In another embodiment, a computer program product is embodied in a non-transitory computer-readable medium storing a series of instructions that, when executed by a processor, cause the processor to perform a method for distorting multi-field color virtual content for sequential projection. The method includes acquiring a first color field and a second color field having different first and second colors. The method also includes determining a first time for projecting the distorted first color field. The method further includes predicting a first pose corresponding to the first time. For each of the first colors in the first color field, the method includes: (a) identifying an input representing one of the first colors in the first color field; (b) reconfiguring the input into a series of pulses to create a plurality of per-field inputs; and (c) distorting each of the series of pulses based on the first pose. The method also includes generating the distorted first color field based on the distorted series of pulses. Additionally, the method includes activating pixels on a sequential display based on the distorted series of pulses to display the distorted first color field.

[0021] In one embodiment, a computer-implemented method for distorting multi-field color virtual content for sequential projection includes acquiring a first color field and a second color field having different first and second colors. The method further includes determining a first time for projecting the distorted first color field. The method also includes determining a second time for projecting the distorted second color field. Furthermore, the method includes predicting a first pose at the first time and predicting a second pose at the second time. Additionally, the method includes generating the distorted first color field by distorting the first color field based on the first pose. The method also includes generating the distorted second color field by distorting the second color field based on the second pose.

[0022] In one or more embodiments, the first color field includes first color field information located at the X and Y positions. The first color field information may include a first brightness of a first color. The second color field includes second image information located at the X and Y positions. The second color field information may include a second brightness of a second color.

[0023] In one or more embodiments, the distorted first color field includes distorted first color field information located at a first distorted X, Y position. The distorted second color field includes distorted second color field information located at a second distorted X, Y position. Distorting the first color field based on a first pose may include applying a first transformation to the first color field to generate the distorted first color field. Distorting the second color field based on a second pose may include applying a second transformation to the second color field to generate the distorted second color field.

[0024] In one or more embodiments, the method further includes sending distorted first and second color fields to a sequential projector, which sequentially projects the distorted first color field and the distorted second color field. The distorted first color field can be projected at a first time, and the distorted second color field can be projected at a second time.

[0025] In another embodiment, a system for distorting multi-field color virtual content for sequential projection includes: a distortion unit for receiving a first color field and a second color field having different first and second colors for sequential projection. The distortion unit includes a pose estimator that determines first and second times for projecting the corresponding distorted first and second color fields, and predicts first and second poses at the corresponding first and second times. The distortion unit further includes a transformation unit that generates distorted first and second color fields by distorting the corresponding first and second color fields based on the corresponding first and second poses.

[0026] In yet another embodiment, a computer program product implemented on a non-transient computer-readable medium, the computer-readable medium storing a sequence of instructions that, when executed by a processor, cause the processor to perform a method for distorting multi-field color virtual content for sequential projection. The method includes acquiring first and second color fields having different first and second colors. The method further includes determining a first time for projecting the distorted first color field. The method further includes determining a second time for projecting the distorted second color field. Furthermore, the method includes predicting a first pose at the first time and predicting a second pose at the second time. Additionally, the method includes generating the distorted first color field by distorting the first color field based on the first pose. The method further includes generating the distorted second color field by distorting the second color field based on the second pose.

[0027] In another embodiment, a computer-implemented method for distorting multi-field color virtual content for sequential projection includes acquiring an application frame and an application pose. The method further includes estimating a first pose for a first distortion of the application frame at a first estimated display time. The method also includes performing a first distortion of the application frame using the application pose and the estimated first pose to generate a first distorted frame. Furthermore, the method includes estimating a second pose for a second distortion of the first distorted frame at a second estimated display time. Additionally, the method includes performing a second distortion of the first distorted frame using the estimated second pose to generate a second distorted frame.

[0028] In one or more embodiments, the method includes displaying a second warped frame at approximately a second estimated display time. The method may further include estimating a third pose of a third warp of a first warped frame at a third estimated display time, and using the estimated third pose to perform a third warp of the first warped frame to generate a third warped frame. The third estimated display time may be later than the second estimated display time. The method may also include displaying the third warped frame at approximately a third estimated display time.

[0029] In another embodiment, a computer implementation method for minimizing color separation (“CBU”) artifacts includes predicting CBU artifacts based on received eye or head tracking information, and the method further includes increasing the color field rate based on the predicted CBU artifacts.

[0030] In one or more embodiments, the method includes predicting a second CBU based on received eye or head tracking information and an increased color field rate, and reducing the bit depth based on the predicted second CBU artifacts. The method may also include displaying an image using the increased color field rate and the reduced bit depth.

[0031] Additional and other objects, features and advantages of this disclosure are set forth in the detailed description, drawings and claims. Attached Figure Description

[0032] The accompanying drawings illustrate the design and utility of various embodiments of this disclosure. It should be noted that the drawings are not drawn to scale, and throughout the drawings, elements with similar structures or functions are indicated by similar reference numerals. To better understand how the above and other advantages and objects of the various embodiments of this disclosure are obtained, the disclosure, which has been briefly described above, will be described in more detail with reference to specific embodiments illustrated in the accompanying drawings. It should be understood that these drawings depict only typical embodiments of this disclosure and should therefore not be considered as limiting its scope. The disclosure will be described and explained with additional features and details using the drawings, wherein:

[0033] Figure 1 A user view of augmented reality (AR) via a wearable AR user device is depicted according to some embodiments.

[0034] Figures 2A to 2C An AR system and its subsystems according to some embodiments are schematically depicted.

[0035] Figure 3 and Figure 4 A rendering artifact with rapid head movement is shown according to some embodiments.

[0036] Figure 5 Exemplary virtual content distortions are shown according to some embodiments.

[0037] Figure 6 Depicting according to some embodiments such as Figure 5 The method shown is to distort virtual content.

[0038] Figure 7A and Figure 7B Multi-field (color) virtual content distortion and its results are depicted according to some embodiments.

[0039] Figure 8 A method for distorting multi-field (color) virtual content according to some embodiments is described.

[0040] Figure 9A and Figure 9B Multi-field (color) virtual content distortion and its results are depicted according to some embodiments.

[0041] Figure 10 A graphics processing unit (GPU) according to some embodiments is schematically depicted.

[0042] Figure 11 Virtual objects, stored as primitives according to some embodiments, are described.

[0043] Figure 12 A method for distorting multi-field (color) virtual content according to some embodiments is described.

[0044] Figure 13 This is a block diagram schematically depicting an illustrative computing system according to some embodiments.

[0045] Figure 14 A warp / rendering pipeline for multi-field (color) virtual content is described according to some embodiments.

[0046] Figure 15 Methods for minimizing color separation artifacts in distorted multi-field (color) virtual content, according to some embodiments, are described.

[0047] Figures 16A to 16B The timing aspects of a field sequence display according to some embodiments are described, which displays the uniform subcode bit depth of each field based on head pose.

[0048] Figure 17 The geometric positions of the separated fields within a field sequence display according to some embodiments are depicted.

[0049] Figure 18A It depicts the color scheme of the International Commission on Illumination (CIE) in 1931, based on gray levels.

[0050] Figure 18B The geometric timing aspects of different subcodes within a single field, based on head pose, are described according to some embodiments.

[0051] Figure 19 The geometric positions of field subcodes within a presence sequence display according to some embodiments are depicted.

[0052] Figure 20 Timing aspects related to pixel activation and liquid crystal displays according to some embodiments are described.

[0053] Figure 21 The color contour effect related to the color timing in a field sequence display is depicted.

[0054] Figure 22 The text describes adjusting color subcodes to common timing or common time relationships according to some embodiments.

[0055] Figure 23 A sequence of pulses with bit depth generated within a field based on a time center, according to some embodiments, is described.

[0056] Figure 24 The adverse effects of asymmetric subcode illumination are described.

[0057] Figure 25Methods for distorting multi-field (color) virtual content according to some embodiments are described. Detailed Implementation

[0058] Various embodiments of this disclosure relate to systems, methods, and articles of manufacture for distorting virtual content from a source in one or more embodiments. Further objects, features, and advantages of this disclosure are described in the detailed description, drawings, and claims.

[0059] Various embodiments will now be described in detail with reference to the accompanying drawings, which are provided as illustrative examples to enable those skilled in the art to practice this disclosure. It is important to note that the following drawings and examples are not intended to limit the scope of this disclosure. Where certain elements of this disclosure may be partially or fully implemented using known components (or methods or processes), only those portions of such known components (or methods or processes) necessary for understanding this disclosure will be described, and detailed descriptions of other portions of these known components (or methods or processes) will be omitted so as not to obscure this disclosure. Furthermore, the various embodiments include current and future known equivalents of the components mentioned herein by way of illustration.

[0060] The virtual content distortion system can be implemented independently of the mixed reality system, but some of the embodiments described below are for illustrative purposes only and pertain to AR systems. Furthermore, the virtual content distortion system described herein can also be used in the same manner as a VR system.

[0061] Illustrative mixed reality scenarios and systems

[0062] The following description relates to an illustrative augmented reality system utilized in implementing the distortion system. However, it should be understood that the embodiments are also applicable to applications in other types of display systems (including other types of mixed reality systems), and therefore the embodiments are not limited to the illustrative systems disclosed herein.

[0063] Mixed reality (e.g., VR or AR) scenarios typically involve the presentation of virtual content (e.g., color images and sounds) corresponding to virtual objects associated with real-world objects. For example, reference... Figure 1 The illustration depicts an augmented reality (AR) scene 100, in which users of AR technology see a real-world physical park-like setting 102 characterized by people, trees, and buildings in the background, as well as a real-world physical entity platform 104. In addition to these items, AR users also feel that they "see" a virtual robot statue 106 standing on the physical entity platform 104, and a virtual cartoon-style avatar character 108 that looks like an avatar of a flying bee, even though these virtual objects 106 and 108 do not exist in the real world.

[0064] Similar to AR scenarios, VR scenarios must also take into account the poses used to generate / render virtual content. Accurately warping the virtual content to the AR / VR display reference frame and warping the warped virtual content can improve the AR / VR scenario, or at least not degrade it.

[0065] The following description relates to an illustrative AR system that can be implemented using the methods disclosed herein. However, it should be understood that this disclosure is also applicable to applications in other types of augmented reality and virtual reality systems, and therefore this disclosure is not limited to the illustrative systems disclosed herein.

[0066] Reference Figure 2A This describes one embodiment of an AR system 200 according to some embodiments. The AR system 200 can operate in conjunction with a projection subsystem 208 to provide an image of a virtual object that is blended with a physical object in the field of view of a user 250. The method employs one or more at least partially transparent surfaces through which the surrounding environment, including the physical object, can be seen, and through which the AR system 200 generates an image of the virtual object. The projection subsystem 208 is housed in a control subsystem 201, which is operatively coupled to a display system / subsystem 204 via a link 207. The link 207 can be a wired or wireless communication link.

[0067] For AR applications, it may be desirable to spatially locate various virtual objects relative to individual physical objects within the user's field of vision. Virtual objects can take any of a variety of forms and have any kind of data, information, concept, or logical structure that can be represented as an image. Non-limiting examples of virtual objects may include: virtual text objects, virtual number objects, virtual alphanumeric objects, virtual label objects, virtual field objects, virtual chart objects, virtual map objects, virtual instrument objects, or virtual visual representations of physical objects.

[0068] AR system 200 includes a frame structure 202 worn by user 250; a display system 204 carried by frame structure 202 such that display system 204 is positioned in front of user 250's eyes; and a speaker 206 incorporated into or connected to display system 204. In the illustrated embodiment, speaker 206 is carried by frame structure 202 such that speaker 206 is positioned near (in or around) user 250's ear canal, for example, in an earplug or earphone.

[0069] Display system 204 is designed to present a photo-based radial pattern to the eyes of user 250, which can be comfortably perceived as an enhancement of the surrounding environment, including both two-dimensional and three-dimensional content. Display system 204 presents a sequence of frames at a high frequency, providing the perception of a single coherent scene. To this end, display system 204 includes a projection subsystem 208 and a partially transparent display screen through which the projection subsystem 208 projects images. The display screen is located within the field of view of user 250, between the eyes of user 250 and the surrounding environment.

[0070] In some embodiments, the projection subsystem 208 takes the form of a scan-based projection device, and the display screen takes the form of a waveguide-based display. Scanning light from the projection subsystem 208 is injected into the waveguide-based display to produce images, for example, at a single optical viewing distance (e.g., the length of an arm) relatively close to infinity, images at multiple discrete optical viewing distances or focal planes, and / or stacked image layers at multiple viewing distances or focal planes to represent volumetric 3D objects. These layers in the light field can be stacked tightly enough to appear continuous to the human visual subsystem (e.g., one layer within the cone of confusion of adjacent layers). Additionally or alternatively, image elements can be blended across two or more layers to increase the perceived continuity of transitions between layers in the light field, even if the layers are stacked more sparsely (e.g., one layer outside the cone of confusion of adjacent layers). The display system 204 can be monocular or binocular. The scanning component includes one or more light sources that generate the beam (e.g., emitting light of different colors in a defined pattern). The light source can take any of a variety of forms, such as a set of RGB sources (e.g., laser diodes capable of outputting red, green, and blue light), which can operate to generate coherent collimated red, green, and blue light respectively according to a defined pixel pattern specified in each frame of pixel information or data. Lasers provide high color saturation and high energy efficiency. The optical coupling subsystem includes an optical waveguide input device, such as one or more reflective surfaces, diffraction gratings, mirrors, dichroic mirrors, or prisms, to optically couple light to an end of the display screen. The optical coupling subsystem also includes a collimating element for collimating light from the optical fiber. Optionally, the optical coupling subsystem includes an optical modulation device configured to converge light from the collimating element toward a focal point located at the center of the optical waveguide input device, thereby allowing for minimization of the size of the optical waveguide input device. Therefore, the display subsystem 204 generates a series of composite image frames of pixel information, which presents an undistorted image of one or more virtual objects to the user. The display subsystem 204 also generates a series of color composite sub-image frames containing pixel information, which present the user with an undistorted color image of one or more virtual objects.Further details describing the display subsystem are provided in U.S. Utility Patent Application No. 14 / 212,961 entitled “Display System and Method” (Attorney General’s File No. ML.20006.00) and U.S. Utility Patent Application No. 14 / 331,218 entitled “Planar Waveguide Apparatus With Diffraction Element(s) and Subsystem Employing Same” (Attorney General’s File No. ML.20020.00), the contents of which are expressly incorporated herein by reference in their entirety as if fully described.

[0071] AR system 200 also includes one or more sensors mounted to frame structure 202 for detecting the position (including orientation) and motion of user 250's head and / or the position and interpupillary distance of user 250's eyes. Such sensors may include image capture devices, microphones, inertial measurement units (IMUs), accelerometers, compasses, GPS units, radios, gyroscopes, etc. For example, in one embodiment, AR system 200 includes a head-mounted transducer subsystem comprising one or more inertial transducers to capture inertial measurements indicative of the motion of user 250's head. These devices can be used to sense, measure, or collect information about the motion of user 250's head. For example, these devices can be used to detect / measure the motion, velocity, acceleration, and / or position of user 250's head. The position (including orientation) of user 250's head is also referred to as user 250's "head pose."

[0072] Figure 2A The AR system 200 may include one or more forward-facing cameras. The cameras can be used for any number of purposes, such as recording images / videos from the forward direction of the system 200. Additionally, the cameras can be used to capture information about the environment in which the user 250 is located, such as information indicating the distance, orientation, and / or angular position of the user 250 relative to the environment and specific objects within that environment.

[0073] AR system 200 may also include a rear-facing camera to track the angular position of user 250's eyes (the direction in which the eyes or both eyes are looking), blinking, and depth of focus (by detecting eye convergence). This eye-tracking information can be discerned, for example, by projecting light onto the end user's eyes and detecting the return or reflection of at least some of the projected light.

[0074] The augmented reality system 200 also includes a control subsystem 201, which can take any of a variety of forms. The control subsystem 201 includes multiple controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers such as application-specific integrated circuits (ASICs), programmable gate arrays (PGAs) such as field-programmable gate arrays (FPGAs), and / or programmable logic controllers (PLUs). The control subsystem 201 may include a digital signal processor (DSP), a central processing unit (CPU) 251, a graphics processing unit (GPU) 252, and one or more frame buffers 254. The CPU 251 controls the overall operation of the system, while the GPU 252 renders frames (i.e., converts a three-dimensional scene into a two-dimensional image) and stores these frames in one or more frame buffers 254. Although not shown, one or more additional integrated circuits can control the reading of frames into and / or out of the frame buffers 254, as well as the operation of the display system 204. Reading into and / or from frame buffer 254 can employ dynamic addressing, for example, where frames are over-rendered. Control subsystem 201 also includes read-only memory (ROM) and random access memory (RAM). Control subsystem 201 also includes a 3D database 260 from which GPU 252 can access 3D data for rendering frames from one or more scenes, as well as synthesized sound data associated with virtual sound sources contained within the 3D scene.

[0075] The augmented reality system 200 also includes a user orientation detection module 248. The user orientation module 248 detects the instantaneous position of the user 250's head and can predict the position of the user 250's head based on position data received from sensors. The user orientation module 248 also tracks the user 250's eyes, specifically tracking the direction and / or distance of the user 250's focus based on tracking data received from sensors.

[0076] Figure 2B An AR system 200' according to some embodiments is depicted. Figure 2B The AR system 200' depicted in the text is similar to Figure 2A The AR system 200 described above is illustrated. For example, AR system 200' includes a frame structure 202, a display system 204, a speaker 206, and a control subsystem 201' operatively coupled to the display system subsystem 204 via a link 207. Figure 2B The control subsystem 201' depicted in the text is similar to Figure 2AThe control subsystem 201 described above is illustrated in the figure. For example, the control subsystem 201' includes a projection subsystem 208, an image / video database 271, a user orientation module 248, a CPU 251, a GPU 252, a 3D database 260, ROM, and RAM.

[0077] Figure 2B The control subsystem 201' described herein, and further, the AR system 200', are related to... Figure 2A The differences between the corresponding systems / system components described are, Figure 2B The control subsystem 201' depicted contains a twist unit 280. The twist unit 280 is a separate twist block independent of the GPU 252 or CPU 251. In other embodiments, the twist unit 280 may be a component within a separate twist block. In some embodiments, the twist unit 280 may be internal to the GPU 252. In some embodiments, the twist unit 280 may be internal to the CPU 251. Figure 2C The distortion unit 280 is shown to include a pose estimator 282 and a transformation unit 284.

[0078] Various processing components of AR systems 200, 200' can be contained within distributed subsystems. For example, AR systems 200, 200' include local processing and data modules (i.e., control subsystems 201, 201') operatively coupled to a portion of display system 204, such as via wired or wireless connection 207. The local processing and data modules can be installed in various configurations, such as being fixedly attached to frame structure 202, fixedly attached to a helmet or hat, embedded in headphones, removably attached to the torso of user 250, or removably attached to the hip of user 250 in a belt-coupled configuration. AR systems 200, 200' may also include remote processing modules and remote data repositories operatively coupled to the local processing and data modules, such as via wired or wireless connections, such that these remote modules are operatively coupled to each other and can be used as resources for the local processing and data modules. The local processing and data module may include a power-efficient processor or controller, and digital memory, such as flash memory, both of which can be used to assist in processing, caching, and storing data captured from sensors and / or data acquired and / or processed using remote processing modules and / or remote data repositories, possibly for transmission to display system 204 after such processing or acquisition. The remote processing module may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. The remote data repository may include relatively large-scale digital data storage facilities available via the Internet or other networked configurations in a “cloud” resource configuration. In some embodiments, all data is stored and all computations are performed in the local processing and data module, allowing for fully autonomous use from the remote module. Coupling between the various components described above may include one or more wired interfaces or ports to provide wired or optical communication; or one or more wireless interfaces or ports, such as those via RF, microwave, and IR, to provide wireless communication. In some embodiments, all communication may be wired, while in others, all communication may be wireless, except for fiber optics.

[0079] Summary of problems and solutions

[0080] When an optical system generates / renders colored virtual content, it can use a source reference frame that may be related to the system's pose during the rendering of the virtual content. In an AR system, the rendered virtual content can have a predefined relationship with real physical objects. For example, Figure 3An AR scenario 300 is illustrated, which includes a virtual flowerpot 310 positioned on top of a real physical base 312. The AR system renders the virtual flowerpot 310 based on a source reference frame whose position of the real base 312 is known, making the virtual flowerpot 310 appear to be resting on top of the real base 312. The AR system can render the virtual flowerpot 310 using the source reference frame at a first time, and then display / project the rendered virtual flowerpot 310 onto an output reference frame at a second time after the first time. If the source reference frame and the output reference frame are the same, the virtual flowerpot 310 will appear in the desired location (e.g., on top of the real physical base 312).

[0081] However, if the reference frame of the AR system changes during the interval between the first time of rendering the virtual flowerpot 310 and the second time of displaying / projecting the rendered virtual flowerpot 310 (e.g., with rapid user head movement), the mismatch / difference between the source and output reference frames can lead to visual artifacts / anomalies / short-term interference (glitch). For example, Figure 4 An AR scenario 400 is shown, comprising a virtual flowerpot 410 rendered on top of a real physical base 412. However, because the AR system is rapidly moved to the right after the virtual flowerpot 410 is rendered but before it is displayed / projected, the virtual flowerpot 410 is displayed to the right of its intended position 410' (shown in dashed lines). Thus, the virtual flowerpot 410 appears to float in mid-air to the right of the real physical base 412. This artifact is corrected when the virtual flowerpot is re-rendered in the output reference frame (assuming the AR system movement stops). However, some users may still see the artifact, where the virtual flowerpot 410 briefly interferes by temporarily jumping to an unwanted position. This brief interference, and other similar cases, can have a detrimental effect on the illusion of continuity in the AR scene.

[0082] Some optical systems may include a warping system that warps or transforms the reference frame of the source virtual content from the source reference frame that generates the virtual content to the output reference frame that will display the virtual content. For example, in Figure 4 In the depicted example, the AR system can detect and / or predict (e.g., using an IMU or eye tracking) the output reference frame and / or pose. The AR system can then distort or transform the rendered virtual content from the source reference frame into distorted virtual content in the output reference frame.

[0083] Color Virtual Content Distortion System and Method

[0084] Figure 5The distortion of virtual content is illustrated according to some embodiments. The source virtual content 512 in the source reference frame (rendering pose) represented by ray 510 is distorted into the distorted virtual content 512' in the output reference frame (estimated pose) represented by ray 510'. Figure 5 The distortion depicted can represent the head rotating 520 degrees to the right. When the source virtual content 512 is set at the source X, Y position, the distorted virtual content 512' is transformed to the output X', Y' position.

[0085] Figure 6 A method for distorting virtual content according to some embodiments is described. In step 612, the distortion unit 280 receives virtual content, a base pose (i.e., the current pose of AR systems 200, 200' (current reference frame)), a rendering pose (i.e., the pose used by AR systems 200, 200' to render the virtual content (source reference frame)), and an estimated illumination time (i.e., an estimated time when display system 204 will be illuminated (estimated output reference frame)). In some embodiments, the base pose may be more recent / more recent than the rendering pose. In step 614, the pose estimator 282 uses the base pose and information about AR systems 200, 200' to estimate the pose at the estimated illumination time. In step 616, the transformation unit 284 uses the estimated pose (from the estimated illumination time) and the rendering pose to generate distorted virtual content from the received virtual content.

[0086] When virtual content includes color, some distortion systems use a single X', Y' position in a single output reference frame (e.g., a single estimated pose from a single estimated illumination time) to distort all color sub-images or fields corresponding to / forming a color image. However, some projection display systems (e.g., sequential projection display systems), such as those in some AR systems, do not project all color sub-images / fields simultaneously. For example, there may be some delay between the projection of each color sub-image / field. This delay between the projection of each color sub-image / field, i.e., the difference in illumination time, can cause color stripe artifacts in the final image during rapid head movements.

[0087] For example, Figure 7AThe distortion of colored virtual content using some distortion systems according to some embodiments is schematically illustrated. The source virtual content 712 has three color portions: a red portion 712R; a green portion 712G; and a blue portion 712B. In this example, each color portion corresponds to a colored sub-image / field 712R", 712G", 712B". Some distortion systems use a single output reference frame (e.g., estimated pose) represented by ray 710" (e.g., reference frame 710 corresponding to the green sub-image and its illumination time t1) to distort all three colored sub-images 712R", 712G", 712B". However, some projection systems do not project the colored sub-images 712R", 712G", 712B" simultaneously. Instead, the colored sub-images 712R", 712G", 712B" are projected at three slightly different times (represented by rays 710', 710", 710"' at times t0, t1, and t2). The magnitude of the delay between the projection of sub-images can depend on the frame / refresh rate of the projection system. For example, if the projection system has a frame rate of 60Hz or lower (e.g., 30Hz), the delay may cause color stripe artifacts with fast-moving viewers or objects.

[0088] Figure 7B It is shown that, according to some embodiments, it is composed of similar Figure 7A The color stripe artifacts generated by the virtual content distortion system / method described in the text. Because the red sub-image 712R uses... Figure 7A Ray 710' represents the output reference frame (e.g., estimated pose) to warp, but it is projected at time t0, as represented by ray 710', so the red sub-image 712R" appears to overshoot the expected warp. This overshoot manifests as... Figure 7B The right stripe image 712R is in the middle. Because the green sub-image 712G uses the image from the middle. Figure 7A The output reference frame (e.g., the estimated pose) represented by ray 710” is warped and projected at time t1 represented by ray 710”, thus the green sub-image 712G” is projected with the expected warp. This is caused by Figure 7B The central image 712G is represented by this. Because the blue sub-image 712B uses a design by... Figure 7A The output reference frame (e.g., the estimated pose) represented by ray 710” is warped, but it is projected at time t2 represented by ray 710”', so the blue sub-image 712B” appears to be undershooted. This undershoot manifests as Figure 7B The left stripe image 712B in the text. Figure 7B The illustration shows distorted virtual content reconstructed in the user's mind, including a body with three overlapping R, G, B color fields (i.e., a body rendered in color). Figure 7BIncludes a red right stripe image color separation (“CBU”) artifact 712R”, a center image 712G”, and a blue left stripe image CBU artifact 712B”.

[0089] For the purpose of explanation, Figure 7B The overshoot and undershoot effects are exaggerated. The magnitude of these effects depends on the frame / field rate of the projection system and the relative velocity of the virtual content and the output reference frame (e.g., the estimated pose). When these overshoot and undershoot effects are small, they may appear as colored / rainbow stripes. For example, at a sufficiently slow frame rate, a white virtual object such as a baseball may have colored (e.g., red, green, and / or blue) stripes. Instead of having stripes, virtual objects with a selected solid color (e.g., red, green, and / or blue) that matches the sub-image may exhibit short-term interference (i.e., appearing to jump to an unwanted position during rapid movement and then jump back to the expected position after the rapid movement). Such solid-color virtual objects may also appear to vibrate during rapid movement.

[0090] To address these and other limitations, the system described herein uses multiple reference frames corresponding to multiple color sub-images / fields to distort the color virtual content. For example, Figure 8 A method for distorting color virtual content according to some embodiments is described. In step 812, the distortion unit 280 receives the virtual content, a base pose (i.e., the current pose of the AR systems 200, 200' (current reference frame)), a rendering pose (i.e., the pose of the AR systems 200, 200' for rendering the virtual content (source reference frame)), and an estimated illumination time for each sub-image / color field (R, G, B) associated with the display system 204 (i.e., an estimated illumination time for the display system 204 for each sub-image (estimated output reference frame for each sub-image)). In step 814, the distortion unit 280 divides the virtual content into each sub-image / color field (R, G, B).

[0091] In steps 816R, 816G, and 816B, pose estimator 282 uses a base pose (e.g., the current reference frame) and information about AR systems 200, 200' to estimate the pose at the corresponding estimated illumination time for the R, G, B sub-images / fields. In steps 818R, 818G, and 818B, transformation unit 284 uses the corresponding estimated R, G, and B poses and rendering pose (e.g., the source reference frame) to generate R, G, and B distorted virtual content based on the received virtual content sub-images / color fields (R, G, B). In step 820, transformation unit 284 combines the distorted R, G, and B sub-images / fields for sequential display.

[0092] Figure 9AThis illustration schematically depicts the use of a distortion system to distort colored virtual content according to some embodiments. Source virtual content 912 and... Figure 7A The source virtual content 712 is identical. The source virtual content 912 has three color portions: a red portion 912R; a green portion 912G; and a blue portion 912B. Each color portion corresponds to a color sub-image / field 912R', 912G", 912B"'. The warping system according to embodiments herein uses the corresponding output reference frame (e.g., estimated pose) represented by rays 910', 910", 910"' to warp each corresponding color sub-image / field 912R', 912G", 912B"'. These warping systems take into account the timing (i.e., t0, t1, t2) of the projection of the color sub-images 912R', 912G", 912B"' when warping the color virtual content. The timing of the projection depends on the frame / field rate of the projection system, which is used to calculate the timing of the projection.

[0093] Figure 9B It shows a composition similar to Figure 9A The virtual content warping system / method described herein generates warped color sub-images 912R', 912G”, 912B”'. Because the red, green, and blue sub-images 912R', 912G”, 912B”' are warped using the corresponding output reference frames (e.g., estimated poses) represented by rays 910', 910”, 910”', and projected at times t0, t1, t2 represented by the same rays 910', 910”, 910”', the sub-images 912R', 912G”, 912B”' are projected with the expected warping. Figure 9B The reconstruction of distorted virtual content according to some embodiments is shown, which includes a body in the user's mind with three overlapping R, G, B color fields (i.e., a body rendered in color). Figure 9B The body is rendered in color with essentially accurate detail because the three sub-images / fields 912R', 912G"', and 912B"' are projected at the appropriate time with the expected distortion.

[0094] The warping system according to the embodiments herein uses corresponding reference frames (e.g., estimated poses) that take into account the projection / irradiation time to warp the sub-images / fields 912R', 912G", 912B"', instead of using a single reference frame. Therefore, the warping system according to the embodiments herein warps the colored virtual content into separate sub-images of different colors / fields while minimizing warp-related color artifacts such as CBU. More accurate warping of the colored virtual content contributes to more realistic and believable AR scenes.

[0095] Explanatory graphics processing unit

[0096] Figure 10An exemplary graphics processing unit (GPU) 252 according to one embodiment is schematically depicted, which warps color virtual content to an output reference frame corresponding to various color sub-images or fields. GPU 252 includes input memory 1010 for storing the generated color virtual content to be warped. In one embodiment, the color virtual content is stored as primitives (e.g., Figure 11 (Triangle 1100 in the image). GPU 252 also includes a command processor 1012, which (1) receives / reads color virtual content from input memory 1010, (2) divides the color virtual content into color sub-images and divides these color sub-images into scheduling units, and (3) sends the scheduling units along the rendering pipeline in a wave or warp for parallel processing. GPU 252 also includes a scheduler 1014 for receiving scheduling units from command processor 1012. Scheduler 1014 also determines whether “new work” from command processor 1012 or “old work” returning from downstream in the rendering pipeline (described below) should be sent down the rendering pipeline at any given time. In effect, scheduler 1014 determines the sequence of various input data that GPU 252 processes.

[0097] GPU 252 includes GPU core 1016, which has multiple parallel executable cores / units (“shader cores”) 1018 for parallel processing scheduling units. Command processor 1012 divides the colored virtual content into a number equal to the number of shader cores 1018 (e.g., 32). GPU 252 also includes a “first-in, first-out” (“FIFO”) memory 1020 to receive output from GPU core 1016. From FIFO memory 1020, output can be routed back to scheduler 1014 as “old work” for insertion into the rendering pipeline for additional processing performed by GPU core 1016.

[0098] GPU 252 also includes a raster operation unit (“ROP”) that receives output from FIFO memory 1020 and rasterizes the output for display. For example, primitives of the color virtual content can be stored as coordinates of triangle vertices. After processing by GPU core 1016 (during which the three vertices 1110, 1112, and 1114 of triangle 1100 can be warped), ROP 1022 determines which pixels 1116 are inside triangle 1100 defined by the three vertices 1110, 1112, and 1114, and fills those pixels 1116 in the color virtual content. ROP 1022 can also perform depth testing on the color virtual content. To process the color virtual content, GPU 252 may include one or more ROPs 1022R, 1022B, and 1022G to process sub-images of different primary colors in parallel.

[0099] GPU 252 also includes buffer memory 1024 for temporarily storing distorted color virtual content from ROP 1022. The distorted color virtual content in buffer memory 1024 may include luminance / color and depth information at one or more X, Y positions in the field of view in the output reference frame. Output from buffer memory 1024 may be routed back to scheduler 1014 as "old work" for insertion into additional processing in the rendering pipeline performed by GPU core 1016, or for display in the corresponding pixels of the display system. Each fragment of color virtual content in input memory 1010 is processed at least twice by GPU core 1016. GPU core 1016 first processes vertices 1110, 1112, and 1114 of triangle 1100, and then processes pixel 1116 inside triangle 1100. When all fragments of color virtual content in input memory 1010 have been distorted and depth tested (if necessary), buffer memory 1024 will include all luminance / color and depth information required to display the field of view in the output reference frame.

[0100] Color Virtual Content Distortion System and Method

[0101] In standard image processing without head pose changes, the GPU 252's processing output is color / luminance and depth values ​​at individual X, Y values ​​(e.g., at each pixel). However, in the case of head pose changes, the virtual content is distorted to conform to the head pose variation. For color virtual content, each color sub-image is distorted individually. In existing methods for distorting color virtual content, a single output reference frame (e.g., corresponding to the green sub-image) is used to distort the color sub-image corresponding to the color image. As mentioned above, this can lead to color stripes and other visual artifacts, such as color blotches (CBU).

[0102] Figure 12 A method 1200 for distorting colored virtual content while minimizing visual artifacts such as CBU is described. In step 1202, a distortion system (e.g., GPU core 1016 and / or its distortion unit 280) determines the projection / irradiation time of the R, G, and B sub-images. This determination uses the frame rate and other characteristics associated with the projection system. Figure 9A In the example, the projection times correspond to t0, t1, and t2, and the rays 910', 910", and 910”'.

[0103] In step 1204, the warping system (e.g., GPU core 1016 and / or its pose estimator 282) predicts the pose / reference frame corresponding to the projection time of the R, G, and B sub-images. This prediction uses various system inputs, including the current pose, system IMU velocity, and system IMU acceleration. Figure 9AIn the example, the R, G, B poses / reference frames correspond to rays t0, t1, and t2, and 910', 910", and 910"'.

[0104] In step 1206, the warping system (e.g., GPU core 1016, ROP 1022, and / or its transform unit 284) warps the R sub-image using the R pose / reference frame predicted in step 1204. In step 1208, the warping system (e.g., GPU core 1016, ROP 1022, and / or its transform unit 284) warps the G sub-image using the G pose / reference frame predicted in step 1204. In step 1210, the warping system (e.g., GPU core 1016, ROP 1022, and / or its transform unit 284) warps the B sub-image using the B pose / reference frame predicted in step 1204. Using appropriate poses / reference frames to warp individual sub-images / fields distinguishes these embodiments from existing methods for warping colored virtual content.

[0105] In step 1212, the projection system operatively coupled to the distortion system projects the R, G, and B sub-images at the projection times of the R, G, and B sub-images determined in step 1202.

[0106] As mentioned above, Figure 10 The method 1000 depicted can also be executed on a separate twisting unit 290, which is independent of any GPU 252 or CPU 251. In yet another embodiment, Figure 10 The method 1000 described herein can be executed on CPU 251. In other embodiments, Figure 10 The method 1000 described herein can be executed on various combinations / sub-combinations of GPU 252, CPU 251 and individual twisting units 280. Figure 10 The method 1000 described herein is an image processing pipeline that can be executed using various execution models based on the availability of system resources at a specific time.

[0107] Using predicted poses / reference frames corresponding to each color sub-image / field to warp the color virtual content reduces color stripes and other visual anomalies. Reducing these anomalies results in more realistic and immersive mixed reality scenarios.

[0108] System Architecture Overview

[0109] Figure 13This is a block diagram of an illustrative computing system 1300 according to some embodiments. The computer system 1300 includes a bus 1306 or other communication mechanisms for transmitting information, which interconnects subsystems and devices, such as a processor 1307, system memory 1308 (e.g., RAM), static storage device 1309 (e.g., ROM), (e.g., magnetic or optical) disk drive 1310, communication interface 1314 (e.g., modem or Ethernet card), display 1311 (e.g., CRT or LCD), input device 1312 (e.g., keyboard), and cursor control.

[0110] According to some embodiments, computer system 1300 performs a specific operation by executing one or more sequences of one or more instructions contained in system memory 1308 via processor 1307. These instructions may be read into system memory 1308 from another computer-readable / usable medium (e.g., static storage device 1309 or disk drive 1310). In alternative embodiments, hardwired circuitry may be used instead of or in combination with software instructions to implement this disclosure. Therefore, embodiments are not limited to any particular combination of hardware circuitry and / or software. In one embodiment, the term "logic" should refer to any combination of software or hardware used to implement all or part of this disclosure.

[0111] As used herein, the term "computer-readable medium" or "computer-usable medium" refers to any medium that participates in providing instructions to processor 1307 for execution. Such media can take many forms, including but not limited to non-volatile media and volatile media. Non-volatile media include, for example, optical discs or magnetic disks, such as disk drive 1310. Volatile media include dynamic memory, such as system memory 1308.

[0112] Common forms of computer-readable media include, for example, floppy disks, hard disks, magnetic tapes, any other magnetic media, CD-ROMs, any other optical media, punched cards, paper tapes, any other physical media with a perforated pattern, RAM, PROMs, EPROMs, FLASH-EPROMs (e.g., NAND flash memory, NOR flash memory), any other memory chips or cassette tapes, or any other media that a computer can read.

[0113] In some embodiments, the execution of the instruction sequence for practicing this disclosure is performed by a single computer system 1300. According to some embodiments, two or more computer systems 1300 coupled via a communication link 1315 (e.g., a LAN, PTSN, or wireless network) can coordinate with each other to execute the instruction sequence required for practicing this disclosure.

[0114] Computer system 1300 can send and receive messages, data, and instructions, including programs, i.e., application code, via communication link 1315 and communication interface 1314. Received program code can be executed by processor 1307 upon receipt and / or stored in disk drive 1310 or other non-volatile memory for later execution. Database 1332 in storage medium 1331 can be used to store data accessed by system 1300 via data interface 1333.

[0115] Alternative warp / rendering pipeline

[0116] Figure 14 A warp / rendering pipeline 1400 for multi-field (color) virtual content is depicted according to some embodiments. Pipeline 1400 comprises two aspects: (1) multi-stage / decoupled warping and (2) cadence variation between applied frames and illumination frames.

[0117] (1) Multi-stage / decoupling distortion

[0118] Pipeline 1400 includes one or more warping stages. At 1412, the application CPU (“client”) generates virtual content, which is processed by the application GPU 252 into one or more (e.g., R, G, B) frames and pose 1414. At 1416, the warping / compositing CPU and its GPU 252 perform a first warp using a first estimated pose for each frame. Later in pipeline 1400 (i.e., closer to illumination), warping unit 1420 performs a second warp for each frame 1422R, 1422G, 1422B using a second estimated pose for each frame. The second estimated pose can be more accurate than the corresponding first estimated pose because the second estimated pose is determined to be closer to illumination. The frames 1422R, 1422G, 1422B, which have undergone two warps, are displayed at t0, t1, and t2.

[0119] The first warp can be the best guess, which can be used to align frames of the virtual content for later warps. This can be a computationally intensive warp. The second warp can be a sequential correction warp of the corresponding first warp frames. The second warp can be a less computationally intensive warp to reduce the time between the second estimated pose and the display / irradiation, thereby improving accuracy.

[0120] (2) Rhythm variation

[0121] In some embodiments, the client or application may not match the display or illumination rhythm (i.e., frame rate). In some embodiments, the illumination frame rate may be twice the application frame rate. For example, the illumination frame rate may be 60Hz, and the application frame rate may be 30Hz.

[0122] To address this mismatched warp issue, pipeline 1400 generates two sets of twice-warped frames 1422R, 1422G, 1422B (for projection at t0-t2) and 1424R, 1424G, 1424B (for projection at t3-t5) per frame 1414 from application CPU 1412 and GPU 252. Using the same frame 1414 and the first warped frame 1418, warping unit 1420 sequentially generates the first and second sets of twice-warped frames 1422R, 1422G, 1422B and 1424R, 1424G, 1424B. This provides twice the number of warped frames 1422, 1424 for each application frame 1414. The second warp can be a less computationally intensive warp to further reduce processor / power requirements and heat generation.

[0123] Although pipeline 1400 depicts an illumination / application ratio of 2:1, this ratio can vary in other embodiments. For example, the illumination / application ratio could be 3:1, 4:1, 2.5:1, etc. In embodiments with a fractional ratio, the most recently generated application frame 1414 can be used in the pipeline.

[0124] Alternative color separation minimization methods

[0125] Figure 15 A method 1500 is described, according to some embodiments, for minimizing color separation (CBU) artifacts in distorted multi-field (color) virtual content for sequential display. In step 1512, the CPU receives eye and / or head tracking information (e.g., from an eye-tracking camera or IMU). In step 1514, the CPU analyzes the eye and / or head tracking information to predict CBU artifacts (e.g., based on characteristics of the display system). In step 1516, if a CBU is predicted, method 1500 proceeds to step 1518, where the CPU increases the color field rate (e.g., from 180Hz to 360Hz). In step 1516, if no CBU is predicted, method 1500 proceeds to step 1526, where the image (e.g., segmented and distorted field information) is displayed using the system default color field rate and bit depth (e.g., 180Hz and 8 bits).

[0126] After increasing the color field rate in step 1518, the system re-analyzes the eye and / or head tracking information in step 1520 to predict CBU artifacts. In step 1522, if a CBU is predicted, method 1500 proceeds to step 1524, where the CPU reduces the bit depth (e.g., from 8 bits to 4 bits). After reducing the bit depth, the increased color field rate and reduced bit depth (e.g., 360 Hz and 4 bits) are used to display the image (e.g., segmented and distorted field information).

[0127] In step 1522, if CBU is not predicted, method 1500 proceeds to step 1526, where an increased color field rate and system default bit depth (e.g., 180 Hz and 8 bits) are used to display the image (e.g., segmented and distorted field information).

[0128] After displaying the image (e.g., segmented and distorted field information) using an adjusted or system default color field rate and bit depth, the CPU resets the color field rate and bit depth to the system default values ​​in step 1528 to repeat method 1500 before returning to step 1512.

[0129] Color field rate and bit depth are adjusted in response to the predicted CBU. Figure 15 Method 1500, as depicted herein, illustrates a method for minimizing CBU artifacts. Method 1500 can be combined with other methods described herein (e.g., method 800) to further reduce CBU artifacts. Although Figure 15 Most of the steps in the method 1500 described in the figure are executed by the CPU, and some or all of these steps may be executed by the GPU or a dedicated component.

[0130] Color virtual content distortion using in-field subcode timing in the field sequence display system.

[0131] Now refer to Figure 16A According to some embodiments, illustrative field sequence illumination sequences are shown relative to changes in head posture. (As in conjunction with...) Figure 9A The input image 1610 discussed has three color portions: a red portion; a green portion; and a blue portion. Each color portion corresponds to a corresponding color sub-image / field 1620, 1630, 1640 of the input image 1610. In some embodiments, when distorting the color virtual content, the distortion system considers the projection timing t0, t1, and t2 of the color field.

[0132] In the Red, Green, and Blue (RGB) color system, various colors can be formed based on combinations of red, green, and blue fields. Each color can be represented using a code that includes integers representing each of the red, green, and blue color fields. Red, green, and blue can each use 8 bits, with integer values ​​from 0 to 255 corresponding to the subcode. For example, red can be represented as (R=255, G=0, B=0), green as (0, 255, 0), and blue as (0, 0, 255). Various hues can be formed by modifying the integer values ​​representing the number of primary color fields (red, green, blue). This will be discussed in more detail below.

[0133] Figure 16BThe diagram illustrates a field depth pattern in an S-shaped growth to plateau to decay form for all subcodes of each constituent color field. For example, for the red field, all subcodes include all colors with codes (255, X, Y), where x and y can each take any value between 0 and 255. The S-shaped function (e.g., the field depth pattern) 1620′ corresponds to all subcodes of the red field, the S-shaped function 1630′ corresponds to all subcodes of the green field, and the S-shaped function 1640′ corresponds to all subcodes of the blue field. As shown, each S-shaped function 1620′, 1630′, and 1640′ has an S-shaped growth segment 1602, a plateau segment 1604, and a decay segment 1606.

[0134] Given a source input image 1610, as the user's head moves, the red, green, and blue color fields should display appropriate distortions corresponding to a given time in the sequence for the respective fields. In some embodiments, for a given positioning depth of the color field, the timing is positioned at the centroid of the display sequence of that color field. For example, the centroid of the red field displaying the S-curve function 1620' is aligned with the head pose position at a first time (t0); the centroid of the green field displaying the S-curve function 1630' is aligned with the head pose position at a second time (t1) after the first time; and the centroid of the blue field displaying the S-curve function 1640' is aligned with the head pose position at a third time (t2) after the first and second times.

[0135] Figure 17 The geometric relationships of different temporal sequences of the corresponding fields are shown as the head pose changes. Although the geometric positions of the red, green, and blue fields are offset from each other, the degree of change is consistent with the degree of change in head pose, thus presenting a more uniform image with overlapping fields at a given pixel to produce the desired clean color field.

[0136] Figures 16 and 17 each show the field depth pattern in the form of S-shaped growth to plateau to decay for all subcodes constituting the color field.

[0137] However, it will be understood that colors are not simply created as combinations of equal constituent subcodes, and that different colors require different numbers of red, green, and blue subcodes. For example, consider the International Commission on Illumination (CIE) 1931 color scheme. Figure 18A In the 1810 grayscale representation, any color is a combination of multiple field inputs represented by subcodes. Figure 16B The sigmoid functions 1620′, 1630′, and 1640′ represent the maximum potential of each field (e.g., (255, 0, 0) for red, (0, 255, 0) for green, and (0, 0, 255) for blue – sub-encoding via scheme 1810).

[0138] Specific colors may not share such a uniform subcode. For example, pink can have a combination of red 255, green 192, and blue 203 represented as (255, 192, 203); while orange can have a combination of red 255, green 165, and blue 0 represented as (255, 165, 0).

[0139] The subcodes that make up the color will correspondingly have a varying S-shape. Using the red color field as an example set, in Figure 18B In the diagram, various subcodes of the red color field are shown using sigmoid functions 1822, 1824, and 1826, each corresponding to a different subcode. For example, the first subcode of red (e.g., (255, 10, 15)) represented by sigmoid function 1822 can be red for the entire field time in the sequence, while sigmoid functions 1824 and 1826 represent different subcodes of red (i.e., a second subcode (e.g., (255, 100, 100)) and a third subcode (e.g., (255, 150, 200)), which correspond to a shorter activation time of a given pixel under the pulse of the spatial light modulator within the field time allocated in the sequence. Typically in display technology, the start of the growth phase is the same for any subcode, but the decay phase begins at different times. Thus, when a subcode is started at a common start time in the field time sequence, the specific sigmoid pattern and the resulting centroid of any given subcode will move relative to each other.

[0140] In a conventional on-site sequential display system, sub-codes start at a common time, causing the centroids of the S-shaped patterns of the sub-codes to shift from one another. For example... Figure 18B As shown, the centroid of the first red subcode, represented by the sigmoid function 1822, appears at t0, but the centroids of the second and third red subcodes, represented by the sigmoid functions 1824 and 1826, respectively, appear at t0. 0-n and t 0--n-m Group 1850 illustrates the range of possible head pose positions, which may require each subcode to be distorted to that range of possible head pose positions for effective viewing during head movements of the head-mounted display device.

[0141] When a user's head pose changes, the different centroid times of a subcode within a single field (i.e., color) manifest as different positions, even though any distortion of that field may have occurred previously in other ways. This could lead to intracolor separation because the distortion would be applied to the offset position of that subcode. In other words, because the timing of the head pose does not match the timing of the centroid pattern of the subcode, a pixel intended to be pink may be geometrically offset from a pixel intended to be orange.

[0142] Figure 19This principle is illustrated more specifically for a single field with various subcode possibilities, since the user's head position at t0 is at x, y, which can be correctly aligned with the first subcode represented by the sigmoid function 1822, but the second and third subcodes, represented by the sigmoid functions 1824 and 1826 respectively, correspond geometrically to x1, y1 and x2, y2. If the spatial light modulator carrying this image data is to be activated at the common time t0, the appearance of the pixels transmitting the image data of the second and third subcodes, represented by the sigmoid functions 1824 and 1826 respectively, will appear offset from their proper positions. This problem is equally complicated when extending the green and blue color fields and their corresponding subcodes.

[0143] In some embodiments, this is corrected by having increasingly smaller head pose samples to allow any given color subcode to have an S-shaped centroid that is timed for a given head pose. For example, a centroid for t can be calculated. 0-n-m The specific head pose is applied and then used to the third subcode represented by the sigmoid function 1826, and the specific head pose for t can be calculated. 0-n A new, specific head pose can be calculated and applied to a second subcode represented by the S-function 1824, and a specific head pose for t0 can be calculated and applied to a first subcode represented by the S-function 1822. For believable augmented reality perception, an ideal projector frequency is faster than 120 Hz. For field-sequential displays with three fields, this only allows milliseconds for calculating any single head pose. For each sample of an additional head pose in hundreds of subcodes within each field, the computational power and desired shape factor may be excessively expensive.

[0144] According to some embodiments, the sigmoid function shape of a given subcode can be mixed. Various display systems and spatial light modulators employ media and components that do not respond immediately to input. Figure 20 An exemplary hysteresis that may occur in some systems is illustrated. For example, in a liquid crystal on silicon (LCoS) display, a given liquid crystal layer can cause a delay t during the startup S-curve when a given pixel can be activated. b This lag may exacerbate any head pose changes that already have subcodes as described above, or cause image contouring, where subcodes of a single color scheme appear as stripes across the image. Figure 21 This illustrates the exaggerated effect of image contours in a field-sequential display, where timing issues with subcodes are prone to occur in pixel implementations when the display is moved.

[0145] To mitigate these timing issues without sacrificing excessive computational power, in some embodiments, the centroid of each S-shape representing a subcode is temporally modified to correspond to the common head pose time of all subcodes in the common field. For example... Figure 22The description in the text does not involve starting at a common source time, but rather starting subcodes at different times to present their corresponding bit-depth S-shaped centroids at a common time t0. In some embodiments, the start time of individual or all codes is further offset, such that the S-shape is calculated at time t0-t b Alignment is achieved because the pixel response time will be aligned with the common head pose measurement. In other words, the modulation and timing of each field input value (i.e., red, green, blue) to the spatial light modulator are constructed such that the centroid of the output light for each subcode is the same within the field channel.

[0146] In some embodiments, instead of creating a single subcode input (such as by...), Figure 22 Instead of a single S-shaped function (represented by the second subcode in 1826), a series of pulses creates one or more per-field inputs. Figure 23 In this context, the center pulse 2302 is centered on the temporal sequence (t0) of the fields within the sequentially displayed frames. That is, the center pulse is centered on the time of projection of the color field used for distortion (e.g., the time of the head pose sample used to distort the color field). The centroid of pulse 2302 is at time t0.

[0147] The second pulse 2304 is measured from the centroid of the central pulse 2302 at time t0 (although it occurs before the central pulse 2302, it is called the second pulse because it is measured relative to the central pulse 2302, which can be called the first pulse), so that at time t 0-p This aligns the end of the decay phase of the second pulse 2304 with the beginning of the growth phase of the central pulse 2302 in time. The centroid of the second pulse 2304 is at time t. c2 At that point, the predetermined amount of time preceding time t0 (e.g., Figure 23 t0-t c2 ) appears (i.e., it occurs before time t0).

[0148] The third pulse 2306 (occurring after the central pulse 2302) is measured from the centroid of the central pulse 2302 at time t0, so that at time t 0+r This temporarily aligns the start of the growth phase of the third pulse 2306 with the end of the decay phase of the central pulse 2302 in time. The centroid of the third pulse 2306 is at time t. c3 At that point, the predetermined amount of time after time t0 (e.g., Figure 23 t in c3 -t0) appears (i.e., it appears after time t0).

[0149] In some embodiments, time t c3 The difference between time t0 and time t0 can be equal to the difference between time t0 and time t. c2The difference between them. That is, the centroid of the second pulse 2304 appears before the centroid of the center pulse 2302 by a predetermined amount of time, and the centroid of the third pulse 2306 appears after the centroid of the center pulse 2302 by the same predetermined amount of time. This symmetry of the centroids produces selectable bit depths throughout the field sequence and is more uniformly distributed around the head pose sample. For example, a single pulse for a subcode of the desired bit depth requires precise timing with respect to the head pose time to a specific bit depth; a lower bit depth with a lower pulse spread for the cumulative bit depth around the head pose timing is less easy to colorize due to variations in the direction of the head pose change or the variable velocity, because only one of one or more pulses will be temporally aligned with the head pose sample (e.g., the center pulse 2302).

[0150] like Figure 23 As described in the text, the second pulse 2304 in t 0-p The third pulse 2306 is attached to the center pulse 2302, and the third pulse 2306 is attached to the center pulse 2302. 0+r This is attached to the center pulse 2302. For example... Figure 23 As shown, the growth stage of the second pulse 2304 can occur at time t. 0-y Starting at point t, and the decay phase of the second pulse 2304 can occur at time t. 0-p The process ends at time t. That is, it can be completed at time t. 0-y and time t 0-p The second pulse 2304 is defined between these points. The growth phase of the third pulse 2306 can be defined at time t. 0+r Starting at point t, and the decay phase of the third pulse 2306 can occur at time t. 0+x The process ends at time t. That is, it can be completed at time t. 0+r and time t 0+x The third pulse 2306 is defined between these two pulses. Those skilled in the art will recognize that p and r need not be equal, because the decay of the second pulse 2304 may be longer or shorter than the growth phase of the third pulse 2306, and the corresponding alignment of the centroids may require different timing relative to t0 for each, even if the centroid positions are expected to produce equal distributions in time.

[0151] Figure 23 Three discrete pulses 2302, 2304, and 2306 are shown, which represent a given color subcode (e.g., by...). Figure 22 The centroid of the sigmoid function (representing the color subcode) at time t0 grows toward the edge of the sigmoid function. The center pulse 2302 is used in conjunction with the second pulse 2304 and the third pulse 2306 to create 256 modulation steps for each field (i.e., color).

[0152] Figure 23The pulses 2302, 2304, and 2306 shown can be used in conjunction with a computer-implemented method for distorting multi-field color virtual content for sequential projection. For example, when first and second color fields (e.g., one or more of red, blue, or green) with different first and second colors (e.g., subcodes of red, blue, or green) are obtained, a first time for projection of the first color field to be distorted can be determined. When predicting a first pose corresponding to the first time (e.g., time t0), for each of the first colors in the first color field, a representation of one of the first colors in the first color field (e.g., by...) can be identified. Figure 22 The input can be a single sigmoid function 1824 representing a color subcode, and the input can be reconfigured to create a series of pulses (e.g., a center pulse 2302, a second pulse 2304, and a third pulse 2306 centered at a first time t0) for each field input. Each pulse in the series of pulses can be distorted based on a first pose. Then, a distorted first color field can be generated based on the distorted series of pulses; and pixels on a sequential display can be activated based on the distorted series of pulses to display the distorted first color field.

[0153] In some embodiments, the center pulse 2302 may include a series of short time slots (ts) arranged outward from the center. 1-1 , ts 1-2 , ts 1-3 , ts 1-4 , ts 1-5 ,ts 1-6 That is, time slot ts 1-1 ts 1-2 It is formed as the centroid immediately following time t0. Time slot ts 1-3 ts 1-4 ts 1-5 ts 1-6 Relative to time slot ts 1-1 ts 1-2 Arranged outwards from time t0. In each time slot (ts) 1-1 ts 1-2 ts 1-3 ts 1-4 ts 1-5 ts 1-6 During this period, pixels on the display device (e.g., LCoS pixels) may or may not be activated. That is, pixels on the sequential display may be activated during a subset of the time slots of the center pulse 2302. The activation of pixels on the sequential display may depend on the subcode associated with the center pulse 2302. In some embodiments, only a subset of the time slots may be activated. For example, for the lowest color code, only the center time slot (e.g., ts) may be activated. 1-1 ts1-2 (That is, only the center slot can result in an active pixel on the display device). The higher the color code, the more slots open outwards from the center.

[0154] According to some embodiments, the second pulse 2304 and the third pulse 2306 may include a time slot (ts) that is longer than that of the center pulse 2302. 1-1 ts 1-2 ts 1-3 ts 1-4 ts 1-5 ts 1-6 A longer time slot. For example, the second pulse 2304 may include a time slot (ts) with a duration longer than that of the center pulse 2302. 1-1 ts 1-2 ts 1-3 ts 1-4 ts 1-5 ts 1-6 Longer (i.e., larger) time slots (ts) 2-1 ts 2-2 ts 2-3 ts 2-4 The second pulse 2304 time slot (ts) 2-1 ts 2-2 ts 2-3 ts 2-4 (This can be arranged from later to earlier. That is, time slot ts) 2-1 Relative to the time slot ts within the second pulse 2304 2-2 ts 2-3 ts 2-4 It appears later in time. Similarly, the third pulse 2306 may include a time slot (ts) whose duration is longer than that of the center pulse 2302. 1-1 ts 1-2 ts 1-3 ts 1-4 ts 1-5 ts 1-6 Longer time slots (ts) 3-1 ts 3-2 ts 3-3 ts 3-4 The third pulse 2306 time slot (ts) 3-1 ts 3-2 ts 3-3 ts 3-4 (This can be arranged from earlier to later. That is, time slot ts) 3-1 Relative to the time slot ts within the third pulse 2306 3-2 ts 3-3 ts 3-4It appears earlier in time. Therefore, the pulses can be arranged to grow outward from the central pulse 2302.

[0155] In some embodiments, pixels on the sequential display can be activated during a subset of the time slots of the second pulse 2304 and / or the third pulse 2306. When time slots are opened in the second pulse 2304 and the third pulse 2306 to produce a higher color code, care should be taken to open the corresponding time slots in the second pulse 2304 and the third pulse 2306 together to maintain the overall centroid in the color code. If system constraints require (typically) to open a single time slot in the second pulse 2304 or the third pulse 2306 for adjacent codes, care should be taken to keep that additional time slot short, or to use spatial / temporal jitter to prevent the light energy from shifting too much from the centroid. This also avoids additional contour artifacts caused by head or eye movements.

[0156] The center pulse 2302 can be considered as the least significant bit (LSB) of the digital color code, while the second pulse 2304 and the third pulse 2306 are similar to the most significant bit (MSB) of the digital color code. The combination of the center pulse 2302 with the second pulse 2304 and the third pulse 2306 produces many combinations that can be used to construct 256 modulation steps.

[0157] To achieve maximum brightness, it may be necessary to create a single pulse for the highest modulation step, thus merging the central pulse 2302, the second pulse 2304, and the third pulse 2306. During the transition from three pulses to one pulse, smaller time slots can be opened to keep the step size small. In this case, smaller time slots can be added at the beginning of the second pulse 2304, arranged from later to earlier. For example, as... Figure 23 As shown, time slot ts can be... 2-4 (That is, the time slot at the beginning of the second pulse 2304) is divided into smaller time slots (ts) arranged from later to earlier. 2-4-1 ts 2-4-2 ts 2-4-3 That is, time slot ts 2-4-1 Relative to time slot ts within the second pulse 2304 2-4-2 and ts 2-4-3 It appears later in time. Similarly, smaller time slots are added to the end of the third pulse 2306, arranged from earlier to later. For example, as... Figure 23 As shown, time slot ts can be... 3-4 (That is, the time slot at the end of the third pulse 2306) is divided into smaller time slots arranged from earlier to later (ts 3-4-1 ts 3-4-2 ts 2-4-3 That is, time slot ts 3-4-1 Relative to time slot ts within the third pulse 23063-4-2 and ts 3-4-3 It appears earlier in time. In both cases, the shorter time slot (i.e., ts) 2-4-1 ts 2-4-2 ts 2-4-3 and ts 3-4-1 ts 3-4-2 ts 2-4-3 The arrangement direction of the pulses and their corresponding pulses (i.e., the second pulse 2304 and the third pulse 2306) has a longer time slot (ts). 2-1 ts 2-2 ts 2-3 ts 2-4 and ts 3-1 ts 3-2 ts 3-3 ts 3-4 )same.

[0158] Because many optical modulators (such as LCoS, lasers in scanning displays, digital light processing (DLP), liquid crystal displays (LCDs), and / or other display technologies) have asymmetrical turn-on and turn-off times, the lengths and arrangement of the three pulses may need to be asymmetrical in order to keep the centroid at a fixed point. For example, if the turn-on time is longer than the turn-off time, the centroid will be later in the field than the center time. According to various embodiments, each of the three pulses can be constructed in a similar manner using asymmetrical time slot lengths and arrangements.

[0159] The combination of the pulse length of the center pulse 2302 with the pulse lengths of the second and third pulses 2304 and 2306 can produce more than 256 possible combinations. A subset of these combinations is used to create 256 modulation steps. The combination can be selected based on a number of factors, including: the closest match to the desired luminance response curve (i.e., linear gamma, standard red-green-blue (sRGB) gamma), the minimum change in centroid across all color codes, the minimum change in centroid for adjacent color codes, and the small luminance variation of the combination across temperature and process.

[0160] Because the on / off times can vary with temperature, voltage, process, and other variables, 256 different groups can be selected for different conditions. For example, when the device is first turned on, a first group for low temperatures can be selected, while a different second group can be selected once the device has heated up and reached a steady-state temperature. Any number of groups can be used to constrain the contour and maximize image quality throughout the operating conditions.

[0161] In some embodiments, Figure 23The symmetry of the bit depth timing in the code prevents overly bright or dark streaks because interference between subcodes is mitigated (depending on the head's movement direction from left to right). That is, if the subcodes are not time-adjusted, and the user moves their head in a specific direction, the bits of a particular subcode might appear where color information is presented, where an undesirable timing pattern in the form of an S-shape with respect to the subcode's bit depth is not expected. Figure 24 As shown, region 2250 depicts an area in which head movement can cause a specific subcode 2406 to display color while two other subcodes 2402 and 2404 in the same field are in the decay phase, and unintentionally display pixels when no color in any subcode is intended to be shown to the user based on a given head pose timing sample. Those skilled in the art will understand that additional configurations are possible to establish the desired bit depth for one or more subcodes.

[0162] Figure 25 A method for distorting and coloring virtual content according to some embodiments is described. Figure 25 The steps described can be performed for each color field (R, G, B). In some embodiments, in Figure 25 The steps described herein can be performed as sub-steps of steps 816R, 816G, and / or 816B.

[0163] Each color field (R, G, B) includes one or more colors, each represented by a subcode. For each color (e.g., a subcode) of one or more colors in a selected color field, in step 2502, the pose estimator identifies an input (e.g., an S-shape) representing the subcode of that color field. In step 2504, the pose estimator reconfigures the input into a series of pulses (e.g., three pulses) to create one or more per-field inputs. In step 2506, the transformation unit warps each of the series of pulses based on a first pose. In step 2508, the transformation unit generates a warped first color field based on the warped series of pulses. In step 2510, the transformation unit activates pixels on a sequential display based on the warped series of pulses to display the warped first color field. The same steps 2502-2510 can be performed for all color fields (R, G, B).

[0164] This disclosure includes methods that can be performed using the subject device. These methods may include actions to provide such a suitable device. Such provision can be performed by a user. In other words, the "providing" action only requires the user to acquire, access, approach, locate, set, activate, power on, or otherwise act to provide the necessary device in the subject method. The methods described herein can be performed in any logically possible order of the events and in the sequence of the events.

[0165] The exemplary aspects of this disclosure, as well as details regarding material selection and manufacturing, have been set forth above. Further details regarding this disclosure can be understood in conjunction with the patents and publications cited above, and what is commonly known or understood by those skilled in the art. The same applies to the method-based aspects of this disclosure in terms of additional actions used in a common or logical manner.

[0166] Furthermore, while this disclosure has been described with reference to several examples that optionally incorporate various features, this disclosure is not limited to what is intended to be described or indicated for each variation of this disclosure. Various changes may be made to the described disclosure and equivalents may be substituted (whether or not listed herein or not included for brevity) without departing from the true spirit and scope of this disclosure. Moreover, where numerical ranges are provided, it should be understood that every intermediate value between the upper and lower limits of the range, as well as any other specified value or intermediate value within said range, is included within this disclosure.

[0167] Furthermore, it is conceivable that any optional features of the described inventive alteration may be stated and claimed independently, or in combination with any one or more features described herein. Reference to singular items includes the possibility of multiple identical items. More specifically, as used herein and in the claims associated therewith, unless otherwise specified, the singular forms “a,” “an,” “the,” and “the” include plural indicators. In other words, the use of the article allows for “at least one” of the subject matter items described above and in the claims associated with this disclosure. It should be further noted that these claims may be drafted to exclude any optional elements. Therefore, this statement is intended to serve as a prior basis for the use of exclusive terms such as “merely,” “only,” etc., in conjunction with the statement of the claim elements or for the use of a “negative” limiting term.

[0168] Without using such exclusive terminology, the term "comprising" in the claims associated with this disclosure shall allow for the inclusion of any additional elements—whether or not a given number of elements are enumerated in those claims, or the addition of a feature may be considered as a transformation of the nature of the elements set forth in those claims. Unless otherwise expressly defined herein, all technical and scientific terms used herein shall be understood as broadly as possible while preserving the validity of the claims.

[0169] The breadth of this disclosure is not limited to the examples provided and / or this specification, but is limited to the scope of the language of the claims associated with this disclosure.

[0170] In the foregoing specification, this disclosure has been described with reference to specific embodiments thereof. However, it should be understood that various modifications and changes can be made without departing from the broader spirit and scope of this disclosure. For example, the above process flow has been described with reference to a specific sequence of process actions. However, the order of many described process actions can be changed without affecting the scope or operation of this disclosure. These descriptions and drawings are accordingly considered illustrative rather than restrictive.

Claims

1. A computer-implemented method for distorting multi-field color virtual content for sequential projection, comprising: Obtain a primary color field comprising multiple colors, wherein each of the multiple colors represents a different hue of the primary color field; Determine the first moment for the primary color field used to project the distortion; Predict the pose corresponding to the first time point; For the selection of one color from the multiple colors in the primary color field: The input represents the selected color among the multiple colors in the primary color field; The input is reconfigured into a series of pulses, wherein the series of pulses creates multiple per-field inputs; Based on the posture, distort each pulse in the series of pulses; In this context, one of the selected colors from the plurality of colors in the primary color field is individually distorted; A distorted primary color field is generated based on a series of distorted pulses; and Based on the series of pulses that cause the distortion, pixels on a sequential display are activated to display the distorted primary color field.

2. The method according to claim 1, wherein, The series of pulses includes a central pulse centered at the first time, a second pulse occurring before the central pulse, and a third pulse occurring after the central pulse.

3. The method according to claim 2, wherein, The end of the decay phase of the second pulse is time-aligned with the start of the growth phase of the central pulse, and The start of the growth phase of the third pulse is time-aligned with the end of the decay phase of the central pulse.

4. The method according to claim 2, wherein, The centroid of the central pulse appears at the first time, the centroid of the second pulse appears at the second time before the first time, and the centroid of the third pulse appears at the third time after the first time.

5. The method according to claim 4, wherein, The difference between the first time and the second time is equal to the difference between the first time and the third time.

6. The method according to claim 2, wherein, The center pulse includes a first time slot group, each having a first duration, and the second pulse and the third pulse include a second time slot group, each having a second duration greater than the first duration.

7. The method according to claim 6, wherein, The pixel on the sequential display is activated during either the first time slot group or a subset of the second time slot group.

8. The method according to claim 7, wherein, During the time slot of the center pulse, the pixel on the sequential display is activated according to a color code associated with a selected color among the colors in the primary color field.

9. The method according to claim 7, wherein, For the corresponding time slot in the second pulse and the third pulse, the pixel on the sequential display is activated.

10. The method according to claim 1, wherein, The primary color field is one of the following: red, green, or blue.

11. A system for distorting multi-field color virtual content for sequential projection, comprising: A twisting unit for receiving a primary color field comprising multiple colors, wherein each of the multiple colors represents a different hue of the primary color field, the twisting unit comprising: A pose estimator is used to determine the first time of the projected distorted primary color field and predict the pose corresponding to the first time; and Transformation unit, used for: For the selection of one color from the multiple colors in the primary color field: The input represents the selected color among the multiple colors in the primary color field; The input is reconfigured into a series of pulses, wherein the series of pulses creates multiple per-field inputs; Based on the posture, each of the series of pulses is distorted, wherein one of the selected colors from the plurality of colors in the primary color field is distorted individually; Based on a series of distorted pulses, the distorted primary color field is generated; and Based on the series of pulses that cause the distortion, pixels on a sequential display are activated to display the distorted primary color field.

12. The system according to claim 11, wherein, The series of pulses includes a central pulse centered at the first time, a second pulse occurring before the central pulse, and a third pulse occurring after the central pulse.

13. The system according to claim 12, wherein, The end of the decay phase of the second pulse is time-aligned with the start of the growth phase of the central pulse, and The start of the growth phase of the third pulse is time-aligned with the end of the decay phase of the central pulse.

14. The system according to claim 12, wherein, The centroid of the central pulse appears at the first time, the centroid of the second pulse appears at the second time before the first time, and the centroid of the third pulse appears at the third time after the first time.

15. The system according to claim 12, wherein, The center pulse includes a first time slot group, each having a first duration, and the second pulse and the third pulse include a second time slot group, each having a second duration greater than the first duration.

16. The system according to claim 15, wherein, The pixel on the sequential display is activated during either the first time slot group or a subset of the second time slot group.

17. The system according to claim 16, wherein, During the time slot of the center pulse, the pixel on the sequential display is activated according to a color code associated with a selected color among the colors in the primary color field.

18. The system according to claim 16, wherein, For the corresponding time slot in the second pulse and the third pulse, the pixel on the sequential display is activated.

19. A computer-implemented method for distorting multi-field color virtual content for sequential projection, the method comprising: A first primary color field including multiple first colors and a second primary color field including multiple second colors are obtained, wherein the multiple second colors are different from the multiple first colors of the first primary color field, wherein each of the multiple first colors represents a different hue of the first primary color field, and wherein each of the multiple second colors represents a different hue of the second primary color field; For each of the first primary color field and the second primary color field: Determine the first moment for the primary color field used to project the distortion; Predict the pose corresponding to the first time point; For each of the multiple colors in the primary color field: The input represents the color among the multiple colors in the primary color field; The input is reconfigured into a series of pulses, wherein the series of pulses creates multiple per-field inputs; Based on the posture, each of the series of pulses is distorted, wherein each of the multiple colors in the primary color field is distorted individually; The distorted primary color field is generated based on a series of distorted pulses corresponding to all the multiple colors in the primary color field; and Based on the series of pulses that cause the distortion, pixels on a sequential display are activated to display the distorted primary color field.

Images