Virtual, Augmented, and Mixed Reality Systems and Methods

Abstract

To provide a system and a method for imaging, visualizing and displaying images of virtual reality, augmented reality, and mixed reality.SOLUTION: A method in a virtual reality (VR), augmented reality (AR), or mixed reality (MR) system 800 according to the present invention has a step at which a GPU 802 determines / detects absence of image data, a step of shutting down a portion / component / function of the GPU, a step of shutting down a communication link between the GPU and a display bridge (DB) 806, a step of shutting down a portion / component / function of the DB, a step of shutting down a MIPI link 816 between the DB and left and right display panes 812, and a step of shutting down a portion / component / function of the display panels.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] (Copyright Notice) Part of the disclosure of this patent document contains materials that should be protected by copyright. The copyright owner has no objection to anyone copying this patent document or this patent disclosure as long as it appears in the patent file or records of the Patent and Trademark Office. However, in other cases, the copyright owner retains all copyrights.

[0002] This disclosure relates to virtual reality, augmented reality, and mixed reality imaging, visualization, and display systems and methods.

Background Art

[0003] Modern computing and display technologies are facilitating the development of virtual reality (VR), augmented reality (AR), and mixed reality (MR) systems. VR systems create a simulated environment for users to experience. This can be done by presenting computer-generated images to the user through a head-mounted display. This image creates a sensory experience that immerses the user in the simulated environment. VR scenarios typically involve only the presentation of computer-generated images and not the inclusion of actual real-world images.

[0004] AR systems generally complement the real-world environment with simulated elements. For example, an AR system may provide a user with a view of the surrounding real-world environment via a head-mounted display. However, computer-generated images can also be presented on the display to enhance the real-world environment. The computer-generated images can include elements that are contextually relevant to the real-world environment. Such elements can include simulated text, images, objects, etc. MR systems also introduce simulated objects into the real-world environment, but these objects typically feature a greater degree of interaction than those in AR systems. The simulated elements can often be bidirectional in real time.

[0005] Figure 1 depicts an exemplary AR / MR scene 1, where the user sees a real-world park setting 6 featuring people, trees, buildings in the background, and a concrete platform 20. In addition to these items, computer-generated images are also presented to the user. The computer-generated images can include, for example, a robotic image 10 standing on the real-world platform 20 and an avatar character 12 in the form of a flying cartoon that appears as an anthropomorphic bumblebee, but these elements 12, 10 do not actually exist within the real-world environment.

[0006] Various optical systems generate images at various depths for displaying VR, AR, or MR scenarios. The human visual perception system is complex, and it is difficult to produce VR / AR / MR technologies that facilitate a comfortable, natural, and rich presentation of virtual image elements among other virtual or real-world image elements. For example, techniques for providing control data for controlling how image data is displayed, techniques for correcting optical distortions in the image data, and techniques for warping the image data based on the user's head pose. Improved techniques for processing image data in such systems are needed. VR / AR / MR technologies also have other system and optical challenges that increase the importance of size and portability issues, battery life issues, system overheating issues, and power-efficient image rendering. Improved techniques are needed to address these issues. The systems and methods described herein are configured to address these and other challenges.

[0007] What is needed are techniques or multiple techniques for improving conventional techniques and / or other contemplated approaches. Some of the approaches described in this background section are approaches that can be pursued but are not necessarily approaches that have been previously contemplated or pursued. SUMMARY OF THE INVENTION MEANS FOR SOLVING THE PROBLEM

[0008] In one embodiment, a method in a virtual, augmented, or mixed reality system includes the step of a GPU determining / detecting the absence of image data. The method also includes the step of shutting down a part / component / function of the GPU. The method further includes the step of shutting down the communication link between the GPU and the DB. Further, the method includes the step of shutting down a part / component / function of the DB. Additionally, the method includes the step of shutting down the communication link between the DB and the display panel. The method also further includes the step of shutting down a part / component / function of the display panel.

[0009] In one or more embodiments, the method includes the step of reorganizing frame data to reduce transfer time. The method may also include the step of the GPU DP port sending a custom STP message to the DB. The method may also include the step of the GPU sending an STP message to a DB AUS message. A part / component / function of the GPU may be selected from the group consisting of memory read, compression, and color segmentation. A part / component / function of the DB may be a memory write. A part / component / function of the display panel may be selected from the group consisting of video RAM and MIPI receiver.

[0010] In one or more embodiments, the method includes the step of the GPU sending a wake-up signal to the DB. The GPU may send the wake-up signal via an AUX communication link. The method may also include the step of the GPU sending a wake-up signal to the communication link between the GPU and the DB. A part / component / function of the GPU, the communication link between the GPU and the DB, a part / component / function of the DB, the communication link between the DB and the display panel, and a part / component / function of the display panel may be shut down asynchronously. The method may also include the step of the DB sending a built-in line control message to the display panel.

[0011] In another embodiment, a method in a virtual, augmented, or mixed reality system includes the step of a GPU receiving a frame of image data. The method also includes the step of the GPU identifying a plurality of regions / parts / sections / tiles within the frame of image data that have changed from the previous frame of image data. The method further includes the step of the GPU moving at least some of the plurality of regions / parts / sections / tiles to the start of the frame of data to form a reordered frame of image data. Additionally, the method includes the step of the GPU transmitting the reordered frame of image data to the DB. In addition, the method includes the step of shutting down a part / component / function of the GPU, a communication link between the GPU and the DB, a part / component / function of the DB, a communication link between the DB and the display panel, and a part / component / function of the display panel.

[0012] In one or more embodiments, the method includes the step of the GPU compressing the reordered frame of image data before transmitting the reordered frame of image data to the DB. The reordered frame of image data may be smaller than the frame of image data. The method may also include the step of the DB storing the reordered frame of image data in a buffer.

[0013] In one or more embodiments, the method includes the step of determining the size of the reordered frame of image data. The method further includes the step of shutting down a part / component / function of the GPU, a communication link between the GPU and the DB, a part / component / function of the DB, a communication link between the DB and the display panel, and a part / component / function of the display panel only when the reordered frame of image data is smaller than a predetermined maximum size.

[0014] In one or more embodiments, the method includes the step of the GPU sending an STP message to the DB after sending the rearranged frames of image data to the DB. The method may further include the step of the GPU sending the STP message to the DB via the SDP.

[0015] In one or more embodiments, a part / component / function of the GPU is selected from the group consisting of memory read, compression, and color segmentation. A part / component / function of the DB may be memory write. A part / component / function of the display panel may be selected from the group consisting of video RAM and MIPI receiver.

[0016] In one or more embodiments, the method includes the step of the GPU sending a wake-up signal to the DB. The GPU may send the wake-up signal via the AUX communication link. A part / component / function of the GPU, the communication link between the GPU and the DB, a part / component / function of the DB, the communication link between the DB and the display panel, and a part / component / function of the display panel may be shut down asynchronously.

[0017] In one or more embodiments, the method includes the step of the DB reconstructing a frame of image data from the rearranged frames of image data. The method may further include the step of setting a portion of the frame of image data that is not within a plurality of regions / parts / sections / tiles within the frame of image data to a background color. The method may also include the step of the DB blending the rearranged frame of image data with the previous frame of image data. The method may further include the step of the DB blending the rearranged frame of image data with the image data related to the updated foveation region. The method may also include the step of the DB masking the previous frame of image data before blending it with the rearranged frame of image data.

[0018] In one or more embodiments, the method includes the step of the DB scaling the reordered frames of the image data. The method may further include the step of the DB receiving a scaling factor from the GPU and the step of the DB using the scaling factor to scale the reordered frames of the image data. The scaling may be part of a foveation operation. The method may also include the step of the DB performing a function on the image data, where the function is selected from the group consisting of warping, pixelated dimming, occlusion, chromatic aberration correction, frame rate, and augmentation. The method may further include the step of storing the reordered frames of the image data in a FIFO memory before shutting down a part / component / function of the GPU. The method may also include the step of the DB sending an embedded line control message to the display panel.

[0019] In yet another embodiment, a method in a virtual, augmented, or mixed reality system includes the step of the GPU splitting a first color field into a first partial first color field and a second partial first color field. The method also includes the step of the GPU splitting a second color field into a first partial second color field and a second partial second color field. The method further includes the step of the GPU sending the first partial first color field to the DB. Further, the method includes the step of the GPU sending the first partial second color field to the DB after sending the first partial first color field. Additionally, the method includes the step of the GPU sending the second partial first color field to the DB after sending the first partial second color field. The method also includes the step of the GPU sending the second partial second color field to the DB after sending the second partial first color field.

[0020] In one or more embodiments, the method includes the step of the GPU dividing a third color field into a first partial third color field and a second partial third color field. The method may also include the step of the GPU transmitting the first partial third color field to the DB after transmitting the first partial second color field and before transmitting the second partial first color field. The method may further include the step of the GPU transmitting the second partial third color field to the DB after transmitting the second partial second color field. The method may also include the step of the GPU transmitting the first partial first and second color fields and the second partial first and second color fields as a single vertically encoded data set.

[0021] In yet another embodiment, a method in a virtual, augmented, or mixed reality system includes the GPU dividing a first color field into a first partial first color field, a second partial first color field, and a third partial first color field. The method also includes the GPU dividing a second color field into a first partial second color field, a second partial second color field, and a third partial second color field. The method further includes the GPU dividing a third color field into a first partial third color field, a second partial third color field, and a third partial third color field. Further, the method includes the GPU transmitting the first partial first color field to the DB. Additionally, the method includes the GPU transmitting the first partial second color field to the DB after transmitting the first partial first color field. The method also includes the GPU transmitting the first partial third color field to the DB after transmitting the first partial second color field. The method further includes the GPU transmitting the second partial first color field to the DB after transmitting the first partial third color field. Further, the method includes the GPU transmitting the second partial second color field to the DB after transmitting the second partial first color field. Additionally, the method includes the GPU transmitting the second partial third color field to the DB after transmitting the second partial second color field. The method also includes the GPU transmitting the third partial first color field to the DB after transmitting the second partial third color field. The method further includes the GPU transmitting the third partial second color field to the DB after transmitting the third partial first color field. Further, the method includes the GPU transmitting the third partial third color field to the DB after transmitting the third partial second color field.

[0022] In one or more embodiments, the method includes the step of the GPU transmitting the first partial first, second, and third color fields, the second partial first, second, and third color fields, and the third partial first, second, and third color fields as a single vertically encoded data set.

[0023] In one or more embodiments, the method includes the steps of the GPU sending first pose data to the DB, and the DB using the first pose data to warp a first partial first color field. The method also includes, after sending the first pose data, the step of the GPU sending second pose data to the DB, and the DB using the second pose data to warp a first partial second color field. The method further includes, after sending the second pose data, the step of the GPU sending third pose data to the DB, and the DB using the third pose data to warp a first partial third color field. Further, the method includes, after sending the third pose data, the step of the GPU sending fourth pose data to the DB, and the DB using the fourth pose data to warp a second partial first color field. Additionally, the method includes, after sending the fourth pose data, the step of the GPU sending fifth pose data to the DB, and the DB using the fifth pose data to warp a second partial second color field. The method also includes, after sending the fifth pose data, the step of the GPU sending sixth pose data to the DB, and the DB using the sixth pose data to warp a second partial third color field. The method further includes, after sending the sixth pose data, the step of the GPU sending seventh pose data to the DB, and the DB using the seventh pose data to warp a third partial first color field. Further, the method includes, after sending the seventh pose data, the step of the GPU sending eighth pose data to the DB, and the DB using the eighth pose data to warp a third partial second color field. Additionally, the method includes, after sending the eighth pose data, the step of the GPU sending ninth pose data to the DB, and the DB using the ninth pose data to warp a third partial third color field.

[0024] In one or more embodiments, the GPU transmits at least one of the first, second, third, fourth, fifth, sixth, seventh, eighth, and ninth pose data to the DB through the AUX communication link. The DB may warp each of the first partial first color field, the second partial first color field, the third partial first color field, the first partial second color field, the second partial second color field, the third partial second color field, the first partial third color field, the second partial third color field, and the third partial third color field twice.

[0025] In yet another embodiment, a data format for use in a virtual, augmented, or mixed reality system includes a first signaling row. The data format also includes a plurality of first color field rows. The data format further includes a second signaling row. Additionally, the data format includes a plurality of second color field rows. In addition, the data format includes a third signaling row. The data format also includes a plurality of third color field rows.

[0026] In one or more embodiments, the first signaling row includes a number of active rows for a plurality of first color field rows. The active rows may vary between image frames. The second signaling row may include a number of active rows for a plurality of second color field rows. The third signaling row may include a number of active rows for a plurality of third color field rows. The first signaling row may include the start positions of the plurality of first color field rows. The second signaling row may include the start positions of the plurality of second color field rows. The third signaling row may include the start positions of the plurality of third color field rows. The first, second, and third color field rows may include intensity information without color information. The first, second, and third signaling rows may include color information without intensity information. The first, second, and third signaling rows and the plurality of first, second, and third color field rows may be read at a rate faster than when an image corresponding to the plurality of first, second, and third color field rows is displayed.

[0027] In yet another embodiment, a method in a virtual, augmented, or mixed reality system includes detecting an area of focus of a user. The method also includes the GPU rendering a virtual image outside the area of focus at a lower resolution. The method further includes the GPU rendering a virtual image inside the area of focus at a higher resolution. Additionally, the method includes the GPU transmitting the rendered virtual images outside and inside the area of focus to one or more databases. In addition, the method includes one or more databases merging the rendered virtual images outside and inside the area of focus and generating a frame of image data.

[0028] In one or more embodiments, the method includes the steps of the GPU sending the rendered virtual image outside the focal area to a first DB, the GPU sending the rendered virtual image inside the focal area to a second DB, and the first and / or second DB merging the rendered virtual images outside and inside the focal area and generating a frame of image data.

[0029] In yet another embodiment, the method in a virtual, augmented, or mixed reality system includes the step of detecting the user's hand within the FOV. The method also includes the step of the GPU generating a mask corresponding to the location of the user's hand. The method further includes the step of the GPU sending the mask and the frame of image data to the DB. Additionally, the method includes the step of the DB modifying the frame of image data using the mask.

[0030] In one or more embodiments, the mask is a depth mask.

[0031] In yet another embodiment, a method in a virtual, augmented, or mixed reality system includes steps where a GPU transmits first color field image data to a DB, second color field image data to the DB, and third color field image data to the DB. The method also includes steps where the GPU transmits first pose data to the DB, and the DB uses the first pose data to warp the first color field image data to generate warped first color field image data. The method further includes steps where after the GPU transmits the first pose data, the GPU transmits second pose data to the DB, and the DB uses the second pose data to warp the second color field image data to generate warped second color field image data. Additionally, the method includes steps where after the GPU transmits the second pose data, the GPU transmits third pose data to the DB, and the DB uses the third pose data to warp the third color field image data to generate warped third color field image data.

[0032] In one or more embodiments, the method includes a step where the GPU transmits packet pose data to the DB, and the step where the DB uses the first pose data to warp the first color field image data includes the step where the DB calculates a first pose delta from the packet pose and the first pose data. The method may further include a step where the DB commands a display of a first color field image corresponding to the warped first color field image data immediately after generating the warped first color field image data. The method may also include a step where the DB performs a function on the first, second, and third color field image data, and the function is selected from the group consisting of projector light field distortion compensation, pixelated dimming, occlusion, chromatic aberration correction, frame rate, and augmentation.

[0033] In one or more embodiments, the method includes the GPU transmitting fourth pose data to the DB after transmitting third pose data. The method also includes the DB using the fourth pose data to warp first color field image data to generate second warped first color field image data. The method further includes the GPU transmitting fifth pose data to the DB after transmitting the fourth pose data. Additionally, the method includes the DB using the fifth pose data to warp second color field image data to generate second warped second color field image data. In addition, the method includes the GPU transmitting sixth pose data to the DB after transmitting the fifth pose data. The method also includes the DB using the sixth pose data to warp third color field image data to generate second warped third color field image data.

[0034] In yet another embodiment, a method in a virtual, augmented, or mixed reality system includes the GPU obtaining a frame of image data. The method also includes the GPU identifying a section of the frame of image data. The method further includes the direct memory access controller transmitting the identified section of the frame of image data to the DB without further processing of the image data.

[0035] In one or more embodiments, the section of the frame of image data is a row of non-black image data. The method may also include shutting down a part / component / function of the GPU, a part / component / function of the DMA, a communication link between the GPU and the DB, a part / component / function of the DB, a communication link between the DB and the display panel, and / or a part / component / function of the display panel.

[0036] In yet another embodiment, a method in a virtual, augmented, or mixed reality system includes the GPU dividing a first field into a first partial first field, a second partial first field, and a third partial first field. The method also includes the GPU dividing a second field into a first partial second field, a second partial second field, and a third partial second field. The method further includes the GPU dividing a third field into a first partial third field, a second partial third field, and a third partial third field. Further, the method includes the GPU sending the first partial first field to the DB. Additionally, the method includes the GPU sending the first partial second field to the DB after sending the first partial first field. The method also includes the GPU sending the first partial third field to the DB after sending the first partial second field. The method further includes the GPU sending the second partial first field to the DB after sending the first partial third field. Further, the method includes the GPU sending the second partial second field to the DB after sending the second partial first field. Additionally, the method includes the GPU sending the second partial third field to the DB after sending the second partial second field. The method also includes the GPU sending the third partial first field to the DB after sending the second partial third field. The method further includes the GPU sending the third partial second field to the DB after sending the third partial first field. Further, the method includes the GPU sending the third partial third field to the DB after sending the third partial second field.

[0037] In one or more embodiments, the method includes the steps of the GPU sending first pose data to the DB, and the DB using the first pose data to warp a first partial first field. The method also includes, after sending the first pose data, the step of the GPU sending second pose data to the DB, and the DB using the second pose data to warp a first partial second field. The method further includes, after sending the second pose data, the step of the GPU sending third pose data to the DB, and the DB using the third pose data to warp a first partial third field. Further, the method includes, after sending the third pose data, the step of the GPU sending fourth pose data to the DB, and the DB using the fourth pose data to warp a second partial first field. Additionally, the method includes, after sending the fourth pose data, the step of the GPU sending fifth pose data to the DB, and the DB using the fifth pose data to warp a second partial second field. The method also includes, after sending the fifth pose data, the step of the GPU sending sixth pose data to the DB, and the DB using the sixth pose data to warp a second partial third field. The method further includes, after sending the sixth pose data, the step of the GPU sending seventh pose data to the DB, and the DB using the seventh pose data to warp a third partial first field. Further, the method includes, after sending the seventh pose data, the step of the GPU sending eighth pose data to the DB, and the DB using the eighth pose data to warp a third partial second field. Additionally, the method includes, after sending the eighth pose data, the step of the GPU sending ninth pose data to the DB, and the DB using the ninth pose data to warp a third partial third field.

[0038] In yet another embodiment, a method in a virtual, augmented, or mixed reality system includes the GPU acquiring a frame of image data. The method also includes the GPU acquiring occlusion data related to occlusion within the field of view, the data including depth map data. The method further includes the GPU sending the frame of image data and the occlusion data to a DB. Additionally, the method includes the DB masking the frame of image data prior to display using the occlusion data. The present invention provides, for example, the following. (Item 1) A method in a virtual, augmented, or mixed reality system, comprising: the GPU determining / detecting the absence of image data; shutting down a part / component / function of the GPU; shutting down a communication link between the GPU and the DB; shutting down a part / component / function of the DB; shutting down a communication link between the DB and the display panel; and shutting down a part / component / function of the display panel. A method comprising the above. (Item 2) The method according to item 1, further comprising reorganizing frame data to reduce transfer time. (Item 3) The method according to item 1, further comprising the GPU DP port sending a custom STP message to the DB. (Item 4) The method according to item 3, further comprising the GPU sending the STP message to the DB AUX message. (Item 5) The method according to item 1, wherein a part / component / function of the GPU is selected from the group consisting of memory read, compression, and color segmentation. (Item 6) A part / component / function of the DB is the method described in item 1, which is memory writing. (Item 7) A part / component / function of the display panel is the method described in item 1, which is selected from the group consisting of a video RAM and a MIPI receiver. (Item 8) The method according to item 1, further comprising the GPU sending a wake-up signal to the DB. (Item 9) The method according to item 8, wherein the GPU sends the wake-up signal via an AUX communication link. (Item 10) The method according to item 1, further comprising the GPU sending a wake-up signal to a communication link between the GPU and the DB. (Item 11) A part / component / function of the GPU, a communication link between the GPU and the DB, a part / component / function of the DB, a communication link between the DB and the display panel, and a part / component / function of the display panel are shut down asynchronously, according to the method of item 1. (Item 12) The method according to item 1, further comprising the DB sending a built-in line control message to the display panel. (Item 13) A method in a virtual, augmented, or mixed reality system, comprising: The GPU divides a first color field into a first partial first color field and a second partial first color field; The GPU divides a second color field into a first partial second color field and a second partial second color field; The GPU sends the first partial first color field to the DB; After sending the first partial first color field, the GPU sends the first partial second color field to the DB; After the GPU transmits the first partial second color field, the GPU transmits the second partial first color field to the DB, and After the GPU transmits the second partial first color field, the GPU transmits the second partial second color field to the DB, and A method comprising. (Item 14) The GPU divides a third color field into a first partial third color field and a second partial third color field, and After the GPU transmits the first partial second color field and before transmitting the second partial first color field, the GPU transmits the first partial third color field to the DB, and After the GPU transmits the second partial second color field, the GPU transmits the second partial third color field to the DB, and The method according to item 13, further comprising. (Item 15) The method according to item 13, further comprising the GPU transmitting the first partial first and second color fields and the second partial first and second color fields as a single vertically encoded data set. (Item 16) A method in a virtual, augmented, or mixed reality system, The GPU divides a first color field into a first partial first color field, a second partial first color field, and a third partial first color field, and The GPU divides a second color field into a first partial second color field, a second partial second color field, and a third partial second color field, and The GPU divides a third color field into a first partial third color field, a second partial third color field, and a third partial third color field, and The GPU transmits the first partial first color field to the DB, after the GPU transmits the first partial first color field, the GPU transmits the first partial second color field to the DB, after the GPU transmits the first partial second color field, the GPU transmits the first partial third color field to the DB, after the GPU transmits the first partial third color field, the GPU transmits the second partial first color field to the DB, after the GPU transmits the second partial first color field, the GPU transmits the second partial second color field to the DB, after the GPU transmits the second partial second color field, the GPU transmits the second partial third color field to the DB, after the GPU transmits the second partial third color field, the GPU transmits the third partial first color field to the DB, after the GPU transmits the third partial first color field, the GPU transmits the third partial second color field to the DB, after the GPU transmits the third partial second color field, the GPU transmits the third partial third color field to the DB A method comprising. (Item 17) The method according to item 16, further comprising the GPU transmitting the first partial first, second, and third color fields, the second partial first, second, and third color fields, and the third partial first, second, and third color fields as a single vertically encoded data set. (Item 18) The GPU transmits the first attitude data to the DB, The DB uses the first posture data to warp the first partial first color field, after the GPU transmits the first posture data, the GPU transmits second posture data to the DB, The DB uses the second posture data to warp the first partial second color field, after the GPU transmits the second posture data, the GPU transmits third posture data to the DB to do, The DB uses the third posture data to warp the first partial third color field, after the GPU transmits the third posture data, the GPU transmits fourth posture data to the DB, The DB uses the fourth posture data to warp the second partial first color field, after the GPU transmits the fourth posture data, the GPU transmits fifth posture data to the DB, The DB uses the fifth posture data to warp the second partial second color field, after the GPU transmits the fifth posture data, the GPU transmits sixth posture data to the DB, The DB uses the sixth posture data to warp the second partial third color field, after the GPU transmits the sixth posture data, the GPU transmits seventh posture data to the DB, The DB uses the seventh posture data to warp the third partial first color field, after the GPU transmits the seventh posture data, the GPU transmits eighth posture data to the DB, The DB uses the eighth posture data to warp the third partial second color field, after the GPU transmits the eighth posture data, the GPU transmits ninth posture data to the DB, The DB uses the ninth pose data to warp the third partial third color field The method according to item 16, further comprising: (Item 19) The method according to item 18, wherein the method performs frame rate expansion with zero latency using updated warping (Item 20) The method according to item 18, further comprising adjusting at least one of the first to ninth pose data and performing continuous warping (Item 21) The DB calculates first pose data The DB uses the first pose data to warp the first partial first color field After calculating the first pose data, the DB calculates second pose data The DB uses the second pose data to warp the first partial second color field After calculating the second pose data, the DB calculates third pose data The DB uses the third pose data to warp the first partial third color field After calculating the third pose data, the DB calculates fourth pose data The DB uses the fourth pose data to warp the second partial first color field After calculating the fourth pose data, the DB calculates fifth pose data The DB uses the fifth pose data to warp the second partial second color field After calculating the fifth pose data, the DB calculates sixth pose data The DB uses the sixth pose data to warp the second partial third color field After calculating the sixth pose data, the DB calculates seventh pose data; The DB warps the third partial first color field using the seventh pose data; After calculating the seventh pose data, the DB calculates eighth pose data; The DB warps the third partial second color field using the eighth pose data; After calculating the eighth pose data, the DB calculates ninth pose data; The DB warps the third partial third color field using the ninth pose data The method according to item 16, further comprising. (Item 22) The method according to item 18, wherein the GPU transmits at least one of the first, second, third, fourth, fifth, sixth, seventh, eighth, and ninth pose data to the DB through an AUX communication link. (Item 23) The method according to item 18, wherein the DB warps each of the first partial first color field, the second partial first color field, the third partial first color field, the first partial second color field, the second partial second color field, the third partial second color field, the first partial third color field, the second partial third color field, and the third partial third color field twice. (Item 24) A data format for use in a virtual, augmented, or mixed reality system, A first signaling row, A plurality of first color field rows, A second signaling row, A plurality of second color field rows, A third signaling row, And a plurality of third color field rows A data format comprising (Item 25) The first signaling row includes a certain number of active rows for the plurality of first color field rows, the data format according to Item 24. (Item 26) The active rows vary between image frames, the data format according to Item 24. (Item 27) The first, second, and third signaling rows and the plurality of first, second, and third color field rows are read at a rate faster than the images corresponding to the plurality of first, second, and third color field rows are displayed, the data format according to Item 24. (Item 28) The second signaling row includes a certain number of active rows for the plurality of second color field rows, the data format according to Item 24. (Item 29) The third signaling row includes a certain number of active rows for the plurality of third color field rows, the data format according to Item 24. (Item 30) The first signaling row includes the start position of the plurality of first color field rows, the data format according to Item 24. (Item 31) The second signaling row includes the start position of the plurality of second color field rows, the data format according to Item 24. (Item 32) The third signaling row includes the start position of the plurality of third color field rows, the data format according to Item 24. (Item 33) The first, second, and third color field rows include intensity information without color information, the data format according to Item 24. (Item 34) The first, second, and third signaling lines are the data format according to item 33, including color information without intensity information. (Item 35) A method in a virtual, augmented, or mixed reality system, detecting the area of the user's focus, the GPU rendering a virtual image outside the area of the focus at a lower resolution, the GPU rendering a virtual image inside the area of the focus at a higher resolution, the GPU transmitting the rendered virtual images outside and inside the area of the focus to one or more databases, the one or more databases merging the rendered virtual images outside and inside the area of the focus to generate a frame of image data and including a method. (Item 36) the GPU transmitting the rendered virtual image outside the area of the focus to a first database, the GPU transmitting the rendered virtual image inside the area of the focus to a second database, the first and / or second database merging the rendered virtual images outside and inside the area of the focus to generate a frame of image data and further including the method according to item 35. (Item 37) A method in a virtual, augmented, or mixed reality system, detecting the user's hand within the FOV, the GPU generating a mask corresponding to the location of the user's hand, the GPU transmitting the mask and a frame of image data to a database, the database using the mask to correct the frame of the image data and including a method. (Item 38) The method according to item 37, wherein the mask is a depth mask. (Item 39) A method in a virtual, augmented, or mixed reality system, comprising: the GPU transmitting first color field image data to the DB; the GPU transmitting second color field image data to the DB; the GPU transmitting third color field image data to the DB; the GPU transmitting first pose data to the DB; the DB using the first pose data to warp the first color field image data to generate warped first color field image data; after transmitting the first pose data, the GPU transmitting second pose data to the DB; the DB using the second pose data to warp the second color field image data to generate warped second color field image data; after transmitting the second pose data, the GPU transmitting third pose data to the DB; the DB using the third pose data to warp the third color field image data to generate warped third color field image data. A method as described above. (Item 40) The method according to item 39, further comprising the DB performing functions on the first, second, and third color field image data, the functions being selected from the group consisting of projector light field distortion compensation, pixelated dimming, occlusion, chromatic aberration correction, frame rate, and augmentation. (Item 41) The method further comprising the GPU transmitting packet pose data to the DB. The method according to item 39, wherein the DB warping the first color field image data using the first pose data includes the DB calculating a first pose delta from the packet pose and the first pose data. (Item 42) The method according to item 39, further including the DB instructing the display of the first color field image corresponding to the warped first color field image data immediately after the DB generates the warped first color field image data. (Item 43) After the GPU transmits the third pose data, the GPU transmits fourth pose data to the DB, the DB warps the first color field image data using the fourth pose data to generate second warped first color field image data, after the GPU transmits the fourth pose data, the GPU transmits fifth pose data to the DB, the DB warps the second color field image data using the fifth pose data to generate second warped second color field image data, after the GPU transmits the fifth pose data, the GPU transmits sixth pose data to the DB, the DB warps the third color field image data using the sixth pose data to generate second warped third color field image data The method according to item 39, further including the above. (Item 44) A method in a virtual, augmented, or mixed reality system, the GPU obtains a frame of image data, the GPU identifies a section of the frame of the image data, the direct memory access controller transmits the identified section of the frame of the image data to the DB without further processing of the image data. A method including (Item 45) The method according to item 44, wherein a section of a frame of the image data is a row of non-black image data. (Item 46) Shutting down a part / component / function of the GPU, Shutting down a part / component / function of the DMA, Shutting down a communication link between the GPU and the DB, Shutting down a part / component / function of the DB, Shutting down a communication link between the DB and the display panel, and / or Shutting down a part / component / function of the display panel The method according to item 44, further including (Item 47) A method in a virtual, augmented, or mixed reality system, wherein the GPU divides a first field into a first partial first field, a second partial first field, and a third partial first field, the GPU divides a second field into a first partial second field, a second partial second field, and a third partial second field, the GPU divides a third field into a first partial third field, a second partial third field, and a third partial third field, the GPU transmits the first partial first field to the DB, after transmitting the first partial first field, the GPU transmits the first partial second field to the DB, after transmitting the first partial second field, the GPU transmits the first partial third field to the DB, After the GPU transmits the first partial third field, the GPU transmits the second partial first field to the DB. After the GPU transmits the second partial first field, the GPU transmits the second partial second field to the DB. After the GPU transmits the second partial second field, the GPU transmits the second partial third field to the DB. After the GPU transmits the second partial third field, the GPU transmits the third partial first field to the DB. After the GPU transmits the third partial first field, the GPU transmits the third partial second field to the DB. After the GPU transmits the third partial second field, the GPU transmits the third partial third field to the DB. The GPU transmits the first pose data to the DB. The DB uses the first pose data to warp the first partial first field. After the GPU transmits the first pose data, the GPU transmits the second pose data to the DB. The DB uses the second pose data to warp the first partial second field. After the GPU transmits the second pose data, the GPU transmits the third pose data to the DB. The DB uses the third pose data to warp the first partial third field thereby. After the GPU transmits the third pose data, the GPU transmits the fourth pose data to the DB. The DB uses the fourth pose data to warp the second partial first field. After the GPU transmits the fourth pose data, the GPU transmits the fifth pose data to the DB. The DB warps the second partial second field using the fifth pose data, After the GPU transmits the fifth pose data, the GPU transmits sixth pose data to the DB, The DB warps the second partial third field using the sixth pose data, After the GPU transmits the sixth pose data, the GPU transmits seventh pose data to the DB, The DB warps the third partial first field using the seventh pose data, After the GPU transmits the seventh pose data, the GPU transmits eighth pose data to the DB, The DB warps the third partial second field using the eighth pose data, After the GPU transmits the eighth pose data, the GPU transmits ninth pose data to the DB, The DB warps the third partial third field using the ninth pose data A method including the above. (Item 48) A method in a virtual, augmented, or mixed reality system, The GPU obtains a frame of image data, The GPU obtains occlusion data related to occlusion within the field of view, the data including depth map data, The GPU transmits the frame of the image data and the occlusion data to the DB, The DB masks the frame of the image data before display using the occlusion data A method including the above.

Brief Description of the Drawings

[0039] The drawings described below are for illustrative purposes only. The drawings are not intended to limit the scope of the present disclosure. The drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the figures are not drawn to an exact scale, and elements of similar structure or function are represented by like reference numerals throughout the figures. To gain a deeper understanding of the foregoing and other advantages and objects of various embodiments of the present disclosure, a more detailed description of the present disclosure will be given by reference to the specific embodiments illustrated in the accompanying drawings. It is understood that these drawings depict only typical embodiments of the present disclosure and are, therefore, not to be considered as limiting its scope, and the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0040]

Figure 1

[0041]

Figure 2

Figure 3

Figure 4

Figure 5

[0042]

Figure 6

[0043]

Figure 7

[0044]

Figure 8

[0045]

Figure 8A

[0046]

Figure 9

[0047]

Figure 9A

[0048]

Figure 10

[0049]

Figure 10A

[0050]

Figure 11

[0051]

Figure 12

[0052]

Figure 13

[0053]

Figure 14

[0054]

Figure 15

[0055]

Figure 15A

[0056]

Figure 15B

[0057]

Figure 15C

[0058]

Figure 15D

[0059]

Figure 15E

[0060]

Figure 15F

[0061]

Figure 15G

[0062]

Figure 16

[0063]

Figure 16A

[0064]

Figure 17

[0065]

Figure 17A

[0066]

Figure 17B

[0067]

Figure 18

[0068]

Figure 19

[0069]

Figure 20

Figure 21

Figure 22

Figure 22A

Figure 23

Figure 24

Figure 24A

[0070]

Figure 24B

[0071]

Figure 25

[0072]

Figure 26

[0073]

Figure 27

Figure 28

Figure 29

Figure 30

Figure 30A

Figure 31

Figure 31A-1

Figure 31A-2

Figure 31B-1

Figure 31B-2

[0074]

Figure 32

[0075]

Figure 33

[0076]

Figure 34

[0077]

Figure 35

[0078]

Figure 36A

[0079]

Figure 36B

DETAILED DESCRIPTION OF THE INVENTION

[0080] Various embodiments of the present disclosure are directed to systems, methods, and articles of manufacture for VR / AR / MR in a single embodiment or multiple embodiments. Other objects, features, and advantages of the present disclosure are set forth in the detailed description, the figures, and the claims.

[0081] Here, various embodiments will be described in detail with reference to the drawings, which are provided as illustrative examples to enable those skilled in the art to practice the present disclosure. It should be noted that the following figures and examples are not meant to limit the scope of the present disclosure. If certain elements of the present disclosure can be implemented partially or fully using known components (or methods or processes), only those portions of such known components (or methods or processes) necessary for understanding the present disclosure will be described, and detailed descriptions of other portions of such known components (or methods or processes) will be omitted so as not to obscure the present disclosure. Further, various embodiments include present and future known equivalents of the components referenced herein by way of illustration.

[0082] Embodiments according to the present disclosure often address the implementation problems of VR / AR / MR systems that rely on combinations of off-the-shelf components and custom components. In some cases, off-the-shelf components do not possess all of the features or performance characteristics required to implement a desired aspect of the VR / AR / MR system to be deployed. Some embodiments are directed to approaches for adding capabilities and / or reusing resources for other purposes to adapt to the desired features or performance characteristics of the VR / AR / MR system to be deployed. The accompanying figures and the discussion herein present exemplary environments, systems, methods, and computer program products for VR / AR / MR systems.

[0083] The head-mounted visual display system and the power management system can be implemented independently of the AR / MR system, but some of the following embodiments are described in relation to the AR / MR system for illustrative purposes only. The power management system described herein can also be used in a similar manner in VR systems. (Overview of Problems and Solutions)

[0084] VR / AR / MR systems have limitations such as size and portability issues, battery life issues, system overheating issues, processing power, memory, bandwidth, data sources, component latency, and other system and optical issues that can negatively impact VR / AR / MR system performance. These limitations highlight the importance of power-efficient image rendering.

[0085] For example, in some wearables, various components within the image pipeline (e.g., GPU, display bridge, display panel, etc.) consume a significant portion of the system resources (e.g., processing power, memory, bandwidth, battery life). Additionally, these system resource requirements can lead to size and portability issues and system overheating issues. Further, component latency issues can also affect VR / AR / MR system performance. For example, the system latency between the final warping of the rendered image data and the display of the image corresponding to the warped image data can result in artifacts.

[0086] Power management systems according to various embodiments include features such as deactivation of depth planes or color fields within depth planes, time-domain power management, discrete imaging modes, low-power depth plane switching, lower-power short-latency standby / wake-up, lower-power side channels, multiple component low-power modes, and reduction of power to light sources and / or SLMs.

[0087] The embodiments described herein include power management systems and methods for use in conjunction with various VR / AR / MR systems. These power management systems and methods reduce the system resources consumed by the image pipeline, thereby addressing many of the issues described above. The embodiments described herein also include virtual image warping systems and methods for use in conjunction with various VR / AR / MR systems. These virtual image warping systems and methods address some of the issues described above. (Exemplary VR, AR, and / or MR Systems)

[0088] The following description relates to exemplary VR, AR, and / or MR systems in which embodiments of various power management systems may be practiced. However, the embodiments are also suitable for use in other types of display systems (including other types of VR, AR, and / or MR systems), and thus, it should be understood that the embodiments are not limited to the exemplary systems disclosed herein.

[0089] The VR / AR / MR systems disclosed herein can include a display that presents computer-generated images (video / image data) to a user. In some embodiments, the display system is wearable, which can advantageously provide a more immersive VR / AR / MR experience. Various components of the VR, AR, and / or MR virtual image system 100 are depicted in FIGS. 2-5. The virtual image generation system 100 includes a frame structure 102 worn by an end user 50, a display subsystem 110 carried by the frame structure 102 such that the display subsystem 110 is positioned in front of the eyes of the end user 50, and a speaker 106 carried by the frame structure 102 such that the speaker 106 is positioned adjacent to the outer ear canal of the end user 50 (optionally, another speaker (not shown) is positioned adjacent to the other outer ear canal of the end user 50 to provide stereo / adjustable sound control). The display subsystem 110 is designed to present a light pattern that can be comfortably perceived as an augmentation to the physical reality, presenting high-quality imagery and three-dimensional perception to the eyes of the end user 50 and also being capable of presenting two-dimensional content. The display subsystem 110 presents a sequence of frames at a high frequency to provide the perception of a single coherent scene.

[0090] In the illustrated embodiment, the display subsystem 110 employs an “optical see-through” display through which a user can directly view light from real objects through a transparent (or translucent) element. The transparent element is often referred to as a “combiner” and superimposes light from the display across the view of the real-world user. To achieve this purpose, the display subsystem 110 includes a partially transparent display. In some embodiments, the transparent display may be electronically controlled. In some embodiments, the transparent display includes segmented dimming and may control the transparency of one or more portions of the transparent display. In some embodiments, the transparent display includes global dimming and may control the overall transparency of the transparent display. The display is positioned within the field of view of the end user 50 between the end user 50's eye and the surrounding environment such that direct light from the surrounding environment is transmitted through the display to the end user 50's eye.

[0091] In the illustrated embodiment, the image projection assembly provides light to a partially transparent display, whereby it is combined with direct light from the ambient environment and transmitted from the display to the user's eyes 50. The projection subsystem may be a fiber optic scanning-based projection device, and the display may be a waveguide-based display into which scanned light from the projection subsystem is input to produce an image at a single optical viewing distance closer than, for example, infinity (e.g., arm's length), images at multiple discrete optical viewing distances or focal planes, and / or image layers stacked at multiple viewing distances or focal planes to represent a three-dimensional 3D object. These layers in the light field may be stacked sufficiently close together to appear continuously to the human peripheral visual system (i.e., one layer is within the cone of confusion of an adjacent layer). Additionally, or alternatively, the perceived continuity of the transition between layers in the light field may be increased even when picture elements are blended across two or more layers and those layers are stacked more sparsely (i.e., one layer is outside the cone of confusion of an adjacent layer). The display subsystem 110 may be for monocular or binocular use.

[0092] The virtual image generation system 100 may also include one or more sensors (not shown) mounted on the frame structure 102 for detecting the position and movement of the end user 50's head 54 and / or the end user 50's eye position and interpupillary distance. Such sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes. Many of these sensors operate based on the assumption that the frame 102 to which they are attached is, in turn, substantially fixed to the user's head, eyes, and ears.

[0093] The virtual image generation system 100 may also include a user orientation detection module. The user orientation module detects the instantaneous position of the head 54 of the end user 50 (e.g., via a sensor coupled to frame 102) and may predict the position of the head 54 of the end user 50 based on the position data received from the sensor. Detecting the instantaneous position of the head 54 of the end user 50 facilitates the determination of the specific actual object that the end user 50 is looking at, thereby providing an indication of the specific virtual object to be generated in relation to that actual object and further providing an indication of the position where the virtual object should be displayed therein. The user orientation module may also track the eyes of the end user 50 based on the tracking data received from the sensor.

[0094] The virtual image generation system 100 may also include a control subsystem that may take any of a variety of forms. The control subsystem includes several controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers such as application specific integrated circuits (ASICs), display bridge chips, display controllers, programmable gate arrays (PGAs), e.g., field PGAs (FPGAs), and / or programmable logic controllers (PLUs).

[0095] The control subsystem of the virtual image generation system 100 may include a central processing unit (CPU), a graphics processing unit (GPU), one or more frame buffers, and a three-dimensional database for storing three-dimensional scene data. The CPU may control the overall operation, while the GPU may render frames from the three-dimensional data stored in the three-dimensional database (i.e., convert a three-dimensional scene into a two-dimensional image) and store these frames in the frame buffer. One or more additional integrated circuits may control the reading of frames into and out of the frame buffer and the operation of the image projection assembly of the display subsystem 110.

[0096] The various processing components of the virtual image generation system 100 may be physically contained within a distributed subsystem. For example, as illustrated in FIGS. 2-5, the virtual image generation system 100 may include a local processing and data module 130 that is operatively coupled to a local display bridge 142, a display subsystem 110, and a sensor by means of a wired conductor or wireless connectivity 136, etc. The local processing and data module 130 may be mounted in various configurations, such as fixedly attached to the frame structure 102 (FIG. 2), fixedly attached to a helmet or cap 56 (FIG. 3), removably attached to the torso 58 of the end user 50 (FIG. 4), or removably attached to the waist 60 of the end user 50 in a belt-coupled configuration (FIG. 5). The virtual image generation system 100 may also include remote processing modules 132 and a remote data repository 134 that are operatively coupled to the local processing and data module 130 and the local display bridge 142 by means of a wired conductor or wireless connectivity 138, 140, etc., such that these remote modules 132, 134 are operatively coupled to each other and are available as resources to the local processing and data module 130 and the local display bridge 142.

[0097] Local processing and data module 130 and local display bridge 142 may each include a power - efficient processor or controller and digital memory such as flash memory, both of which may be used, possibly after processing or reading, to assist in the processing, caching, and storing of data captured from sensors and / or obtained and / or processed using remote processing module 132 and / or remote data repository 134 for passage to display subsystem 110. Remote processing module 132 may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. Remote data repository 134 may include a relatively large - scale digital data storage facility, which may be available through other networking configurations in an Internet or "cloud" resource configuration. In some embodiments, all data is stored and all calculations are performed in local processing and data module 130 and local display bridge 142, enabling completely autonomous use from any remote module.

[0098] The couplings 136, 138, 140 between the various components described above may include one or more wired interfaces or ports for providing wire or optical communication, or one or more wireless interfaces or ports via RF, microwave, IR, etc. for providing wireless communication. In some implementations, all communication may be wired, while in other implementations, all communication may be wireless. Still further, in other implementations, the options for wired and wireless communication may differ from those illustrated in FIGS. 2 - 5. Thus, the specific choice of wired or wireless communication should not be considered limiting.

[0099] In some embodiments, the user orientation module is contained within the local processing and data module 130 and / or the local display bridge 142, while the CPU and GPU are contained within the remote processing module. In an alternative embodiment, the CPU, GPU, or a portion thereof may be contained within the local processing and data module 130 and / or the local display bridge 142. The 3D database can be associated with the remote data repository 134 or can be located locally.

[0100] Some VR, AR, and / or MR systems use multiple volume phase holograms, surface relief holograms, or light guiding optical elements that incorporate depth plane information to generate images that appear to originate from individual depth planes. In other words, a diffraction pattern or diffraction optical element ("DOE") is incorporated within or imprinted / embossed on a light guiding optical element ("LOE", e.g., a planar waveguide) such that collimated light (a light beam with a substantially planar wavefront) intersects the diffraction pattern at multiple locations as it is substantially totally internally reflected along the LOE and exits towards the user's eye. The DOE is configured such that the light exiting the LOE through it is focused so as to appear to originate from a particular depth plane. The collimated light may be generated using an optical condenser lens ("condenser").

[0101] For example, the first LOE may be configured to deliver collimated light to the eye that appears to originate from an optically infinite depth plane (0 diopters). Another LOE may be configured to deliver collimated light that appears to originate from a distance of 2 meters (1 / 2 diopter). Yet another LOE may be configured to deliver collimated light that appears to originate from a distance of 1 meter (1 diopter). By using a stacked LOE assembly, it should be understood that multiple depth planes can be created, and each LOE is configured to display an image that appears to originate from a particular depth plane. It should be understood that the stack may include any number of LOEs. However, at least N stacked LOEs are required to generate N depth planes. Further, N, 2N, or 3N stacked LOEs may be used to generate an RGB color image on N depth planes.

[0102] To present 3-D virtual content to a user, a VR, AR, and / or MR system projects an image of the virtual content into the user's eye such that it appears to originate from various depth planes in the Z direction (i.e., orthogonal to and away from the user's eye). In other words, the virtual content can vary not only in the X and Y directions (i.e., the 2D plane orthogonal to the user's central line of sight), but also in the Z direction such that the user can perceive the object as being very close, or at infinite distance, or at any distance in between. In other embodiments, the user can perceive multiple objects at different depth planes simultaneously. For example, to the user, a virtual dragon may appear to come running towards the user from infinity. Alternatively, to the user, a virtual bird at a distance of 3 meters from the user and a virtual coffee cup at arm's length (about 1 meter) from the user may be visible simultaneously.

[0103] The multi-plane focus system creates a perception of variable depth by projecting an image onto some or all of a plurality of depth planes located at individual fixed distances in the Z direction from the user's eye. Referring now to FIG. 6, it should be understood that the multi-plane focus system may display the frame on a fixed depth plane 150 (e.g., the six depth planes 150 shown in FIG. 6). The MR system can include any number of depth planes 150, but one exemplary multi-plane focus system has six fixed depth planes 150 in the Z direction. When generating virtual content on one or more of the six depth planes 150, a 3-D perception is created such that the user perceives one or more virtual objects at variable distances from the user's eye. Assuming that the human eye is more sensitive to objects that are closer in distance than objects that appear to be farther away, more depth planes 150 are generated closer to the eye, as shown in FIG. 6. In other embodiments, the depth planes 150 may be equidistantly spaced apart from each other.

[0104] The depth plane position 150 may be measured in diopters, which is a unit of refractive power equal to the reciprocal of the focal length measured in meters. For example, in some embodiments, depth plane 1 may be 1 / 3 diopter apart, depth plane 2 may be 0.3 diopter apart, depth plane 3 may be 0.2 diopter apart, depth plane 4 may be 0.15 diopter apart, depth plane 5 may be 0.1 diopter apart, and depth plane 6 may represent infinity (i.e., 0 diopter apart). It should be understood that other embodiments may generate the depth plane 150 at other distances / diopters. Thus, when generating virtual content at the strategically placed depth plane 150, the user is able to perceive virtual objects in three dimensions. For example, the user may perceive a first virtual object as being nearby when it is displayed within depth plane 1 while another virtual object appears at infinity in depth plane 6. Alternatively, the virtual object may first be displayed at depth plane 6, then at depth plane 5, and so on until the virtual object appears very close to the user. It should be understood that the above examples are significantly simplified for illustrative purposes. In another embodiment, all six depth planes may be concentrated on a particular focal length away from the user. For example, if the virtual content to be displayed is a coffee cup that is 0.5 meters away from the user, all six depth planes may be generated at various cross-sections of the coffee cup, providing the user with a high granularity 3-D view of the coffee cup.

[0105] In some embodiments, the VR, AR, and / or MR system may function as a multi-plane focus system. In other words, all six LOEs may be illuminated simultaneously such that images resulting from six fixed depth planes are generated rapidly and continuously, where the light source rapidly transmits the image information to LOE1, then LOE2, then LOE3, etc. For example, a portion of a desired image that includes an empty image at optical infinity may be input at time 1, and an LOE that preserves the collimation of light (e.g., depth plane 6 from FIG. 6) may be utilized. Next, an image of a closer tree branch may be input at time 2, and an LOE (e.g., depth plane 5 from FIG. 6) configured to create an image that appears to originate from a depth plane 10 meters away may be utilized. Next, an image of a pen may be input at time 3, and an LOE configured to create an image that appears to originate from a depth plane 1 meter away may be utilized. This type of paradigm can be repeated in a high-speed time-sequential manner such that the user's eyes and brain (e.g., visual cortex) perceive the input as being all parts of the same image.

[0106] The VR, AR, and / or MR system may project an image that appears to originate from various locations along the Z-axis (i.e., depth plane) to generate an image for a 3-D experience / scenario (i.e., by diverging or converging a light beam). As used herein, a light beam includes a directional projection of light energy (including visible and invisible light energy) emitted from a light source, among other things. Generating an image that appears to originate from various depth planes matches the convergence / divergence movement and accommodation of the user's eyes for that image and minimizes or eliminates convergence / divergence movement-accommodation conflict.

[0107] Referring now to FIG. 7, an exemplary embodiment of an AR or MR system 700 (hereinafter referred to as "system 700") is illustrated. System 700 uses stacked light guiding optical elements (hereinafter referred to as "LOE 790"). System 700 generally includes one or more image generation processors 710, one or more light sources 720, one or more controller / display bridges (DB) 730, one or more spatial light modulators (SLM) 740, and one or more sets of stacked LOE 790 that function as a plurality of planar focusing systems. System 700 may also include an eye tracking subsystem 750.

[0108] The image generation processor 710 is configured to generate virtual content for display to a user. The image generation processor 710 may convert an image or video associated with the virtual content into a format that can be projected in 3D to the user. For example, when generating 3D content, the virtual content may need to be formatted such that a portion of a particular image is displayed on a particular depth plane while the rest is displayed on other depth planes. In one embodiment, all of the images may be generated on a particular depth plane. In another embodiment, the image generation processor 710 may be programmed to provide slightly different images to the right and left eyes such that when viewed together, the virtual content appears to the user's eyes coherently and comfortably.

[0109] The image generation processor 710 may further include a memory 712, a GPU 714, a CPU 716, and other circuitry for image generation and processing. The image generation processor 710 may be programmed such that the desired virtual content is presented to the user of the system 700. It should be understood that in some embodiments, the image generation processor 710 may be stored within the system 700. In other embodiments, the image generation processor 710 and other circuitry may be stored within a belt pack that is coupled to the system 700. In some embodiments, the image generation processor 710 or one or more of its components may be part of a local processing and data module (e.g., local processing and data module 130). As described above, the local processing and data module 130 may be fixedly attached to the frame structure 102 (FIG. 2), fixedly attached to a helmet or cap 56 (FIG. 3), removably attached to the body 58 of the end user 50 (FIG. 4), or removably attached to the waist 60 of the end user 50 in a belt-coupled configuration (FIG. 5), etc., and may be mounted in various configurations.

[0110] The image generation processor 710 is operably coupled to a light source 720 that projects light associated with the desired virtual content and one or more spatial light modulators 740. The light source 720 is compact and has high resolution. The light source 720 is operably coupled to a controller / DB 730. The light source 720 may include color-specific LEDs and lasers arranged in various geometric configurations. Alternatively, the light source 720 may include LEDs or lasers of the same color, each linked to a specific region of the field of view of the display. In another embodiment, the light source 720 may include an area emitter such as an incandescent or fluorescent lamp with a mask overlay for segmentation of the emission area and position. The light source 720 is directly connected to the system 700 in FIG. 2B, but the light source 720 may be connected to the system 700 via an optical fiber (not shown). The system 700 may also include a condenser (not shown) configured to collimate the light from the light source 720.

[0111] In various exemplary embodiments, the SLM740 may be reflective (e.g., LCOS, FLCOS, DLP DMD, or MEMS mirror system), transmissive (e.g., LCD), or emissive (e.g., FSD or OLED). The type of SLM740 (e.g., speed, size, etc.) can be selected to improve the creation of 3D perception. A DLP DMD operating at a higher refresh rate can be easily incorporated into the stationary system 700, while the wearable system 700 may use a smaller size and power DLP. The power of the DLP changes the way the 3D depth plane / focal plane is created. The image generation processor 710 is operably coupled to the SLM740, which encodes the light from the light source 720 with the desired virtual content. The light from the light source 720 may be encoded with image information when it is reflected from, emitted from, or passes through the SLM740.

[0112] The light from the SLM740 is directed to the LOE790 such that a light beam encoded with image data for one depth plane and / or color by the SLM740 propagates along a single LOE790 for delivery to the user's eye, in effect. Each LOE790 is configured to project an image or sub-image that appears to originate from a desired depth plane or FOV angular position onto the user's retina. The light source 720 and the LOE790 can thus selectively project (synchronized and encoded by the SLM740 under the control of the controller / DB730) images that appear to originate from various depth planes or positions within the space. By sequentially projecting the images at a sufficiently high frame rate (e.g., 360 Hz for six depth planes at an effective full volume frame rate of 60 Hz) using each of the light source 720 and the LOE790, the system 700 can generate 3D images of virtual objects at various depth planes that appear to be present within the 3D image simultaneously.

[0113] The Controller / DB730 communicates with and is operably coupled to an image generation processor 710, a light source 720, and an SLM 740, and coordinates the synchronized display of images by instructing the SLM 740 to encode an optical beam from the light source 720 with appropriate image information from the image generation processor 710. The system includes an image generation processor 710, although in some embodiments the Controller / DB730 may also perform at least some of the image generation processes, including, for example, the processes of memory 712, GPU 714, and / or CPU 716. In some embodiments, the Controller / DB730 may include one or more components shown in the image generation processor 710, such as, for example, memory 712, GPU 714, and / or CPU 716.

[0114] System 700 also includes an optional eye tracking subsystem 750 configured to track a user's eyes and determine the user's focus. In one embodiment, System 700 is configured to illuminate a subset of LOE 790 based on input from the eye tracking subsystem 750 such that the image is generated at a desired depth plane that coincides with the user's focus / focus adjustment. For example, if the user's eyes are parallel to each other, System 700 may illuminate LOE 790 configured to deliver collimated light to the user's eyes such that the image appears to originate from optical infinity. In another example, if the eye tracking subsystem 750 determines that the user's focus is one meter away, LOE 790, configured to focus approximately within that range, may instead be illuminated.

[0115] Figure 8 schematically depicts an image generation component of a VR / AR / MR system 800 according to some embodiments. The VR / AR / MR system 800 includes a graphics processing unit (GPU) 802 having a DisplayPort (DP) source 804 and coupled to a display bridge (DB) 806 having a DP sink 808 via a DP link 810. The GPU 802 may be located within a belt pack of the VR / AR / MR system 800, the DB 806 may be located within a wearable component of the VR / AR / MR system 800, and the DP link 810 may be an optical fiber-based link. In some embodiments, the DP link 810 may be a PCIe link or other, e.g., proprietary high-speed link. In some embodiments, both the GPU 802 and the DB 806 may be within the belt pack of the VR / AR / MR system 800. In some embodiments, both the GPU 802 and the DB 806 may be within a wearable component of the VR / AR / MR system 800. In some embodiments, the DB 806 may include the GPU 802. The DB 806 is also coupled to left and right display panels 812 and left and right light field chips 814 via four MIPI links 816. In some embodiments, the MIPI links 816 may be DP links, PCIe links, or other, e.g., proprietary high-speed links. In these embodiments, the DB 806, the left and right display panels 812, and the left and right field chips 814 may include appropriate ports for the links. The DB 806 may also be coupled to a wearable processor 818 via the MIPI links 816 and an I2C or SPI connector 820 (coupled to an MIPI / I2C 822 or SPI receiver 824 within the DB 806). The MIPI links 816 within the system 800 (i.e., between the DB 806 and the left and right display panels 812, the left and right light field chips 814, and the wearable processor 818) are coupled to an MIPI transmitter 832 and an MIPI receiver 824 within the transmit and receive components.The MIPI link 816 may include one or more data lanes (e.g., four video data lanes) and a clock lane. The DB806 may also include a memory 826 (e.g., stacked low-power DDR memory or embedded DRAM) and a memory 828 (e.g., stacked flash memory). The left and right display panels 812 may also include individual left and right decompression engines 830. In some embodiments, the DB806 may include a video processor (e.g., a pixel engine) for pixel operations. The video processor customizes data received through a data stream (e.g., a video stream from the GPU 802) for a primary panel (e.g., the left and right panels 812 and / or the left and right light field chips 814).

[0116] Figure 8A schematically depicts the image generation components of a VR / AR / MR system 800' according to some embodiments. The VR / AR / MR system 800' depicted in Figure 8A includes many of the components of the VR / AR / MR system 800 depicted in Figure 8. The VR / AR / MR system 800' includes a graphics processing unit (GPU) 802 having a DisplayPort (DP) source 804 and coupled to a display bridge (DB) 806' having a DP sink 808 via a DP link 810. The DB 806' is also coupled to left and right display panels 812' and left and right light field chips 814' via four MIPI links 816. In some embodiments, the DB 806' may be coupled to the left and right display panels 812' and left and right light field chips 814' via SPI and / or I2C connectors. Additionally, the DB 806' is further coupled to left and right LED / laser drivers 834 via left and right buses (e.g., I2C buses) 836. The left and right buses 836 also couple both the DB 806' and the left and right LED / laser drivers 834 to the left and right display panels 812'. The DB 806' may also be coupled to a wearable processor 818' via MIPI links 816, I2C connectors 820, and / or SPI connectors 838 that may be coupled to a MIPI receiver 824, an I2C receiver 822, or an SPI receiver 840 within the DB 806'. The DB 806' also includes a memory 826 (e.g., built-in SRAM) and a memory 828 (e.g., stacked quad SPI flash memory).

[0117] The MIPI link 816 described above may have a bi - directional capability (e.g., for returning read data, acknowledgement responses, and / or error information from the left and right display panels 812, 812' to the DBs 806, 806'). Lane 0 of the MIPI link 816 may be used for transmission to the DBs 806, 806', while the other lanes may be unidirectional. The MIPI link 816 may be used for transmission to the DBs 806, 806' during the low - power transmission mode, so that the DBs 806, 806' can initiate a general - purpose read from the left and right display panels 812, 812'.

[0118] Figure 9 schematically depicts a DB900 for use in conjunction with a VR / AR / MR system according to some embodiments. The DB900 includes a DP receiver 902 configured to receive one or more streams of image data. The DP receiver 902 is communicatively coupled to a DSC decoder 904, a DDR controller 906, a pixel engine 908, output FIFOs for primary and secondary streams 910, and pixel distribution modules for primary and secondary panels (the "pixel distribution module") 912, which in turn are communicatively coupled to four MIPI channels 914. The DB900 also includes a MIPI receiver 916, a memory 918 (e.g., a low - power DDR stack die), a CPU module 920, and an RGB / gray - scale segmentation block 922, all of which are communicatively coupled to the DDR controller 906. The RGB / gray - scale segmentation block 922 receives data from the DSC decoder 904 and transmits the output to the memory 918'. The DB900 is configured to process two video streams. The first video stream 924 is processed by the DSC decoder 904 and the RGB / gray - scale segmentation block 922 and then transmitted to the DDR controller 906 for writing to the memory 918. The second video stream 926 is transmitted directly from the DP receiver 902 to the DDR controller 906 for writing to the memory 918.

[0119] Figure 9A schematically depicts a DB900' for use in conjunction with a VR / AR / MR system according to some embodiments. The DB900', depicted in Figure 9A, includes many of the components of the DB900 depicted in Figure 9. The DB900' includes a DP receiver 902 configured to receive one or more streams of image data. The DP receiver 902 is communicatively coupled to a DSC decoder 904, a pixel engine 908, output FIFOs primary and secondary streams 910, and pixel distribution primary and secondary panels 912, which in turn are communicatively coupled to four MIPI channels 914. The DB900' also includes a MIPI receiver 916 and an RGB / grayscale segmentation block 922, both of which are coupled to a memory 918' (e.g., SRAM). The RGB / grayscale segmentation block 922 receives data from the DSC decoder 904 and transmits the output to the memory 918' as described herein. The DB900' further includes a splash screen decompressor 928 coupled to the memory 918'. The DB900' is configured to process two video streams. The first video stream 924 is processed by the DSC decoder 904 and the RGB / grayscale segmentation block 922 and then written to the memory 918'. The second video stream 926 is transmitted directly from the DP receiver 902 and written to the memory 918'.

[0120] FIG. 10 schematically depicts a pixel engine 1000 for use in conjunction with a DB according to some embodiments. The pixel engine 1000 includes a core 1002 that executes code for a color field and generates pixels for display based on input parameters. The pixel engine 1000 reads information regarding the color field from memory (e.g., low power DDR) and generates pixels (e.g., one output line at a time). The core 1002 may be programmable and may include SRAM and instruction RAM. The core 1002 also has an AXI communication channel 1004 (e.g., for communicating with memory), a pair of FIFO outputs 1006 (e.g., for communicating with a display and a light field chip), a secondary data packet / auxiliary data first in first out (SDP / AUXFIFO) input 1008 (e.g., for communicating with a DP source), and may have a dual bank of SRAM 1010 that is tightly coupled to contain pixel data for processing. In some embodiments, the pixel engine 1000 is involved in compensating for optical distortion associated with an optical system and a projector, persistent warping reprojection, frame rate extension, dynamic occlusion, segmented foveation blending, pixelated dimming control, pixel chromatic aberration correction, partial display support, custom dimming and partial dimming modes, content fold-out, and performing many other AR / VR algorithms using a vector engine, e.g., using custom instructions. In some embodiments, the pixel engine 1000 calculates a head pose update directly from sensor data provided by a sensor or a wearable processor 818 that is communicatively coupled to a DB (e.g., DB806).

[0121] FIG. 10A schematically depicts a pixel engine 1000' for use in conjunction with a DB according to some embodiments. The pixel engine 1000' includes a core 1002' that executes code for a color field and generates pixels for display based on input parameters. The pixel engine 1000' reads information regarding the color field from memory (e.g., low power DDR or SRAM) and generates pixels (e.g., one output line at a time). The core 1002' may be programmable and may include SRAM and instruction RAM. The core 1002' may also have an AXI communication channel 1004' (e.g., for communicating with memory), a pair of FIFO outputs 1006' (e.g., for communicating with a display and a light field chip), a secondary data packet / auxiliary data first-in first-out (SDP / AUXFIFO) input 1008' (e.g., for communicating with a DP source), and a dual bank of SRAM 1010' that is tightly coupled to contain pixel data for processing. The tightly coupled SRAM 1010' reduces the need to communicate with other memories within the system. Further, the core 1002' may have a connection 1012 to an SRAM controller and HSYNC / VSYNC / scanning line interrupt 1014 (e.g., for receiving input from a pixel segmentation block). Further, the core 1002' may include an APB register bank, an APB / AXI slave port 1018, and an SPI / I2C master port. The APB / AXI slave port 1018 may be used to copy firmware from flash memory to the DB.

[0122] In some embodiments, pixel engine 1000’ may be involved in compensating for optical distortion associated with the optics and the projector, continuous warping reprojection, frame rate enhancement, dynamic occlusion, segmented foveation blending, pixelated dimming control, pixel chromatic aberration correction, partial display support, custom dimming and partial dimming modes, content unfolding, and implementing many other AR / VR algorithms using the vector engine, e.g., custom instructions. In some embodiments, pixel engine 1000’ calculates the head pose update directly from sensor data provided by a sensor communicatively coupled directly to the DB (e.g., DB806) or a wearable processor 818.

[0123] FIG. 11 schematically depicts a display panel 1100 for use in conjunction with a VR / AR / MR system according to some embodiments. The display panel 1100 includes a MIPI receiver 1102 communicatively coupled to a built-in line decoder 1104, partial screen refresh logic 1106, various buffers 1108, various gamma correction modules 1110, and a display memory 1112. The MIPI receiver 1102 may be configured to operate with an MIPI DSI-2 interface, a DP link interface, and / or other, e.g., proprietary serial links with one or more data lanes (e.g., four video data lanes) and one clock lane. The MIPI receiver 1102 may have various features including burst mode with synchronous events, clock / lane deskewing, and manual configuration of resolution / timing via registers. The built-in line decoder 1104 may be configured to read the first line of frame data and extract the data embedded therein. The embedded data may define the operating mode and format for the frame. The partial screen refresh logic 1104 may be configured to operate in two modes, namely, blank screen and partial display (both described below). The buffer 1108 may have a maximum latency for one color field. The gamma correction module 1110 may support a gamma table and convert 8-bit input video to 10-bit output. The display panel 1100 may include a table / offset synchronization module and a sequencer table. (Image data format) (Segmented color sequential format)

[0124] FIG. 12 schematically depicts two input data formats that can be supported by a DP port (see, e.g., DP SRC 804 and DP sink 808 in FIGS. 8 and 8A). The upper data format 1200 is an RGB 24-bit color (8 bits per field), and the lower data format 1202 is an RGB 30-bit color (10 bits per field). The input data may be compressed with Display Stream Compression (DSC). The DP port may also support a sequential grayscale data format (described below).

[0125] FIG. 13 schematically depicts a color image data format for use in conjunction with a VR / AR / MR system according to some embodiments. In this color image data format, color image data (e.g., 24-bit or 30-bit RGB data) typically organized into three primary color fields 1302, 1304, 1306 is reorganized into three images 1312, 1314, 1316. Each of the three images 1312, 1314, 1316 is divided into thirds, and each third includes a partial color field (e.g., 1312R, 1312G, 1312B). (Color Sequential Grayscale Format)

[0126] The color image data is generated within the GPU and reorganized into three images 1312, 1314, 1316. The three images 1312, 1314, 1316 can be transmitted from the GPU to the DB via the DP. The reorganization of the color image data into the three images 1312, 1314, 1316 results in portions of each color field (e.g., partial color fields 1312R, 1312G, 1312B) arriving at the DB earlier within the image pipeline. This earlier access to the image data may enable the DB to begin acting on the image data earlier within the image pipeline without the reorganized color image data.

[0127] FIG. 14 schematically depicts a color image 1402 that has been reorganized into three partial color fields. Additionally, the three partial color fields have been converted into three sequential grayscale images 1422, 1424, 1426. The positions of the three sequential grayscale images 1422, 1424, 1426 are used to encode the color of each partial field. Converting the color fields to grayscale can reduce the amount of data transmitted from the GPU to the DB. The conversion to grayscale can occur in the GPU or the DB. (Color Segmentation)

[0128] FIG. 15 schematically depicts color segmentation of color image data in a DB for use in conjunction with a VR / AR / MR system, e.g., where color image data is transmitted from a GPU to the DB for further processing, according to some embodiments. When the DB receives a register indicating a color sequential grayscale format, the DB converts the packed color image data to color sequential grayscale data as described herein. The DB receives color image data 1502 from the GPU through the DP. The DB then optionally segments the color image data 1502 into three fields 1512, 1514, 1516. The three fields 1512, 1514, 1516 are stored in memory, e.g., DDR dual buffer memory, for further processing into a color sequential grayscale format. This segmentation can occur in real time as the color image data stream is received. The register that receives the color image data 1502 stream may also indicate an address in the memory (e.g., low power DDR) where the image data will be stored.

[0129] FIG. 15A schematically depicts the color segmentation of color image data in a database for use in conjunction with a VR / AR / MR system, e.g., where color image data is transmitted from a GPU to the database for further processing. When the database receives a register indicating a color sequential grayscale format, the database converts the packed color image data to color sequential grayscale data as described herein. The database receives color image data 1502 from the GPU through a DP. Next, the database or an optional segmentation hardware block 1520 segments the color image data 1502 into three fields 1522, 1524, 1526. The three fields 1522, 1524, 1526 are stored in nine output buffers (BUF0 - BUF8). A subset of the buffers in which the three fields 1522, 1524, 1526 are stored varies between even frames (i.e., 0, 2, 4, 6, …) and odd frames (i.e., 1, 3, 5, 7, …). An incoming message from the segmentation block 1520 or the database indicates the currently active buffer state for the segmentation block output. The segmentation block 1520 separates the color image data 1502 into three primary color grayscale fields (i.e., red, green, blue). This segmentation can occur in real time as the color image data 1502 stream is received.

[0130] When the segmentation block 1520 is disabled, the color image data 1502 is copied to nine output buffers (BUF0 - BUF8) in RGB packing mode using a numerical order that matches the arrival time, as shown in FIG. 15B. In this mode, the nine output buffers (BUF0 - BUF8) are in order because there is no segmentation.

[0131] When the color image data 1502 is in the raw 8-bit form, the segmentation block 1520 may convert the pixels on-the-fly to raw 10 bits by adding two redundant zero bits within the least significant bit for each color component before storing the segment data. Even if segmentation is not enabled, the color image data 1502 may be stored in the raw 10-bit format for system consistency.

[0132] FIG. 15C schematically depicts the "RGB30" data format for storing color image data 1502. FIG. 15D schematically depicts the input display panel resolution of 1,440 pixels × 1,440 pixels, which is stored in SRAM as 128-bit aligned data. Each left and right segmented color includes 1,800 bytes with 8 bytes of padding. Using the data format in FIG. 15D, it can be assumed that the start address for every nine buffers of the segmented block 1520 will always be 128-bit aligned. FIG. 15E schematically depicts the input display panel resolution of 512 pixels × 512 pixels, which is stored in SRAM as 128-bit aligned data. Each left and right segmented color includes 640 bytes with 1,168 bytes of padding. Thus, the color image data 1502 can still be stored in SRAM as 128-bit aligned data. FIG. 15F schematically depicts the input display panel resolution of 512 pixels × 512 pixels for a special image (e.g., a splash screen), which is stored in SRAM as 128-bit aligned data. In this particular embodiment, the left panel has 512-color pixels, while the right panel has 512 pixels that are all set to black. As shown in FIG. 15E, each left and right segmented "color" includes 640 bytes with 1,168 bytes of padding. Thus, the color image data 1502 can still be stored in SRAM as 128-bit aligned data. Although 1,440 pixels × 1,440 pixels and 512 pixels × 512 pixels resolutions are described, similar data formats can also be used to store color image data 1502 with different resolutions.

[0133] FIG. 15G schematically depicts the data structure after color segmentation of a frame. The data structure shown in FIG. 15G does not include padding, and thus, the data structure has a fixed destination pitch. That is, the byte address distance from one row to the next has a fixed number of bytes.

[0134] Each color field requires three separate buffers, and each buffer holds one-third of the color field image. With respect to the display resolution having a number of lines that is not evenly divisible by 3, the buffer can include one or two additional lines so that the number of buffers is evenly divisible by 3. The surplus lines / buffers can be used to store image metadata. (Multi-stream mode)

[0135] As shown in FIGS. 9 and 9A above, DB900, 900' may support multi-stream mode, and the image data includes two streams 924, 926. In some embodiments, video stream 1 924 may include image data for display, and video stream 2 926 may not include image data for display, but rather may include configuration data for, for example, one or more light field chips (e.g., the left and right light field chips 814 in FIGS. 8 and 8A). In some embodiments, video stream 2 926 may also include light field panel data separated by pixel engine 908 for use by the light field chips. As shown in FIGS. 9 and 9A, this configuration data may be communicated to and thereby used by DDR controller 906, memories 918, 918', pixel engine 908, and pixel distribution module 912. In some embodiments, the configuration data can include panel resolution, light field controller data, warping data, distortion data, occlusion data, foveation region data, bridge control data, and the like.

[0136] Video stream 1 924 and video stream 2 926 may be synchronized. Video stream 2 926 may support partial lines per frame. The partial lines facilitate the definition of the timing of video stream 1 924 and video stream 2 926. The partial lines may be placed at the end of the blanking section. The DB may be configured to generate a MIPI output for video stream 2 926 and facilitate the spacing within memories 918, 918'. In the full darkening mode (described herein), video stream 2 926 will not be transmitted by the DP source. In the partial darkening mode (described herein), the DB may be configured to generate an output for video stream 1 924 and keep the link active until video stream 2 926 is transmitted. Video stream 2 926 may shift the line phase once per V blanking to facilitate the synchronization of the two streams 924, 926.

[0137] FIG. 16 schematically depicts a secondary display stream 1600 (e.g., video stream 2926) and its flow through the DB. The secondary display stream 1600 includes light field processor data 1602 for controlling left and right light field processors and other custom data 1604. The light field processor data 1602 and the custom data 1604 are separated by a programmable row 1606. The DB splits the data separated by the programmable row 1606 such that the custom data 1604 is stored in a memory, e.g., memory 1608 (e.g., low power DDR memory), and the light field processor data 1602 is transmitted to a pixel distribution 1614 (e.g., pixel distribution module 912) through a pixel engine 1610 and an output FIFO memory 1612. In the pixel distribution 1614, the light field processor data 1602 is split into left and right channels before being transmitted to an MIPI link for communication to a display panel (not shown). If custom data 1604 exists for a particular frame, the output MIPI timing may be adapted to the custom data 1604, thereby adjusting to reduce the resolution of the light field processor data 1602.

[0138] There may be no relationship between the input resolution of the secondary display stream 1600 and the corresponding MIPI output resolution. Locations in memories 918, 918' corresponding to the secondary display stream 1600 may be double buffered so that information about a new frame can be written without deleting information about the previous frame. The double buffer of memories 918, 918' for the secondary display stream 1600 may be selectively activated.

[0139] In some embodiments, a secondary display stream (e.g., video stream 2 926) may be larger than necessary. As a result, the DB may compress the secondary display stream for transmission to the display. FIG. 16A schematically depicts the secondary display stream 1600’ (e.g., video stream 2 926) and its flow through the DB. The secondary display stream 1600’ includes light field processor data 1602’ having custom data 1604’. The light field processor data 1602’ may be stored in a memory 1608’ (e.g., SRAM), from which the light field processor data 1602’ is transmitted through a pixel engine 1610’, an output FIFO memory 1612’, and a pixel distribution 1614’ (e.g., pixel distribution module 912). The pixel engine 1610’ may copy only the sub-pixels within the light field processor data 1602’ from the memory 1608’ according to the output. Removing the data splitting operation in the secondary display stream 1600 from the secondary display stream 1600’ depicted in FIG. 16A simplifies the hardware design but adds complexity to the pixel engine 1610’ firmware design. The final resolution for the MIPI output of video stream 2 926 may not be fixed. (Secondary Data Packet (SDP))

[0140] In addition to the image data for display, video stream 1 also includes one or more SDP messages from the GPU to the DB. In some embodiments that use the VESA packet format, the SDP messages may be encoded within the video image data during vertical blanking before any video image data for display is communicated. The SDP messages may also be encoded within the video image data during horizontal blanking. The SDP messages are received by the DB and stored in a buffer (e.g., for access by the pixel engine). The SDP messages may include pose data updates.

[0141] Figure 17 schematically depicts a large chained VSC_EXT_VESA packet 1700 having a plurality of encoded SDP messages therein, according to some embodiments. Up to two maximum-sized KB of SDP messages may be incorporated within the vertical blanking segment 1702 of the packet 1700 due to its relatively large size. On the other hand, the relatively smaller size of the horizontal blanking segment 1704 facilitates only relatively smaller SDP messages. Figure 17 depicts six horizontal blanking packets that may be partitioned into three chained packets. The use of chained packets enables somewhat larger SDP messages. In embodiments with relatively smaller SDP messages, chaining is not required.

[0142] Figure 17A schematically depicts a VSC_EXT_VESA SDP message 1750 associated with vertical blanking. The VSC_EXT_VESA SDP message 1750 may be included only within video stream 1 924. Thus, the VSC_EXT_VESA SDP message 1750 may include information regarding both video streams 1 and 2. In some embodiments, the VSC_EXT_VESA SDP message 1750 includes a header 1752 (20 bytes), a main DP control block 1754 (5 bytes), partial MSA information (18 bytes each) regarding video streams 1 and 2 1756, 1758, and a proprietary data block 1760 (variable size, e.g., 173 bytes).

[0143] Figure 17B schematically depicts a horizontal blanking SDP message 1770 associated with horizontal blanking. In some embodiments, the horizontal blanking SDP message 1770 includes a header 1772 (4 bytes), a timestamp 1774 (4 bytes), and a proprietary data block 1776 (28 bytes). (SDP and AUX Message FIFO Memory)

[0144] Figure 18 schematically depicts the flow of data from SDP messages and auxiliary (AUX) channel messages in a DB according to some embodiments. Since the DP can be shut down to conserve power (explained below), various configuration data may be communicated via the AUX channel. Configuration data may be communicated using register writes via the AUX channel. In some embodiments, the AUX channel is used to communicate configuration data with the DP.

[0145] Figure 18 depicts SDP message 1802 from the DP receiver and configuration data from AUX channel 1804. The SDP and AUX messages 1802, 1804 are stored on FIFO memory 1806. From the FIFO memory 1806, the SDP and AUX messages 1802, 1804 are sent to pixel engine 1808 and can thereby be used. (Output video stream compression)

[0146] Configuration data for the display panel is most efficiently communicated directly from the controller to the display panel. In a VR / AR / MR system where the DB is interposed between the CPU and the display panel, the configuration data can be sent to the display panel within the video stream as an embedded signal.

[0147] Figure 19 schematically depicts video stream 1900 with pixel data 1902 interleaved with embedded signal 1904 that includes configuration data. The embedded signal 1904 can be placed anywhere within the video stream 1900 (e.g., start, middle, end). The embedded signal 1904 can also be full or partial lines.

[0148] The DB can receive configuration data from the AUX channel register write, SDP metadata, and the custom data section of video stream 2. The pixel engine will receive the configuration data and generate an output MIPI stream that includes an embedded signal with the configuration data related to a specific display panel. The output timing of the MIPI stream may be extended to match the embedded signaling. (Full Darkening Low Power Mode)

[0149] In some embodiments, for a particular VR / AR / MR user pose / view, there will be no virtual image data. Thus, the GPU output will be sent for black / full darkening. When the GPU detects a full darkening frame, shutting down the image pipeline for video stream 1 924 will save system power. Sources of power savings include, but are not limited to, the DP link, DP source processor, DP source memory, DP optical link, GPU compression, DB decompression, DB color segmentation, DB memory write, pixel engine memory read, pixel processing, MIPI transmitter, MIPI receiver, display panel memory read and write, etc. Turning off video stream 1 924 and its display panel will save power. In some embodiments, the light field chip will continue to operate from previously received data.

[0150] When the GPU detects a fully darkened frame, the GPU notifies the DP source MCU (e.g., via an "STP" SDP message or a k-code power-down sequence on the main DP link). In some embodiments, the GPU synthesizer creates, writes, and initiates a fully darkened mode by creating a vertical blank SDP message. The DP source MCU then bypasses the memory transfer of the fully darkened frame to the DP source pipeline and places most of the DP source pipeline components (e.g., DSC encoder) in a low-power mode. The DP source MCU disables various registers and shuts down downstream components of the imaging pipeline such as the display bridge and the display.

[0151] The GPU may initiate the fully darkened mode by transmitting an SDP message to the DB during vertical blanking. The DP receiver decodes the SDP message and places itself in the fully darkened low-power mode. The DP receiver also transmits a message to place the pixel engine in the fully darkened low-power mode. Various other components of the imaging pipeline, including the MIPI data lanes that will be placed in the LP11 mode, are also placed in the fully darkened low-power mode. The DP source CPU will shut down the physical components of the DP. The display may be shut down by setting the number of active lines to zero. The display may also transition to a black / self-refresh mode. After receiving the vertical blank SDP message, the DP sink bridge may not consider any data transmitted by the source until it receives a wake-up signal. Various components of the image pipeline are shut down asynchronously as the previous non-darkened image is processed and displayed through the pipeline. In some embodiments, the display pipeline, including the DP port, will remain in the shutdown mode as long as non-frame data is required.

[0152] When the GPU detects an incomplete dark frame after a full dark frame, the GPU starts a wake-up sequence. In some embodiments, the DP source MCU may power on the DP transmitter and receiver. The GPU compositor may create, write, and start either partial dimming or normal mode for the vertical blank SDP message. Next, the DP source MCU may send a wake-up message via the AUX channel. The display may wake up by setting the number of active lines to a non-zero value. Optional high-speed link training may occur. In some embodiments, the high-speed link training may be stored on the DB, and the stored high-speed link training may be used for fast wake-up.

[0153] Figure 32 schematically depicts the MIPI timing of a display system transitioning to and sustaining a full dark mode 3200, according to some embodiments. The system transitions to the full dark mode at 3210. In some embodiments, the display panel is actually operating in self-refresh but only displaying black pixels. Thus, a significant portion of the power savings from the full dark mode is derived from the shutdown of the MIPI receiver, memory, DB, light source, and optical fiber system connectors.

[0154] Figure 33 schematically depicts the transition of a display system to the full - darkening mode 3300, its duration, and its termination according to some embodiments. Figure 33 depicts the MIPI data 3310 from the DB and the display (e.g., LCOS) output video 3320 over time. The MIPI data 3310 transitions from the normal mode to full - darkening in frame 3 3312. The video data is full - darkening (i.e., blank or zero), but the MIPI data 3310 includes one frame of full - darkening 3312 that contains built - in control lines to set the system to the full - darkening mode. In response to the reception of the built - in control lines, the display output video 3320 displays the full - darkening frame in frame 3 3322. The system then sustains the full - darkening mode over a variable number of frames as required by the video data. In frame N 3314 of the MIPI data 3310, the MIPI data returns to the normal mode, thereby waking up the system. However, since the display is no longer synchronized with the DB MIPI output, the display does not consider the first frame of the normal mode and instead displays the last black frame 3324. Starting from frame N + 1 3326, the display output video wakes up, synchronizes, and operates in the normal mode.

[0155] Figure 34 schematically depicts the transition of a display system to the full - darkening mode for panel self - refresh 3400, its duration, and its termination according to some embodiments. Figure 34 depicts the MIPI data 3410 from the DB and the display (e.g., LCOS) output video 3420 over time. The MIPI data 3410 is in frame 3 At 3412, panel self - refresh is started. The frame 3 3412 MIPI data 3410 includes an indicator 3430 for panel self - refresh. In response to receiving the panel self - refresh indicator, the panel stores the MIPI data 3410 in one or more of its buffers. The display output video 3420 indicates a frame 3422 corresponding to the frame 3 3412 of the MIPI data 3410 until the panel self - refresh is completed. During the panel self - refresh, the SRAM buffer can be shut down because the video data is stored in a buffer within the display panel. In the frame N 3414 of the MIPI data 3410, the MIPI data returns to the normal mode, thereby waking up the system. However, since the display is no longer synchronized with the DB MIPI output, the display does not consider the first frame of the normal mode by displaying the last copy of the third frame 3424. Starting from the N + 1 frame 3426, the display output video wakes up, synchronizes, and operates in the normal mode. The display panel self - refresh may occur during the display of a splash screen, during which the content is typically stationary. (Partial Dimming Low - Power Mode)

[0156] The trigger for the partial dimming low power mode is the presence of a threshold amount of non-black content within a particular VR / AR / MR user pose / view. FIG. 20 depicts a first partial dimming low power mode where the content is placed only in the upper portion 2002 of the screen 2000. Examples of such content include a status bar, battery charge level, volume settings, etc. The DP source in this mode will initiate a power down sequence (similar to that described above for the full dimming low power mode) after a predetermined number of lines defined within the SDP message. The SDP message may indicate the first partial dimming low power mode and the number of active lines within the frame. This first partial dimming low power mode can conserve power by reducing the amounts of, for example, DB writes to memory, pixel engine fetches from memory, pixel engine processing, MIPI output, etc.

[0157] FIG. 21 depicts a second partial dimming low power mode where the content is placed in the upper portion 2102 and the center 2104 of the screen 2100. In this second partial dimming low power mode, the GPU may reorder the output data and minimize the number of active lines, as shown in FIG. 22. In FIG. 22, the content at the center 2104 of the screen 2100 in FIG. 21 is moved as image data 2204’ displaced adjacent to the content at the upper portion 2202 of the screen 2200. Reordering the image data reduces the number of active lines (depicted in FIG. 22A), thereby reducing the overall power consumption of the display pipeline components.

[0158] After the upper image data 2202 and the displaced image data 2204' have advanced through the image pipeline, the components of the image pipeline are placed in the partial dimming low power mode and conserve power, as described herein. The pixel engine in the DB performs segmentation of the image data, remaps the displaced image data 2204' using the data configuration from the SDP, and reformats the source image 2100 shown in FIG. 21. Reducing the number of active lines (see FIG. 22A) facilitates earlier shutdown of the DP link.

[0159] FIG. 23 depicts a third partial dimming low power mode in which the content is placed only at the center 2304 of the screen 2300. Similar to the modes shown in FIGS. 21 and 22, the GPU may reorganize the central data 2304 by moving it as displaced image data 2404' to the upper part of the screen 2400, as shown in FIGS. 24 and 24A.

[0160] After the displaced image data 2404' has advanced through the image pipeline, the components of the image pipeline are placed in the partial dimming low power mode and conserve power, as described herein. The pixel engine in the DB performs segmentation of the image data, remaps the displaced image data 2404' using the configuration data from the SDP, and reformats the source image 2300 shown in FIG. 23.

[0161] In the three partial dimming low power modes depicted in FIGS. 20 - 24 and described above, the SDP message informs the DP receiver of the number of active lines for both modes. This enables the DP receiver to shut down after the active lines have been communicated from the GPU to the DB. In embodiments where compression is enabled, the compression may optionally be disabled depending on the number of active lines. For example, a minimum set of lines may be required to enable compression, or the compression may simply vary the number of VESA DSC slices to always be enabled.

[0162] In other embodiments, the horizontal packing depicted in FIGS. 22-24 and described above may be replaced or combined with vertical packing. In some embodiments, FIGS. 22-24 illustrate several modes, but one or more separate regions on a single stream may be consolidated to reduce the data rate. In some embodiments, the number of separate regions may be limited to make the implementation more practical based on other constraints.

[0163] FIG. 24B schematically depicts the MIPI timing of a display system that transitions to and sustains a partial dimming mode 2450 according to some embodiments. The system transitions to the partial dimming mode at 2452 after a non-black theme within a frame is displayed. In some embodiments, the display panel is actually in self-refresh but only displays black pixels. Thus, a significant portion of the power savings from the full dimming mode is due to the shutdown of the MIPI receiver, memory, DB, light source, and optical fiber system connectors. The partial dimming mode 2450 may only act on video stream 1 924. Thus, video stream 2 926 cannot be affected by the partial dimming mode 2450. (Custom Packing for Partial Screen Refresh Mode)

[0164] In AR / MR mode, potentially, there are a number of black pixels that do not represent a correction to light from the real world. However, the black pixels and corresponding non-black pixels may be arranged in a complex pattern that is difficult to handle with simple horizontal or vertical packing.

[0165] In custom packing for partial frame refresh mode, the GPU may reorganize the image data such that only the regions / parts / sections / tiles (e.g., square-shaped tiles) that have changed from the previous frame to the currently rendered frame are sent from the GPU to the DB via the DP. The SDP message may include information indicating that custom packing for partial frame refresh mode has been enabled, the locations of various changed tiles within the FOV, etc. The DB can use the locations of the various changed tiles within the FOV to store the various changed tiles in the appropriate locations within the output buffer.

[0166] As shown in FIG. 25, the GPU will pack the changed tiles such that they form the minimum number of rows of image data. The source frame 2502 shows a virtual image that includes various black pixels and non-black pixels. Sending the source frame 2502 would require a certain number (e.g., 960) of rows to be sent from the GPU to the DB. The folded frame 2504 indicates that the tiles, which contain the changed image data, are packed at the top of the folded frame 2504 here. Sending the folded frame 2504 would require a smaller number (e.g., 160) of rows to be sent from the GPU to the DP. The compressed frame 2506 indicates that the tiles at the top of the folded frame 2504 are compressed into an even smaller number of rows. Sending the compressed frame 2506 would require an even smaller number (e.g., 80) of rows to be sent from the GPU to the DP. The compression of the folded frame 2504 resulting in the tiled and compressed frame 2506 enables various components of the image pipeline to be shut down earlier in the frame transfer / rendering interval, increasing power savings (as described herein).

[0167] As described herein, after the compressed frame 2506 is transmitted and processed, the GPU may initiate the shutdown of various image pipeline components by transmitting a k-code power-down sequence or a DPCD power state command on the main DP link. The shutdown may include the power-down of the DP. The power-down of the image pipeline components, including the DP link, may be signaled by the GPU via the AUX channel. During wake-up, the GPU powers on the DP link driver and optionally trains the link using a high-speed link training pattern.

[0168] The GPU will instruct the image pipeline components to shut down only if the number of lines to be transmitted is less than a threshold, based on the wake-up time and a cost-benefit analysis of the power savings from component shutdown versus the power usage during wake-up. Cyclic Redundancy Check (CRC) may be performed at the end of the frame transferred / rendered using custom packing for the partial screen refresh mode and may not be. The CRC information may be transmitted via the AUX channel. (Custom partial display mode / Custom dimming mode)

[0169] The custom packing for the partial screen refresh mode has two operating modes. In the custom partial display mode, the changed tiles are transmitted from the GPU to the DB and displayed by the display panel.

[0170] In the custom dimming mode, the SDP message includes a specific background color (mostly black in most cases). Any area of the visual field that does not belong to the changing tiles associated with the content is set to the specific background color. The custom dimming mode can be enabled when the entire image changes but only a small number of tiles contain content. In other words, many tiles that previously contained image data are converted to black in the current frame. First, by setting the areas without content to black, only the tiles with actual content will be sent from the GPU to the DB, thereby saving power in the image pipeline with a faster component shutdown. The SDP message may also include the number of dimming or background tiles. (Custom packing for blending and scaling)

[0171] If multi-stream implementation is not supported, the custom packing mode may send tiles and blend them with the tiles from the previous frame (rather than being copied across). A square or radio blend mass may be pre-programmed on the DB to blend the new and previous tiles for output.

[0172] The tiles may also be scaled before being copied to the output frame buffer. The SDP message may include a scaling factor for use when scaling the tiles. In some embodiments, this approach may be used for foveation implementation. (Custom pixel operations and rate conversion)

[0173] The pixel engine can minimize artifacts resulting from user pose changes (e.g., rapid head rotation) after the image data has been warped but before the image based on the image data is displayed to the user.

[0174] In one embodiment of custom pixel warping without rate conversion, the GPU sends the SDP to the DB at the start of a frame. The SDP contains information such as compression enable / disable, custom packing enable / disable, custom grayscale mode enable / disable, packet orientation, etc. The GPU also sends the image data to the DB. If compression is enabled, the DB uses an appropriate algorithm to decompress the image data. If the image data requires color segmentation, the DB performs color segmentation and stores the result in memory. The pixel engine receives configuration data via AUX register writes, via the SDP message, or as part of a second video stream. In some embodiments, the configuration data may include updated head pose information. In some embodiments, the DB may receive updated information from a wearable processor (e.g., IMU data) in order for the DB to generate a head pose update. In some embodiments, the IMU may be connected to the DB bridge SPI port and the DB may directly access the raw IMU samples from the IMU. The configuration data may be stored in the message FIFO memory. The pixel engine reads the field data from the low-power DDR, performs the necessary conversions, provides the output pixels, and drives the display panel. The DB then repeats the steps described above for each supported color field. After all color fields have been processed, the DB proceeds to the next frame.

[0175] FIG. 26A shows the input to and output from the DB (i.e., related to the image data within video stream 1924) when rate conversion is not enabled, according to some embodiments. Since the input image data was not formatted in color sequential mode, the DB must receive the first complete frame of the image data ("DP input frame 0") 2602 before segmenting the image data 2602 and transmitting the first output frame of the image data, "DP output frame 0" 2604, to the display panel. Similarly, the DB must receive the entire second complete frame of the image data 2606 before it can transmit the second output frame 2608 to the display panel. This results in a latency of 8.33 ms between the reception of each image data and the transmission of the corresponding output frame to the display panel. This latency is undesirable.

[0176] FIG. 26B shows the input to and output from the DB (i.e., related to the image data within video stream 1924) when rate conversion is not enabled, according to some embodiments. Since the input image data was formatted in color sequential mode, the first color field (e.g., red) image data 2602R is received well ahead of the end of the image frame. Therefore, the output frame image data corresponding to the first color field 2604R can be transmitted to the display panel earlier within the image pipeline. Using the grayscale color segmentation mode, the latency can be reduced from 8.33 ms to 2.78 ms. (Rate up conversion)

[0177] FIG. 26C shows the input to and output from the DB (i.e., related to the image data within video stream 1924) when rate conversion is enabled, according to some embodiments. The DB receives the image data at 60 Hz but outputs the image data at 120 Hz. In some embodiments, the DB receives the image data at 45 Hz but outputs the image data at 90 Hz. Regardless of the image data frequency, the output image data doubles the frame rate. In the embodiment depicted in FIG. 26C, the input image data was not formatted in color sequential mode as in the embodiment depicted in FIG. 26A. Thus, there is an 8.33 ms latency between the reception 2602 of the frames of image data by the DB and the transmission 2604 of the frames of image data by the DB. The difference in this rate up-conversion image pipeline is that the output frame 2604 of the image data includes each color field that doubles in speed and doubles the effective refresh rate from 60 Hz to 120 Hz, and the pixel engine will update the head pose because each field warps the incoming image and has a zero latency effect for each of the two output frames. The pipeline depicted in FIG. 26C also includes a DP offset 2610 during the step of sequentially receiving the frames of image data to conserve power. Operating the GPU at a lower refresh rate (e.g., 60 Hz) can also conserve power. Compression can be used to reduce the latency. (Rate up-conversion using warping)

[0178] Figure 26D shows the input to and output from the DB (i.e., related to the image data within video stream 1924) when rate conversion and warping are enabled, according to some embodiments. The image pipeline depicted in Figure 26D is similar to the image pipeline depicted in Figure 26C. The difference is that Figure 26D also shows that SDP and AUX messages are being sent from the GPU to the pixel engine within the DB. The SDP and AUX messages may include current pose information that can be used to warp the rendered image data for more accurate image rendering. The current pose information may be provided by an IMU that is operating at 1,000 Hz. The DB calculates a pose delta from the packet pose and the current pose for use in warping. Minimizing the time between the receipt of the current pose information and the warping of the rendered image data increases accuracy. The AUX channel is used instead of the SDP channel when the DP is powered down. The warping depicted in Figure 26D can also use the distortion information within the SDP / AUX messages to account for the physical distortion of the various components within the optical stack.

[0179] In all of the above image pipelines, the receipt, processing, and transmission of output in parallel minimize the latency between the input and the display and the delay within the image pipeline.

[0180] The composite video stream 2926 can be an unpacked RGB888 data stream. Thus, the rate conversion for video stream 2926 can be as simple as duplicating the data stream for two frame rate up-conversions. (Partial Dimming Tracking Mask)

[0181] In some embodiments, the GPU may segment each frame into a group of tiles (e.g., rows), and generate a mask indicating whether a particular group of tiles (e.g., rows) has non-black content. The GPU may use the mask to generate a rendering output to a downstream component of the image pipeline. A new mask may be generated for each new frame. The image pipeline components may be communicatively coupled by a dynamic circuit network (DCN).

[0182] In one embodiment, a frame may have 1,760 rows of pixels, and each row of tiles may have 16 rows of pixels. In this embodiment, 110 bits may be used to store the black / non-black status of 110 tile rows within each frame. Bits corresponding to all-black tile rows may be set to zero, and bits corresponding to non-black tile rows may be set to one. The bits may be default set to zero and set to one when a tile row is non-black. The GPU or another image pipeline component may use the information within these bits to select non-black tile rows to be transmitted over the DCN. The GPU or another image pipeline component may use the information within these bits to pack and unpack non-black tile rows as described herein.

[0183] As described herein, transmitting only non-black tile rows and packing these rows increases the opportunity to implement power optimizations, including, but not limited to, the following. · Reduce display port link power (e.g., do not transmit rows, or only a subset of rows) · Reduce DP source processor memory bandwidth and power (e.g., the DCN does not need to fetch all black pixels from DRAM, which saves DRAM bandwidth, DRAM power, and DCN power) · Eliminate the need for the DSC to compress the full frame on the DP source ·Eliminate the need for the DP sink bridge to the DSC to decompress the complete frame ·Eliminate the need for the DP sink bridge device to perform color segmentation of the complete frame ·Eliminate the need for the DP sink bridge device to write the complete frame to LPDDR and save power ·The pixel engine block will not need to fetch the complete frame from LPDDR and will save power ·The pixel engine will not need to perform pixel processing on the complete frame and will save power ·Reduce the MIPI-TX power to the panel using reduced signaling ·Reduce the MIPI-RX power within the panel ·Prevent the need to store or read data from its SRAM and reduce panel power by supporting partial black screens

[0184] The DCN may include a direct memory access controller (DMA) and fetch non-black lines into the DCN without any accompanying processor. The DMA may read a mask and identify non-black lines. The non-black lines may be transmitted to the display engine via the DCN and DP without being reorganized or reconfigured (e.g., folded or compacted), and power may be conserved

[0185] As described herein, if non-black lines form a significant portion of the frame, frame reconstruction or partial dimming modes may not conserve power. Such frames may be transmitted directly to the DP sink bridge via the DCN (Additional aspect)

[0186] In addition to the claimed invention, as non-limiting examples, further embodiments or aspects of the present invention are described herein

[0187] 1. A method in a virtual, augmented, or mixed reality system, comprising The step where the GPU receives a frame of image data, The step where the GPU identifies multiple regions / parts / sections / tiles within the frame of image data that have changed from the previous frame of image data, The step where the GPU moves at least some of the multiple regions / parts / sections / tiles to the start of the frame of data to form a rearranged frame of image data, The step where the GPU sends the rearranged frame of image data to the DB, The step of shutting down a part / component / function of the GPU, The step of shutting down the communication link between the GPU and the DB, The step of shutting down a part / component / function of the DB, The step of shutting down the communication link between the DB and the display panel, The step of shutting down a part / component / function of the display panel, A method including

[0188] 2. The method according to aspect 1, further including the step where the GPU compresses the rearranged frame of image data before sending the rearranged frame of image data to the DB.

[0189] 3. The method according to aspect 1, where the rearranged frame of image data is smaller than the frame of image data.

[0190] 4. The method according to aspect 1, further including the step where the DB stores the rearranged frame of image data in a buffer.

[0191] 5. The step of determining the size of the rearranged frame of image data, Steps to shut down a part / component / function of the GPU, a communication link between the GPU and the DB, a part / component / function of the DB, a communication link between the DB and the display panel, and a part / component / function of the display panel only when the rearranged frame of the image data is smaller than a predetermined maximum size, and The method according to aspect 1, further comprising

[0192] 6. The method according to aspect 1, wherein the GPU further comprises a step of sending an STP message to the DB after sending the rearranged frame of the image data to the DB.

[0193] 7. The method according to aspect 6, wherein the GPU further comprises a step of sending an STP message to the DB via the SDP.

[0194] 8. A part / component / function of the GPU is selected from the group consisting of memory reading, compression, and color segmentation, according to the method described in aspect 1.

[0195] 9. A part / component / function of the DB is memory writing, according to the method described in aspect 1.

[0196] 10. A part / component / function of the display panel is selected from the group consisting of video RAM and MIPI receiver, according to the method described in aspect 1.

[0197] 11. The method according to aspect 1, wherein the GPU further comprises a step of sending a wake-up signal to the DB.

[0198] 12. The GPU sends a wake-up signal via an AUX communication link, according to the method described in aspect 11.

[0199] 13. A part / component / function of the GPU, a communication link between the GPU and the DB, a part / component / function of the DB, a communication link between the DB and the display panel, and a part / component / function of the display panel are shut down asynchronously by the method described in aspect 1.

[0200] 14. The method according to aspect 1, further comprising the step of the DB reconstructing a frame of image data from the rearranged frame of image data.

[0201] 15. The method according to aspect 1, further comprising the step of setting a portion of the frame of image data that is not within a plurality of regions / parts / sections / tiles within the frame of image data to a background color.

[0202] 16. The method according to aspect 1, further comprising the step of the DB blending the rearranged frame of image data with the previous frame of image data.

[0203] 17. The method according to aspect 16, further comprising the step of the DB masking the previous frame of image data before blending it with the rearranged frame of image data.

[0204] 18. The method according to aspect 1, further comprising the step of the DB blending the rearranged frame of image data with the image data related to the updated foveation region.

[0205] 19. The method according to aspect 1, further comprising the step of the DB scaling the rearranged frame of image data.

[0206] 20. The DB receiving a scaling factor from the GPU and the DB scaling the rearranged frame of image data using the scaling factor, and further comprising the method according to aspect 18.

[0207] 21. Scaling is the method described in aspect 18, which is part of the foveation operation.

[0208] 22. The method according to aspect 1, further comprising the step of the DB performing a function on the image data, wherein the function is selected from the group consisting of warping, pixelated dimming, occlusion, chromatic aberration correction, frame rate, and enhancement.

[0209] 23. The method according to aspect 1, further comprising the step of storing the reordered frames of the image data in the FIFO memory before shutting down a part / component / function of the GPU.

[0210] 24. The method according to aspect 1, further comprising the step of the DB sending an embedded line control message to the display panel. (MIPI Output Data Configuration)

[0211] As described herein, the DB processes the image data received from the GPU and sends the processed image data to the display panel for display to the user. In the grayscale format, the display panel receives the image data through the MIPI link format where all red pixels are followed by all green pixels and then all blue pixels for color sequential display, otherwise standard packed RGB is used. Assuming a display resolution of Y×X, the output timing from the MIPI receiver will be Y / 3×3X. Each row will have one-third of the pixels as only one-third of the colors are transmitted, but the number of rows will be tripled. Also, an additional 24 lines for signaling and 3 lines for VSA, VBP, and VFP result in 5,307 lines.

[0212] FIG. 27 depicts sample MIPI receiver output data 2700 according to some embodiments. The output data 2700 includes a VSYNC start packet 2702, followed by a first blanking packet / VBP 2704, followed by image data generated by a pixel engine 2706, followed by a second blanking packet / VFP 2708 and an EOT packet 2710.

[0213] As shown in FIG. 28, the image data 2706 generated by the pixel engine consists of an HSYNC start packet 2812, followed by a first blanking packet / HBP 2814, followed by one line of image data and / or embedded signaling data in a long packet format 2816, followed by a second blanking packet / HFP 2818.

[0214] Figure 29 depicts sample MIPI receiver output data 2900 corresponding to a video frame with differentiated red, green, and blue pixel rows when compression is not used. Each color field includes 1,760 individual rows with 480 pixels within each row. The output data 2900 includes a VSYNC start packet 2902, followed by a first blanking packet / VBP 2904, followed by image data generated by a pixel engine 2906, followed by a second blanking packet / VFP 2908 and an EOT packet 2910. The image data generated by the pixel engine 2906 consists of an HSYNC start packet 2912, followed by a first blanking packet / HBP 2914, followed by one row of image data and / or embedded signaling data in a long packet format 2916, followed by a second blanking packet / HFP 2918. The image data 2916 includes a first signaling row 2920, followed by a first color (red) field row 2922, followed by a second signaling row 2924, followed by a second color (green) field row 2926, followed by a third signaling row 2928, followed by a third color (blue) field row 2930. (Full Darkening Low Power Mode)

[0215] In the full darkening low power mode (described above), the DB will communicate its transition to the low power mode to the display panel using a first signaling row 3020 as shown in Figure 30. After the transition to the low power mode is communicated, the DB will transition to the low power mode (LP11). The display panel will also conserve power by placing the MIPI receiver in the LP11 mode and will not store any information in the panel video RAM and will not read any information from the panel video RAM.

[0216] FIG. 38 schematically depicts the MIPI timing of the image frame data 3030 that is communicated to the DB to initiate the full darkening low power mode. As described herein, the image frame data 3030 is similar to that of the normal image mode. The exception is that the GPU writes a vertical blank SDP message into the image frame data 3030 to initiate the full darkening mode. (Partial darkening low power mode)

[0217] In the various partial darkening low power modes (described above), the DB will communicate the parameters of the partial darkening low power mode (e.g., low power mode enable and the number of active lines) for each color field in the corresponding signaling lines 3120, 3124, 3128 that precede the color field lines 3122, 3126, 3130, as shown in FIG. 31. The timing of the frame for the partial darkening low power mode is similar to that for the color field display depicted in FIG. 29, with the exception that only a programmable percentage of the 1,760 lines containing image data per color field is transmitted from the DB to the display panel through the MIPI link. After the lines containing the image data are received, the MIPI link is placed in the low power or LP11 mode between all the black lines to conserve power and ends LP11 prior to each field.

[0218] Figure 31A schematically depicts MIPI timing 3150 for two cycles of a DB operating in a partial dimming low power mode, such as those described herein. At 3152, during the display of frame N, the pixel engine sets an appropriate value for the number of active lines in the upper wires for frame N+1. At 3154, during the display of frame N, the DB MIPI system fetches and latches those wires for frame N+1. Similarly, at 3156, during the display of frame N+1, the pixel engine sets an appropriate value for the number of active lines in the upper wires for frame N+2. At 3158, during the display of frame N+1, the DB MIPI system fetches and latches those wires for frame N+2.

[0219] Figure 31B schematically depicts MIPI timing 3150 for two cycles of a DB transitioning from a partial dimming low power mode to full dimming low power. At 3162, during the display of frame N, the pixel engine sets a value for the number of active lines in the upper wires corresponding to full dimming (i.e., zero) for frame N+1. At 3164, during the display of frame N, the DB MIPI system fetches and latches those wires for frame N+1. Due to the full dimming values in the wires, frame N+1 is fully dimmed.

[0220] FIG. 35 schematically depicts MIPI timing 3500 for two cycles of a DB operating in a partial dimming low power mode according to some embodiments such as those described above. In each cycle / frame, there is a portion where the DB is operating in the low power mode. During this time, the DP is turned off and in response, the optical fiber system connector is also turned off. Turning off these two components of the DB during all cycles / frames saves system power. In some embodiments, the low power portion of the cycle / frame is about half of the cycle. Possible reasons for considering shutting down the components of the DB include, but are not limited to, the following. 1) The normal display subsystem shuts down 2) The normal display subsystem transitions to the low power mode 3) During normal operation, after the active lines are transmitted per display frame, power savings requires that the lines be shut down 4) During the partial dimming mode after the required active lines are transmitted, as will be described herein 5) To transition to the full dimming mode as described herein 6) To transition to the frame repetition mode as described herein (Reduced resolution rendering using scaling (foveation rendering per section))

[0221] In some embodiments, the VR / AR / MR system can conserve power by using per-section foveation rendering to render the virtual image corresponding to the user's fovea area at a higher resolution while rendering the background virtual image at a lower resolution. Using per-section foveation rendering can reduce GPU calculations and the power consumption of the system. The DB on the wearable uses the pixel engine and the latest eye-tracking data to merge the background and foveated content at a higher resolution for display. Per-section foveation rendering is optimized for multi-stream implementations. In such implementations, each stream has its own image pipeline and DB. Alternatively, custom packing can be used when a multi-stream implementation is not available, such that multiple streams share a single video stream in VESA SST mode. (Dynamic occlusion)

[0222] In some embodiments, the VR / AR / MR system can conserve power by excluding occluded virtual content from the image pipeline, for example, at the final stage in a possible pipeline. The DB can use the depth map from the wearable to mask virtual content as needed (e.g., for occlusion by a fast-moving hand). The DP can mask the virtual content immediately before display and reduce latency compared to masking on the GPU. The GPU still performs occlusion on the content, but may not be able to occlude everything due to pipeline latency. The rendered content is sent from the GPU to the DB, where it is masked using the depth map with a latency of, for example, 3 ms or less and then sent to the display panel for display. (Uncompressed high-bandwidth data)

[0223] The decompressed high-bandwidth data can be transmitted from the GPU to the DB using various channels (e.g., MST of the VESA standard). This decompressed high-bandwidth data can be transmitted as part of a combined stream for use in occlusion, segmented dimming, control, geometric distortion LUT, and foveation. In some embodiments, a first video stream can be used to transmit compressed image data for the primary panel, and a second video stream can be used to transmit the decompressed high-bandwidth data. The DB with built-in MIPI pre-emphasis support compensates for RX EQ issues and the greater distance between the display and the DB. (Repeating the previous frame with an offset)

[0224] In some embodiments, rather than changing the image content frame by frame as the user's head moves slightly, instead, some pixels are offset up or down. Instead of tasking the GPU to redraw all the pixels again, the mode of repeating the previous frame with an offset allows the GPU to bypass re-rendering to move the content and perform a simple transformation using the offset values already stored in the DB memory. In this mode, the GPU calculates the offset values using only the head pose update, thereby saving significant GPU power, DP power, and fiber optic channel power since not much data is transferred to the wearable (i.e., head pose update). (Black part mask)

[0225] Figure 36A depicts an AR view 3600 according to some embodiments. The graphical user interface 3610 can change as the AR system operates. However, the outer frame portion of the graphical user interface 3610 will typically not change and will thus be rendered as the black / matte portion of the frame 3620 (see FIG. 36B). The black portion of the frame 3620 shown in FIG. 36B will, in some embodiments, remain the same for each frame. Thus, the DB will not fetch or act (e.g., warp) on the black portion of the frame 3620. Using a mask made from the black portion of the frame 3620, the GPU may generate a per-row start column / end column pair that is sent to the DB via the DP. In response, the DB will fetch only the image information between the start column and the end column per row from memory. This technique substantially reduces the interleaved split-multiple access (IDMA) active time, which reduces the time before the DB can transition to the low power mode.

[0226] Certain aspects, advantages, and features of the present disclosure are described herein. It should be understood that not necessarily all such advantages can be achieved in accordance with any particular embodiment of the present disclosure. Thus, the present disclosure may be embodied or practiced so as to achieve one advantage or group of advantages taught herein without necessarily achieving other advantages that may be taught or suggested herein.

[0227] Embodiments are described in connection with the accompanying drawings. However, it should be understood that the figures are not drawn to exact scale. Distances, angles, etc. are merely illustrative and do not necessarily have an exact relationship to the actual dimensions and layout of the devices shown. In addition, the foregoing embodiments are described in detail to enable one of ordinary skill in the art to make and use the devices, systems, methods, and equivalents described herein. Various modifications are also possible. Components, elements, and / or steps may be modified, added, removed, or rearranged.

[0228] The devices and methods described herein can advantageously be implemented, at least in part, using, for example, computer software, hardware, firmware, or any combination of software, hardware, and firmware. A software module can include computer-executable code stored in a computer's memory for performing the functions described herein. In some embodiments, the computer-executable code is executed by one or more general-purpose computers. However, one of ordinary skill in the art will understand, in light of the present disclosure, that any module that can be implemented using software for execution on a general-purpose computer can also be implemented using different combinations of hardware, software, or firmware. For example, such a module can be implemented entirely in hardware using a combination of integrated circuits. Alternatively, or in addition, such a module can be implemented using a special-purpose computer designed to perform the specific functions described herein, rather than by a general-purpose computer. In addition, when a method is described that is performed, or can be performed, at least in part, by computer software, it should be understood that such a method can be provided on a non-transitory computer-readable medium that, when read by a computer or other processing device, causes the method to be performed.

[0229] One embodiment has been explicitly described, but other embodiments will also be apparent to those skilled in the art based on the present disclosure.

[0230] The various processors and other electronic components described herein are suitable for use in combination with any optical system for projecting light. The various processors and other electronic components described herein are also suitable for use in combination with any audio system for receiving voice commands.

[0231] Various exemplary embodiments of the present disclosure are described herein. These examples are referred to in a non-limiting sense. They are provided to illustrate more broadly applicable aspects of the present disclosure. Various changes may be made to the present disclosure described, and equivalents may be substituted without departing from the true spirit and scope of the present disclosure. Additionally, many modifications may be made to adapt a particular situation, material, composition, process, process act, or step to the purpose, spirit, or scope of the present disclosure. Furthermore, as will be understood by those skilled in the art, each of the individual variations described and illustrated herein can be readily separated from or combined with features of any of several other embodiments without departing from the scope or spirit of the present disclosure, having discrete components and features. All such modifications are intended to be within the scope of the claims associated with the present disclosure.

[0232] The present disclosure includes methods that may be implemented using a subject device. The method may include the act of providing such a suitable device. Such providing may be performed by an end user. In other words, the act of "providing" simply requires that the end user acts to obtain, access, approach, locate, configure, activate, power on, or otherwise provide the device required in the method of the present subject matter. The methods recited herein may be performed in any order of the recited logical possible events and in the order of the events described.

[0233] Exemplary aspects of the present disclosure are described above, along with details regarding material selection and manufacturing. Regarding other details of the present disclosure, these are understood in relation to the patents and publications referenced above and are generally known or understandable by those of ordinary skill in the art. The same may apply to the method-based aspects of the present disclosure from the perspective of additional acts that are commonly or logically employed.

[0234] In addition, the present disclosure is described with reference to several embodiments that optionally incorporate various features, but the present disclosure is not limited to what is described and indicated as being considered for each variation of the present disclosure. Various changes may be made to the present disclosure described, and equivalents (whether recited herein or not for purposes of some brevity) may be substituted without departing from the true spirit and scope of the present disclosure. In addition, when ranges of values are provided, it is to be understood that all intervening values between the upper and lower limits of that range and any other stated value or intervening value within the stated range are included within the present disclosure.

[0235] Also, any optional features of the described variations are contemplated to be described and claimed either independently or in combination with any one or more of the features described herein. References to singular items include the possibility that multiple identical items exist. More specifically, as used in this specification and the claims associated herewith, the singular forms "a", "an", "said", and "the" include plural references unless specifically stated otherwise. In other words, the use of the article enables "at least one" of the items of the subject matter in the claims associated with the above description and this disclosure. Further, note that such claims may be drafted to exclude any optional elements. Accordingly, the text is intended to serve as a precedent for the use of exclusive terms such as "merely", "only", and equivalents in relation to the listing of elements of the claims, or the use of "negative" limitations.

[0236] Without using such exclusive terms, the term "comprising" in the claims associated with this disclosure shall be taken to enable the inclusion of any additional elements regardless of the number of elements given in such claims, or the addition of features may be considered to transform the nature of the elements described in such claims. Unless specifically defined herein, all technical and scientific terms used herein should be given the broadest generally understood meaning possible while maintaining the validity of the claims.

[0237] The scope of this disclosure is not limited to the provided examples and / or the specification of the subject matter, but rather is limited only by the scope of the terms of the claims associated with this disclosure.

[0238] In the foregoing specification, the present disclosure has been described with reference to its specific embodiments. However, it will be apparent that various modifications and changes can be made thereto without departing from the broader spirit and scope of the present disclosure. For example, the foregoing process flow is described with reference to a particular order of process actions. However, many of the orders of the described process actions may be changed without affecting the scope or operation of the present disclosure. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a limiting sense.

Claims

1. A method in a wearable virtual, augmented, or mixed reality system, comprising: the wearable virtual, augmented, or mixed reality system comprises a GPU for converting image data into a virtual image, a display panel for displaying the virtual image, and a display bridge (DB) for transmitting the virtual image from the GPU to the display panel, the DB communicating with the GPU via a communication link and an auxiliary channel, and communicating with the display panel via a communication link and an auxiliary channel; The method comprises: the GPU obtaining a frame of image data and determining that an output of the GPU is a fully darkened frame having no virtual image; In response to determining that the output of the GPU is the fully darkened frame, the wearable virtual, augmented, or mixed reality system shutting down a portion, component, or function of the GPU; the wearable virtual, augmented, or mixed reality system shutting down the communication link between the GPU and the DB; the wearable virtual, augmented, or mixed reality system shutting down a portion, component, or function of the DB; the wearable virtual, augmented, or mixed reality system shutting down the communication link between the DB and the display panel; the wearable virtual, augmented, or mixed reality system shutting down a portion, component, or feature of the display panel; A method comprising:

2. The method of claim 1 , further comprising reorganizing frame data to reduce transfer time.

3. The method of claim 1 , further comprising the GPU sending an STP message to the DB.

4. The method of claim 3 , further comprising the GPU sending the STP message to the DB via an AUX message communicated over an auxiliary channel.

5. 2. The method of claim 1, wherein shutting down a portion, component, or function of the GPU does not include shutting down the entire GPU, and the portion, component, or function of the GPU is selected from the group consisting of memory reading, compression, and color segmentation.

6. The method of claim 1 , wherein the portion, component, or function of the DB is a memory write.

7. The method of claim 1 , wherein the portion, component, or function of the display panel is selected from the group consisting of a video RAM and a MIPI receiver.

8. The method of claim 1 , further comprising the GPU sending a wake-up signal to the DB.

9. The method of claim 8 , wherein the GPU transmits the wake-up signal over an AUX communications link.

10. The method of claim 1 , further comprising the GPU sending a wake-up signal to the communication link between the GPU and the DB.

11. 2. The method of claim 1, wherein the portion, component, or function of the GPU, the communication link between the GPU and the DB, the portion, component, or function of the DB, the communication link between the DB and a display panel, and the portion, component, or function of the display panel are shut down asynchronously.

12. The method of claim 1 , further comprising the DB sending a built-in line control message to the display panel.

13. A method in a wearable virtual, augmented, or mixed reality system, comprising: in the wearable virtual, augmented, or mixed reality system, no virtual image data exists for a particular user pose or a particular user field of view; The wearable virtual, augmented, or mixed reality system comprises a GPU for converting image data into a virtual image, a display panel for displaying the virtual image, a direct memory access controller (DMA), and a display bridge (DB) for transmitting the virtual image from the GPU to the display panel, the DB communicating with the GPU via a communication link and an auxiliary channel, and communicating with the display panel via a communication link and an auxiliary channel; The method comprises: the GPU acquiring a frame of image data; the GPU identifying a section of the frame of image data; the DMA sending the identified section of the frame of image data to the DB without further processing of the image data; the GPU obtaining another frame of image data and determining that the output of the GPU is a fully darkened frame having no virtual image; In response to determining that the output of the GPU is the fully darkened frame, the wearable virtual, augmented, or mixed reality system shutting down a portion, component, or function of the GPU; the wearable virtual, augmented, or mixed reality system shutting down a portion, component, or function of the DMA; the wearable virtual, augmented, or mixed reality system shutting down the communication link between the GPU and the DB; the wearable virtual, augmented, or mixed reality system shutting down a portion, component, or function of the DB; the wearable virtual, augmented, or mixed reality system shutting down the communication link between the DB and a display panel; the wearable virtual, augmented, or mixed reality system shutting down a portion, component, or feature of the display panel; A method comprising:

14. The method of claim 13 , wherein the sections of the frame of image data are rows of non-black image data.

15. The GPU transmits a wake-up signal to the DB. The method of claim 13 further comprising:

Images