Virtual, augmented, and mixed reality system and method

Abstract

To provide a virtual reality, augmented reality, and mixed reality image formation, visualization, and display system and method.SOLUTION: A method in a virtual, augmented, or mixed reality system includes a step in which a GPU determines / detects the absence of image data. The method also includes a step of shutting down a portion / component / function of the GPU. The method further includes a step of shutting down a communication link between the GPU and a DB. The method further includes a step of shutting down a portion / component / function of the DB. Additionally, the method includes a step of shutting down a communication link between the DB and a display panel. The method still further includes a step of shutting down a portion / component / function of the display panel.SELECTED DRAWING: Figure 10

Description

[Technical Field]

[0001] (Copyright Notice) A portion of the disclosure of this patent document contains material that is subject to copyright protection. The copyright owner has no objection to anyone copying this patent document or the patent disclosure, as it appears in the Patent and Trademark Office patent file or records, but otherwise reserves all copyright rights whatsoever.

[0002] The present disclosure relates to virtual reality, augmented reality, and mixed reality imaging, visualization, and display systems and methods. [Background technology]

[0003] Modern computing and display technologies have facilitated the development of virtual reality (VR), augmented reality (AR), and mixed reality (MR) systems. VR systems create simulated environments for users to experience. This can be done by presenting computer-generated images to the user through a head-mounted display. The images create a sensory experience that immerses the user within the simulated environment. VR scenarios typically involve only the presentation of computer-generated images, rather than including actual real-world images.

[0004] AR systems generally supplement real-world environments with simulated elements. For example, an AR system may provide a user with a view of the surrounding real-world environment via a head-mounted display. However, computer-generated imagery may also be presented on the display to enhance the real-world environment. The computer-generated imagery may include elements that are contextually related to the real-world environment. Such elements may include simulated text, images, objects, etc. MR systems also introduce simulated objects into the real-world environment, but these objects typically feature a degree of interactivity that exceeds that in AR systems. The simulated elements can often be interactive in real time.

[0005] 1 depicts an exemplary AR / MR scene 1 in which a user sees a real-world park setting 6 featuring people, trees, a building in the background, and a concrete platform 20. In addition to these items, computer-generated imagery is also presented to the user. The computer-generated imagery may include, for example, a robotic figure 10 standing on the real-world platform 20 and a flying, cartoon-like avatar character 12 that appears to be an anthropomorphic bumblebee, although these elements 12, 10 do not actually exist in the real-world environment.

[0006] Various optical systems generate images at various depths to display VR, AR, or MR scenarios. The human visual perception system is complex, making it difficult to produce VR / AR / MR technologies that facilitate comfortable, natural-feeling, and rich presentations of virtual image elements among other virtual or real-world image elements. Improved techniques for processing image data in such systems are needed, including, for example, techniques for providing control data to control how the image data is displayed, techniques for correcting optical distortions in the image data, and techniques for warping the image data based on the user's head pose. VR / AR / MR technologies also have size and portability issues, battery life issues, system heating issues, and other system and optical challenges that increase the importance of power-efficient image rendering. Improved techniques are needed to address these issues. The systems and methods described herein are configured to address these and other challenges.

[0007] What is needed is a technique or techniques for improving upon older techniques and / or other contemplated approaches. Some of the approaches described in this Background section are approaches that could be pursued, but not necessarily approaches that have been previously conceived or pursued. Summary of the Invention [Means for solving the problem]

[0008] In one embodiment, a method in a virtual, augmented, or mixed reality system includes a step in which a GPU determines / detects an absence of image data. The method also includes a step of shutting down a portion / component / function of the GPU. The method further includes a step of shutting down a communication link between the GPU and a DB. The method further includes a step of shutting down a portion / component / function of the DB. Additionally, the method includes a step of shutting down a communication link between the DB and a display panel. The method still further includes a step of shutting down a portion / component / function of the display panel.

[0009] In one or more embodiments, the method includes reorganizing frame data to reduce transfer time. The method may also include the GPU DP port sending a custom STP message to the DB. The method may also include the GPU sending the STP message to the DB AUS message. The part / component / function of the GPU may be selected from the group consisting of memory read, compression, and color segmentation. The part / component / function of the DB may be memory write. The part / component / function of the display panel may be selected from the group consisting of video RAM and a MIPI receiver.

[0010] In one or more embodiments, the method includes the GPU sending a wake-up signal to the DB. The GPU may send the wake-up signal via an AUX communication link. The method may also include the GPU sending the wake-up signal to a communication link between the GPU and the DB. A portion / component / function of the GPU, the communication link between the GPU and the DB, a portion / component / function of the DB, the communication link between the DB and the display panel, and a portion / component / function of the display panel may be shut down asynchronously. The method may also include the DB sending an internal line control message to the display panel.

[0011] In another embodiment, a method in a virtual, augmented, or mixed reality system includes a GPU receiving a frame of image data. The method also includes the GPU identifying a plurality of regions / portions / sections / tiles in the frame of image data that have changed from a previous frame of image data. The method further includes the GPU moving at least some of the plurality of regions / portions / sections / tiles to a beginning of the frame of data to form a reordered frame of image data. The method further includes the GPU sending the reordered frame of image data to a DB. In addition, the method includes shutting down portions / components / functions of the GPU, a communication link between the GPU and the DB, portions / components / functions of the DB, a communication link between the DB and a display panel, and portions / components / functions of the display panel.

[0012] In one or more embodiments, the method includes the GPU compressing the reordered frames of image data before sending the reordered frames of image data to the DB. The reordered frames of image data may be smaller than the frames of image data. The method may also include the DB storing the reordered frames of image data in a buffer.

[0013] In one or more embodiments, the method includes determining a size of the reordered frame of image data, and further includes shutting down portions / components / functions of the GPU, a communication link between the GPU and the DB, portions / components / functions of the DB, a communication link between the DB and the display panel, and portions / components / functions of the display panel only when the reordered frame of image data is smaller than a predetermined maximum size.

[0014] In one or more embodiments, the method includes, after the GPU sends the reordered frames of image data to the DB, sending an STP message to the DB. The method may further include, by the GPU, sending the STP message to the DB via SDP.

[0015] In one or more embodiments, the part / component / function of the GPU may be selected from the group consisting of memory read, compression, and color segmentation. The part / component / function of the DB may be memory write. The part / component / function of the display panel may be selected from the group consisting of video RAM and MIPI receiver.

[0016] In one or more embodiments, the method includes the GPU sending a wake-up signal to the DB. The GPU may send the wake-up signal over an AUX communication link. Portions / components / functions of the GPU, the communication link between the GPU and the DB, portions / components / functions of the DB, the communication link between the DB and the display panel, and portions / components / functions of the display panel may be shut down asynchronously.

[0017] In one or more embodiments, the method includes a step of the DB reconstructing a frame of image data from the reordered frame of image data. The method may further include a step of setting portions of the frame of image data that are not within multiple regions / portions / sections / tiles in the frame of image data to a background color. The method may also include a step of the DB blending the reordered frame of image data with a previous frame of image data. The method may further include a step of the DB blending the reordered frame of image data with image data associated with the updated foveation region. The method may also include a step of the DB masking the previous frame of image data before blending it with the reordered frame of image data.

[0018] In one or more embodiments, the method includes the DB scaling the reordered frames of image data. The method may further include the DB receiving a scaling factor from the GPU and the DB scaling the reordered frames of image data using the scaling factor. The scaling may be part of a foveation operation. The method may also include the DB performing a function on the image data, the function being selected from the group consisting of warping, pixelated dimming, occlusion, chromatic aberration correction, frame rate, and dilation. The method may further include storing the reordered frames of image data in a FIFO memory before shutting down a portion / component / function of the GPU. The method may also include the DB sending an internal line control message to the display panel.

[0019] In yet another embodiment, a method in a virtual, augmented, or mixed reality system includes a GPU dividing a first color field into a first partial first color field and a second partial first color field. The method also includes the GPU dividing the second color field into a first partial second color field and a second partial second color field. The method further includes the GPU sending the first partial first color field to a DB. The method further includes the GPU sending the first partial second color field to the DB after transmitting the first partial first color field. In addition, the method includes the GPU sending the second partial first color field to the DB after transmitting the first partial second color field. The method also includes the GPU sending the second partial second color field to the DB after transmitting the second partial first color field.

[0020] In one or more embodiments, the method includes the GPU splitting the third color field into a first partial third color field and a second partial third color field. The method may also include the GPU sending the first partial third color field to the DB after sending the first partial second color field and before sending the second partial first color field. The method may further include the GPU sending the second partial third color field to the DB after sending the second partial second color field. The method may also include the GPU sending the first partial first and second color fields and the second partial first and second color fields as a single vertically encoded data set.

[0021] In yet another embodiment, a method in a virtual, augmented, or mixed reality system includes a GPU dividing a first color field into a first partial first color field, a second partial first color field, and a third partial first color field. The method also includes a GPU dividing the second color field into a first partial second color field, a second partial second color field, and a third partial second color field. The method further includes a GPU dividing the third color field into a first partial third color field, a second partial third color field, and a third partial third color field. The method further includes a GPU sending the first partial first color field to a DB. In addition, the method includes a GPU sending the first partial second color field to the DB after sending the first partial first color field. The method also includes a step in which the GPU, after transmitting the first partial second color field, transmits the first partial third color field to the DB. The method further includes a step in which the GPU, after transmitting the first partial third color field, transmits the second partial first color field to the DB. The method further includes a step in which the GPU, after transmitting the second partial first color field, transmits the second partial second color field to the DB. In addition, the method includes a step in which the GPU, after transmitting the second partial second color field, transmits the second partial third color field to the DB. The method also includes a step in which the GPU, after transmitting the second partial third color field, transmits the third partial first color field to the DB. The method further includes a step in which the GPU, after transmitting the third partial first color field, transmits the third partial second color field to the DB. The method further includes the step of, after transmitting the third partial second color field, the GPU transmitting the third partial third color field to the DB.

[0022] In one or more embodiments, the method includes the GPU transmitting the first partial first, second, and third color field, the second partial first, second, and third color field, and the third partial first, second, and third color field as a single vertically encoded data set.

[0023] In one or more embodiments, the method includes the GPU sending first pose data to the DB and the DB warping the first partial first color field using the first pose data. The method also includes, after sending the first pose data, sending second pose data to the DB and the DB warping the first partial second color field using the second pose data. The method further includes, after sending the second pose data, sending third pose data to the DB and the DB warping the first partial third color field using the third pose data. The method further includes, after sending the third pose data, sending fourth pose data to the DB and the DB warping the second partial first color field using the fourth pose data. Additionally, the method includes, after transmitting the fourth pose data, transmitting fifth pose data to the DB, and the DB using the fifth pose data to warp the second partial second color field. The method also includes, after transmitting the fifth pose data, transmitting sixth pose data to the DB, and the DB using the sixth pose data to warp the second partial third color field. The method further includes, after transmitting the sixth pose data, transmitting seventh pose data to the DB, and the DB using the seventh pose data to warp the third partial first color field. The method further includes, after transmitting the seventh pose data, transmitting eighth pose data to the DB, and the DB using the eighth pose data to warp the third partial second color field. Additionally, the method includes a step of transmitting ninth attitude data to the DB after transmitting the eighth attitude data, and a step of the DB warping the third partial third color field using the ninth attitude data.

[0024] In one or more embodiments, the GPU transmits at least one of the first, second, third, fourth, fifth, sixth, seventh, eighth, and ninth pose data to the DB over the AUX communication link. The DB may warp each of the first partial first color field, the second partial first color field, the third partial first color field, the first partial second color field, the second partial second color field, the third partial second color field, the first partial third color field, the second partial third color field, and the third partial third color field twice.

[0025] In yet another embodiment, a data format for use in a virtual, augmented, or mixed reality system includes a first signaling row. The data format also includes a plurality of first color field rows. The data format further includes a second signaling row. Further, the data format includes a plurality of second color field rows. Additionally, the data format includes a third signaling row. The data format also includes a plurality of third color field rows.

[0026] In one or more embodiments, the first signaling row includes a number of active rows for the plurality of first color-field rows. The active rows may vary between image frames. The second signaling row may include a number of active rows for the plurality of second color-field rows. The third signaling row may include a number of active rows for the plurality of third color-field rows. The first signaling row may include starting positions for the plurality of first color-field rows. The second signaling row may include starting positions for the plurality of second color-field rows. The third signaling row may include starting positions for the plurality of third color-field rows. The first, second, and third color-field rows may include intensity information without color information. The first, second, and third signaling rows may include color information without intensity information. The first, second, and third signaling rows and the plurality of first, second, and third color field rows may be read at a rate faster than the rate at which an image corresponding to the plurality of first, second, and third color field rows is displayed.

[0027] In yet another embodiment, a method in a virtual, augmented, or mixed reality system includes detecting an area of ​​focus of a user. The method also includes a GPU rendering a virtual image outside the area of ​​focus at a lower resolution. The method further includes the GPU rendering a virtual image inside the area of ​​focus at a higher resolution. The method further includes the GPU sending the rendered virtual images outside and inside the area of ​​focus to one or more DBs. Additionally, the method includes the one or more DBs merging the rendered virtual images outside and inside the area of ​​focus to generate a frame of image data.

[0028] In one or more embodiments, the method includes the steps of: a GPU sending a rendered virtual image outside the area of ​​focus to a first DB; a GPU sending a rendered virtual image inside the area of ​​focus to a second DB; and the first and / or second DB merging the rendered virtual images outside and inside the area of ​​focus to generate a frame of image data.

[0029] In yet another embodiment, a method in a virtual, augmented, or mixed reality system includes detecting a user's hand within an FOV. The method also includes a GPU generating a mask corresponding to the location of the user's hand. The method further includes the GPU sending the mask and the frame of image data to a DB. The method further includes the DB modifying the frame of image data using the mask.

[0030] In one or more embodiments, the mask is a depth mask.

[0031] In yet another embodiment, a method in a virtual, augmented, or mixed reality system includes a GPU transmitting first color field image data to a DB, second color field image data to the DB, and third color field image data to the DB. The method also includes the GPU transmitting first pose data to the DB, and the DB warping the first color field image data using the first pose data to generate warped first color field image data. The method further includes the GPU transmitting second pose data to the DB after transmitting the first pose data, and the DB warping the second color field image data using the second pose data to generate warped second color field image data. The method further includes the GPU transmitting third pose data to the DB after transmitting the second pose data, and the DB warping the third color field image data using the third pose data to generate warped third color field image data.

[0032] In one or more embodiments, the method includes the GPU sending packet attitude data to the DB, and the DB warping the first color field image data using the first attitude data includes the DB calculating a first attitude delta from the packet attitude and the first attitude data. The method may further include the DB instructing display of a first color field image corresponding to the warped first color field image data immediately after the DB generates the warped first color field image data. The method may also include the DB performing a function on the first, second, and third color field image data, the function selected from the group consisting of projector light field distortion compensation, pixelated dimming, occlusion, chromatic aberration correction, frame rate, and enhancement.

[0033] In one or more embodiments, the method includes the GPU sending fourth orientation data to the DB after sending the third orientation data. The method also includes the DB warping the first color field image data using the fourth orientation data to generate second warped first color field image data. The method further includes the GPU sending fifth orientation data to the DB after sending the fourth orientation data. The method further includes the DB warping the second color field image data using the fifth orientation data to generate second warped second color field image data. In addition, the method includes the GPU sending sixth orientation data to the DB after sending the fifth orientation data. The method also includes the DB warping the third color field image data using the sixth orientation data to generate second warped third color field image data.

[0034] In yet another embodiment, a method in a virtual, augmented, or mixed reality system includes a GPU obtaining a frame of image data. The method also includes the GPU identifying a section of the frame of image data. The method further includes a direct memory access controller sending the identified section of the frame of image data to a DB without further processing of the image data.

[0035] In one or more embodiments, the section of the frame of image data is a row of non-black image data. The method may also include shutting down a portion / component / function of the GPU, a portion / component / function of the DMA, a communication link between the GPU and the DB, a portion / component / function of the DB, a communication link between the DB and the display panel, and / or a portion / component / function of the display panel.

[0036] In yet another embodiment, a method in a virtual, augmented, or mixed reality system includes a GPU dividing a first field into a first partial first field, a second partial first field, and a third partial first field. The method also includes a GPU dividing the second field into a first partial second field, a second partial second field, and a third partial second field. The method further includes a GPU dividing the third field into a first partial third field, a second partial third field, and a third partial third field. The method further includes a GPU sending the first partial first field to a DB. In addition, the method includes a GPU sending the first partial second field to the DB after sending the first partial first field. The method also includes a GPU sending the first partial third field to the DB after sending the first partial second field. The method further includes a step in which the GPU, after transmitting the first partial third field, sends a second partial first field to the DB. Further, the method includes a step in which the GPU, after transmitting the second partial first field, sends a second partial second field to the DB. In addition, the method includes a step in which the GPU, after transmitting the second partial second field, sends a second partial third field to the DB. The method also includes a step in which the GPU, after transmitting the second partial third field, sends a third partial first field to the DB. The method further includes a step in which the GPU, after transmitting the third partial first field, sends a third partial second field to the DB. Further, the method includes a step in which the GPU, after transmitting the third partial second field, sends a third partial third field to the DB.

[0037] In one or more embodiments, the method includes the GPU sending first attitude data to the DB, and the DB warping the first partial first field using the first attitude data. The method also includes, after sending the first attitude data, sending second attitude data to the DB, and the DB warping the first partial second field using the second attitude data. The method further includes, after sending the second attitude data, sending third attitude data to the DB, and the DB warping the first partial third field using the third attitude data. Furthermore, the method includes, after sending the third attitude data, sending fourth attitude data to the DB, and the DB warping the second partial first field using the fourth attitude data. Additionally, the method includes, after sending the fourth attitude data, sending fifth attitude data to the DB, and the DB warping the second partial second field using the fifth attitude data. The method also includes, after transmitting the fifth attitude data, transmitting sixth attitude data to the DB, and the DB using the sixth attitude data to warp the second partial third field. The method further includes, after transmitting the sixth attitude data, transmitting seventh attitude data to the DB, and the DB using the seventh attitude data to warp the third partial first field. The method further includes, after transmitting the seventh attitude data, transmitting eighth attitude data to the DB, and the DB using the eighth attitude data to warp the third partial second field. In addition, the method includes, after transmitting the eighth attitude data, transmitting ninth attitude data to the DB, and the DB using the ninth attitude data to warp the third partial third field.

[0038] In yet another embodiment, a method in a virtual, augmented, or mixed reality system includes a GPU obtaining a frame of image data. The method also includes the GPU obtaining occlusion data related to occlusion in a field of view, the data including depth map data. The method further includes the GPU sending the frame of image data and the occlusion data to a DB. The method further includes the DB masking the frame of image data prior to display using the occlusion data. The present invention provides, for example, the following. (Item 1) 1. A method in a virtual, augmented, or mixed reality system, comprising: the GPU determining / detecting the absence of image data; shutting down a portion / component / function of said GPU; Shutting down a communication link between the GPU and DB; Shutting down parts / components / functions of said DB; and shutting down a communication link between the DB and a display panel; shutting down a portion / component / function of said display panel; A method comprising: (Item 2) Item 10. The method of item 1, further comprising reorganizing frame data to reduce transfer time. (Item 3) 2. The method of claim 1, further comprising the GPU DP port sending a custom STP message to the DB. (Item 4) 4. The method of claim 3, further comprising the GPU sending the STP message to the DB AUX message. (Item 5) Item 10. The method of claim 1, wherein the part / component / function of the GPU is selected from the group consisting of memory reading, compression, and color segmentation. (Item 6) The method according to item 1, wherein the part / component / function of the DB is memory writing. (Item 7) Item 10. The method of claim 1, wherein the part / component / function of the display panel is selected from the group consisting of a video RAM and a MIPI receiver. (Item 8) 2. The method of claim 1, further comprising the GPU sending a wake-up signal to the DB. (Item 9) Item 9. The method of item 8, wherein the GPU transmits the wake-up signal over an AUX communication link. (Item 10) Item 10. The method of item 1, further comprising the GPU sending a wake-up signal to a communication link between the GPU and a DB. (Item 11) The method described in item 1, wherein a portion / component / function of the GPU, a communication link between the GPU and DB, a portion / component / function of the DB, a communication link between the DB and a display panel, and a portion / component / function of the display panel are shut down asynchronously. (Item 12) Item 10. The method of item 1, further comprising the DB sending a built-in line control message to the display panel. (Item 13) 1. A method in a virtual, augmented, or mixed reality system, comprising: a GPU dividing the first color field into a first partial first color field and a second partial first color field; the GPU splitting the second color field into a first partial second color field and a second partial second color field; the GPU sending the first partial first color field to a DB; the GPU sends the first partial first color field and then sends the first partial second color field to the DB; the GPU sends the first partial second color field, and then sends the second partial first color field to the DB; the GPU transmits the second partial first color field and then transmits the second partial second color field to the DB; A method comprising: (Item 14) the GPU splitting the third color field into a first partial third color field and a second partial third color field; the GPU sending the first partial third color field to the DB after sending the first partial second color field and before sending the second partial first color field; the GPU transmits the second partial second color field and then transmits the second partial third color field to the DB; Item 14. The method of item 13, further comprising: (Item 15) Item 14. The method of item 13, further comprising the GPU transmitting the first partial first and second color field and the second partial first and second color field as a single vertically encoded data set. (Item 16) 1. A method in a virtual, augmented, or mixed reality system, comprising: a GPU dividing the first color field into a first partial first color field, a second partial first color field, and a third partial first color field; the GPU dividing the second color field into a first partial second color field, a second partial second color field, and a third partial second color field; the GPU dividing the third color field into a first partial third color field, a second partial third color field, and a third partial third color field; the GPU sending the first partial first color field to a DB; the GPU sends the first partial first color field and then sends the first partial second color field to the DB; the GPU sending the first partial second color field and then sending the first partial third color field to the DB; the GPU sends the first partial third color field and then sends the second partial first color field to the DB; the GPU sends the second partial first color field and then sends the second partial second color field to the DB; the GPU sends the second partial second color field and then sends the second partial third color field to the DB; the GPU sends the second partial third color field, and then sends the third partial first color field to the DB; the GPU sends the third partial first color field and then sends the third partial second color field to the DB; After the GPU transmits the third partial second color field, the GPU transmits the third partial third color field to the DB; A method comprising: (Item 17) Item 17. The method of item 16, further comprising the GPU transmitting the first partial first, second, and third color field, the second partial first, second, and third color field, and the third partial first, second, and third color field as a single vertically encoded data set. (Item 18) The GPU transmits first attitude data to the DB; the DB warping the first partial first color field using the first pose data; the GPU transmits the first attitude data and then transmits second attitude data to the DB; the DB warping the first partial second color field using the second pose data; the GPU transmits third attitude data to the DB after transmitting the second attitude data; the DB warping the first partial third color field using the third pose data; the GPU transmits fourth attitude data to the DB after transmitting the third attitude data; the DB warping the second partial first color field using the fourth pose data; the GPU transmits fifth attitude data to the DB after transmitting the fourth attitude data; the DB warping the second partial second color field using the fifth pose data; the GPU transmits sixth attitude data to the DB after transmitting the fifth attitude data; the DB warping the second partial third color field using the sixth pose data; the GPU transmits seventh attitude data to the DB after transmitting the sixth attitude data; the DB warping the third partial first color field using the seventh pose data; the GPU transmits eighth attitude data to the DB after transmitting the seventh attitude data; the DB warping the third partial second color field using the eighth pose data; the GPU transmits the eighth attitude data and then transmits ninth attitude data to the DB; the DB warping the third partial third color field using the ninth pose data; Item 17. The method of item 16, further comprising: (Item 19) Item 19. The method according to item 18, wherein the method uses updated warping to perform frame rate enhancement with zero latency appearance. (Item 20) Item 19. The method of item 18, further comprising adjusting at least one of the first to ninth pose data and performing persistent warping. (Item 21) The DB calculates first attitude data; the DB warping the first partial first color field using the first pose data; The DB calculates second attitude data after calculating the first attitude data; the DB warping the first partial second color field using the second pose data; The DB calculates third attitude data after calculating the second attitude data; the DB warping the first partial third color field using the third pose data; The DB calculates fourth attitude data after calculating the third attitude data; the DB warping the second partial first color field using the fourth pose data; The DB calculates fifth attitude data after calculating the fourth attitude data; the DB warping the second partial second color field using the fifth pose data; The DB calculates sixth attitude data after calculating the fifth attitude data; the DB warping the second partial third color field using the sixth pose data; The DB calculates seventh attitude data after calculating the sixth attitude data; the DB warping the third partial first color field using the seventh pose data; The DB calculates eighth attitude data after calculating the seventh attitude data; the DB warping the third partial second color field using the eighth pose data; The DB calculates ninth attitude data after calculating the eighth attitude data; the DB warping the third partial third color field using the ninth pose data; Item 17. The method of item 16, further comprising: (Item 22) Item 19. The method of item 18, wherein the GPU transmits at least one of the first, second, third, fourth, fifth, sixth, seventh, eighth, and ninth attitude data to the DB through an AUX communication link. (Item 23) Item 19. The method of item 18, wherein the DB warps each of the first partial first color field, the second partial first color field, the third partial first color field, the first partial second color field, the second partial second color field, the third partial second color field, the first partial third color field, the second partial third color field, and the third partial third color field twice. (Item 24) 1. A data format for use in a virtual, augmented, or mixed reality system, comprising: a first signaling line; a plurality of first color field rows; a second signaling line; and a plurality of second color field rows; a third signaling line; and Multiple third color field rows and A data format comprising: (Item 25) Item 25. The data format of item 24, wherein the first signaling row includes a number of active rows for the plurality of first color field rows. (Item 26) Item 25. The data format of item 24, wherein the active row changes between image frames. (Item 27) Item 25. The data format of item 24, wherein the first, second, and third signaling lines and the plurality of first, second, and third color field lines are read at a rate faster than an image corresponding to the plurality of first, second, and third color field lines is displayed. (Item 28) Item 25. The data format of item 24, wherein the second signaling row includes a number of active rows for the plurality of second color field rows. (Item 29) Item 25. The data format of item 24, wherein the third signaling row includes a number of active rows for the plurality of third color field rows. (Item 30) 25. The data format of item 24, wherein the first signaling row includes the starting positions of the plurality of first color field rows. (Item 31) 25. The data format of item 24, wherein the second signaling row includes the starting positions of the plurality of second color field rows. (Item 32) 25. The data format of item 24, wherein the third signaling row includes the starting positions of the plurality of third color field rows. (Item 33) 25. The data format of claim 24, wherein the first, second, and third color field rows contain intensity information without color information. (Item 34) Item 34. The data format of item 33, wherein the first, second, and third signaling rows include color information without intensity information. (Item 35) 1. A method in a virtual, augmented, or mixed reality system, comprising: Detecting an area of ​​focus of a user; a GPU rendering a virtual image outside the area of ​​focus at a lower resolution; the GPU rendering a virtual image within the area of ​​focus at a higher resolution; the GPU sending the rendered virtual images outside and inside the area of ​​focus to one or more DBs; the one or more DBs merging the rendered virtual images outside and inside the area of ​​focus to generate a frame of image data; A method comprising: (Item 36) The GPU sends the rendered virtual image outside the area of ​​focus to a first DB; the GPU sending the rendered virtual image inside the area of ​​focus to a second DB; the first and / or second DB merging the rendered virtual images outside and inside the area of ​​focus to generate a frame of image data; Item 36. The method of item 35, further comprising: (Item 37) 1. A method in a virtual, augmented, or mixed reality system, comprising: Detecting a user's hand within the FOV; generating a mask corresponding to the location of the user's hands; the GPU sending the mask and the frame of image data to a DB; the DB modifies the frame of image data using the mask; and A method comprising: (Item 38) Item 38. The method of item 37, wherein the mask is a depth mask. (Item 39) 1. A method in a virtual, augmented, or mixed reality system, comprising: The GPU sends the first color field image data to the DB; The GPU sends the second color field image data to the DB; The GPU sends the third color field image data to the DB; The GPU transmits first attitude data to the DB; the DB warping the first color field image data using the first pose data to generate warped first color field image data; the GPU transmits the first attitude data and then transmits second attitude data to the DB; the DB warping the second color field image data using the second orientation data to generate warped second color field image data; the GPU transmits third attitude data to the DB after transmitting the second attitude data; the DB warps the third color field image data using the third orientation data to generate warped third color field image data; A method comprising: (Item 40) Item 39. The method of item 39, wherein the DB further comprises performing functions on the first, second, and third color field image data, the functions being selected from the group consisting of projector light field distortion compensation, pixelated dimming, occlusion, chromatic aberration correction, frame rate, and enhancement. (Item 41) The GPU further includes transmitting packet posture data to the DB; Item 39. The method of item 39, wherein the DB warping the first color field image data using the first orientation data includes the DB calculating a first orientation delta from the packet orientation and the first orientation data. (Item 42) Item 39. The method of item 39, further comprising the DB instructing display of a first color field image corresponding to the warped first color field image data immediately after the DB generates the warped first color field image data. (Item 43) the GPU transmits fourth attitude data to the DB after transmitting the third attitude data; the DB warping the first color field image data using the fourth orientation data to generate second warped first color field image data; the GPU transmits fifth attitude data to the DB after transmitting the fourth attitude data; the DB warping the second color field image data using the fifth orientation data to generate second warped second color field image data; the GPU transmits sixth attitude data to the DB after transmitting the fifth attitude data; the DB warps the third color field image data using the sixth orientation data to generate second warped third color field image data; Item 39. The method of item 39, further comprising: (Item 44) 1. A method in a virtual, augmented, or mixed reality system, comprising: the GPU acquiring a frame of image data; the GPU identifying a section of the frame of image data; a direct memory access controller transmitting the identified section of the frame of image data to a DB without further processing of the image data; A method comprising: (Item 45) Item 45. The method of item 44, wherein the section of the frame of image data is a row of non-black image data. (Item 46) Shutting down some parts / components / functions of said GPU; Shutting down some / components / functions of said DMA; shutting down a communication link between the GPU and DB; Shutting down parts / components / functions of said DB; shutting down the communication link between the DB and the display panel; and / or Shutting down a portion / component / function of said display panel Item 45. The method of item 44, further comprising: (Item 47) 1. A method in a virtual, augmented, or mixed reality system, comprising: the GPU dividing the first field into a first partial first field, a second partial first field, and a third partial first field; the GPU dividing the second field into a first partial second field, a second partial second field, and a third partial second field; the GPU dividing the third field into a first partial third field, a second partial third field, and a third partial third field; The GPU sends the first partial first field to a DB; The GPU transmits the first partial first field and then transmits the first partial second field to the DB; the GPU sending the first partial second field and then the first partial third field to the DB; the GPU sends the first partial third field and then the second partial first field to the DB; After the GPU sends the second partial first field, the GPU sends the second partial second field to the DB; The GPU transmits the second partial second field and then transmits the second partial third field to the DB; The GPU transmits the second partial third field and then transmits the third partial first field to the DB; the GPU sends the third partial first field and then sends the third partial second field to the DB; After the GPU transmits the third partial second field, the GPU transmits the third partial third field to the DB; The GPU transmits first attitude data to the DB; the DB warping the first partial first field using the first pose data; the GPU transmits the first attitude data and then transmits second attitude data to the DB; the DB warping the first partial second field using the second pose data; the GPU transmits third attitude data to the DB after transmitting the second attitude data; the DB warping the first partial third field using the third pose data; the GPU transmits fourth attitude data to the DB after transmitting the third attitude data; the DB warping the second partial first field using the fourth attitude data; the GPU transmits fifth attitude data to the DB after transmitting the fourth attitude data; the DB warping the second partial second field using the fifth attitude data; the GPU transmits sixth attitude data to the DB after transmitting the fifth attitude data; the DB warping the second partial third field using the sixth attitude data; the GPU transmits seventh attitude data to the DB after transmitting the sixth attitude data; the DB warping the third partial first field using the seventh attitude data; the GPU transmits eighth attitude data to the DB after transmitting the seventh attitude data; the DB warping the third partial second field using the eighth attitude data; the GPU transmits the eighth attitude data and then transmits ninth attitude data to the DB; the DB warps the third partial third field using the ninth attitude data; and A method comprising: (Item 48) 1. A method in a virtual, augmented, or mixed reality system, comprising: the GPU acquiring a frame of image data; the GPU obtaining occlusion data relating to occlusions in a field of view, the data including depth map data; the GPU sending the frame of image data and the occlusion data to a DB; the DB masking the frame of image data before displaying it using the occlusion data; A method comprising: [Brief explanation of the drawings]

[0039] The drawings described below are for illustrative purposes only. The drawings are not intended to limit the scope of the present disclosure. The drawings illustrate the design and utility of various embodiments of the present disclosure. It should be noted that the drawings are not drawn to scale, and elements of similar structure or function are represented by like reference numerals throughout the drawings. To better understand how the foregoing and other advantages and objects of various embodiments of the present disclosure are obtained, a more detailed description of the present disclosure will be given by reference to specific embodiments thereof that are illustrated in the accompanying drawings. With the understanding that these drawings depict only exemplary embodiments of the present disclosure and therefore should not be considered limiting of its scope, the present disclosure will be described and explained with additional specificity and detail through the use of the accompanying drawings.

[0040] [Figure 1] FIG. 1 illustrates a user's view of an AR / MR scene using an exemplary AR system.

[0041] [Figure 2] 2-5 diagrammatically depict a user using a VR / AR / MR system, according to some embodiments. [Figure 3] 2-5 diagrammatically depict a user using a VR / AR / MR system, according to some embodiments. [Figure 4] 2-5 diagrammatically depict a user using a VR / AR / MR system, according to some embodiments. [Figure 5] 2-5 diagrammatically depict a user using a VR / AR / MR system, according to some embodiments.

[0042] [Figure 6] FIG. 6 diagrammatically depicts various planes of a multi-plane focusing system, according to some embodiments.

[0043] [Figure 7] FIG. 7 diagrammatically depicts a VR / AR / MR system, according to some embodiments.

[0044] [Figure 8] FIG. 8 diagrammatically depicts image generation components of a VR / AR / MR system, according to some embodiments.

[0045] [Figure 8A] FIG. 8A diagrammatically depicts image generation components of a VR / AR / MR system, according to some embodiments.

[0046] [Figure 9] FIG. 9 diagrammatically depicts a display bridge for use with a VR / AR / MR system, according to some embodiments.

[0047] [Figure 9A] FIG. 9A diagrammatically depicts a display bridge for use with a VR / AR / MR system, according to some embodiments.

[0048] [Figure 10] FIG. 10 diagrammatically depicts a pixel engine for use with a display bridge in a VR / AR / MR system, according to some embodiments.

[0049] [Figure 10A] FIG. 10A diagrammatically depicts a pixel engine for use with a display bridge in a VR / AR / MR system, according to some embodiments.

[0050] [Figure 11] FIG. 11 diagrammatically depicts a display panel for use with a VR / AR / MR system, according to some embodiments.

[0051] [Figure 12] FIG. 12 diagrammatically depicts two input data formats for use with a VR / AR / MR system, according to some embodiments.

[0052] [Figure 13] FIG. 13 diagrammatically depicts a color image data format for use with a VR / AR / MR system, according to some embodiments.

[0053] [Figure 14] FIG. 14 diagrammatically depicts a grayscale image data format for use with a VR / AR / MR system, according to some embodiments.

[0054] [Figure 15] FIG. 15 graphically depicts color segmentation of color image data for use with a VR / AR / MR system, according to some embodiments.

[0055] [Figure 15A] FIG. 15A graphically depicts color segmentation of color image data in a DB for use with a VR / AR / MR system, according to some embodiments.

[0056] [Figure 15B] FIG. 15B diagrammatically depicts packing of image data without color segmentation in a DB for use with a VR / AR / MR system, according to some embodiments.

[0057] [Figure 15C] FIG. 15C graphically depicts the RGB30 data format for storing color image data 1502.

[0058] [Figure 15D] FIG. 15D graphically depicts an input display panel resolution of 1,440 pixels by 1,440 pixels, stored in SRAM as 128-bit coherent data.

[0059] [Figure 15E]FIG. 15E graphically depicts an input display panel resolution of 512 pixels by 512 pixels, stored in SRAM as 128-bit coherent data.

[0060] [Figure 15F] FIG. 15F graphically depicts an input display panel resolution of 512 pixels by 512 pixels for a special image, stored in SRAM as 128-bit coherent data.

[0061] [Figure 15G] FIG. 15G graphically depicts the data structure after color segmentation of the frame.

[0062] [Figure 16] FIG. 16 diagrammatically depicts a secondary display stream flow through a display bridge for use with a VR / AR / MR system, according to some embodiments.

[0063] [Figure 16A] FIG. 16A diagrammatically depicts a secondary display stream flow through a display bridge for use with a VR / AR / MR system, according to some embodiments.

[0064] [Figure 17] FIG. 17 diagrammatically depicts an image data packet, according to some embodiments.

[0065] [Figure 17A] FIG. 17A graphically depicts a VSC_EXT_VESA SDP message 1750 associated with vertical blanking.

[0066] [Figure 17B] FIG. 17B graphically depicts a horizontal blanking SDP message 1770 associated with horizontal blanking.

[0067] [Figure 18] FIG. 18 diagrammatically depicts data flow through a display bridge for use with a VR / AR / MR system, according to some embodiments.

[0068] [Figure 19] FIG. 19 diagrammatically depicts a video stream, according to some embodiments.

[0069] [Figure 20] 20-24A diagrammatically depict a partially dark low power mode, according to some embodiments. [Figure 21] 20-24A diagrammatically depict a partially dark low power mode, according to some embodiments. [Figure 22] 20-24A diagrammatically depict a partially dark low power mode, according to some embodiments. [Figure 22A] 20-24A diagrammatically depict a partially dark low power mode, according to some embodiments. [Figure 23] 20-24A diagrammatically depict a partially dark low power mode, according to some embodiments. [Figure 24] 20-24A diagrammatically depict a partially dark low power mode, according to some embodiments. [Figure 24A] 20-24A diagrammatically depict a partially dark low power mode, according to some embodiments.

[0070] [Figure 24B] FIG. 24B diagrammatically depicts MIPI timing of a display system transitioning into and sustaining partial dimming mode, according to some embodiments.

[0071] [Figure 25] FIG. 25 graphically depicts packing of partially darkened image data according to some embodiments.

[0072] [Figure 26] FIG. 26 diagrammatically depicts inputs to and outputs from a display bridge for use with a VR / AR / MR system, according to some embodiments.

[0073] [Figure 27] 27-31B graphically depict MIPI receiver output data, according to some embodiments. [Figure 28] 27-31B graphically depict MIPI receiver output data, according to some embodiments. [Figure 29] 27-31B graphically depict MIPI receiver output data, according to some embodiments. [Figure 30] 27-31B graphically depict MIPI receiver output data, according to some embodiments. [Figure 30A] 27-31B graphically depict MIPI receiver output data, according to some embodiments. [Figure 31] 27-31B graphically depict MIPI receiver output data, according to some embodiments. [Figure 31A-1] 27-31B graphically depict MIPI receiver output data, according to some embodiments. [Figure 31A-2] 27-31B graphically depict MIPI receiver output data, according to some embodiments. [Figure 31B-1] 27-31B graphically depict MIPI receiver output data, according to some embodiments. [Figure 31B-2] 27-31B graphically depict MIPI receiver output data, according to some embodiments.

[0074] [Figure 32] FIG. 32 graphically depicts MIPI timing of a display system transitioning into and sustaining full dark mode 3200, according to some embodiments.

[0075] [Figure 33] FIG. 33 diagrammatically depicts a display system's transition into, duration of, and exit from full dark mode, according to some embodiments.

[0076] [Figure 34] FIG. 34 diagrammatically depicts a display system's transition into, duration of, and exit from full dark mode for panel self-refresh, according to some embodiments.

[0077] [Figure 35] FIG. 35 diagrammatically depicts MIPI timing for two cycles of DB operating in partial dimming low power mode, in accordance with some embodiments.

[0078] [Figure 36A] FIG. 36A depicts an AR view, according to some embodiments.

[0079] [Figure 36B] FIG. 36B depicts an AR view with a black partial mask applied thereon, according to some embodiments. DETAILED DESCRIPTION OF THE INVENTION

[0080] Various embodiments of the present disclosure are directed to systems, methods, and articles of manufacture for VR / AR / MR, in a single embodiment or multiple embodiments. Other objects, features, and advantages of the present disclosure are set forth in the detailed description, drawings, and claims.

[0081] Various embodiments will now be described in detail with reference to the drawings, which are provided as illustrative examples to enable those skilled in the art to practice the present disclosure. It should be noted that the following figures and examples are not intended to limit the scope of the present disclosure. Where certain elements of the present disclosure can be partially or fully implemented using known components (or methods or processes), only those portions of such known components (or methods or processes) necessary for understanding the present disclosure will be described, and detailed descriptions of other portions of such known components (or methods or processes) will be omitted so as not to obscure the present disclosure. Furthermore, various embodiments encompass present and future known equivalents of the components referenced herein by way of example.

[0082] Embodiments according to the present disclosure address the problem of implementing VR / AR / MR systems, which often rely on a combination of off-the-shelf and custom components. In some cases, the off-the-shelf components do not possess all of the features or performance characteristics needed to implement certain desired aspects of the VR / AR / MR system to be deployed. Some embodiments are directed to approaches for adding capabilities and / or repurposing resources to accommodate the desired features or performance characteristics of the VR / AR / MR system to be deployed. The accompanying figures and discussion herein present example environments, systems, methods, and computer program products for VR / AR / MR systems.

[0083] Although the head-mounted audiovisual display system and power management system can be implemented independently of the AR / MR system, some embodiments below are described in relation to an AR / MR system for illustrative purposes only. The power management system described herein can also be used in a similar manner in a VR system. (Problem and Solution Overview)

[0084] VR / AR / MR systems have limitations such as size and portability issues, battery life issues, system heating issues, processing power, memory, bandwidth, data sources, component latency, and other system and optical challenges that can adversely affect VR / AR / MR system performance. These limitations increase the importance of power-efficient image rendering.

[0085] For example, in some wearables, various components in the image pipeline (e.g., GPU, display bridge, display panel, etc.) consume a significant portion of system resources (e.g., processing power, memory, bandwidth, battery life). Furthermore, these system resource demands can lead to size and portability issues and system overheating issues. Furthermore, component latency issues can also affect VR / AR / MR system performance. For example, system latency between the final warping of rendered image data and the display of an image corresponding to the warped image data can result in artifacts.

[0086] According to various embodiments, the power management system includes features such as deactivation of depth planes or color fields within depth planes, time domain power management, discrete imaging modes, low power depth plane switching, lower power low latency standby / wake-up, lower power side channels, multiple component low power modes, and reduced power to the light source and / or SLM.

[0087] Embodiments described herein include power management systems and methods for use with various VR / AR / MR systems. These power management systems and methods reduce system resources consumed by an image pipeline, thereby addressing many of the issues described above. Embodiments described herein also include virtual image warping systems and methods for use with various VR / AR / MR systems. These virtual image warping systems and methods address some of the issues described above. (Illustrative VR, AR, and / or MR Systems)

[0088] The following description relates to illustrative VR, AR, and / or MR systems in which various power management system embodiments may be practiced. However, it should be understood that the embodiments are also suitable for use in other types of display systems (including other types of VR, AR, and / or MR systems), and thus the embodiments are not limited to only the illustrative systems disclosed herein.

[0089] The VR / AR / MR systems disclosed herein can include a display that presents computer-generated images (video / image data) to a user. In some embodiments, the display system is wearable, which can advantageously provide a more immersive VR / AR / MR experience. Various components of a VR, AR, and / or MR virtual image system 100 are depicted in FIGS. 2-5. The virtual image generation system 100 includes a frame structure 102 worn by an end user 50, a display subsystem 110 carried by the frame structure 102 such that the display subsystem 110 is positioned directly in front of the eyes of the end user 50, and a speaker 106 carried by the frame structure 102 such that the speaker 106 is positioned adjacent to the ear canal of the end user 50 (optionally, another speaker (not shown) is positioned adjacent to the other ear canal of the end user 50 to provide stereo / adjustable sound control). Display subsystem 110 is designed to present light patterns to the eyes of end user 50 that can be comfortably perceived as an augmentation to physical reality with a high level of image quality and three-dimensional perception, and is also capable of presenting two-dimensional content. Display subsystem 110 presents a sequence of frames at a high frequency that provides the perception of a single coherent scene.

[0090] In the illustrated embodiment, the display subsystem 110 employs an "optical see-through" display, through which a user can view light from real objects directly through a transparent (or translucent) element. The transparent element, often referred to as a "combiner," superimposes light from the display over the user's view of the real world. To this end, the display subsystem 110 includes a partially transparent display. In some embodiments, the transparent display may be electronically controlled. In some embodiments, the transparent display may include segmented dimming, controlling the transparency of one or more portions of the transparent display. In some embodiments, the transparent display may include global dimming, controlling the overall transparency of the transparent display. The display is positioned within the field of view of the end user 50, between the eyes of the end user 50 and the surrounding environment, such that direct light from the surrounding environment is transmitted through the display to the eyes of the end user 50.

[0091] In the illustrated embodiment, the image projection assembly provides light to a partially transparent display, whereby it is combined with direct light from the surrounding environment and transmitted from the display to the user's eye 50. The projection subsystem may be a fiber optic scanning-based projection device, and the display may be a waveguide-based display into which scanned light from the projection subsystem is injected to produce, for example, an image at a single optical viewing distance closer than infinity (e.g., arm's length), images at multiple discrete optical viewing distances or focal planes, and / or image layers stacked at multiple viewing distances or focal planes to represent a stereoscopic 3D object. These layers within the light field may be stacked closely enough together (i.e., one layer is within the distraction cone of an adjacent layer) to appear persistent to the human accessory visual system. Additionally or alternatively, picture elements may be blended across two or more layers to increase the perceived continuity of transitions between layers within the light field, even when the layers are more loosely stacked (i.e., one layer is outside the distraction cone of an adjacent layer). The display subsystem 110 may be monocular or binocular.

[0092] The virtual image generation system 100 may also include one or more sensors (not shown) mounted to the frame structure 102 for detecting the position and movement of the head 54 of the end user 50 and / or the eye position and interocular distance of the end user 50. Such sensors may include image capture devices (such as cameras), microphones, inertial measurement units, accelerometers, compasses, GPS units, wireless devices, and / or gyroscopes. Many of these sensors operate under the assumption that the frame 102 onto which they are affixed is, in turn, substantially fixed to the user's head, eyes, and ears.

[0093] The virtual image generation system 100 may also include a user orientation detection module. The user orientation module may detect the instantaneous position of the end user's 50's head 54 (e.g., via sensors coupled to the frame 102) and predict the position of the end user's 50's head 54 based on the position data received from the sensors. Detecting the instantaneous position of the end user's 50's head 54 facilitates determining the specific real object the end user 50 is looking at, thereby providing an indication of the specific virtual object to be generated in relation to that real object, and further providing an indication of the location within which the virtual object should be displayed. The user orientation module may also track the eyes of the end user 50 based on tracking data received from the sensors.

[0094] The virtual image generation system 100 may also include a control subsystem, which may take any of a wide variety of forms. The control subsystem may include several controllers, such as one or more microcontrollers, microprocessors or central processing units (CPUs), digital signal processors, graphics processing units (GPUs), other integrated circuit controllers such as application specific integrated circuits (ASICs), display bridge chips, display controllers, programmable gate arrays (PGAs), e.g., field programmable gate arrays (FPGAs), and / or programmable logic controllers (PLUs).

[0095] The control subsystem of virtual image generation system 100 may include a central processing unit (CPU), a graphics processing unit (GPU), one or more frame buffers, and a 3D database for storing 3D scene data. The CPU may control overall operation, while the GPU may render frames from the 3D data stored in the 3D database (i.e., convert the 3D scene into a 2D image) and store these frames in the frame buffer. One or more additional integrated circuits may control the loading and retrieval of frames into and from the frame buffer and the operation of the image projection assembly of display subsystem 110.

[0096] The various processing components of virtual image generation system 100 may be physically contained within a distributed subsystem. For example, as illustrated in Figures 2-5, virtual image generation system 100 may include a local processing and data module 130 operably coupled to a local display bridge 142, a display subsystem 110, and sensors, such as by wired or wireless connectivity 136. Local processing and data module 130 may be mounted in a variety of configurations, such as fixedly attached to frame structure 102 (Figure 2), fixedly attached to a helmet or hat 56 (Figure 3), removably attached to a torso 58 of end user 50 (Figure 4), or removably attached to a waist 60 of end user 50 in a belt-type configuration (Figure 5). The virtual image generation system 100 may also include a remote processing module 132 and a remote data repository 134 operably coupled to the local processing and data module 130 and the local display bridge 142 by wired or wireless connectivity 138, 140, etc., such that these remote modules 132, 134 are operably coupled to each other and available as resources to the local processing and data module 130 and the local display bridge 142.

[0097] The local processing and data module 130 and the local display bridge 142 may each include a power-efficient processor or controller and digital memory, such as flash memory, both of which may be utilized to aid in processing, caching, and storing data captured from sensors and / or obtained and / or processed using the remote processing module 132 and / or the remote data repository 134, possibly for passage to the display subsystem 110 after processing or retrieval. The remote processing module 132 may include one or more relatively powerful processors or controllers configured to analyze and process data and / or image information. The remote data repository 134 may include a relatively large-scale digital data storage facility, which may be available through the Internet or other networking configuration in a “cloud” resource configuration. In some embodiments, all data is stored and all calculations are performed in the local processing and data module 130 and the local display bridge 142, allowing for fully autonomous use from any remote module.

[0098] The couplings 136, 138, 140 between the various components described above may include one or more wired interfaces or ports for providing wire or optical communication, or one or more wireless interfaces or ports via RF, microwave, IR, and the like for providing wireless communication. In some implementations, all communication may be wired, while in other implementations, all communication may be wireless. In still further implementations, the wired and wireless communication options may differ from those illustrated in Figures 2-5. Thus, the particular options for wired or wireless communication should not be considered limiting.

[0099] In some embodiments, the user orientation module is contained within the local processing and data module 130 and / or the local display bridge 142, while the CPU and GPU are contained within the remote processing module. In alternative embodiments, the CPU, GPU, or portions thereof may be contained within the local processing and data module 130 and / or the local display bridge 142. The 3D database can be associated with the remote data repository 134 or located locally.

[0100] Some VR, AR, and / or MR systems use multiple volume phase holograms, surface relief holograms, or light-directing optical elements that incorporate depth plane information to generate images that appear to originate from distinct depth planes. In other words, a diffraction pattern or diffractive optical element ("DOE") may be incorporated into or imprinted / embossed on a light-directing optical element ("LOE," e.g., a planar waveguide) such that collimated light (a light beam with a substantially planar wavefront) intersects the diffraction pattern at multiple locations as it is substantially totally internally reflected along the LOE and exits toward the user's eye. The DOE is configured such that light exiting the LOE through it is converged to appear to originate from a particular depth plane. The collimated light may be generated using an optical focusing lens ("concentrator").

[0101] For example, a first LOE may be configured to deliver collimated light to the eye that appears to originate from the optical infinity depth plane (0 diopters). Another LOE may be configured to deliver collimated light that appears to originate from a distance of 2 meters (½ diopters). Yet another LOE may be configured to deliver collimated light that appears to originate from a distance of 1 meter (1 diopter). It should be understood that by using stacked LOE assemblies, multiple depth planes can be created, with each LOE configured to display an image that appears to originate from a particular depth plane. It should be understood that a stack may include any number of LOEs. However, at least N stacked LOEs are required to generate N depth planes. Furthermore, N, 2N, or 3N stacked LOEs may be used to generate an RGB color image at N depth planes.

[0102] To present 3-D virtual content to a user, a VR, AR, and / or MR system projects images of the virtual content into the user's eye so that they appear to originate from different depth planes in the Z direction (i.e., orthogonal away from the user's eye). In other words, the virtual content may vary not only in the X and Y directions (i.e., 2D planes orthogonal to the central visual axis of the user's eye), but also in the Z direction, so that the user may perceive objects as being very close, at infinite distance, or any distance in between. In other embodiments, the user may perceive multiple objects simultaneously at different depth planes. For example, a virtual dragon may appear to appear from infinity and running toward the user. Alternatively, the user may simultaneously see a virtual bird at a distance of 3 meters away from the user and a virtual coffee cup at arm's length (approximately 1 meter) from the user.

[0103] A multi-plane focusing system creates the perception of variable depth by projecting images onto some or all of multiple depth planes located at discrete fixed distances in the Z direction from the user's eyes. Referring now to FIG. 6 , it should be understood that the multi-plane focusing system may display frames at fixed depth planes 150 (e.g., the six depth planes 150 shown in FIG. 6 ). While an MR system can include any number of depth planes 150, one exemplary multi-plane focusing system has six fixed depth planes 150 in the Z direction. When generating virtual content in one or more of the six depth planes 150, a 3-D perception is created in which the user perceives one or more virtual objects at variable distances from the user's eyes. Given that the human eye is more sensitive to objects that are closer than objects that appear farther away, more depth planes 150 are generated closer to the eyes, as shown in FIG. 6 . In other embodiments, the depth planes 150 may be positioned equidistant from each other.

[0104] Depth plane positions 150 may be measured in diopters, a unit of refractive power equal to the reciprocal of focal length measured in meters. For example, in some embodiments, depth plane 1 may be 1 / 3 diopter away, depth plane 2 may be 0.3 diopters away, depth plane 3 may be 0.2 diopters away, depth plane 4 may be 0.15 diopters away, depth plane 5 may be 0.1 diopters away, and depth plane 6 may represent infinity (i.e., 0 diopters away). It should be understood that other embodiments may generate depth planes 150 at other distances / diopters. Thus, when generating virtual content at strategically placed depth planes 150, a user may perceive virtual objects in three dimensions. For example, a user may perceive a first virtual object as nearby when it is displayed in depth plane 1 while another virtual object appears at infinity in depth plane 6. Alternatively, the virtual object may be displayed first at depth plane 6, then at depth plane 5, and so on until the virtual object appears very close to the user. It should be understood that the above example is significantly simplified for illustrative purposes. In another embodiment, all six depth planes may be centered on a particular focal distance away from the user. For example, if the virtual content to be displayed is a coffee cup 0.5 meters away from the user, all six depth planes may be generated at various cross sections of the coffee cup, giving the user a highly granular 3-D view of the coffee cup.

[0105] In some embodiments, the VR, AR, and / or MR system may function as a multi-planar focusing system. In other words, all six LOEs may be illuminated simultaneously, with the light source rapidly transmitting image information to LOE 1, then LOE 2, then LOE 3, etc., so that images appearing to arise from six fixed depth planes are generated in rapid succession. For example, a portion of a desired image comprising an image of the sky at optical infinity may be injected at time 1, and an LOE that retains light collimation (e.g., depth plane 6 from FIG. 6) may be utilized. An image of a closer tree branch may then be injected at time 2, and an LOE configured to create an image that appears to arise from a depth plane 10 meters away (e.g., depth plane 5 from FIG. 6) may be utilized. An image of a pen may then be injected at time 3, and an LOE configured to create an image that appears to arise from a depth plane 1 meter away may be utilized. This type of paradigm can be repeated in a rapid time-sequential manner so that the user's eyes and brain (eg, visual cortex) perceive the inputs as all part of the same image.

[0106] The VR, AR, and / or MR system may project images (i.e., by diverging or converging light beams) that appear to originate from various locations along the Z axis (i.e., depth planes) and generate images for a 3-D experience / scenario. As used herein, light beam includes, but is not limited to, the directional projection of light energy (including visible and invisible light energy) emitted from a light source. Generating images that appear to originate from various depth planes matches the vergence and accommodation of the user's eyes for the images, minimizing or eliminating vergence-accommodation conflicts.

[0107] 7, an exemplary embodiment of an AR or MR system 700 (hereinafter referred to as "system 700") is illustrated. System 700 uses stacked light directing optical elements (hereinafter referred to as "LOEs 790"). System 700 generally includes one or more image generation processors 710, one or more light sources 720, one or more controller / display bridges (DBs) 730, one or more spatial light modulators (SLMs) 740, and one or more sets of stacked LOEs 790 that function as a multiple plane focus system. System 700 may also include an eye tracking subsystem 750.

[0108] The image generation processor 710 is configured to generate virtual content for display to a user. The image generation processor 710 may convert images or videos associated with the virtual content into a format that can be projected to a user in 3D. For example, when generating 3D content, the virtual content may need to be formatted so that portions of a particular image are displayed at a particular depth plane, while others are displayed at other depth planes. In one embodiment, the images may all be generated at a particular depth plane. In another embodiment, the image generation processor 710 may be programmed to provide slightly different images to the right and left eyes so that, when viewed together, the virtual content appears coherent and comfortable to the user's eyes.

[0109] Image generation processor 710 may further include memory 712, a GPU 714, a CPU 716, and other circuitry for image generation and processing. Image generation processor 710 may be programmed to present desired virtual content to a user of system 700. It should be understood that in some embodiments, image generation processor 710 may be stored within system 700. In other embodiments, image generation processor 710 and other circuitry may be stored within a belt pack coupled to system 700. In some embodiments, image generation processor 710 or one or more components thereof may be part of a local processing and data module (e.g., local processing and data module 130). As noted above, the local processing and data module 130 may be mounted in a variety of configurations, such as fixedly attached to the frame structure 102 (FIG. 2), fixedly attached to the helmet or hat 56 (FIG. 3), removably attached to the torso 58 of the end user 50 (FIG. 4), or removably attached to the waist 60 of the end user 50 in a belt-connected configuration (FIG. 5).

[0110] The image generation processor 710 is operably coupled to a light source 720 and one or more spatial light modulators 740, which project light associated with the desired virtual content. The light source 720 is compact and has high resolution. The light source 720 is operably coupled to the controller / DB 730. The light source 720 may include color-specific LEDs and lasers arranged in various geometric configurations. Alternatively, the light source 720 may include similar colored LEDs or lasers, each linked to a specific region of the display's field of view. In another embodiment, the light source 720 may include a wide-area emitter, such as an incandescent or fluorescent lamp, with a mask overlay for segmenting the emission area and position. Although the light source 720 is directly connected to the system 700 in FIG. 2B , the light source 720 may also be connected to the system 700 via an optical fiber (not shown). The system 700 may also include a concentrator (not shown) configured to collimate the light from the light source 720.

[0111] In various exemplary embodiments, the SLM 740 may be reflective (e.g., LCOS, FLCOS, DLP DMD, or MEMS mirror system), transmissive (e.g., LCD), or emissive (e.g., FSD or OLED). The type of SLM 740 (e.g., speed, size, etc.) can be selected to improve the creation of the 3D perception. While a DLP DMD operating at a higher refresh rate can be easily incorporated into a stationary system 700, a wearable system 700 may use a DLP of smaller size and power. The power of the DLP changes the way the 3D depth plane / focal plane is created. The image generation processor 710 is operably coupled to the SLM 740, which encodes light from the light source 720 with desired virtual content. The light from the light source 720 may be encoded with image information as it reflects from, emits from, or passes through the SLM 740.

[0112] Light from the SLM 740 is directed to the LOE 790 so that a light beam encoded by the SLM 740 with image data for one depth plane and / or color effectively propagates along a single LOE 790 for delivery to the user's eye. Each LOE 790 is configured to project an image or sub-image onto the user's retina that appears to originate from a desired depth plane or FOV angular location. The light sources 720 and LOE 790 can thus selectively project images (synchronously encoded by the SLM 740 under control of the controller / DB 730) that appear to originate from various depth planes or locations in space. By sequentially projecting images using each of the light sources 720 and LOE 790 at a sufficiently high frame rate (e.g., 360 Hz for six depth planes at an effective full-volume frame rate of 60 Hz), the system 700 can generate 3D images of virtual objects at various depth planes that appear to exist simultaneously in the 3D image.

[0113] Controller / DB 730 communicates with and is operably coupled to image generation processor 710, light source 720, and SLM 740, and coordinates the synchronous display of images by instructing SLM 740 to encode the light beam from light source 720 with appropriate image information from image generation processor 710. While the system includes image generation processor 710, controller / DB 730 may, in some embodiments, also perform at least some of the image generation processes, including, for example, processes in memory 712, GPU 714, and / or CPU 716. In some embodiments, controller / DB 730 may include one or more components shown in image generation processor 710, such as, for example, memory 712, GPU 714, and / or CPU 716.

[0114] System 700 also includes an optional eye tracking subsystem 750 configured to track the user's eyes and determine the user's focus. In one embodiment, system 700 is configured to illuminate a subset of LOEs 790 based on input from eye tracking subsystem 750 so that images are generated at a desired depth plane consistent with the user's focus / accommodation. For example, if the user's eyes are parallel to one another, system 700 may illuminate LOEs 790 configured to deliver collimated light to the user's eyes so that images appear to originate from optical infinity. In another example, if eye tracking subsystem 750 determines that the user's focus is one meter away, LOEs 790 configured to focus within approximately that range may instead be illuminated.

[0115] FIG. 8 diagrammatically depicts image generation components of a VR / AR / MR system 800, according to some embodiments. The VR / AR / MR system 800 includes a graphics processing unit (GPU) 802 having a display port (DP) source 804 coupled to a display bridge (DB) 806 having a DP sink 808 via a DP link 810. The GPU 802 may be located within a belt pack of the VR / AR / MR system 800, and the DB 806 may be located within a wearable component of the VR / AR / MR system 800, and the DP link 810 may be a fiber-optic-based link. In some embodiments, the DP link 810 may be a PCIE link or other, e.g., proprietary, high-speed link. In some embodiments, the GPU 802 and the DB 806 may both be within the belt pack of the VR / AR / MR system 800. In some embodiments, the GPU 802 and the DB 806 may both be within the wearable component of the VR / AR / MR system 800. In some embodiments, the DB 806 may include a GPU 802. The DB 806 is also coupled to the left and right display panels 812 and the left and right light field chips 814 via four MIPI links 816. In some embodiments, the MIPI links 816 may be DP links, PCIE links, or other, e.g., proprietary high-speed links. In these embodiments, the DB 806, the left and right display panels 812, and the left and right light field chips 814 may include appropriate ports for the links. The DB 806 may also be coupled to the wearable processor 818 via the MIPI links 816 and an I2C or SPI connector 820 (coupled to a MIPI / I2C 822 or SPI receiver 824 within the DB 806). The MIPI link 816 in the system 800 (i.e., between the DB 806 and the left and right display panels 812, the left and right light field chips 814, and the wearable processor 818) is coupled to a MIPI transmitter 832 and a MIPI receiver 824 in the transmitting and receiving components.The MIPI link 816 may include one or more data lanes (e.g., four video data lanes) and a clock lane. The DB 806 also includes memory 826 (e.g., stacked low-power DDR memory or embedded DRAM) and memory 828 (e.g., stacked flash memory). The left and right display panels 812 also include separate left and right decompression engines 830. In some embodiments, the DB 806 may include a video processor (e.g., a pixel engine) for pixel manipulation. The video processor customizes data received through a data stream (e.g., a video stream from the GPU 802) for the primary panels (e.g., the left and right panels 812 and / or the left and right light field chips 814).

[0116] FIG. 8A schematically depicts image generation components of a VR / AR / MR system 800′, according to some embodiments. The VR / AR / MR system 800′ depicted in FIG. 8A includes many of the components of the VR / AR / MR system 800 depicted in FIG. 8. The VR / AR / MR system 800′ includes a graphics processing unit (GPU) 802 having a DisplayPort (DP) source 804 coupled via a DP link 810 to a display bridge (DB) 806′ having a DP sink 808. The DB 806′ is also coupled to left and right display panels 812′ and left and right light field chips 814′ via four MIPI links 816. In some embodiments, the DB 806′ may be coupled to the left and right display panels 812′ and left and right light field chips 814′ via SPI and / or I2C connectors. Additionally, DB 806′ is further coupled to left and right LED / laser drivers 834 via left and right buses (e.g., I2C buses) 836. Left and right buses 836 also couple both DB 806′ and left and right LED / laser drivers 834 to left and right display panels 812′. DB 806′ may also be coupled to wearable processor 818′ via MIPI link 816, I2C connector 820, and / or SPI connector 838, which may be coupled to MIPI receiver 824, I2C receiver 822, or SPI receiver 840 within DB 806′. DB 806′ also includes memory 826 (e.g., built-in SRAM) and memory 828 (e.g., stacked quad SPI flash memory).

[0117] The MIPI link 816 described above may have bidirectional capabilities (e.g., to return read data, acknowledgments, and / or error information from the left and right display panels 812, 812′ to the DB 806, 806′). Lane 0 of the MIPI link 816 may be used for transmission to the DB 806, 806′, while the other lanes may be unidirectional. The MIPI link 816 may be used for transmission to the DB 806, 806′ during a low-power transmission mode, for the DB 806, 806′ to initiate general-purpose readings from the left and right display panels 812, 812′.

[0118] FIG. 9 diagrammatically depicts a DB 900 for use with a VR / AR / MR system, according to some embodiments. The DB 900 includes a DP receiver 902 configured to receive one or more streams of image data. The DP receiver 902 is communicatively coupled to a DSC decoder 904, a DDR controller 906, a pixel engine 908, output FIFO primary and secondary streams 910, and a pixel distribution primary and secondary panel (“pixel distribution module”) 912, which is in turn communicatively coupled to four MIPI channels 914. The DB 900 also includes a MIPI receiver 916, a memory 918 (e.g., a low-power DDR stacked die), a CPU module 920, and an RGB / grayscale segmentation block 922, all of which are communicatively coupled to the DDR controller 906. The RGB / grayscale segmentation block 922 receives data from the DSC decoder 904 and sends output to memory 918′, as described herein. The DB 900 is configured to process two video streams. The first video stream 924 is processed by the DSC decoder 904 and the RGB / grayscale segmentation block 922 and then sent to the DDR controller 906 for writing to memory 918. The second video stream 926 is sent directly from the DP receiver 902 to the DDR controller 906 for writing to memory 918.

[0119] FIG. 9A schematically depicts a DB 900′ for use with a VR / AR / MR system, according to some embodiments. The DB 900′ depicted in FIG. 9A includes many of the components of the DB 900 depicted in FIG. 9. The DB 900′ includes a DP receiver 902 configured to receive one or more streams of image data. The DP receiver 902 is communicatively coupled to a DSC decoder 904, a pixel engine 908, output FIFO primary and secondary streams 910, and pixel distribution primary and secondary panels 912, which are in turn communicatively coupled to four MIPI channels 914. The DB 900′ also includes an MIPI receiver 916 and an RGB / grayscale segmentation block 922, both of which are coupled to memory 918′ (e.g., SRAM). The RGB / grayscale segmentation block 922 receives data from the DSC decoder 904 and sends output to the memory 918′, as described herein. The DB 900' further includes a splash screen decompressor 928 coupled to the memory 918'. The DB 900' is configured to process two video streams. The first video stream 924 is processed by the DSC decoder 904 and the RGB / grayscale segmentation block 922 and then written to the memory 918'. The second video stream 926 is sent directly from the DP receiver 902 and written to the memory 918'.

[0120] FIG. 10 diagrammatically depicts a pixel engine 1000 for use with a DB, according to some embodiments. The pixel engine 1000 includes a core 1002 that executes code for a color field and generates pixels for display based on input parameters. The pixel engine 1000 reads information about the color field from memory (e.g., low-power DDR) and generates pixels (e.g., one output line at a time). The core 1002 is programmable and may include SRAM and instruction RAM. The core 1002 may also have an AXI communication channel 1004 (e.g., for communicating with memory), a pair of FIFO outputs 1006 (e.g., for communicating with a display and light field chip), a secondary data packet / auxiliary data first-in-first-out (SDP / AUXFIFO) input 1008 (e.g., for communicating with a DP source), and dual banks of tightly coupled SRAM 1010 to contain pixel data for processing. In some embodiments, pixel engine 1000 may be responsible for compensating for optical distortions associated with optics and projectors, persistence warping reprojection, frame rate enhancement, dynamic occlusion, segmented foveation blending, pixelated dimming control, pixel chromatic aberration correction, partial display support, custom darkening and partial darkening modes, content unfolding, and implementing many other AR / VR algorithms using the vector engine, e.g., with custom instructions. In some embodiments, pixel engine 1000 computes head pose updates directly from sensor data provided by a sensor or wearable processor 818 communicatively coupled to a DB (e.g., DB 806).

[0121] FIG. 10A diagrammatically depicts a pixel engine 1000′ for use with a DB, according to some embodiments. The pixel engine 1000′ includes a core 1002′ that executes code for a color field and generates pixels for display based on input parameters. The pixel engine 1000′ reads information about the color field from memory (e.g., low-power DDR or SRAM) and generates pixels (e.g., one output line at a time). The core 1002′ is programmable and may include SRAM and instruction RAM. The core 1002′ may also have an AXI communication channel 1004′ (e.g., for communicating with memory), a pair of FIFO outputs 1006′ (e.g., for communicating with a display and light field chip), a secondary data packet / auxiliary data first-in-first-out (SDP / AUXFIFO) input 1008′ (e.g., for communicating with a DP source), and dual banks of tightly coupled SRAM 1010′ to contain pixel data for processing. The tightly coupled SRAM 1010' reduces the need to communicate with other memories in the system. Additionally, the core 1002' may have connections 1012 to an SRAM controller and an HSYNC / VSYNC / scan line interrupt 1014 (e.g., to receive input from the pixel segmentation block). Additionally, the core 1002' may include an APB register bank, an APB / AXI slave port 1018, and an SPI / I2C master port. The APB / AXI slave port 1018 may be used to copy firmware from flash memory to the DB.

[0122] In some embodiments, pixel engine 1000' may be responsible for compensating for optical distortions associated with optics and projectors, persistence warping reprojection, frame rate enhancement, dynamic occlusion, segmented foveation blending, pixelated dimming control, pixel chromatic aberration correction, partial display support, custom darkening and partial darkening modes, content unfolding, and implementing many other AR / VR algorithms using the vector engine, e.g., with custom instructions. In some embodiments, pixel engine 1000' computes head pose updates directly from sensor data provided by a sensor or wearable processor 818 communicatively coupled to a DB (e.g., DB 806).

[0123] FIG. 11 diagrammatically depicts a display panel 1100 for use with a VR / AR / MR system, according to some embodiments. The display panel 1100 includes a MIPI receiver 1102 communicatively coupled to a built-in line decoder 1104, partial screen refresh logic 1106, various buffers 1108, various gamma correction modules 1110, and display memory 1112. The MIPI receiver 1102 may be configured to operate with a MIPI DSI-2 interface, a DP link interface, and / or other, e.g., proprietary serial links, with one or more data lanes (e.g., four video data lanes) and one clock lane. The MIPI receiver 1102 may have various features, including burst mode with synchronization events, clock / lane deskewing, and manual configuration of resolution / timing via registers. The built-in line decoder 1104 may be configured to read the first line of frame data and extract the data embedded therein. The built-in data may define the operating mode and format for the frame. The partial screen refresh logic 1104 may be configured to operate in two modes: blank screen and partial display (both described below). The buffer 1108 may have a maximum latency of one color field. The gamma correction module 1110 may support gamma tables and convert 8-bit input video to 10-bit output. The display panel 1100 may include a table / offset synchronization module and a sequencer table. (Image data format) (Segmented Color Sequential Format)

[0124] Figure 12 diagrammatically depicts two input data formats that may be supported by a DP port (see, for example, DP SRC 804 and DP sink 808 in Figures 8 and 8A). The top data format 1200 is RGB 24-bit color (8 bits per field), and the bottom data format 1202 is RGB 30-bit color (10 bits per field). The input data may be compressed with Display Stream Compression (DSC). The DP port may also support a sequential grayscale data format (described below).

[0125] 13 schematically depicts a color image data format for use with a VR / AR / MR system, according to some embodiments. In this color image data format, color image data (e.g., 24-bit or 30-bit RGB data), which is typically organized into three primary color fields 1302, 1304, 1306, is reorganized into three images 1312, 1314, 1316. Each of the three images 1312, 1314, 1316 is divided into thirds, with each third containing a partial color field (e.g., 1312R, 1312G, 1312B). (Color sequential grayscale format)

[0126] Color image data is generated within the GPU and reorganized into three images 1312, 1314, and 1316. The three images 1312, 1314, and 1316 can be sent from the GPU to the DB via the DP. Reorganizing the color image data into three images 1312, 1314, and 1316 results in portions of each color field (e.g., partial color fields 1312R, 1312G, and 1312B) arriving at the DB earlier in the image pipeline. This earlier access to the image data may allow the DB to begin operating on the image data earlier in the image pipeline without the reorganized color image data.

[0127] 14 diagrammatically depicts a color image 1402 being reorganized into three partial color fields. In addition, the three partial color fields are converted into three sequential grayscale images 1422, 1424, and 1426. The positions of the three sequential grayscale images 1422, 1424, and 1426 are used to encode the color of each partial field. Converting the color fields to grayscale may reduce the amount of data transmitted from the GPU to the DB. The conversion to grayscale may occur in the GPU or the DB. (Color Segmentation)

[0128] FIG. 15 diagrammatically depicts color segmentation of color image data in a DB for use with a VR / AR / MR system, for example, where color image data is sent from a GPU to a DB for further processing, according to some embodiments. When the DB receives a register indicating a color sequential grayscale format, the DB converts the packed color image data to color sequential grayscale data as described herein. The DB receives color image data 1502 from the GPU through a DP. The DB then optionally segments the color image data 1502 into three fields 1512, 1514, and 1516. The three fields 1512, 1514, and 1516 are stored in memory, for example, a DDR double-buffered memory, for further processing into the color sequential grayscale format. This segmentation can occur in real time as the color image data stream is received. The register receiving the color image data 1502 stream may also indicate an address in memory (e.g., a low-power DDR) where the image data will be stored.

[0129] 15A diagrammatically depicts color segmentation of color image data in a DB for use with a VR / AR / MR system, for example, where the color image data is sent from a GPU to a DB for further processing, according to some embodiments. When the DB receives a register indicating a color sequential grayscale format, the DB converts the packed color image data to color sequential grayscale data as described herein. The DB receives color image data 1502 from the GPU through a DP. The DB or an optional segmentation hardware block 1520 then segments the color image data 1502 into three fields 1522, 1524, and 1526. The three fields 1522, 1524, and 1526 are stored in nine output buffers (BUF0-BUF8). The subset of buffers in which the three fields 1522, 1524, 1526 are stored varies between even frames (i.e., zero, two, four, six, ...) and odd frames (i.e., one, three, five, seven, ...). Incoming messages from the segmentation block 1520 or DB indicate the currently active buffer state for the segmentation block output. The segmentation block 1520 separates the color image data 1502 into three primary grayscale fields (i.e., red, green, blue). This segmentation can occur in real time as the color image data 1502 stream is received.

[0130] When segmentation block 1520 is disabled, color image data 1502 is copied to nine output buffers (BUF0-BUF8) in RGB packing mode using a numeric order that matches the time of arrival, as shown in Figure 15B. In this mode, the nine output buffers (BUF0-BUF8) are in order because there is no segmentation.

[0131] When the color image data 1502 is in raw 8-bit form, the segmentation block 1520 may convert the pixel to raw 10-bit on the fly by adding two extra zero bits in the least significant bits per color component before storing the segment data. Even if segmentation is not enabled, the color image data 1502 may be stored in raw 10-bit format for system consistency.

[0132] FIG. 15C schematically depicts an "RGB30" data format for storing color image data 1502. FIG. 15D schematically depicts an input display panel resolution of 1,440 pixels by 1,440 pixels stored in SRAM as 128-bit aligned data. Each left and right segmented color includes 1,800 bytes, with 8 bytes of padding. Using the data format in FIG. 15D, the segmentation block 1520 can assume that the starting address of every nine buffers will always be 128-bit aligned. FIG. 15E schematically depicts an input display panel resolution of 512 pixels by 512 pixels stored in SRAM as 128-bit aligned data. Each left and right segmented color includes 640 bytes, with 1,168 bytes of padding. Thus, color image data 1502 can still be stored in SRAM as 128-bit aligned data. Figure 15F diagrammatically depicts an input display panel resolution of 512 pixels by 512 pixels for a special image (e.g., a splash screen) stored in SRAM as 128-bit coherent data. In this particular embodiment, the left panel has 512 colored pixels, while the right panel has 512 pixels that are all set to black. As shown in Figure 15E, each left and right segmented "color" comprises 640 bytes, with 1,168 bytes of padding. Thus, color image data 1502 can still be stored in SRAM as 128-bit coherent data. While 1,440 pixel by 1,440 pixel and 512 pixel by 512 pixel resolutions are described, similar data formats can also be used to store color image data 1502 with different resolutions.

[0133] Figure 15G graphically depicts the data structure after color segmentation of the frame. The data structure shown in Figure 15G does not include padding, and therefore the data structure has a fixed destination pitch (i.e., the byte address distance from one row to the next has a fixed number of bytes).

[0134] Each color field requires three separate buffers, with each buffer holding one-third of the color field image. For display resolutions with a number of rows that is not evenly divisible by three, the buffers may contain one or two additional rows so that the number of buffers is evenly divisible by three. The extra rows / buffers may be used to store image metadata. (Multi-stream mode)

[0135] As shown in FIGS. 9 and 9A above, the DB 900, 900′ may support a multi-stream mode, in which the image data includes two streams 924, 926. In some embodiments, video stream 1 924 may include image data for display, and video stream 2 926 may not include image data for display, but rather may include configuration data, for example, for one or more light field chips (e.g., the left and right light field chips 814 in FIGS. 8 and 8A). In some embodiments, video stream 2 926 may also include light field panel data that is separated by the pixel engine 908 for use by the light field chips. As shown in FIGS. 9 and 9A, this configuration data may be communicated to and used by the DDR controller 906, memory 918, 918′, pixel engine 908, and pixel distribution module 912. In some embodiments, the configuration data may include panel resolution, light field controller data, warping data, distortion data, occlusion data, foveation region data, bridge control data, and the like.

[0136] Video stream 1 924 and video stream 2 926 may be synchronized. Video stream 2 926 may support partial lines per frame. The partial lines facilitate defining the timing of video stream 1 924 and video stream 2 926. The partial lines may be placed at the end of the blanking section. The DB may be configured to generate a MIPI output for video stream 2 926 and facilitate spacing within memories 918, 918′. In full darkening mode (described herein), video stream 2 926 will not be transmitted by the DP source. In partial darkening mode (described herein), the DB may be configured to generate an output for video stream 1 924 and keep the link active until video stream 2 926 is transmitted. Video stream 2 926 may shift line phase once per V blanking to facilitate synchronization of the two streams 924, 926.

[0137] FIG. 16 diagrammatically depicts a secondary display stream 1600 (e.g., video stream 2 926) and its flow through a DB. The secondary display stream 1600 includes light field processor data 1602 for controlling left and right light field processors and other custom data 1604. The light field processor data 1602 and custom data 1604 are separated by programmable rows 1606. The DB splits the data separated by programmable rows 1606 so that the custom data 1604 is stored in a memory, for example, memory 1608 (e.g., low-power DDR memory), and the light field processor data 1602 is sent through a pixel engine 1610 and output FIFO memory 1612 to pixel distribution 1614 (e.g., pixel distribution module 912). In pixel distribution 1614, the light field processor data 1602 is split into left and right channels before being sent over a MIPI link for communication to a display panel (not shown). If custom data 1604 is present for a particular frame, the output MIPI timing may be adjusted to accommodate the custom data 1604, thereby reducing the resolution of the light field processor data 1602.

[0138] There may be no relationship between the secondary display stream 1600 input resolution and the corresponding MIPI output resolution. The locations in the memories 918, 918' corresponding to the secondary display stream 1600 may be double buffered so that information about new frames can be written without erasing information about previous frames. The double buffering of the memories 918, 918' for the secondary display stream 1600 may be selectively activated.

[0139] In some embodiments, a secondary display stream (e.g., video stream 2) 926) may be larger than necessary. As a result, the DB may compress the secondary display stream for transmission to the display. FIG. 16A diagrammatically depicts a secondary display stream 1600′ (e.g., video stream 2 926) and its flow through the DB. The secondary display stream 1600′ includes light field processor data 1602′ with custom data 1604′. The light field processor data 1602′ may be stored in memory 1608′ (e.g., SRAM), from which the light field processor data 1602′ is transmitted through a pixel engine 1610′, an output FIFO memory 1612′, and a pixel distribution module 1614′ (e.g., pixel distribution module 912). The pixel engine 1610′ may copy only sub-pixels in the light field processor data 1602′ from memory 1608′ according to the output. Removing the data splitting operation in secondary display stream 1600 from secondary display stream 1600' depicted in Figure 16A simplifies the hardware design but adds complexity to pixel engine 1610' firmware design. The final resolution for the MIPI output of video stream 2 926 does not have to be fixed. (Secondary Data Packet (SDP))

[0140] In addition to image data for display, video stream 1 also includes one or more SDP messages from the GPU to the DB. In some embodiments using the VESA packet format, the SDP messages may be encoded within the video image data during vertical blanking before any video image data for display is communicated. The SDP messages may also be encoded within the video image data during horizontal blanking. The SDP messages are received by the DB and stored in a buffer (e.g., for access by the pixel engine). The SDP messages may include attitude data updates.

[0141] FIG. 17 diagrammatically depicts a large chained VSC_EXT_VESA packet 1700 having multiple SDP messages encoded therein, according to some embodiments. Up to 2 KB of an SDP message may be contained in the vertical blanking segment 1702 of the packet 1700 due to its relatively large size. On the other hand, the relatively smaller size of the horizontal blanking segment 1704 facilitates only smaller SDP messages. FIG. 17 depicts six horizontal blanking packets that can be broken down into three chained packets. The use of chained packets allows for slightly larger SDP messages. In embodiments with smaller SDP messages, chaining is not required.

[0142] 17A diagrammatically depicts a VSC_EXT_VESA SDP message 1750 associated with vertical blanking. The VSC_EXT_VESA SDP message 1750 may be included only in video stream 1 924. Thus, the VSC_EXT_VESA SDP message 1750 may include information about both video streams 1 and 2. In some embodiments, the VSC_EXT_VESA SDP message 1750 includes a header 1752 (20 bytes), a main DP control block 1754 (5 bytes), partial MSA information (18 bytes each) for video streams 1 and 2 1756, 1758, and a proprietary data block 1760 (variable size, e.g., 173 bytes).

[0143] 17B diagrammatically depicts a horizontal blanking SDP message 1770 associated with horizontal blanking. In some embodiments, the horizontal blanking SDP message 1770 includes a header 1772 (4 bytes), a timestamp 1774 (4 bytes), and a proprietary data block 1776 (28 bytes). (SDP and AUX message FIFO memory)

[0144] Figure 18 diagrammatically depicts the flow of data from SDP messages and auxiliary (AUX) channel messages within a DB, according to some embodiments. Because the DP may be shut down to save power (described below), various configuration data may be communicated over the AUX channel. Configuration data may be communicated using register writes over the AUX channel. In some embodiments, the AUX channel is used to communicate configuration data along with the DP.

[0145] 18 depicts an SDP message 1802 from a DP receiver and configuration data from an AUX channel 1804. The SDP and AUX messages 1802, 1804 are stored on a FIFO memory 1806. From the FIFO memory 1806, the SDP and AUX messages 1802, 1804 can be sent to and used by a pixel engine 1808. (Output video stream compression)

[0146] Configuration data for the display panel is most efficiently communicated directly from the controller to the display panel. In VR / AR / MR systems where a DB is interposed between the CPU and the display panel, the configuration data can be sent to the display panel as a built-in signal within the video stream.

[0147] 19 diagrammatically depicts a video stream 1900 with pixel data 1902 interleaved with a built-in signal 1904, which contains configuration data. The built-in signal 1904 can be located anywhere within the video stream 1900 (e.g., beginning, middle, end). The built-in signal 1904 can also be a full or partial row.

[0148] The DB can receive configuration data from AUX channel register writes, SDP metadata, and the custom data section of video stream 2. The pixel engine will receive the configuration data and generate an output MIPI stream that includes built-in signaling with the configuration data associated with the specific display panel. The output timing of the MIPI stream may be extended to match the built-in signaling. (Fully darkened low power mode)

[0149] In some embodiments, for certain VR / AR / MR user poses / viewpoints, no virtual image data will be present. Therefore, the GPU output will be sent to black / full black. When the GPU detects a full black frame, Video Stream 1 Shutting down the image pipeline for 924 will save system power. Sources of power savings include, but are not limited to, the DP link, DP source processor, DP source memory, DP optical link, GPU compression, DB decompression, DB color segmentation, DB memory writes, pixel engine memory reads, pixel processing, MIPI transmitter, MIPI receiver, display panel memory reads and writes, etc. Turning off Video Stream 1 924 and its display panel will conserve power. In some embodiments, the light field chip will continue to operate from previously received data.

[0150] When the GPU detects a full dark frame, it notifies the DP source MCU (e.g., via an "STP" SDP message or a k-code power-down sequence on the main DP link). In some embodiments, the GPU compositor creates and writes a vertical blank SDP message to initiate full dark mode. The DP source MCU then bypasses memory transfers of the full dark frame to the DP source pipeline and places most of the DP source pipeline components (e.g., the DSC encoder) in a low-power mode. The DP source MCU disables various registers and shuts down downstream components of the imaging pipeline, such as the display bridge and display.

[0151] The GPU may initiate full dark mode by sending an SDP message to the DB during vertical blanking. The DP receiver decodes the SDP message and places itself in full dark low-power mode. The DP receiver also sends a message to place the pixel engine in full dark low-power mode. Various other components of the imaging pipeline, including the MIPI data lanes, which will be placed in LP11 mode, will also be placed in full dark low-power mode. The DP source CPU will shut down the DP physical components. The display may be shut down by setting the number of active lines to zero. The display may transition to black / self-refresh mode. After receiving the vertical blank SDP message, the DP sink bridge may not consider any data sent by the source until it receives a wake-up signal. Various components of the image pipeline are shut down asynchronously as the previous undarkened image is processed through the pipeline and displayed. In some embodiments, the display pipeline, including the DP port, will remain in shutdown mode as long as non-frame data is needed.

[0152] When the GPU detects a non-full dark frame after a full dark frame, the GPU initiates a wake-up sequence. In some embodiments, the DP source MCU may power up the DP transmitter and receiver. The GPU compositor may create and write a vertical blank SDP message to initiate either partial darkening or normal mode. The DP source MCU may then send a wake-up message over the AUX channel. The display may wake up by setting the number of active lines to a non-zero value. Optional fast link training may occur. In some embodiments, fast link training may be stored on a DB, and the stored fast link training may be used for fast wake-up.

[0153] 32 diagrammatically depicts MIPI timing of a display system transitioning into and sustaining full dark mode 3200, according to some embodiments. The system transitions into full dark mode at 3210. In some embodiments, the display panel actually self-refreshes but displays only black pixels. Therefore, a significant portion of the power savings from full dark mode comes from shutting down the MIPI receiver, memory, DB, light source, and fiber-optic connector.

[0154] Figure 33 diagrammatically depicts a display system's transition to, duration of, and exit from full dark mode 3300, in accordance with some embodiments. Figure 33 depicts MIPI data 3310 from the DB and display (e.g., LCOS) output video 3320 over time. The MIPI data 3310 transitions from normal mode to full dark in frame 3 3312. While the video data is full dark (i.e., blanks or zeros), the MIPI data 3310 includes one frame of full dark 3312, which includes a built-in control line to place the system in full dark mode. In response to receiving the built-in control line, the display output video 3320 displays a full dark frame in frame 3 3322. The system then persists in full dark mode for a variable number of frames as required by the video data. In frame N 3314 of the MIPI data 3310, the MIPI data returns to normal mode, thereby waking up the system. However, because the display is no longer synchronized with the DB MIPI output, the display does not consider the first frame in normal mode by displaying the final black frame 3324. Starting with frame N+1 3326, the display output video wakes up, synchronizes, and operates in normal mode.

[0155] Figure 34 diagrammatically depicts the transition, duration, and exit of a display system into full dark mode for panel self-refresh 3400, according to some embodiments. Figure 34 depicts MIPI data 3410 from the DB and display (e.g., LCOS) output video 3420 over time. The MIPI data 3410 is At 3412, a panel self-refresh begins. Frame 3 3412 MIPI data 3410 includes an indicator 3430 for a panel self-refresh. In response to receiving the panel self-refresh indicator, the panel stores MIPI data 3410 in one or more buffers therein. Display output video 3420 shows frame 3422, which corresponds to frame 3 3412 of MIPI data 3410, until the panel self-refresh is complete. During the panel self-refresh, the SRAM buffer can be shut down as video data is stored in a buffer within the display panel. At frame N 3414 of MIPI data 3410, the MIPI data returns to normal mode, thereby waking up the system. However, because the display is no longer synchronized with the DB MIPI output, the display does not consider the first frame in normal mode by displaying the last copy of frame 3 3424. Starting with frame N+1 3426, the display output video wakes up, synchronizes, and operates in normal mode. The display panel self-refresh may occur during the display of a splash screen, during which the content is mostly stationary. (Partial darkening low power mode)

[0156] The trigger for the partial darkening low power mode is the presence of a threshold amount of non-black content within a particular VR / AR / MR user posture / field of view. FIG. 20 depicts a first partial darkening low power mode in which content is placed only on the top 2002 of the screen 2000. Examples of such content include a status bar, battery charge level, volume settings, etc. The DP source in this mode will initiate a power-down sequence (similar to that described above for the full darkening low power mode) after a predetermined number of rows specified in the SDP message. The SDP message may indicate the first partial darkening low power mode and the number of active rows in the frame. This first partial darkening low power mode can conserve power by reducing the amount of DB writes to memory, pixel engine fetches from memory, pixel engine processing, MIPI output, etc.

[0157] Figure 21 depicts a second partial dimming low power mode in which content is located at the top 2102 and center 2104 of screen 2100. In this second partial dimming low power mode, the GPU may reorganize the output data to minimize the number of active rows, as shown in Figure 22. In Figure 22, the content in the center 2104 of screen 2100 in Figure 21 has been moved as displaced image data 2204' adjacent to the content at the top 2202 of screen 2200. Reorganizing the image data reduces the number of active rows (depicted in Figure 22A), thereby reducing the overall power consumption of the display pipeline components.

[0158] After the top image data 2202 and the displaced image data 2204' progress through the image pipeline, the components of the image pipeline are placed in a partial darkening low power mode as described herein to conserve power. The pixel engine in the DB performs segmentation of the image data and remaps the displaced image data 2204' using the data structure from the SDP to recreate the source image 2100 shown in Figure 21. Reducing the number of active rows (see Figure 22A) facilitates earlier shutdown of the DP link.

[0159] Figure 23 depicts a third partial dimming low power mode in which content is placed only in the center 2304 of the screen 2300. As in the modes shown in Figures 21 and 22, the GPU may rearrange the center data 2304 by moving it as displaced image data 2404' to the top of the screen 2400, as shown in Figures 24 and 24A.

[0160] After the displaced image data 2404' advances through the image pipeline, the components of the image pipeline are placed in a partial darkening low power mode as described herein to conserve power. The pixel engine in the DB performs segmentation of the image data and uses configuration data from the SDP to remap the displaced image data 2404' to recreate the source image 2300 shown in Figure 23.

[0161] In the three partial dimming low power modes depicted in Figures 20-24 and described above, the SDP message informs the DP receiver of both modes in terms of the number of active rows. This allows the DP receiver to shut down after the active rows have been communicated from the GPU to the DB. In embodiments where compression is enabled, compression may optionally be disabled depending on the number of active rows. For example, a minimum set of rows may be required to enable compression, or compression may simply be disabled by the VESA to improve power efficiency. A varying number of DSC slices may be enabled at any one time.

[0162] In other embodiments, the horizontal packing depicted in Figures 22-24 and described above may be replaced or combined with vertical packing. In some embodiments, although Figures 22-24 illustrate some modes, one or more distinct regions on a single stream may be compacted to reduce the data rate. In some embodiments, the number of distinct regions may be limited to make implementation more practical based on other constraints.

[0163] 24B schematically depicts MIPI timing of a display system transitioning into and sustaining partial dimming mode 2450, according to some embodiments. The system transitions into partial dimming mode at 2452 after non-black subject matter in a frame is displayed. In some embodiments, the display panel actually self-refreshes but displays only black pixels. Thus, a significant portion of the power savings from full dimming mode comes from shutting down the MIPI receiver, memory, DB, light source, and fiber-optic connector. Partial dimming mode 2450 may only affect video stream 1 924. Therefore, video stream 2 926 may not be affected by partial dimming mode 2450. (Custom packing for partial screen refresh mode)

[0164] In AR / MR modes, there are potentially many black pixels that do not represent modifications to real-world lighting. However, the black pixels and corresponding non-black pixels may be arranged in complex patterns that make simple horizontal or vertical packing unwieldy.

[0165] In custom packing for partial screen refresh mode, the GPU may reorganize the image data so that only regions / portions / sections / tiles (e.g., square-shaped tiles) that have changed from the previous frame to the currently rendered frame are sent from the GPU to the DB via the DP. The SDP message may include information indicating that custom packing for partial screen refresh mode has been enabled, the locations of the various changed tiles within the FOV, etc. The DB can use the locations of the various changed tiles within the FOV to store the various changed tiles in the appropriate locations within the output buffer.

[0166] As shown in FIG. 25, the GPU will pack the transformed tiles so that they form the minimum number of rows of image data. Source frame 2502 shows a virtual image containing various black and non-black pixels. Sending source frame 2502 would require a certain number of rows (e.g., 960) to be sent from the GPU to the DB. Folded frame 2504 shows that the tiles containing the transformed image data are now packed onto the top of folded frame 2504. Sending folded frame 2504 would require a smaller number of rows (e.g., 160) to be sent from the GPU to the DP. Compressed frame 2506 shows that the tiles at the top of folded frame 2504 have been compressed into an even fewer number of rows. Sending compressed frame 2506 would require an even smaller number of rows (e.g., 80) to be sent from the GPU to the DP. The folding of the tiles and compression of the resulting folded frame 2504 to form compressed frame 2506 allows various components of the image pipeline to be shut down earlier in the frame transfer / rendering interval, increasing power savings (as described herein).

[0167] After the compressed frame 2506 has been transmitted and processed as described herein, the GPU may initiate a shutdown of various image pipeline components by sending a k-code power-down sequence or a DPCD power state command on the main DP link. The shutdown may include a power-down of the DP. The power-down of image pipeline components, including the DP link, may be signaled by the GPU over the AUX channel. During wake-up, the GPU powers up the DP link driver and optionally trains the link using a fast link training pattern.

[0168] The GPU will instruct image pipeline components to shut down only if the number of rows to be transmitted is less than a threshold, based on wake-up time and a cost-benefit analysis of power savings from component shutdown versus power usage during wake-up. A cyclic redundancy check (CRC) may or may not be performed at the end of a frame transmitted / rendered using custom packing for partial screen refresh mode. The CRC information may be sent over the AUX channel. (Custom partial display mode / Custom darkening mode)

[0169] Custom packing for partial screen refresh mode has two modes of operation: In custom partial display mode, the changed tiles are sent from the GPU to the DB and displayed by the display panel.

[0170] In custom darkening mode, the SDP message contains a specific background color (black in most cases). Any areas of the field of view that do not belong to changing tiles with content are set to the specific background color. Custom darkening mode can be enabled when the entire image is changing, but only a small number of tiles contain content. In other words, many tiles that previously contained image data are being converted to black in the current frame. By setting areas without content to black initially, only tiles with actual content will be sent from the GPU to the DB, thereby saving power in the image pipe with earlier component shutdown. The SDP message may also contain the number of darkened or background tiles. (Custom packing for mixing and scaling)

[0171] If a multi-stream implementation is not supported, a custom packing mode may send tiles and blend them with tiles from the previous frame (rather than copying over them). Square or radio blending masses may be pre-programmed on the DB to blend new and previous tiles for output.

[0172] The tiles may also be scaled before being copied to the output frame buffer. The SDP message may include a scaling factor to use in scaling the tiles. In some embodiments, this approach may be used for foveation implementations. (Custom pixel manipulation and rate conversion)

[0173] The pixel engine can warp the image data to minimize artifacts resulting from user pose changes (e.g., rapid head rotation) after the image data is rendered but before an image based on the image data is displayed to the user.

[0174] In one embodiment of custom pixel warping without rate conversion, the GPU sends an SDP to the DB at the start of a frame. The SDP contains information such as compression enable / disable, custom packing enable / disable, custom grayscale mode enable / disable, and packet pose. The GPU also sends image data to the DB. If compression is enabled, the DB decompresses the image data using the appropriate algorithm. If the image data requires color segmentation, the DB performs the color segmentation and stores the results in memory. The pixel engine receives configuration data via an AUX register write, an SDP message, or as part of the second video stream. In some embodiments, the configuration data may include updated head pose information. In some embodiments, the DB may receive updated information from the wearable processor (e.g., IMU data) for the DB to generate head pose updates. In some embodiments, the IMU may be connected to the DB bridge SPI port, and the DB may access raw IMU samples directly from the IMU. The configuration data may be stored in a message FIFO memory. The pixel engine reads the field data from the low-power DDR, performs the necessary conversions, and provides the output pixels to drive the display panel. The DB then repeats the steps described above for each supported color field. After all color fields have been processed, the DB proceeds to the next frame.

[0175] FIG. 26A illustrates inputs to and outputs from a DB (i.e., relating to image data in video stream 1 924) when rate conversion is not enabled, according to some embodiments. Because the input image data was not formatted in color sequential mode, the DB must receive the first complete frame of image data (“DP Input Frame 0”) 2602 before segmenting the image data 2602 and sending the first output frame of image data “DP Output Frame 0” 2604 to the display panel. Similarly, the DB must receive the entire second complete frame of image data 2606 before it can send the second output frame 2608 to the display panel. This results in a latency of 8.33 ms between the receipt of each image data and the sending of the corresponding output frame to the display panel. This delay is non-ideal.

[0176] Figure 26B shows inputs to and outputs from the DB (i.e., relating to image data in video stream 1 924) when rate conversion is not enabled, according to some embodiments. Because the input image data was formatted in color sequential mode, the first color field (e.g., red) image data 2602R is received well before the end of the image frame. Therefore, the output frame image data corresponding to the first color field 2604R can be sent to the display panel earlier in the image pipeline. Using grayscale color segmentation mode, latency can be reduced from 8.33 ms to 2.78 ms. (Rate Upward Conversion)

[0177] FIG. 26C shows inputs to and outputs from the DB (i.e., relating to image data in video stream 1 924) when rate conversion is enabled, according to some embodiments. The DB receives image data at 60 Hz but outputs image data at 120 Hz. In some embodiments, the DB receives image data at 45 Hz but outputs image data at 90 Hz. Regardless of the image data frequency, the output image data doubles the frame rate. In the embodiment depicted in FIG. 26C, the input image data was not formatted in color sequential mode, as in the embodiment depicted in FIG. 26A. Thus, there is a latency of 8.33 ms between the DB receiving 2602 a frame of image data and the DB transmitting 2604 the frame of image data. The difference in this rate upconversion image pipeline is that the output frame 2604 of image data contains each color field twice as fast, doubling the effective refresh rate from 60 Hz to 120 Hz, and the pixel engine will update the head pose for each field to warp the incoming image and have zero latency effects for each of the two output frames. The pipeline depicted in Figure 26C also includes a DP off segment 2610 between steps of sequentially receiving frames of image data to save power. Running the GPU at a lower refresh rate (e.g., 60 Hz) also saves power. Compression can be used to reduce latency. (Rate upconversion using warping)

[0178] Figure 26D shows inputs to and outputs from the DB (i.e., related to image data in video stream 1 924) when rate conversion and warping are enabled, according to some embodiments. The image pipeline depicted in Figure 26D is similar to the image pipeline depicted in Figure 26C. The difference is that Figure 26D also shows SDP and AUX messages being sent from the GPU to the pixel engine in the DB. The SDP and AUX messages contain current attitude information that can be used to warp rendered image data for more accurate image rendering. The current attitude information may be provided by an IMU running at 1,000 Hz. The DB calculates an attitude delta from the packet attitude and the current attitude for use in warping. Minimizing the time between receiving the current attitude information and warping the rendered image data increases accuracy. The AUX channel is used instead of the SDP channel when the DP is powered down. The warping depicted in Figure 26D can also account for physical distortions of various components in the optical stack using distortion information in the SDP / AUX messages.

[0179] In all of the above image pipelines, parallel input reception, processing, and output transmission minimizes input-to-display latency and delay within the image pipeline.

[0180] Composite video stream 2 926 may be a decompressed RGB888 packed data stream, so rate conversion for video stream 2 926 may be as simple as duplicating the data stream for two frame rate upconversions. (Partial Darkening Tracking Mask)

[0181] In some embodiments, the GPU may segment each frame into groups of tiles (e.g., rows) and generate a mask that indicates whether a particular group of tiles (e.g., row) has non-black content. The GPU may use the mask to generate rendering output to downstream components in the image pipeline. A new mask may be generated for each new frame. The image pipeline components may be communicatively coupled by a dynamic circuit network (DCN).

[0182] In one embodiment, a frame may have 1,760 rows of pixels, and each row of tiles may have 16 rows of pixels. In this embodiment, 110 bits may be used to store the black / non-black status of the 110 tile rows in each frame. Bits corresponding to all-black tile rows may be set to zero, and bits corresponding to non-black tile rows may be set to one. A bit may default to zero and be set to one when the tile row is non-black. A GPU or another image pipeline component may use the information in these bits to select non-black tile rows to be sent over the DCN. A GPU or another image pipeline component may use the information in these bits to pack and unpack non-black tile rows as described herein.

[0183] As described herein, transmitting only non-black tile rows and packing these rows increases the opportunities to implement power optimizations, including, but not limited to: Reduce the DisplayPort link power (e.g., send no rows or only a subset of rows) Reduces DP source processor memory bandwidth and power (e.g., the DCN does not need to fetch all black pixels from DRAM, which saves DRAM bandwidth, DRAM power, and DCN power) Eliminates the need for DSCs to compress full frames on DP sources DP Sync Bridge to DSC eliminates the need to decompress full frames Eliminates the need for DP SyncBridge devices to perform full-frame color segmentation Eliminates the need for DP SyncBridge devices to write complete frames to LPDDR, saving power The pixel engine block will not need to fetch the complete frame from LPDDR, saving power. The pixel engine will not need to perform pixel processing on the full frame, saving power. Reduced signaling reduces MIPI-TX power to the panel Reduces MIPI-RX power consumption in the panel Reduces panel power by preventing the need to store or read data from its SRAM and supporting partial black screens

[0184] The DCN may include a direct memory access controller (DMA) to fetch non-black rows into the DCN without involving any processor. The DMA may read a mask to identify the non-black rows. The non-black rows may be sent to the display engine via the DCN and DP without reorganization or reconfiguration (e.g., folding or compacting) to conserve power.

[0185] As described herein, if non-black rows form a significant portion of a frame, frame reconfiguration or partial darkening mode may not conserve power: such frames may be sent directly to the DP sink bridge via the DCN. (Additional Aspects)

[0186] In addition to the claimed invention, further embodiments or aspects of the invention are described herein by way of non-limiting example.

[0187] 1. A method in a virtual, augmented, or mixed reality system, comprising: receiving a frame of image data by a GPU; a GPU identifying a number of regions / portions / sections / tiles within a frame of image data that have changed from a previous frame of image data; a GPU moving at least some of the plurality of regions / portions / sections / tiles to a start of a frame of data to form a reordered frame of image data; the GPU sending the reordered frames of image data to the DB; Shutting down parts / components / functions of the GPU; shutting down the communication link between the GPU and the DB; A step to shut down parts / components / functions of the DB, shutting down the communication link between the DB and the display panel; shutting down a portion / component / function of the display panel; A method comprising:

[0188] 2. The method of aspect 1, further comprising the step of the GPU compressing the reordered frames of image data before transmitting the reordered frames of image data to the DB.

[0189] 3. The method of aspect 1, wherein the reordered frame of image data is smaller than the frame of image data.

[0190] 4. The method of aspect 1, further comprising the step of the DB storing the reordered frames of image data in a buffer.

[0191] 5. Determining the size of the reordered frame of image data; shutting down a portion / component / function of the GPU, a communication link between the GPU and the DB, a portion / component / function of the DB, a communication link between the DB and the display panel, and a portion / component / function of the display panel only when the reordered frame of image data is smaller than a predetermined maximum size; 2. The method of aspect 1, further comprising:

[0192] 6. The method of aspect 1, further comprising the step of the GPU sending an STP message to the DB after sending the reordered frame of image data to the DB.

[0193] 7. The method of aspect 6, further comprising the step of the GPU sending an STP message to the DB via SDP.

[0194] 8. The method of aspect 1, wherein the part / component / function of the GPU is selected from the group consisting of memory reading, compression, and color segmentation.

[0195] 9. The method of aspect 1, wherein a part / component / function of the DB is memory write.

[0196] 10. The method of aspect 1, wherein the part / component / function of the display panel is selected from the group consisting of a video RAM and a MIPI receiver.

[0197] 11. The method of aspect 1, further comprising the step of the GPU sending a wake-up signal to the DB.

[0198] 12. The method of aspect 11, wherein the GPU sends a wake-up signal over an AUX communication link.

[0199] 13. The method of aspect 1, wherein a portion / component / function of the GPU, a communication link between the GPU and the DB, a portion / component / function of the DB, a communication link between the DB and the display panel, and a portion / component / function of the display panel are shut down asynchronously.

[0200] 14. The method of aspect 1, further comprising the DB reconstructing frames of image data from the reordered frames of image data.

[0201] 15. The method of aspect 1, further comprising setting portions of the frame of image data that are not within multiple regions / portions / sections / tiles within the frame of image data to a background color.

[0202] 16. The method of aspect 1, further comprising the step of DB blending the reordered frame of image data with a previous frame of image data.

[0203] 17. The method of aspect 16, further comprising the step of the DB masking the previous frame of image data before blending it with the reordered frame of image data.

[0204] 18. The method of aspect 1, further comprising the step of the DB blending the reordered frames of image data with image data associated with the updated foveated region.

[0205] 19. The method of aspect 1, further comprising the step of the DB scaling the reordered frames of image data.

[0206] 20. The DB receives a scaling factor from the GPU; the DB scaling the reordered frame of image data using a scaling factor; 20. The method of aspect 18, further comprising:

[0207] 21. The method of aspect 18, wherein the scaling is part of a foveation operation.

[0208] 22. The method of aspect 1, further comprising a step in which the DB performs a function on the image data, the function being selected from the group consisting of warping, pixelated dimming, occlusion, chromatic aberration correction, frame rate, and dilation.

[0209] 23. The method of aspect 1, further comprising storing the reordered frames of image data in a FIFO memory before shutting down the portion / component / function of the GPU.

[0210] 24. The method of aspect 1, further comprising the step of the DB sending a built-in line control message to the display panel. (MIPI output data configuration)

[0211] As described herein, the DB processes image data received from the GPU and sends the processed image data to the display panel for display to the user. In grayscale format, the display panel receives image data over the MIPI link format, where all red pixels are followed by all green pixels, then all blue pixels, for color-sequential display; otherwise, standard packed RGB is used. Assuming a display resolution of Y x X, the output timing from the MIPI receiver would be Y / 3 x 3X. Each row would have one-third of the pixels, but the number of rows would be tripled, since only one-third of the colors are being transmitted. Also, an additional 24 rows for signaling and three rows for VSA, VBP, and VFP result in 5,307 rows.

[0212] 27 depicts sample MIPI receiver output data 2700, according to some embodiments. The output data 2700 includes a VSYNC start packet 2702, followed by a first blanking packet / VBP 2704, followed by image data generated by a pixel engine 2706, followed by a second blanking packet / VFP 2708 and an EOT packet 2710.

[0213] As shown in FIG. 28, the image data 2706 generated by the pixel engine consists of an HSYNC start packet 2812, followed by a first blanking packet / HBP 2814, followed by one row of image data and / or embedded signaling data in a long packet format 2816, followed by a second blanking packet / HFP 2818.

[0214] Figure 29 depicts sample MIPI receiver output data 2900 corresponding to a video frame with differentiated red, green, and blue pixel rows according to some embodiments when no compression is used. Each color field includes 1,760 individual rows with 480 pixels within each row. The output data 2900 includes a VSYNC start packet 2902, followed by a first blanking packet / VBP 2904, followed by image data generated by the pixel engine 2906, followed by a second blanking packet / VFP 2908 and an EOT packet 2910. The image data generated by the pixel engine 2906 consists of a HSYNC start packet 2912, followed by a first blanking packet / HBP 2914, followed by one row of image data and / or embedded signaling data in a long packet format 2916, followed by a second blanking packet / HFP 2918. The image data 2916 includes a first signaling row 2920, followed by a first color (red) field row 2922, followed by a second signaling row 2924, followed by a second color (green) field row 2926, followed by a third signaling row 2928, followed by a third color (blue) field row 2930. (Fully darkened low power mode)

[0215] In the full dark low power mode (described above), the DB will communicate its transition to low power mode to the display panel using the first signaling line 3020, as shown in Figure 30. After the transition to low power mode is communicated, the DB will transition to low power mode (LP11). The display panel will also conserve power by placing the MIPI receiver in LP11 mode and will not store any information into or read any information from the panel video RAM.

[0216] 38 schematically depicts the MIPI timing of image frame data 3030 communicated to the DB to initiate full dark low power mode. As described herein, the image frame data 3030 is similar to that of normal image mode. The exception is that the GPU writes a vertical blank SDP message into the image frame data 3030 to initiate full dark mode. (Partial darkening low power mode)

[0217] In the various partial darkening low power modes (described above), the DB will communicate the partial darkening low power mode parameters (e.g., low power mode enable and number of active rows) for each color field in the corresponding signaling rows 3120, 3124, 3128 preceding the color field rows 3122, 3126, 3130, as shown in Figure 31. The frame timing for the partial darkening low power mode is similar to that for the color field display depicted in Figure 29, except that only a programmable percentage of the 1,760 rows containing image data per color field are sent from the DB to the display panel over the MIPI link. After the rows containing image data are received, the MIPI link is placed in low power, or LP11, mode for all black rows to conserve power and terminate LP11 prior to each field.

[0218] 31A diagrammatically depicts MIPI timing 3150 for two cycles of a DB operating in partial dimming low power mode, according to some embodiments such as those described herein. At 3152, during the display of frame N, the pixel engine sets the appropriate value for the number of active rows in the upper wires for frame N+1. At 3154, during the display of frame N, the DB MIPI system fetches and latches those wires for frame N+1. Similarly, at 3156, during the display of frame N+1, the pixel engine sets the appropriate value for the number of active rows in the upper wires for frame N+2. At 3158, during the display of frame N+1, the DB MIPI system fetches and latches those wires for frame N+2.

[0219] 31B diagrammatically depicts MIPI timing 3150 for two cycles of a DB transitioning from partial darkening low power mode to full darkening low power mode, in accordance with some embodiments. At 3162, during the display of frame N, the pixel engine sets a value for the number of active rows corresponding to full darkening (i.e., zero) in the upper wires for frame N+1. At 3164, during the display of frame N, the DB MIPI system fetches and latches those wires for frame N+1. Due to the full darkening value in the wires, frame N+1 is fully darkened.

[0220] FIG. 35 diagrammatically depicts MIPI timing 3500 for two cycles of a DB operating in partial darkened low-power mode, according to some embodiments such as those described above. In each cycle / frame, there is a portion during which the DB is operating in low-power mode. During this time, the DP is turned off, and in response, the fiber-optic connector is also turned off. Turning off these two components of the DB during the entire cycle / frame saves system power. In some embodiments, the low-power portion of the cycle / frame is approximately half the cycle. Possible reasons for shutting down components of the DB include, but are not limited to, the following: 1) The display subsystem shuts down normally. 2) Normal display subsystem goes into low power mode 3) After an active line is transmitted every display frame, power savings require the line to be shut down during normal operation. 4) During partial darkening mode after the requested active line has been transmitted, as will be described herein. 5) To transition to full dark mode as described herein 6) To transition to frame repeat mode as described herein (Reduced resolution rendering with scaling (section-wise foveated rendering))

[0221] In some embodiments, a VR / AR / MR system can conserve power by using section-by-section foveated rendering to render a virtual image corresponding to the user's area of ​​focus at a higher resolution while rendering a background virtual image at a lower resolution. Using section-by-section foveated rendering can reduce GPU computation and system power usage. The DB on the wearable uses a pixel engine and up-to-date eye tracking data to merge the background and foveated content at a higher resolution for display. Section-by-section foveated rendering is optimized for multi-stream implementations. In such implementations, each stream has its own image pipeline and DB. Alternatively, custom packing can be used to allow multiple streams to share a single video stream in VESA SST mode if a multi-stream implementation is not available. (Dynamic Occlusion)

[0222] In some embodiments, the VR / AR / MR system can conserve power by, for example, eliminating occluded virtual content from the image pipeline, potentially at the final stage in the pipeline. The DB can use depth maps from the wearable to mask virtual content as needed (e.g., for occlusion by a fast-moving hand). The DP can mask virtual content just before display, reducing latency compared to masking on the GPU. The GPU still performs occlusion on the content, but may not be able to occlude everything due to pipeline latency. Rendered content is sent from the GPU to the DB, where it is masked using the depth map and sent to the display panel for display, with latency of, for example, 3 ms or less. (Uncompressed high bandwidth data)

[0223] The decompressed high-bandwidth data can be transmitted from the GPU to the DB using various channels (e.g., MST in the VESA standard). This decompressed high-bandwidth data can be transmitted as part of a combined stream for use in occlusion, segmented dimming, control, geometric distortion LUTs, and foveation. In some embodiments, a first video stream can be used to carry compressed image data for the primary panel, and a second video stream can be used to carry the decompressed high-bandwidth data. The DB has built-in MIPI pre-emphasis support to compensate for RX EQ issues and larger distances between the display and the DB. (Repeat previous frame with offset)

[0224] In some embodiments, the image content does not change from frame to frame as the user's head moves slightly, but rather shifts a few pixels up or down. Rather than tasking the GPU with redrawing all the pixels, the repeat previous frame with offset mode allows the GPU to bypass re-rendering and perform a simple transformation using the offset values ​​already stored in DB memory to move the content. In this mode, the GPU calculates the offset values ​​using only head pose updates, which can save significant GPU power, DP power, and fiber optics channel power because not much data is transferred to the wearable (i.e., head pose updates). (Black part mask)

[0225] FIG. 36A depicts an AR view 3600 according to some embodiments. The graphical user interface 3610 may change as the AR system operates. However, the portion of the frame outside the graphical user interface 3610 typically does not change and would therefore be rendered as a black / unlit portion of the frame 3620 (see FIG. 36B). The black portion of the frame 3620 shown in FIG. 36B would remain the same from frame to frame in some embodiments. Thus, the DB would not fetch or operate on (e.g., warp) the black portion of the frame 3620. Using a mask created from the black portion of the frame 3620, the GPU may generate start / end column pairs per row, which are sent to the DB via the DP. In response, the DB would fetch from memory only the image information between the start and end columns per row. This technique substantially reduces interleaved division multiple access (IDMA) active time, which reduces the time before the DB can transition to a low-power mode.

[0226] Certain aspects, advantages, and features of the present disclosure are described herein. It should be understood that not necessarily all such advantages may be achieved in accordance with any particular embodiment of the present disclosure. Thus, the present disclosure may be embodied or carried out to achieve or optimize one advantage or group of advantages as taught herein, without necessarily achieving other advantages as may be taught or suggested herein.

[0227] The embodiments are described in conjunction with the accompanying drawings. However, it should be understood that the figures are not drawn to scale. Distances, angles, and the like are merely illustrative and do not necessarily bear precise relationships to the actual dimensions and layout of the devices shown. Additionally, the foregoing embodiments have been described with a level of detail to enable those skilled in the art to make and use the devices, systems, methods, and equivalents described herein. Various modifications are possible. Components, elements, and / or steps may be altered, added, removed, or rearranged.

[0228] The devices and methods described herein can advantageously be implemented, at least in part, using, for example, computer software, hardware, firmware, or any combination of software, hardware, and firmware. Software modules can include computer-executable code stored in a computer's memory to perform functions described herein. In some embodiments, the computer-executable code is executed by one or more general-purpose computers. However, those skilled in the art will understand in light of this disclosure that any module that can be implemented using software to run on a general-purpose computer can also be implemented using a different combination of hardware, software, or firmware. For example, such modules can be implemented entirely in hardware using a combination of integrated circuits. Alternatively, or in addition, such modules can be implemented, entirely or partially, not by a general-purpose computer, but by a specialized computer designed to perform the particular functions described herein. Additionally, where a method is described that is or can be performed, at least in part, by computer software, it should be understood that such a method can be provided on a non-transitory computer-readable medium that, when read by a computer or other processing device, causes the method to be performed.

[0229] Certain embodiments are explicitly described; however, other embodiments will be apparent to those skilled in the art based on this disclosure.

[0230] The various processors and other electronic components described herein are suitable for use with any optical system for projecting light. The various processors and other electronic components described herein are also suitable for use with any audio system for receiving voice commands.

[0231] Various exemplary embodiments of the present disclosure are described herein. These examples are referred to in a non-limiting sense. They are provided to illustrate the more broadly applicable aspects of the present disclosure. Various changes may be made to the disclosed embodiments, and equivalents may be substituted without departing from the true spirit and scope of the present disclosure. In addition, many modifications may be made to adapt a particular situation, material, composition, process, process act, or step to the objective, spirit, or scope of the present disclosure. Furthermore, as will be understood by those skilled in the art, each of the individual variations described and illustrated herein has discrete components and features that can be readily separated from or combined with the features of any of the other several embodiments without departing from the scope or spirit of the present disclosure. All such modifications are intended to be within the scope of the claims associated with this disclosure.

[0232] The present disclosure includes methods that may be implemented using the subject devices. The methods may include the act of providing such a suitable device. Such provisioning may be performed by an end user. In other words, the act of "providing" merely requires the end user to obtain, access, approach, locate, configure, activate, power on, or otherwise act to provide the device required in the subject methods. The methods recited herein may occur in any order of the recited events that is logically possible, and in the stated order of events.

[0233] Exemplary aspects of the present disclosure are described above, along with details regarding material selection and manufacturing. As for other details of the present disclosure, these may be understood in connection with the above-referenced patents and publications and are generally known or may be understood by those skilled in the art. The same may be true with respect to the method-based aspects of the present disclosure in terms of additional acts as commonly or logically adopted.

[0234] Additionally, while the present disclosure has been described with reference to several embodiments that optionally incorporate various features, the present disclosure is not limited to what has been described and indicated as being contemplated with respect to each variation of the present disclosure. Various modifications may be made to the present disclosure as described, and equivalents (whether recited herein or not included for purposes of brevity to some extent) may be substituted without departing from the true spirit and scope of the present disclosure. Additionally, when a range of values ​​is provided, it is understood that all intervening values ​​between the upper and lower limits of that range, and any other stated value or intervening values ​​within the stated range, are encompassed within the present disclosure.

[0235] It is also contemplated that any optional features of the described variations may be described and claimed independently or in combination with any one or more of the features described herein. Reference to a singular item includes the possibility that multiple identical items are present. More specifically, as used in this specification and the claims associated therewith, the singular forms "a," "an," "said," and "the" include plural references unless specifically stated otherwise. In other words, the use of articles allows for "at least one" of the items of the present subject matter in the above description and in the claims associated with this disclosure. Furthermore, it should be noted that such claims may be drafted to exclude any optional element. Accordingly, this language is intended to serve as a predicate for the use of exclusive terminology such as "solely," "only," and the like in connection with the recitation of claim elements, or the use of a "negative" limitation.

[0236] Without the use of such exclusive terminology, the term "comprising" in a claim associated with this disclosure shall be construed as permitting the inclusion of any additional elements, regardless of the number of elements recited in such a claim, or the addition of features may be considered to change the nature of the elements recited in such a claim. Except as specifically defined herein, all technical and scientific terms used herein should be given the broadest commonly understood meaning possible while maintaining claim legitimacy.

[0237] The scope of the present disclosure is not intended to be limited to the examples provided and / or the specification of the present subject matter, but rather is intended to be limited only by the scope of the terms of the claims associated with this disclosure.

[0238] In the foregoing specification, the present disclosure has been described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes can be made therein without departing from the broader spirit and scope of the present disclosure. For example, the foregoing process flows are described with reference to a particular order of process actions. However, the order of many of the described process actions may be changed without affecting the scope or operation of the present disclosure. The specification and drawings are, therefore, to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A data format for use in a virtual, augmented, or mixed reality system, comprising: a first signaling row; a plurality of first color field rows; a second signaling row; and a plurality of second color field rows; a third signaling row; and a plurality of third color field rows; A data format comprising:

2. The data format of claim 1, wherein the first signaling row includes several active rows for the plurality of first color field rows.

3. The data format of claim 1, wherein the active row changes between image frames.

4. The data format of claim 1, wherein the first, second, and third signaling lines and the plurality of first, second, and third color field lines are read at a rate faster than the images corresponding to the plurality of first, second, and third color field lines are displayed.

5. The data format of claim 1, wherein the second signaling row includes several active rows for the plurality of second color field rows.

6. The data format of claim 1, wherein the third signaling row includes several active rows for the plurality of third color field rows.

7. The data format of claim 1, wherein the first signaling row includes the starting positions of the plurality of first color field rows.

8. The data format of claim 1, wherein the second signaling row includes the starting positions of the plurality of second color field rows.

9. The data format of claim 1, wherein the third signaling row includes the starting positions of the plurality of third color field rows.

10. The data format of claim 1, wherein the first, second, and third color field rows contain intensity information without color information.

11. The data format of claim 10, wherein the first, second, and third signaling lines include color information without intensity information.

Images