Method and system for dynamic depth-based reprojection

Abstract

A method for generating a reprojected image includes receiving motion data and determining, based on the motion data, whether a motion threshold has been exceeded. The method also includes generating a depth-based reprojection if the motion threshold has been exceeded, or generating a non-depth-based reprojection if the motion threshold has not been exceeded.

Description

Technical Field

[0001]

[0001] Cross - reference to Related Applications This application claims priority to U.S. Provisional Patent Application No. 63 / 453,412, titled "Method and System for Dynamic Depth - Based Reprojection," filed on March 20, 2023, and U.S. Provisional Patent Application No. 63 / 453,376, titled "Method and System for Performing Foveated Image Compression Based on Eye Gaze," filed on March 20, 2023, the disclosures of which are hereby incorporated by reference in their entireties for all purposes.

Background Art

[0002]

[0002] Modern computing and display technologies have facilitated the development of systems for so - called virtual reality or augmented reality experiences, where digitally reproduced images or portions thereof are presented to viewers in a way that they appear to be real or can be perceived as real. Virtual reality, i.e., VR scenarios, typically involve presenting digital or virtual image information without transparency to other actual real - world visual inputs, and augmented reality, i.e., AR scenarios, typically involve presenting digital or virtual image information as an extension to the visualization of the actual world around the viewer.

[0003]

[0003] Referring to Figure 1, an augmented reality scene 100 is depicted. The user of the AR technology sees a setting like a real-world park, featuring people, trees, buildings, and a concrete platform 120 in the background. The user also perceives that they are "seeing" "virtual content," such as a robot statue 110 standing on the real-world concrete platform 120, and a flying cartoonish avatar character 102 that appears to be an anthropomorphic bumblebee. These elements 110 and 102 are "virtual" in the sense that they do not exist in the real world. Because the human visual system is complex, it is difficult to produce AR technology that facilitates a rich presentation of virtual image elements surrounded by other virtual or real-world image elements in a comfortable and natural sense.

[0004]

[0004] Despite these advances in display technology, there is a need in the art for improved methods and systems related to augmented reality systems, particularly display systems. [Overview of the Initiative]

[0005]

[0005] The present invention generally relates to methods and systems related to projection display systems, including wearable displays. More specifically, embodiments of the present invention provide methods and systems that provide dynamic control of image reprojection. The present invention is applicable to a variety of applications in computer vision and image display systems.

[0006]

[0006] Some embodiments of the present invention provide a headset rendering system having two different reprojection systems. Each of the reprojection systems is characterized by a different power profile. The system can implement a determination of which reprojection system to use based on the positional difference between the headset position and orientation (i.e., headset posture) corresponding to the original rendered image and the current, i.e., actual or physical headset posture corresponding to the display of the image. Additionally, the determination can be based at least in part on the time difference between when the original image was rendered and when the reprojected image is displayed to the user. Thus, either positional data, i.e., the difference between the headset posture corresponding to the rendering of the original image and the headset posture corresponding to the display of the reprojected image, time data, i.e., the time difference between when the original image was rendered and when the reprojected image is displayed, or a combination of positional and time data can be used when selecting the reprojection system to be used to carry out the reprojection. As will be described more fully herein, both high-power and low-power reprojection systems are provided by embodiments of the present invention, and the low-power reprojection system can obtain data from either the output of the high-power reprojection system or the original source image.

[0007]

[0007] Embodiments of the present invention adjust system power by using low-power, non-depth-based reprojection under conditions where limited headset movement is observed, and high-power, depth-based reprojection under conditions where increased headset movement is observed. As a result, embodiments of the present invention provide a high-quality user experience in which the image is reprojected to align with objects in the world, but with variable power consumption to reduce system power when appropriate.

[0008]

[0008] The present invention achieves many advantages over conventional techniques. For example, embodiments of the present invention provide a method and system for reducing power consumption by utilizing low-power non-depth-based reprojection when the time difference and / or positional difference between rendering and reprojection for display is below a threshold, and by utilizing high-power depth-based reprojection when the time difference and / or positional difference between rendering and reprojection for display is above a threshold. Embodiments of the present invention make it possible to avoid the implementation of a custom depth-based reprojection ASIC solution and enable the use of a general-purpose GPU while maintaining the low overall power consumption typically provided by a custom depth-based reprojection ASIC solution. Embodiments of the present invention also provide the additional advantage of retaining a local secondary GPU for device computation needs, thereby enabling flexibility in the design and use of image processing algorithms. These and other embodiments of the present invention, along with many of their advantages and features, will be described in more detail below in conjunction with the accompanying drawings. [Brief explanation of the drawing]

[0009] [Figure 1] This diagram shows a user's view of augmented reality (AR) through an AR device. [Figure 2A] These are cross-sectional side views of an example of a stacked set of waveguides, each containing an internally coupled optical element. [Figure 2B] Figure 2A is a perspective view of an example of one or more stacked waveguides. [Figure 2C] Figures 2A and 2B are top views of an example of one or more stacked waveguides. [Figure 3] This is a simplified diagram of an eyepiece waveguide with combined pupil expanders according to one embodiment of the present invention. [Figure 4] This figure shows an example of a wearable display system according to one embodiment of the present invention. [Figure 5] This is a perspective view of a wearable device according to one embodiment of the present invention. [Figure 6] This is a simplified schematic diagram illustrating a dynamic depth-based reprojection system according to one embodiment of the present invention. [Figure 7] This is a simplified flowchart illustrating how to perform dynamic depth-based reprojection using one embodiment of the present invention. [Figure 8] This is a simplified schematic diagram illustrating a dynamic depth-based reprojection system according to another embodiment of the present invention. [Figure 9] This is a simplified flowchart illustrating a method for performing dynamic depth-based reprojection according to another embodiment of the present invention. [Figure 10] This is a simplified schematic diagram illustrating a dynamic depth-based reprojection system including foveal image compression according to one embodiment of the present invention. [Figure 11] This diagram shows a fovealed image having three fovealed regions according to one embodiment of the present invention. [Figure 12] This is a fovealed 3D generated image having three fovealed regions, according to yet another embodiment of the present invention. [Figure 13] This is a diagram showing an image that can be used with multiple foveal maps according to one embodiment of the present invention. [Figure 14] This is a simplified flowchart illustrating a method for compressing images using one embodiment of the present invention. [Figure 15] The compression levels obtained as a function of time, expressed by consecutive frames versus frequency, for both a sparse compression system implementation and a DSC-SPARSE system implementation according to one embodiment of the present invention are shown. [Figure 16] The following shows a histogram of frame count versus compression for a sparse compression system implementation and a DSC-SPARSE system implementation according to one embodiment of the present invention. [Figure 17] This is a simplified flowchart illustrating a method for compressing image frames using an alternating compression algorithm according to one embodiment of the present invention. [Figure 18]A simplified image showing an image frame divided into a high-quality region and a low-quality region according to an embodiment of the present invention. [Figure 19] A simplified flowchart showing a method of compressing an image using different compression ratios for a high-quality region and a low-quality region according to an embodiment of the present invention. [Figure 20] A simplified image showing an image frame divided into high-quality tiles and low-quality tiles according to an embodiment of the present invention. [Figure 21] A simplified flowchart showing a method of compressing an image using different compression ratios for high-quality tiles and low-quality tiles according to an embodiment of the present invention. [Figure 22] A simplified schematic diagram showing components of an AR system according to an embodiment of the present invention.

Mode for Carrying Out the Invention

[0010]

[0033] The present invention generally relates to methods and systems related to projection display systems including wearable displays. More particularly, embodiments of the present invention provide methods and systems for providing dynamic control of image reprojection. The present invention is applicable to various applications in computer vision and image display systems.

[0011]

[0034] Referring now to the drawings, like reference numerals refer to like parts throughout. Unless otherwise indicated, the drawings are schematic diagrams not necessarily drawn to scale.

[0012]

[0035] Referring here to Figure 2A, in some embodiments, light impacting a waveguide may need to be redirected to internally couple to the waveguide. Internal coupling optics may be used to redirect and internally couple the light to its corresponding waveguide. Although referred to as “internal coupling optics” throughout this specification, internal coupling optics do not necessarily have to be optical elements and may be non-optical elements. Figure 2A shows a cross-sectional side view of an example of a set of stacked waveguides 200, each containing an internal coupling optics. Each waveguide may be configured to output light of one or more different wavelengths, or one or more different wavelength ranges. Light from a projector is fed into the set of stacked waveguides 200 and externally coupled to the user, as will be described more fully below.

[0013]

[0036] The illustrated set of stacked waveguides 200 includes waveguides 202, 204, and 206. Each waveguide includes associated internally coupled optical elements (which may also be referred to as optical input areas on the waveguide), for example, an internally coupled optical element 203 disposed on the main surface (e.g., upper main surface) of waveguide 202, an internally coupled optical element 205 disposed on the main surface (e.g., upper main surface) of waveguide 204, and an internally coupled optical element 207 disposed on the main surface (e.g., upper main surface) of waveguide 206. In some embodiments, one or more of the internally coupled optical elements 203, 205, and 207 may be disposed on the bottom main surface of each waveguide 202, 204, and 206 (particularly when one or more internally coupled optical elements are reflective deflection optical elements). As shown in the figures, the internally coupled optical elements 203, 205, and 207 may be located on the upper main surface of their respective waveguides 202, 204, and 206 (or on the upper part of the subsequent lower waveguide), particularly if those internally coupled optical elements are transmissive deflection optical elements. In some embodiments, the internally coupled optical elements 203, 205, and 207 may be located within the body of their respective waveguides 202, 204, and 206. In some embodiments, as considered herein, the internally coupled optical elements 203, 205, and 207 are wavelength-selective, selectively redirecting light of one or more wavelengths while transmitting light of other wavelengths. Although the internally coupled optical elements 203, 205, and 207 are shown on one side or corner of their respective waveguides 202, 204, and 206, it will be recognized that in some embodiments they may be located in other areas of their respective waveguides 202, 204, and 206.

[0014]

[0037] As shown in the figure, the internally coupled optical elements 203, 205, and 207 may be offset laterally from each other. In some embodiments, each internally coupled optical element may be offset so that light does not pass through another internally coupled optical element. For example, each internally coupled optical element 203, 205, and 207 may be configured to receive light from different projectors and may be separated (e.g., laterally spaced) from the other internally coupled optical elements 203, 205, and 207 so as not to receive light from the other internally coupled optical elements 203, 205, and 207.

[0015]

[0038] Each waveguide also includes associated optical distribution elements, for example, an optical distribution element 210 disposed on the main surface (e.g., upper main surface) of waveguide 202, an optical distribution element 212 disposed on the main surface (e.g., upper main surface) of waveguide 204, and an optical distribution element 214 disposed on the main surface (e.g., upper main surface) of waveguide 206. In some other embodiments, the optical distribution elements 210, 212, and 214 may each be disposed on the bottom main surface of the associated waveguides 202, 204, and 206. In some other embodiments, the optical distribution elements 210, 212, and 214 may each be disposed on both the top and bottom main surfaces of the associated waveguides 202, 204, and 206. Alternatively, the optical distribution elements 210, 212, and 214 may be arranged on different surfaces of the upper and lower main surfaces of different associated waveguides 202, 204, and 206, respectively.

[0016]

[0039] Waveguides 202, 204, and 206 may be separated and isolated by layers of material, for example, gas, liquid, and / or solid. For example, as shown, layer 208 may separate waveguides 202 and 204, and layer 209 may separate waveguides 204 and 206. In some embodiments, layers 208 and 209 are formed from low refractive index materials (i.e., materials with a lower refractive index than the materials forming directly adjacent waveguides among waveguides 202, 204, and 206). Preferably, the refractive index of the materials forming layers 208 and 209 is 0.05 or more, or 0.10 or less, lower than the refractive index of the materials forming waveguides 202, 204, and 206. Advantageously, layers 208 and 209 with lower refractive indices can function as cladding layers that facilitate total internal reflection (TIR) ​​of light passing through waveguides 202, 204, and 206 (e.g., TIR between the upper and lower main surfaces of each waveguide). In some embodiments, layers 208 and 209 are formed from air. Although not shown, it will be recognized that the upper and lower parts of the illustrated set of waveguides 200 may include directly adjacent cladding layers.

[0017]

[0040] Preferably, for ease of manufacture and other considerations, the materials forming waveguides 202, 204, and 206 are similar or identical, and the materials forming layers 208, 209 are similar or identical. In some embodiments, the materials forming waveguides 202, 204, and 206 may differ between one or more waveguides, and / or the materials forming layers 208, 209 may differ while still maintaining the various refractive index relationships described above.

[0018]

[0041] Continuing to refer to Figure 2A, rays 218, 219, and 220 are incident on waveguide set 200. It will be recognized that rays 218, 219, and 220 may also be introduced into waveguides 202, 204, and 206 by one or more projectors (not shown).

[0019]

[0042] In some embodiments, the light rays 218, 219, and 220 have different characteristics, such as different wavelengths or different wavelength ranges, which may correspond to different colors. Each internally coupled optical element 203, 205, and 207 deflects the incident light so that it propagates through waveguides 202, 204, and 206 respectively by TIR. In some embodiments, each of the internally coupled optical elements 203, 205, and 207 selectively deflects one or more specific wavelengths of light, while allowing other wavelengths to pass through the underlying waveguide and associated internally coupled optical elements.

[0020]

[0043] For example, the internally coupled optical element 203 may be configured to deflect a ray 218 having a first wavelength or wavelength range, and to transmit rays 219 and 220 having different second and third wavelengths or wavelength ranges, respectively. The transmitted ray 219 collides with and is deflected by an internally coupled optical element 205 configured to deflect light of the second wavelength or wavelength range. The ray 220 is deflected by an internally coupled optical element 207 configured to selectively deflect light of the third wavelength or wavelength range.

[0021]

[0044] Continuing to refer to Figure 2A, the deflected rays 218, 219, and 220 are deflected so that they propagate through their corresponding waveguides 202, 204, and 206. That is, the internal coupling optical elements 203, 205, and 207 of each waveguide deflect the light into their corresponding waveguides 202, 204, and 206, thereby internally coupling the light within their corresponding waveguides. The rays 218, 219, and 220 are deflected by TIR at an angle that causes the light to propagate through their respective waveguides 202, 204, and 206. The rays 218, 219, and 220 propagate through their respective waveguides 202, 204, and 206 by TIR until they collide with the corresponding optical distribution elements 210, 212, and 214 of the waveguides, where they are externally coupled to provide the externally coupled ray 216.

[0022]

[0045] Referring now to Figure 2B, a perspective view of an example of the stacked waveguides in Figure 2A is shown. As described above, the internally coupled rays 218, 219, and 220 are deflected by the internally coupled optical elements 203, 205, and 207, respectively, and then propagate through waveguides 202, 204, and 206 by TIR, respectively. The rays 218, 219, and 220 then collide with the optical distribution elements 210, 212, and 214, respectively. The optical distribution elements 210, 212, and 214 deflect the rays 218, 219, and 220 so that they propagate toward the externally coupled optical elements 222, 224, and 226, respectively.

[0023]

[0046] In some embodiments, the light distribution elements 210, 212, and 214 are orthogonal pupil expanders (OPEs). In some embodiments, the OPEs deflect or distribute light to externally coupled optics 222, 224, and 226, and in some embodiments, they can also increase the beam or spot size of this light as it propagates to the externally coupled optics. In some embodiments, the light distribution elements 210, 212, and 214 may be omitted, and the internally coupled optics 203, 205, and 207 may be configured to deflect light directly to the externally coupled optics 222, 224, and 226. For example, referring to Figure 2A, the light distribution elements 210, 212, and 214 may be replaced by externally coupled optics 222, 224, and 226, respectively. In some embodiments, the externally coupled optics 222, 224, and 226 are exit pupils (EPs) or exit pupil expanders (EPEs) that direct light towards the user's eye. It will be recognized that the OPE may be configured to increase the dimensions of the eyebox on at least one axis, and the EPE may be configured to increase the eyebox on an axis intersecting the axis of the OPE, for example, an orthogonal axis. For example, each OPE may be configured to redirect a portion of the light that hits the OPE to the EPE in the same waveguide, while allowing the rest of the light to continue propagating through the waveguide. Upon hitting the OPE again, another portion of the remaining light is redirected to the EPE, and the rest of that portion continues propagating through the waveguide, and so on. Similarly, upon hitting the EPE, a portion of the colliding light is redirected out of the waveguide toward the user, and the rest of that light continues propagating through the waveguide until it hits the EPE again, at which point another portion of the colliding light is redirected out of the waveguide, and so on. As a result, a single beam of internally coupled light can be "duplicated" each time a portion of its light is redirected by the OPE or EPE, thereby forming a field of cloned light beams. In some embodiments, the OPE and / or EPE may be configured to resize the light beam. In some embodiments, the functions of the light distribution elements 210, 212, and 214 and the externally coupled optical elements 222, 224, and 226 are combined in a combined pupil expander, as discussed in relation to Figure 2E.

[0024]

[0047] Therefore, referring to Figures 2A and 2B, in some embodiments, the set of waveguides 200 includes waveguides 202, 204, 206 for each component color, internally coupled optics 203, 205, 207, optical distribution elements (e.g., OPE) 210, 212, 214, and externally coupled optics (e.g., EP) 222, 224, 226. Waveguides 202, 204, 206 may be stacked with air gaps / cladding layers between them. The internally coupled optics 203, 205, 207 redirect or deflect the incident light into their waveguides (by different internally coupled optics receiving light of different wavelengths). The light then propagates within each waveguide 202, 204, 206 at an angle that yields a TIR. In the illustrated example, ray 218 (e.g., blue light) is deflected by the first internal coupling optical element 203, then bounces along the waveguide, interacting with the optical distribution element (e.g., OPE) 210 and then the external coupling optical element (e.g., EP) 222 in the manner described above. Rays 219 and 220 (e.g., green light and red light, respectively) pass through waveguide 202, with ray 219 colliding with the internal coupling optical element 205 and being deflected. Ray 219 then bounces along waveguide 204 via TIR, proceeding to its optical distribution element (e.g., OPE) 212, and then to the external coupling optical element (e.g., EP) 224. Finally, ray 220 (e.g., red light) passes through waveguide 206 and collides with the optical internal coupling optical element 207 of waveguide 206. The internal optical coupling element 207 deflects the light ray 220 so that after the light ray propagates to the optical distribution element (e.g., OPE) 214 by TIR, it propagates to the external coupling optical element (e.g., EP) 226 by TIR. The external coupling optical element 226 then finally externally couples the light ray 220 to the viewer, who also receives externally coupled light from the other waveguides 202 and 204.

[0025]

[0048] Figure 2C is a top view of an example of the stacked waveguides shown in Figures 2A and 2B. As shown, waveguides 202, 204, and 206 may be vertically aligned with their associated optical distribution elements 210, 212, and 214 and associated external coupling optics 222, 224, and 226. However, as discussed herein, the internal coupling optics 203, 205, and 207 are not vertically aligned. Rather, the internal coupling optics are preferably non-overlapping (e.g., laterally spaced as seen in the top or plan view). As further discussed herein, this non-overlapping spatial arrangement facilitates one-to-one input of light from different resources to different waveguides, thereby enabling the unique coupling of a particular light source to a particular waveguide. In some embodiments, arrangements including non-overlapping, spatially separated internal coupling optics may be referred to as a pupil-shifting system, where the internal coupling optics in these arrangements can correspond to sub-pupils.

[0026]

[0049] Figure 3 is a simplified diagram of an eyepiece waveguide having a combined pupil expander according to one embodiment of the present invention. In the example shown in Figure 3, the eyepiece 310 utilizes a combined OPE / EPE region in a one-sided configuration. Referring to Figure 3, the eyepiece 310 includes a substrate 320 on which an internally coupled optical element 322 and a combined OPE / EPE region 324, also referred to as a combined pupil expander (CPE), are provided. The incident ray 330 is internally coupled via the internally coupled optical element 322 and externally coupled as an output ray 332 via the combined OPE / EPE region 324.

[0027]

[0050] The combined OPE / EPE region 324 includes grids corresponding to both OPE and EPE that spatially overlap in the x and y directions. In some embodiments, the grids corresponding to both OPE and EPE are located on the same side of the substrate 320 such that the OPE grid is superimposed on the EPE grid, or the EPE grid is superimposed on the OPE grid (or both). In other embodiments, the OPE grid is located on the opposite side of the substrate 320 from the EPE grid such that the grids spatially overlap in the x and y directions but are separated from each other in the z direction (i.e., in different planes). Thus, the combined OPE / EPE region 324 can be implemented in either a one-sided or two-sided configuration.

[0028]

[0051] Figure 4 shows an example of a wearable display system 430 in which various waveguides and associated systems disclosed herein may be integrated. Continuing with reference to Figure 4, the display system 430 includes a display 432 and various mechanical and electronic modules and systems to support the functionality of the display 432. The display 432 is wearable by a user 440 (also referred to as the viewer) of the display system and can be coupled to a frame 434 configured to position the display 432 in front of the user 440's eyes. In some embodiments, the display 432 may be considered eyewear. In some embodiments, a speaker 436 is coupled to the frame 434 and configured to be positioned adjacent to the user 440's ear canal (in some embodiments, another speaker, not shown, may be optionally positioned adjacent to the user's other ear canal to provide stereo / shapeable acoustic control). The display system 430 may also include one or more microphones or other devices for detecting sound. In some embodiments, the microphone is configured to allow the user to provide input or commands to the system 430 (e.g., selection of voice menu commands, natural language questions, etc.) and / or to enable voice communication with other people (e.g., with other users of a similar display system). The microphone may further be configured as a peripheral sensor for collecting voice data (e.g., sounds from the user and / or the environment). In some embodiments, the display system 430 may further include one or more outward-oriented environmental sensors configured to detect objects, stimuli, people, animals, places, or other aspects of the world around the user. For example, the environmental sensors may include one or more cameras, which may be positioned outward, for example, to capture images similar to at least a portion of the user 440's normal field of view.In some embodiments, the display system may also be separate from the frame 434 and may include peripheral sensors that can be attached to the user 440's body (e.g., the user 440's head, torso, limbs, etc.). In some embodiments, the peripheral sensors may be configured to acquire data characterizing the user 440's physiological state. For example, the sensors may be electrodes.

[0029]

[0052] The display 432 is operably coupled to a local data processing module by a communication link such as a wired lead wire or a wireless connection, and the local data processing module can be mounted in various configurations, such as being fixedly mounted to a frame 434, fixedly mounted to a helmet or hat worn by the user, embedded in headphones, or otherwise detachably mounted to the user 440 (e.g., in a backpack configuration, in a belt-mounted configuration). Similarly, sensors can be operably coupled to the local processor and data module by a communication link such as a wired lead wire or a wireless connection. The local processing and data module may include a hardware processor and digital memory such as non-volatile memory (e.g., flash memory or a hard disk drive), both of which can be used to assist in data processing, caching, and storage. Optionally, the local processor and data module may include one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. The data may include a) data captured from sensors such as an image capture device (such as a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, a gyroscope, and / or other sensors disclosed herein (which may, for example, be operably coupled to frame 434 or otherwise attached to user 440), and / or b) data acquired and / or processed using a remote processing module 452 and / or a remote data repository 454 (including data relating to virtual content), possibly for passage to display 432 after such processing or retrieval. The local processing and data module may be operably coupled to the remote processing and data module 450 by a communication link 438, such as via a wired or wireless link, and the remote processing and data module may include the remote processing module 452, the remote data repository 454, and a battery 460.The remote processing module 452 and the remote data repository 454 can be coupled to the remote processing and data module 450 by communication links 456 and 458 so that these remote modules are operablely coupled to each other and available as resources to the remote processing and data module 450. In some embodiments, the remote processing and data module 450 may include one or more of the following: an image capture device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope. In some other embodiments, one or more of these sensors may be mounted on the frame 434 or may be a standalone structure communicating with the remote processing and data module 450 by a wired or wireless communication path.

[0030]

[0053] Continuing to refer to Figure 4, in some embodiments, the remote processing and data module 450 may include one or more processors configured to analyze and process data and / or image information, such as one or more central processing units (CPUs), graphics processing units (GPUs), and dedicated processing hardware. In some embodiments, the remote data repository 454 may include digital data storage facilities that may be available via the Internet or other networking configurations in a “cloud” resource configuration. In some embodiments, the remote data repository 454 may include one or more remote servers that provide information, such as information for generating augmented reality content, to the local processing and data module and / or the remote processing and data module 450. In some embodiments, all data is stored, all calculations are performed on the local processing and data module, and fully autonomous use from the remote module is possible. Optionally, an external system including a CPU, GPU, etc. (e.g., one or more processors, one or more computer systems) may perform at least part of the processing (e.g., generating image information, processing data) and provide and receive information from the illustrated module, for example, via a wireless or wired connection.

[0031]

[0054] Figure 5 shows a perspective view of a wearable device 500 according to one embodiment of the present invention. The wearable device 500 includes a frame 502 configured to support one or more projectors 504 at various positions along a surface facing the interior of the frame 502, as shown in the figure. In some embodiments, the projectors 504 can be mounted near the temples 506. Alternatively, or in addition, another projector can be placed at position 508. Such a projector may include, for example, one or more liquid crystal on silicon (LCoS) modules, micro-LED displays, or fiber scanning devices, or may operate in conjunction with them. In some embodiments, light from projectors 504 or projectors placed at position 508 can be directed into an eyepiece 510 for display to the user's eye. A projector placed at position 512 can be somewhat smaller due to the proximity this brings to the waveguide system. The closer the position, the less light can be lost when the waveguide system directs the light from the projector to the eyepiece 510. In some embodiments, the projector at position 512 can be used in conjunction with projector 504 or a projector positioned at position 508. Although not depicted, in some embodiments, the projector can also be positioned below the eyepiece 510. The wearable device 500 is depicted including sensors 514 and 516. Sensors 514 and 516 may take the form of forward-facing and lateral-facing optical sensors configured to characterize the real-world environment surrounding the wearable device 500.

[0032]

[0055] Embodiments of the present invention utilize an eye-tracking system to determine the user's gaze position and to utilize the gaze position for an image compression process. Referring to Figure 5, an eye-tracking camera 505 is positioned on frame 502 and can be used to track the user's gaze position using a wearable device 500. In other embodiments, the gaze position is determined using other eye-tracking systems, and the eye-tracking camera 505 shown in Figure 5 is merely illustrative. Among other functions, as will be described more fully herein, the image compression process, internal communications, and display used to compress and decompress virtual content for storage in memory can be modified according to the gaze position. For example, portions of an image or video stream corresponding to a gaze position can be compressed using a higher-quality compression process compared to other portions of the image or video stream located further away from the gaze position. Since these further portions of the image or video stream are in the user's peripheral vision, any impact on the user experience resulting from reduced compression quality may be smaller than the benefits achieved in terms of memory and processing efficiency and / or requirements. Those skilled in the art will recognize many variations, modifications, and substitutions.

[0033]

[0056] Embodiments of the present invention utilize a combination of a general-purpose GPU, generally associated with higher power consumption, configured to perform 6-degree-of-freedom (6DOF) depth-based reprojection, and a non-depth-based 6DOF or 3DOF reprojection processor operating at lower power consumption levels. Using motion data (e.g., current inertial measurement unit (IMU) measurements, headset attitude information, eye-tracking information, etc.), the system can utilize the general-purpose GPU to perform either 6DOF, depth-based reprojection, or a 6DOF / 3DOF, non-depth-based reprojection system, depending on the motion data. As a result, the system can conserve resources when motion data indicates that non-depth-based reprojection is appropriate, but can use the GPU at higher power consumption levels to perform depth-based reprojection when appropriate.

[0034]

[0057] Therefore, embodiments of the present invention provide similar performance and power profiles to custom ASIC implementations, while adding the flexibility of having a fully accessible GPU as needed, thereby reducing or eliminating the use of custom depth-based ASIC implementations.

[0035]

[0058] In augmented reality (AR) systems where wearable devices overlay computer-generated images onto the existing world, image correction is performed to provide consistent fixation characteristics to the images. Therefore, when an image is placed in a real-world location, it preferably does not move or jitter relative to its real-world placement. This characteristic can be referred to as pixel sticking.

[0036]

[0059] During the use of an AR system, image correction is performed because the headset's position may change from the time the image is generated until the final image is displayed. Therefore, the computer system (e.g., GPU) that generates the original display image can also predict the future position of the headset device to reduce or minimize errors caused by headset movement. Some AR systems also perform a final additional correction based on the actual headset position before displaying the image on the headset.

[0037]

[0060] In some implementations, the processor that initially generates the content is located close to the headset. However, cloud-based implementations can perform cloud-based rendering, also known as remote rendering, instead of using a local computer system. In these cloud-based implementations, the headset prediction process can be severely affected by transmission latency that occurs during data communication.

[0038]

[0061] Remote rendering systems can have extremely high latency, sometimes requiring a complete reprojection. This reprojection process, known as depth-based reprojection, typically utilizes a significant amount of computational power on traditional GPUs and results in considerable power consumption.

[0039]

[0062] To reduce the overall size and power consumption of the headset device, the concept of a remote computing implementation combined with local low-latency computing has been implemented. This allows latency-prone algorithms to be kept close to the device. This effort has led to the implementation of a custom ASIC depth-based reprojection system to reduce the overall power consumption of the remote device system.

[0040]

[0063] In remote rendering systems, if the connection is lost (for example, for a few seconds), a depth-based reprojection system (e.g., using the GPU) provided as a component of the headset may be able to continue rendering. Several use cases may arise if the connection is lost. 1) The headset moves significantly, and depth-based reprojection is utilized. 2) The headset is not working, and a non-depth-based solution will be implemented. 3) The headset has not moved from the last depth-based reprojection (i.e., use case 1), and non-depth-based reprojection is implemented. 4) The headset takes over full rendering because, for example, in the example of segmented rendering discussed below, sufficient information is available on the wearable device to perform reprojection.

[0041]

[0064] Therefore, embodiments of the present invention allow the GPU to be power-gated and disabled when depth-based reprojection is not required. The GPU can perform reprojection if the difference between the headset pose corresponding to the initial rendering and the actual headset pose (i.e., the temporal headset predicted delta) indicates that the required correction is sufficiently large. Thus, if limited headset movement occurs, the GPU can be power-gated and maintained in low-power hold mode. If significant movement occurs, the GPU can be used to reproject the image. If the GPU is not required, non-depth-based 6DOF or 3DOF reprojection or correction can be performed. Additionally, non-depth-based reprojection can be performed after the initial reprojection by the GPU.

[0042]

[0065] For example, based on motion data, also known as motion information or positional information, available from the IMU of an AR system headset, embodiments of the present invention utilize the GPU as needed, or at a lower frame rate. If the GPU is not utilized, a lower-power, non-depth-based 6DOF / 3DOF warp reprojection processor can be used.

[0043]

[0066] In some implementations, embodiments of the present invention enable complete headset rendering. For example, in one use case where all information used for rendering is provided to the headset, complete rendering can be performed on the headset. In another use case, segmented rendering can be performed. In this segmented rendering use case, instead of sending the rendered image to the headset (or the image being reprojected), only a list of items to be rendered or reprojected is transmitted to the headset. This configuration, which can be called a segmented GPU implementation, is performed remotely using all conventional setup operations, while the complete, for example, conventional rendering process is performed locally on the headset. By using the segmented rendering technique, a reduction in wireless bandwidth can be achieved because only a highly compressed list of items to be rendered is transmitted to the headset.

[0044]

[0067] In a third use case, providing additional GPUs within the system allows for the use of additional GPUs when needed for other processing functions in the absence of local display control or a complete system.

[0045]

[0068] Figure 6 is a simplified schematic diagram showing a dynamic depth-based reprojection system according to one embodiment of the present invention. The dynamic depth-based reprojection system 600 includes a CPU / control device 610 that controls system power at least in part based on motion data 615. In some implementations, the CPU / control device 610 acts as a throttle for system power and provides system control power control by prompting either a GPU 630 or a warp reprojection processor 640 to perform the reprojection task.

[0046]

[0069] Figure 6 shows how the CPU / control device 610, or other preferred control logic, takes motion data, such as positional information provided by the system IMU, and based on this motion data, the CPU / control device 610 determines which reprojection system will handle the reprojection. Both systems are power-gated and placed in a special holding mode for minimal power savings. The GPU 630 receives the depth map stored in the depth map memory 620 and the color map stored in the color map memory 622 as input. Thus, this depth map and color map information is received from system memory, and the GPU 630 performs depth-based reprojection, which is stored in the secondary intermediate system memory 632 (system memory can be located anywhere in the system). Thus, the GPU 630 can use both the depth map data and the color map data to provide depth-based reprojection. Generally, the power used to perform depth-based reprojection is greater than the power used to perform non-depth-based reprojection, as will be discussed below in relation to the warp reprojection processor 640.

[0047]

[0070] The reprojection generated by the GPU 630 can be made available by the external display 650. Thus, the depth-based reprojection can be sent to the external display 650 for display to the user, as indicated by the optional data path 628, or sent to the warp reprojection processor 640 for further processing before being displayed to the user using the external display 650. As an example of further processing, the depth-based reprojection can be generated at 60 Hz, and the warp reprojection processor 640 can generate the reprojection at 360 Hz. As another example, if the headset has not moved significantly during a period of time, the image previously rendered using the GPU 630 and stored in the secondary intermediate system memory 632 can be updated by the warp reprojection processor 640 before being displayed using the external display 650. As will be apparent to those skilled in the art, other image processing operations can also be performed using the warp reprojection processor 640.

[0048]

[0071] Referring again to Figure 6, low-power, non-depth-based warp reprojection can be performed by a warp reprojection processor 640, which can be implemented as a custom 6DOF or 3DOF logic device, i.e., a custom ASIC. The warp reprojection processor 640 can utilize image data stored in the secondary intermediate system memory 632, or in the color map memory 622 (i.e., the original color map buffer location) as indicated by the data path 626. Using the warp reprojection processor 640 to perform the reprojection reduces system power consumption compared to using the GPU to perform the reprojection.

[0049]

[0072] As indicated by the optional data path 628, the reprojected image can be sent from the GPU 630 to the external display 650 without further processing by the warp reprojection processor 640.

[0050]

[0073] Figure 7 is a simplified flowchart illustrating a method for performing dynamic depth-based reprojection according to one embodiment of the present invention. Method 700 includes generating motion data (710). Motion data, such as head pose information which can be determined using IMU data, photogrammetry, etc., head tracking information which can be determined using IMU data, eye tracking information which can be determined using an eye tracking system provided as a component of the headset, such as photogrammetry, or a combination of these datasets, can be received from various sources, including an IMU located in the headset. Motion information includes time data corresponding to the motion information.

[0051]

[0074] Method 700 also includes determining whether the time difference between the time the image was last rendered and the time the reprojected image is displayed to the user is greater than a threshold (712). If the time difference is less than the threshold, for example less than 2ms, then non-depth-based reprojection can be utilized because the movement of the headset is limited by acceleration and velocity values ​​corresponding to human movement (732). Non-depth-based reprojection can be 6DOF reprojection or 3DOF reprojection, depending on the specific application.

[0052]

[0075] If the time difference is greater than a threshold, for example, greater than 16 ms, method 700 proceeds to determine whether the positional difference (e.g., head pose difference) is greater than a threshold (714). In some cases, there is a large time difference between rendering and reprojection, but the headset has not experienced a large movement. In this case, the determination in 712 is positive, but the determination in 714 is negative, leading to the use of non-depth-based reprojection in 732. In some embodiments, a multi-level threshold is used instead of a single threshold in determination 712. In these embodiments, if the time difference is greater than a second threshold, the method can proceed to the use of depth-based reprojection (722) independently of the positional difference corresponding to the determination in 714. Thus, the time data corresponding to the determination in 712 can be used together with or independently of the positional data corresponding to the determination in 714. Thus, embodiments of the present invention can address latency present in an AR system by utilizing different reprojection techniques depending on the latency between virtual content generation and display to the user. Those skilled in the art will recognize many variations, modifications, and substitutions.

[0053]

[0076] In some cases, initial GPU correction results in significant system latency combined with miscalculated positional location differences. In other words, the remote rendering system miscalculates the correct location due to large discrepancies in the required future time predictions. During that period, the user is moving in a direction not predicted by the remote rendering system. Several possible scenarios can be addressed using embodiments of the present invention. 1) Remote rendering performs predictions, the headset does not move, the predictions are correct, and the GPU is not used. 2) Remote rendering makes predictions, the headset moves a considerable amount, the predictions are inaccurate, and the GPU is used. 3) A GPU is used, and a higher reprojection refresh rate is achieved without a GPU and using a low-power projection system. 4) A higher reprojection refresh rate is achieved without using a GPU and with a low-power projection system.

[0054]

[0077] As indicated by the positive judgment in 714, depth-based reprojection is used when the headset experiences large movements (722).

[0055]

[0078] After a non-depth-based reprojection is utilized based at least partially on the color map 730 (732), or after a depth-based reprojection is utilized based on both the depth map and the color map (720) (722), the content is displayed (740). As shown in Figure 7, in some embodiments, the depth-based reprojection generated in 722 is provided to the non-depth-based reprojection generated in 732 using an optional data path 724 for further processing, e.g., an increase in refresh rate implemented by the non-depth-based reprojection in 732.

[0056]

[0079] Therefore, in order to generate the reprojected image, time data corresponding to the time difference between rendering and reprojection, positional data, i.e., the difference in headset position and orientation (i.e., head posture) in rendering and reprojection, or a combination of time data and positional data, can be used when selecting the reprojection system to be used to perform the reprojection.

[0057]

[0080] It should be noted that the specific steps shown in Figure 7 provide a specific method for performing dynamic depth-based reprojection according to one embodiment of the present invention. Other sets of steps may be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Furthermore, the individual steps shown in Figure 7 may include a plurality of substeps that can be performed in various sequences as appropriate to the individual steps. Additionally, additional steps may be added or removed depending on the specific application. Those skilled in the art will recognize numerous variations, modifications, and alternatives.

[0058]

[0081] Figure 8 is a simplified schematic diagram showing a dynamic depth-based reprojection system according to another embodiment of the present invention. The dynamic depth-based reprojection system 800 shown in Figure 8 shares elements in common with the dynamic depth-based reprojection system 600 shown in Figure 6, and the explanations provided in relation to the dynamic depth-based reprojection system 600 shown in Figure 6 are appropriately applicable to the dynamic depth-based reprojection system 800 shown in Figure 8.

[0059]

[0082] In the dynamic depth-based reprojection system 800 shown in Figure 8, the dynamic depth-based reprojection system 800 includes a CPU / control device 810 that controls system power at least partially based on motion data 815. In some implementations, the CPU / control device 810 acts as a throttle for system power and provides system control power control by requesting either a GPU or an application-specific integrated circuit (ASIC) depth-based reprojection engine 830 or warp reprojection processor 840 to perform the reprojection task.

[0060]

[0083] Figure 8 shows how the CPU / control device 810, or other preferred control logic, takes motion data, such as positional information provided by the system IMU, and based on this motion data, the CPU / control device 810 determines which reprojection system will handle the reprojection. Both systems are power-gated and placed in a special holding mode for minimal power savings. The depth-based reprojection engine 830 of the GPU or ASIC receives the depth map stored in the depth map memory 820 and the color map stored in the color map memory 822 as input. Thus, this depth map and color map information is received from system memory, and the depth-based reprojection engine 830 of the GPU or ASIC performs the depth-based reprojection, which is stored in the secondary intermediate system memory 832 (system memory can be located anywhere in the system). Thus, the depth-based reprojection engine 830 of the GPU or ASIC can provide depth-based reprojection using both the depth map data and the color map data. Generally, the power used to perform depth-based reprojection is less than the power used in GPU-only implementations.

[0061]

[0084] The reprojection generated by the GPU or ASIC depth-based reprojection engine 830 can be made available to the external display 850. Thus, the depth-based reprojection can be sent to the external display 850 for display to the user, as indicated by the optional data path 828, or sent to the warp reprojection processor 840 for further processing before being displayed to the user using the external display 850. As an example of further processing, the depth-based reprojection can be generated at 60 Hz, and the warp reprojection processor 840 can generate the reprojection at 360 Hz. As another example, if the headset has not moved significantly during a period of time, the image previously rendered using the GPU or ASIC depth-based reprojection engine 830 and stored in the secondary intermediate system memory 832 can be updated by the warp reprojection processor 840 before being displayed using the external display 850. As will be apparent to those skilled in the art, other image processing operations can also be performed using the warp reprojection processor 840.

[0062]

[0085] Referring again to Figure 8, low-power, non-depth-based warp reprojection can be performed by a warp reprojection processor 840, which can be implemented as a custom 6DOF or 3DOF logic device, i.e., a custom ASIC. The warp reprojection processor 840 can utilize image data stored in the secondary intermediate system memory 832, or in the color map memory 822 (i.e., the original color map buffer location) as indicated by the data path 826. Using the warp reprojection processor 840 to perform the reprojection reduces system power consumption compared to using a GPU-only implementation to perform the reprojection.

[0063]

[0086] In some embodiments, depth-based reprojection is performed by a depth-based reprojection engine 830 of the GPU or ASIC, and the reprojected image can be sent to an external display 850 for display to the user, as indicated by an optional data path 828. Thus, as indicated by an optional data path 828, the reprojected image can be sent from the depth-based reprojection engine 830 of the GPU or ASIC to the external display 850 without further processing by a warp reprojection processor 840.

[0064]

[0087] Figure 9 is a simplified flowchart illustrating a method for performing dynamic depth-based reprojection according to another embodiment of the present invention. Method 900 includes generating motion data (910). Motion data, such as head pose information which can be determined using IMU data, photogrammetry, etc., head tracking information which can be determined using IMU data, eye tracking information which can be determined using an eye tracking system provided as a component of the headset, such as photogrammetry, or a combination of these datasets, can be received from various sources, including an IMU located in the headset. Motion information includes time data corresponding to the motion information.

[0065]

[0088] Method 900 also includes determining whether the time difference between the time the image was last rendered and the time the reprojected image is displayed to the user is greater than a threshold (912). If the time difference is less than the threshold, for example less than 2ms, then non-depth-based reprojection can be utilized because the movement of the headset is limited by acceleration and velocity values ​​corresponding to human movement (932). Non-depth-based reprojection can be 6DOF reprojection or 3DOF reprojection, depending on the specific application.

[0066]

[0089] If the time difference is greater than a threshold, for example, greater than 16 ms, method 900 proceeds to determine whether the positional difference (e.g., head pose difference) is greater than a threshold (914). In some cases, there is a large time difference between rendering and reprojection, but the headset has not experienced a large movement. In this case, the determination in 912 is positive, but the determination in 914 is negative, leading to the use of non-depth-based reprojection in 932. In some embodiments, a multi-level threshold is used instead of a single threshold in determination 912. In these embodiments, if the time difference is greater than a second threshold, the method can proceed to the use of an ASIC to perform depth-based reprojection (922) independently of the positional difference corresponding to the determination in 914. Thus, the time data corresponding to the determination in 912 can be used together with or independently of the positional data corresponding to the determination in 914. Thus, embodiments of the present invention can address latency present in an AR system by utilizing different reprojection techniques depending on the latency between virtual content generation and display to the user. Those skilled in the art will recognize many variations, modifications, and substitutions.

[0067]

[0090] In some cases, initial ASIC-based corrections result in high system latency combined with miscalculated positional location differences. In other words, the remote rendering system miscalculates the correct location due to large discrepancies in the required future time predictions. During that period, the user is moving in a direction not predicted by the remote rendering system. Several possible scenarios can be addressed using embodiments of the present invention. 1) Remote rendering performs predictions, the headset does not move, the predictions are correct, and no ASICs are used. 2) Remote rendering involves prediction, the headset moves a considerable amount, the prediction is inaccurate, and an ASIC is used. 3) ASICs are used, and higher reprojection refresh rates are achieved without ASICs and using low-power projection systems. 4) ASICs are not used, and higher reprojection refresh rates are achieved without ASICs and using a low-power projection system.

[0068]

[0091] As indicated by the positive judgment in 914, depth-based reprojection using the ASIC is employed when the headset experiences large movements (922).

[0069]

[0092] The content is displayed (940) after a non-depth-based reprojection is utilized (932) based at least partially on the color map 930, or after a depth-based reprojection using an ASIC is utilized (922) based on both the depth map and the color map (920). As shown in Figure 9, in some embodiments, the depth-based reprojection generated in 922 is provided to the non-depth-based reprojection generated in 932 using an optional data path 924 for further processing, e.g., for an increase in refresh rate implemented by the non-depth-based reprojection in 932.

[0070]

[0093] Therefore, in order to generate the reprojected image, time data corresponding to the time difference between rendering and reprojection, positional data, i.e., the difference in headset position and orientation (i.e., head posture) in rendering and reprojection, or a combination of time data and positional data, can be used when selecting the reprojection system to be used to perform the reprojection.

[0071]

[0094] It should be noted that the specific steps shown in Figure 9 provide a specific method for performing dynamic depth-based reprojection according to one embodiment of the present invention. Other sets of steps may be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Furthermore, the individual steps shown in Figure 9 may include a plurality of substeps that can be performed in various sequences as appropriate to the individual steps. Additionally, additional steps may be added or removed depending on the specific application. Those skilled in the art will recognize numerous variations, modifications, and alternatives.

[0072]

[0095] Figure 10 is a simplified schematic diagram showing a dynamic depth-based reprojection system including foveal image compression according to one embodiment of the present invention. The dynamic depth-based reprojection system 1000 including foveal image compression shown in Figure 10 shares elements in common with the dynamic depth-based reprojection system 600 shown in Figure 6, and the explanations provided in relation to Figure 6 are applicable to Figure 10 as appropriate.

[0073]

[0096] Referring to Figure 10, a dynamic depth-based reprojection system 1000, including foveal image compression, can receive image data from a remote source, such as a cloud-based source, as indicated by WiFi data 1001 provided to the decoder 1020. The implications of using a cloud-based source in the context of an augmented reality system are that the latency associated with the WiFi data 1001 can be significant and negatively impact the user experience, because the virtual content generated in the cloud may correspond to a different head pose than the head pose at the time the virtual content is displayed to the user. To compensate for this difference in head pose, the virtual content can be reprojected as discussed herein.

[0074]

[0097] Reprojection can be performed by a warp reprojection processor 1040 that can be compatible with a GPU 630, and / or a warp reprojection processor 640, as discussed in relation to Figure 6. To reduce the size of memory 1032, the image generated using GPU 1030 can be compressed using a foveal image compression process 1035. In particular, using eye-tracking data provided by the eye-tracking unit 1037, the portion of the image corresponding to the user's gaze position can be compressed at a high quality setting (e.g., 100% quality setting), while the portion of the image further away from the user's gaze position can be compressed at a low quality setting (e.g., 70% quality setting), thereby reducing the compressed image size and thus the size of memory 1032.

[0075]

[0098] Referring again to Figure 10, the dynamic depth-based reprojection system 1000 includes a CPU / control device 1010 that controls system power based at least partially on motion data 1005. In some implementations, the CPU / control device 1010 acts as a throttle for system power and provides system control power control by prompting either the GPU 1030 or the warp reprojection processor 1040 to perform the reprojection task.

[0076]

[0099] Figure 10 shows how the CPU / control device 1010, or other preferred control logic, takes motion data 1005, for example, positional information provided by a system inertial motion unit (IMU), and based on this motion data, the CPU / control device 1010 determines which reprojection system will handle the reprojection. Both systems are power-gated and placed in a special holding mode for minimal power savings. The GPU 1030 receives as input a depth map stored in depth map memory 1024 and a color map stored in color map memory 1022. Thus, this depth map and color map information is received from system memory, and the GPU 1030 performs depth-based reprojection, which is stored in memory 1032 (e.g., secondary intermediate system memory, system memory can be located anywhere in the system). Thus, the GPU 1030 can use both depth map data and color map data to provide depth-based reprojection. Generally, the power used to perform depth-based reprojection is greater than the power used to perform non-depth-based reprojection, as will be discussed below in relation to the warp reprojection processor 1040.

[0077]

[0100] The reprojection generated by the GPU 1030 can be made available by the external display 1050. Thus, the depth-based reprojection can be sent to the external display 1050 for display to the user, as indicated by the optional data path 1028, or sent to the warp reprojection processor 1040 for further processing before being displayed to the user using the external display 1050. As an example of further processing, the depth-based reprojection can be generated at 60Hz, and the warp reprojection processor 1040 can generate the reprojection at 360Hz. As another example, if the headset has not moved significantly during a period of time, the image previously rendered using the GPU 1030 and stored in memory 1032 can be updated by the warp reprojection processor 1040 before being displayed using the external display 1050. As will be apparent to those skilled in the art, other image processing operations can also be performed using the warp reprojection processor 1040.

[0078]

[0101] After depth-based reprojection is performed by the GPU 1030, the image can be compressed based on the user's gaze position. That is, the user's gaze information can be obtained, for example, from the gaze tracking unit 1037 or gaze tracking system 2255 shown in Figure 22. Then, using this gaze information, foveal image compression can be performed based on the gaze. This foveal image compression process 1035 will be described in more detail below in relation to Figures 11 to 14. The compressed image generated using the foveal image compression process 1035 is stored in memory 1032. The compressed image can then be decompressed using the image decompression process 1038 to provide input to the warp reprojection processor 1040.

[0079]

[0102] In some embodiments, the external display 1050 is capable of performing image decompression. In these embodiments, compressed data (e.g., images) stored in memory 1032 can be sent to either the warp reprojection processor 1040 or the external display 1050 for decompression.

[0080]

[0103] Referring again to Figure 10 and the foveal image compression process 1035, the line-of-sight based foveal process is shown in relation to Figures 11-14.

[0081]

[0104] Figure 11 is a diagram showing a fovealed image having three fovealed regions according to one embodiment of the present invention. The image in Figure 11 is divided into multiple regions based on the gaze position. In this case, the user is fixated on the center of the image, and as a result, the gaze position is located at the center of the image. As discussed herein, the gaze position can be determined using an eye-tracking system such as those discussed in relation to Figures 5 and 22. Thus, the image can be divided into a central region corresponding to the gaze position and peripheral regions further away from the gaze position. In some embodiments, a foveal map is created based on the gaze position, with portions of the image closer to the gaze position mapped to a high-quality setting and portions of the image further away from the gaze position mapped to a low-quality setting. In Figure 11, the foveal map takes the form of two peripheral regions with lower quality settings and a central region with a higher (e.g., 100%) quality setting.

[0082]

[0105] In the image shown in Figure 11, region 1110, corresponding to the left quarter of the image (i.e., left 1 / 4), is compressed using a first quality setting. Additionally, region 1130, corresponding to the right quarter of the image (i.e., right 1 / 4), is compressed using a first quality setting. However, region 1120, corresponding to the middle half of the image (i.e., central 2 / 4), is compressed using a second quality setting, which is higher than the first quality setting. This division of the image into parts can be referred to as a three-region division: the left quarter (e.g., fovealed with a 70% quality setting), the middle half (e.g., non-fovealed with a 100% quality setting), and the right quarter (e.g., fovealed with a 70% quality setting).

[0083]

[0106] Figure 11 shows the division into three regions using a foveal map containing these three regions, but the present invention is not limited to this implementation, and images can be divided in other ways. By dividing an image into multiple regions, the quality setting of individual blocks or tiles (e.g., 8x8 pixel blocks for JPEG compression) contained in each region can be set to a predetermined quality setting for each block. Thus, in Figure 11, the same quality setting is assigned to all blocks within each region, i.e., the blocks in region 1110 are assigned a first quality setting (e.g., 70%), the blocks in region 1120 are assigned a second quality setting (e.g., 100%), and the blocks in region 1130 are assigned a first quality setting (e.g., 70%), but this is not mandatory, and different quality settings can be assigned to individual blocks within a region. Thus, the foveal map can be more complex than the three-region division shown in Figure 11. In some embodiments, blocks in the peripheral regions are assigned a quality setting that depends on the distance of the block from the line of sight, while blocks in the central region have a uniform quality setting. In other embodiments, the foveal map can be defined such that blocks in the peripheral region are assigned a uniform quality setting, while blocks in the central region are assigned a quality setting that depends on the distance of the block from the line of sight. Those skilled in the art will recognize many variations, modifications, and substitutions.

[0084]

[0107] In the three-region fovealization image shown in Figure 11, an overall reduction of approximately 67% in image / memory size was achieved while maintaining 100% quality in region 1120, i.e., the non-fovealized section. As discussed above, the non-fovealized region (i.e., compressed using an uncompressed or lossless compression algorithm) can be any region identified in the foveal map. Consequently, the three-region division shown in Figure 11 is merely illustrative.

[0085]

[0108] It should be noted that if the gaze position is, for example, on the right side of the image, the foveal map can compress the right side using a higher quality setting and the left side of the image using a lower quality setting. Therefore, in this example, if the gaze position is within region 1130, regions 1110 and 1120 are compressed using a first quality setting, and region 1130 is compressed using a second quality setting higher than the first. In some embodiments, for example, if the gaze position is within region 1130, region 1130 can be compressed using a higher quality setting, e.g., lossless compression; region 1120 can be compressed using an intermediate quality setting lower than the higher quality setting; and region 1110 can be compressed using a minimum quality setting lower than the intermediate quality setting. As a result, the fovea of ​​the image is a function of the gaze position, compressing or encoding the region containing the gaze position with a higher quality setting than one or more regions further away from the gaze position. Those skilled in the art will recognize many variations, modifications, and substitutions.

[0086]

[0109] Furthermore, although Figure 11 shows a set of vertical regions, this is not essential to embodiments of the present invention, and the definition of regions can be carried out in other ways, including horizontally oriented regions, regions defined based on the distance to the line of sight, for example, a set of regions defined radially.

[0087]

[0110] Figure 12 is a foveated 3D generated image having three foveated regions according to yet another embodiment of the present invention. In Figure 12, the regions are defined similarly to those shown in Figure 11. However, in 3D generated images, since the majority of the image is black, much higher compression can be achieved. Using the method described herein, an 87% compression was achieved while maintaining 100% quality at the center of the image corresponding to the gaze position. In this example, region 1220 was compressed using a 100% quality setting (non-foveated at 100% quality setting), while regions 1210 and 1230 were compressed with a lower quality setting (foveated at 20% quality setting). Since in many examples of virtual content the image content is highest near the gaze position and the surrounding regions are dark or black, embodiments of the present invention are particularly well suited for use in virtual reality and augmented reality implementations.

[0088]

[0111] In some examples, all areas of an image can be compressed using a lower quality setting, while non-foveated areas can be compressed using a higher quality setting. Using the example in Figure 12, areas 1210, 1220, and 1230 can be compressed, respectively, using a lower quality setting for the foveated areas. Area 1220 can also be compressed using a high quality setting. When decoding a compressed image (for example, for reconstruction for display to a user), it may be desirable to decode sections of the image in parallel. Therefore, two decoders can be used to decode a compressed image. During image reconstruction, the decoded area 1220 using the high quality setting can be superimposed on the decoded areas 1210, 1220, and 1230 (i.e., the entire image) using the lower quality setting. The encoding may be JPEG (for example, using the quality settings described above), or it may be a technique including DSC or VDC-X (for example, using compression ratios), which are discussed more fully herein.

[0089]

[0112] Figure 13 is a diagram showing an image that can be used with multiple foveal maps according to one embodiment of the present invention. Figure 13 shows an image that includes a person 1306 located in section 1310, a tree 1302 located in sections 1320, 1322, 1330, and 1332, and a house 1304 located in sections 1324, 1326, 1338, and 1340. Different foveal maps can be created based on this image depending on the line of sight.

[0090]

[0113] If the user's gaze position is located in one of sections 1320, 1322, 1330, or 1332, i.e., if the user is looking at tree 1302, a foveal map can be used, and blocks in sections 1320, 1322, 1330, and 1332 are compressed using a 100% quality setting (non-fovealed at 100% quality), while blocks in the remaining sections (i.e., sections 1310, 1312, 1314, 1316, 1324, 1326, 1328, 1334, 1336, 1338, 1340, and 1342) are compressed using a lower quality setting (fovealed at 70% quality). Thus, image compression can be implemented using a foveal map that maintains quality within the region of the image corresponding to the gaze position, while peripheral parts of the image can be compressed using a lower quality setting to save system resources, including memory and processing.

[0091]

[0114] Alternatively, if the user's gaze position is in one of sections 1324, 1326, 1338, or 1340, i.e., if the user is looking at house 1304, the foveal map can be utilized, and the blocks in sections 1324, 1326, 1338, and 1340 are compressed using a 100% quality setting (non-fovealed at 100% quality), while the blocks in the remaining sections (i.e., sections 1310, 1312, 1314, 1316, 1320, 1322, 1328, 1330, 1332, 1334, and 1336, as well as 1342) are compressed using a lower quality setting (fovealed at 70% quality).

[0092]

[0115] Finally, when the user's gaze position is in section 1310, i.e., when the user is looking at person 1306, the foveal map can be utilized, and the block in section 1310 is compressed using a 100% quality setting (non-fovealed at 100% quality setting), while the blocks in the remaining sections (i.e., sections 1312, 1314, 1316, 1320, 1322, 1324, 1326, 1328, 1330, 1332, 1334, and 1336, 1338, 1340, and 1342) are compressed using a lower quality setting (fovealed at 70% quality setting). In some embodiments, the quality setting used for the remaining sections varies, for example, as a function of distance from the gaze position. In these embodiments, the blocks in sections 1312, 1314, and 1316 are compressed at 90% The images can be compressed using the following quality settings: sections 1320, 1322, 1324, 1326, and 1328 can be compressed using the 80% quality setting, while sections 1330, 1332, 1334, and 1336, 1338, 1340, and 1342 can be compressed using the 70% quality setting. In some examples, instead of encoding with JPEG (e.g., using the quality settings described above), sections 1310-1342 may be compressed using techniques including DSC or VDC-X (e.g., using a compression ratio). For example, based on the gaze position, non-tile-based compression techniques such as DSC can be used to compress sections closer to the gaze position at a lower compression ratio, while sections further away from the gaze position can be compressed at a higher compression ratio.

[0093]

[0116] Figure 14 is a simplified flowchart illustrating a method for compressing an image according to one embodiment of the present invention. Method 1400 includes receiving an image (1410), determining the user's gaze position (1412), and generating a foveal map based on the gaze position (1414).

[0094]

[0117] The image may be an image contained within a video stream. Determining the user's gaze position can be achieved by utilizing an eye-tracking system that provides the gaze position as a function of time. The foveal map defines the quality at which blocks are compressed, which varies as a function of their position in the image; blocks in regions closer to the gaze position are compressed using a higher quality setting, and blocks in regions further away from the gaze position are compressed using a lower quality setting. In the example shown in Figure 11, the foveal map includes three regions, but the present invention is not limited to this particular implementation, and two or more regions can be defined. Furthermore, blocks within a given region can be compressed using a uniform quality setting, or they can be compressed with different quality settings depending on the particular implementation. In some embodiments, the foveal map includes a first region of the image and a second region of the image.

[0095]

[0118] The method also includes compressing a first region of an image using a first quality setting and compressing a second region of the image using a second quality setting (1416). In some embodiments, the first quality setting is an uncompressed quality setting or a lossless compression quality setting. Thus, blocks within the first region are compressed with a higher quality than other parts of the image. The second quality setting is a lower quality setting, e.g., a 70% quality setting, which reduces the data corresponding to the compressed image within these regions. As discussed above, since the user's line of sight places these regions in the user's peripheral vision, any loss of quality is offset by savings in memory and processor usage. The data compression processes for the first and second regions can be performed sequentially or in parallel, depending on the specific application.

[0096]

[0119] A compressed image or video, which may be referred to as a foveal image or video, can be transmitted to a display system along with the foveal map (1418), or stored in memory along with the foveal map (1419).

[0097]

[0120] In embodiments in which a compressed image or video is stored in memory together with a foveal map, method 1400 includes retrieving the fovealized image and foveal map from memory (1420), decompressing a first region of the image using a first quality setting, and decompressing a second region of the image using a second quality setting (1440). In embodiments in which a compressed image or video is transmitted to a display system together with a foveal map, method 1400 includes receiving the fovealized image and foveal map (1420), decompressing a first region of the image using a first quality setting, and decompressing a second region of the image using a second quality setting (1440). The decompression processes for the first and second regions can be performed sequentially or in parallel, depending on the specific application. The two regions can be merged to form a final image suitable for display (1442). The final image is then displayed on a display device (1444).

[0098]

[0121] It should be noted that the specific steps shown in Figure 14 provide a particular method for compressing an image according to one embodiment of the present invention. Other sets of steps may be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Furthermore, the individual steps shown in Figure 14 may include a plurality of substeps that can be performed in various sequences as appropriate to the individual steps. Additionally, additional steps may be added or removed depending on the specific application. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0099]

[0122] Referring again to Figure 10, low-power, non-depth-based warp reprojection can be performed by a warp reprojection processor 1040, which can be implemented as a custom 6DOF or 3DOF logic device, i.e., a custom ASIC. The warp reprojection processor 1040 can utilize image data provided after the image decompression process 1038, which uses data read from memory 1032 (e.g., secondary intermediate system memory) or, as indicated by the data path 1026, from color map memory 1022 (i.e., the original color map buffer location). Using the warp reprojection processor 1040 to perform the reprojection reduces system power consumption compared to using a GPU to perform the reprojection.

[0100]

[0123] Although the embodiments described above utilize a tile-based (also called block-based) JPEG compression algorithm, embodiments of the present invention are not limited to this particular compression standard, and other compression standards can be used in conjunction with various embodiments of the present invention. As an example, Figures 15 to 21 describe a technique that uses run-length coding with DSC and VDC-X to compress video data.

[0101]

[0124] Figure 15 shows the compression levels obtained as a function of time, expressed by consecutive frames versus frequency, for both a sparse compression system implementation and a DSC-SPARSE system implementation according to one embodiment of the present invention. In Figure 15, each frame was compressed using either a mask-based compression method or DSC, according to alternating algorithms that implement either a mask-based compression method or fully fixed-frame compression, such as DSC.

[0102]

[0125] As shown in Figure 15, each frame is analyzed to determine the number of lines with pixels that have a brightness level below a threshold. If the mask-based compression method results in a compression level greater than the compression threshold (e.g., 37%), the frame is compressed using the mask-based compression method. In Figure 15, this results in the first approximately 3800 frames being compressed using the mask-based compression method.

[0103]

[0126] The DSC method is used when a mask-based compression method produces compressed frames with a compression level of less than 37%, such as frames with little black content. This results in these frames having a compression value of 37%. Referring to Figure 15, frames represented by blue compression values ​​of less than 37% are compressed using DSC, effectively baseline the minimum compression at 37%. Therefore, the frames in sets A and B have a compression value of 37%, rather than the lower values ​​achieved using the mask-based compression method.

[0104]

[0127] Figure 16 shows a histogram of frame count versus compression for a sparse compression system implementation and a DSC-SPARSE system implementation according to one embodiment of the present invention. As shown in Figure 16, the number of frames with less than approximately 37% compression is reduced to zero because the mask-based compression method was used for frames that could be compressed to a compression level greater than 37%, or the frame-based compression method (e.g., DSC) was used for the remaining frames that could not be compressed to a compression level greater than 37% using the mask-based compression method. Thus, the mask-based compression method operating alone produced some frames with a compression level of less than 37%, but the alternating method provided by embodiments of the present invention limits the minimum compression level to approximately 37%, as shown in Figure 16. For frames with significant black pixel content, the mask-based compression method provides a high level of compression, but for frames with limited black pixel content, the frame-based compression method establishes a lower limit of compression level, for example 37% in this illustrated embodiment. As will be apparent to those skilled in the art, the minimum compression level does not have to be 37%, which is merely illustrative, and other minimum compression levels can be utilized depending on the specific application. Those skilled in the art will recognize many variations, modifications, and substitutions.

[0105]

[0128] Information about the compression method used for each frame can be provided to the endpoint, such as a decoder or display, so that the endpoint can utilize the appropriate decompression method when reconstructing each frame.

[0106]

[0129] Figure 17 is a simplified flowchart illustrating a method for compressing an image frame using an alternating compression algorithm according to one embodiment of the present invention. Method 1700 includes receiving a frame of video data (1710). The method also includes determining the number of lines in the frame that have pixel groups characterized by luminance levels below a threshold (1712).

[0107]

[0130] If the number of lines is greater than or equal to the compression threshold (1714), the frame is compressed using a mask-based compression method (1720). If the number of lines is less than the compression threshold, the frame is compressed using a frame-based compression method (1722). If additional frames exist (1730), the method operates on the next frame of video data by receiving the frame of video data (1710). Otherwise, the method terminates (1740). Thus, embodiments of the present invention alternate between compressing each frame using the respective compression methods, depending on the level of compression that can be achieved by each compression method.

[0108]

[0131] It should be noted that the specific steps shown in Figure 17 provide a particular method for compressing image frames using an alternating compression algorithm according to one embodiment of the present invention. Other sets of steps may be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Furthermore, the individual steps shown in Figure 17 may include a plurality of substeps that can be performed in various sequences as appropriate to the individual steps. Additionally, additional steps may be added or removed depending on the specific application. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0109]

[0132] According to some embodiments of the present invention, for each frame, there is an embedded image line control or alternative control mechanism that provides the endpoint display with information on which system should be used to decode the incoming MIPI frame. In addition, a virtual MIPI channel can be used to indicate the compression ratio used by the endpoint display.

[0110]

[0133] Some embodiments of the present invention modify the compression quality based on target tracking, thereby reducing the quality in the foveal region to give a higher compression ratio. This is done for the MIPI interface, thereby reducing the amount of data transmitted to the LCOS / uLED display via MIPI. Thus, the embodiments also result in power saving.

[0111]

[0134] Embodiments of the present invention reduce the amount of stream-based data transmitted via MIPI compression. Furthermore, embodiments modify the compression quality based on target tracking, thereby providing a higher compression ratio to the foveal region where quality is reduced. Moreover, embodiments enable a higher compression ratio for stream-based compression techniques while maintaining quality in the area observed by the user. As a result, embodiments enable a much higher compression ratio while maintaining quality.

[0112]

[0135] For stream-based compression standards such as DSC and VESA display compression (VDC-X), low-latency implementations are utilized. This low-latency response is used so that any previous spatial warp adjustments performed remain applicable.

[0113]

[0136] Figure 18 is a simplified image showing an image frame divided into a high-quality region and a low-quality region according to one embodiment of the present invention. The image 1800 shown in Figure 18 includes a high-quality region 1810 and a low-quality region 1820. As will be discussed in more detail below, the high-quality region 1810 is compressed and decompressed using a first quality setting or compression level, and the low-quality region 1820 or the entire image is compressed and decompressed using a second quality setting or compression level, providing memory savings and other advantages. As an example, a single decoder can be utilized by not compressing the high-quality region 1810 and compressing the low-quality region using a single decoder. Significant savings can be achieved if the high-quality region 1810 is small compared to the entire image. An additional explanation regarding resizing the high-quality region is provided in U.S. Provisional Patent Application No. 63 / 543,876, filed October 12, 2023, the disclosure of which is incorporated herein by reference in its entirety for all purposes.

[0114]

[0137] DSC

[0138] Conventional DSCs do not offer variable quality compression. Rather, DSCs take 24-bit color coding and compress it to 15 / 12 / 10 / 8 bits. The higher the compression (24→8bpp), the worse the impact on quality. With respect to the quality required for the section the eye is focusing on, embodiments can maintain PSNR quality settings above 60 dB, as discussed above. From the use case analysis shown in Figure 6, the inventors determined that this occurs only at a 37% compression configuration (24→15bpp). However, only the area the eye is currently focusing on actually utilizes that compression setting. The lateral foveal region (e.g., the part of the image further from the line of sight) can have lower quality, for example, a 75% compression level (24→8bpp).

[0115]

[0139] Therefore, in neighborhood-based compression standards such as DSC, which lack the concept of tiles, embodiments divide the main screen into a high-quality region and a low-quality region (as shown in Figure 18) or smaller sections (as shown in Figure 20) each having a different compression ratio. The selected compression ratio is a function of the current viewing position. Thus, referring to Figure 18, where the viewing position is located inside the high-quality region 1810, the high-quality region 1810 can be compressed at a lower compression level (e.g., 24 → 15 bpp), and the low-quality region 1820 can be compressed at a higher compression level (e.g., 24 → 8 bpp). In some examples, the low-quality region 1820 can be compressed at an even higher compression level (e.g., 24 → 6 bpp). In embodiments where the entire image is compressed using a higher compression level, as will be described more fully herein, the high-quality region 1810 can be superimposed on the entire image when the image is reconstructed.

[0116]

[0140] Figure 19 is a simplified flowchart illustrating a method 1900 for compressing an image using different compression ratios for high-quality and low-quality regions, according to one embodiment of the present invention. Method 1900 includes determining the user's line of sight (1910), generating a foveal map containing a first region of the image and a second region of the image (1912), and compressing the first region using a first compression ratio and the second region using a second compression ratio (1914).

[0117]

[0141] The image may be an image contained within a video stream. Determining the user's gaze position can be achieved by utilizing an eye-tracking system that provides the gaze position as a function of time. The foveal map defines the compression ratio to which portions of the image are compressed, which varies as a function of the position in the image relative to the gaze position, with areas(s) closer to the gaze position being compressed using a lower compression ratio and areas(s) further away from the gaze position being compressed using a higher compression ratio. In the example shown in Figure 18, the foveal map contains two regions, but the present invention is not limited to this particular implementation, and three or more regions can be defined. In some embodiments, the foveal map includes a first region of the image and a second region of the image. Method 1900 may be referred to as N-directional compression (e.g., DSC, VDC-X, or JPEG), where N refers to the number of regions determined for the image. For example, based on the gaze position, high-quality regions, medium-quality regions surrounding the high-quality regions, and low-quality regions can be determined for the image. The technique of Method 1900 can then be used as three-directional compression with different compression ratios for each region.

[0118]

[0142] Referring back to Figure 18, in some examples, the low-quality region 1820 may encompass the entire image, including the portion of the image within the high-quality region 1810 characterized by the gaze position. When decoding a compressed image (for example, for reconstruction for display to a user), it may be desirable to decode sections of the image in parallel. For an image divided into a high-quality region 1810 and a low-quality region 1820, as in Figure 18, the low-quality region 1820 may be considered the entire image. For example, in the case of a 2-kilopixel × 2-kilopixel image (4 megapixels total), the low-quality region 1820 may be the entire 4-megapixel image and may be compressed using a high compression level (e.g., 24 → 8 bpp). The high-quality region 1810 may be determined based on the current gaze position and may be, for example, a 1-kilopixel × 1-kilopixel region (1 megapixel total). The high-quality region 1810 can be compressed using a low compression level (e.g., 24 → 15 bpp). Therefore, two DSC decoders can be used to decode a compressed image. During image reconstruction, the decoded high-quality regions can be overlaid on the decoded low-quality regions.

[0119]

[0143] Figure 20 is a simplified image showing an image frame divided into high-quality and low-quality sections according to one embodiment of the present invention. As will be discussed in more detail below, the divided image frame 2000 shown in Figure 20 can be used to define a foveal map that defines the compression ratio at which different sections of the image are compressed, such that the compression ratio or other compression quality metric varies as a function of the position in the image relative to the gaze position. For example, sections closer to the gaze position can be compressed using a lower compression ratio, and sections further from the gaze position can be compressed using a higher compression ratio.

[0120]

[0144] Referring to Figure 20, four sections 2010, 2012, 2014, and 2016, including the high-quality region 2002 (i.e., the region corresponding to the current gaze position), are compressed at a lower compression level (e.g., 24 → 15 bpp), while the remaining sections, which may be referred to as peripheral sections or low-quality sections, are compressed at a higher compression level (e.g., 24 → 8 bpp). As a result, when the compressed image is reconstructed for display to the user, the high-quality region corresponding to the gaze position is characterized by higher quality than the rest of the image further from the gaze position. Consequently, embodiments of the present invention provide gaze-position-based fovealed images with reduced storage and transmission requirements.

[0121]

[0145] In some embodiments of the example shown in Figure 20, all sections 2010–2046 of the image may be compressed at a high compression ratio (e.g., 24 to 8 bpp). Four sections 2010, 2012, 2014, and 2016 containing high-quality regions may also be compressed at a lower compression ratio (e.g., 24 to 15 bpp). By using a decoder, all sections 2010–2046 compressed at a high compression ratio can be decoded according to a higher compression ratio, and the four sections 2010, 2012, 2014, and 2016 compressed at a lower compression ratio can be decoded according to a lower compression ratio. During image reconstruction, the decoded high-quality sections 2010, 2012, 2014, and 2016 can be superimposed on the decoded low-quality sections 2010–2046. In some embodiments, a foveal map can define sections that coincide with high-quality regions. For example, sections 2010–2016 may include only high-quality regions characterized by gaze position, excluding portions of images within low-quality regions.

[0122]

[0146] Similar to N-direction compression, it may be desirable to decode a compressed image using section-based DSC techniques with multiple DSC decoders. For example, a compressed image can be decoded using four DSC decoders: one decoder used to decode high-quality sections 2010-2016, another used to decode sections 2020-2026, a third used to decode sections 2030-2036, and a fourth used to decode sections 2040-2046, with each decoder using a compression ratio for each group of sections based on its proximity to the viewing position. In some embodiments, a single decoder can be implemented with acceptable latency when decoding a compressed image, depending on the memory capacity (e.g., SRAM) of the system used for decoding.

[0123]

[0147] The image may be an image contained within a video stream. Determining the user's gaze position can be achieved by utilizing an eye-tracking system that provides the gaze position as a function of time. The foveal map defines the compression ratios to which different sections of the image (e.g., sections 2010-2016, 2020-2026, 2030-2036, and 2040-2046) are compressed, which vary as a function of their position in the image relative to the gaze position, with sections closer to the gaze position being compressed using lower compression ratios and sections further away from the gaze position being compressed using higher compression ratios. In the example shown in Figure 20, the foveal map contains 16 sections, but the present invention is not limited to this particular implementation, and more or fewer sections may be defined. The methods described herein may be referred to as section-based compression (e.g., DSC, VDC-X, or JPEG) methods.

[0124]

[0148] While some of the examples above show only two compression levels, embodiments of the present invention are not limited to these specific compression levels, and an additional number of compression levels can be utilized. For example, sections 2010–2014 can be compressed using a 37% compression level (i.e., 24→15bpp), while sections 2020, 2022, 2024, and 2026, which are further from the high-quality region, can be compressed using a 50% compression level (i.e., 24→15bpp), sections 2030, 2032, 2034, and 2036, which are even further from the high-quality region than sections 2020–2026, can be compressed using a 58% compression level (i.e., 24→12bpp), and sections 2040, 2042, 2044, and 2046, which are the furthest from the high-quality region than sections 2010–2016, can be compressed using a 67% compression level (i.e., 24→8bpp). Therefore, the use of two compression levels is merely illustrative. Furthermore, for some sections, the compression level may be 0%, i.e., uncompressed, including sections corresponding to the viewing position and high-quality areas. Thus, a compressed image may have both uncompressed and compressed sections. Those skilled in the art will recognize many variations, modifications, and substitutions.

[0125]

[0149] Furthermore, although Figure 20 shows only 16 sections of uniform area, this is not mandatory, and other numbers of sections with different sizes can be used, with smaller sections adjacent to the high-quality areas, and larger sections, such as those compressed at a higher level, being further away from the high-quality areas. Thus, the number of compression levels, the compression levels, the number of sections, and the size of the sections can vary depending on the specific application. Those skilled in the art will recognize many variations, modifications, and substitutions.

[0126]

[0150] When image compression reduces the frame size, the communication interface, such as the MIPI interface, can be changed to enter a low-power data transmission mode, or even an ultra-low-power sleep mode, thereby saving computing resources and reducing power consumption. At the endpoint, the reconstruction of the compressed image can be performed before it is displayed to the user.

[0127]

[0151] Figure 21 is a simplified flowchart illustrating a method 2100 for compressing an image using different compression ratios for high-quality and low-quality sections, according to one embodiment of the present invention. Method 2100 includes determining the user's line of sight (2110), generating a foveal map containing a first section of the image and a second section of the image (2112), and compressing the first region using a first compression ratio and the second region using a second compression ratio (2114).

[0128]

[0152] It should be recognized that the specific steps shown in Figures 19 and 21 provide a specific method for compressing an image according to one embodiment of the present invention. Other sets of steps may be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Furthermore, the individual steps shown in Figures 19 and 21 may include a plurality of substeps that can be performed in various sequences as appropriate to the individual steps. Additionally, additional steps may be added or removed depending on the specific application. Those skilled in the art will recognize many variations, modifications, and alternatives.

[0129]

[0153] VDC-X

[0154] The VDC-X compression standard (e.g., VDC-M) uses a tile-based method instead of the nearest neighbor method. This compression standard encodes different tiles with different quality settings, but the goal of this conventional compression is to maintain a constant frame size (i.e., bitrate) overall. Therefore, the compression ratio, when selected, varies for each tile to maintain a constant bitrate. Using this compression standard in conjunction with embodiments of the present invention, video images are compressed based on the user's gaze position, not solely on the bitrate. As an example, four sections 2010, 2012, 2014, and 2016, which contain high-quality areas (i.e., areas corresponding to the current gaze position), are compressed with a higher quality setting than the remaining sections, which can be called peripheral sections, and the remaining sections are compressed with a lower quality setting than those used for sections 2010-2016.

[0130]

[0155] In some embodiments of the present invention, since a constant bitrate is not maintained, the size of each frame changes over time, and the transport interface, such as MIPI, enters a low-power mode when not in use.

[0131]

[0156] Similar to the DSC-based method discussed above, in the VDC-X tile-based method, the embodiment encodes the quality of each tile based on the current position of the user's gaze. As shown in Figure 20, using gaze information provided by the AR system's gaze tracking system, the tiles are compressed using the VDC-X standard as a function of the distance of the tile from the gaze position.

[0132]

[0157] Therefore, embodiments of the present invention allow for variations in frame size or per-frame bitrate, and use the current line-of-sight information to select which tiles (VDC-X) or sections (DSC) have higher quality than foveal regions with lower quality settings.

[0133]

[0158] In some embodiments, the N-directional compression or section-based compression described above can implement JPEG as a compression standard rather than DSC or VDC-X. In these embodiments, the compression ratio used for high-quality / low-quality regions and / or high-quality / low-quality sections can instead refer to the quality settings of the JPEG standard.

[0134]

[0159] Figure 22 is a simplified block diagram showing the components of an AR system according to one embodiment of the present invention. The AR system 2200 shown in Figure 22 can be incorporated into an AR device as described herein. Figure 22 provides a schematic diagram of one embodiment of the AR system 2200 that can carry out some or all of the steps of the method provided by various embodiments. It should be noted that Figure 22 is intended only to provide a generalized description of the various components, some or all of which may be appropriately utilized. Thus, Figure 22 broadly illustrates how individual system elements may be implemented in a relatively isolated or relatively more integrated manner.

[0135]

[0160] The AR system 2200 is shown as comprising hardware elements that can be electrically coupled via bus 2205 or otherwise communicate as needed. The hardware elements may include, but are not limited to, one or more processors 2210, including one or more general-purpose processors and / or one or more dedicated processors such as digital signal processing chips, graphics accelerators, and / or the like; one or more input devices 2215, which may include, but are not limited to, a mouse, keyboard, camera, and / or the like; and one or more output devices 2220, which may include, but are not limited to, a display device, printer, and / or the like. Additionally, the AR system 2200 includes an eye-tracking system 2255 that can provide the AR system with the user's gaze position. The foveal image compression techniques considered herein can be implemented using the processor 2210.

[0136]

[0161] The AR system 2200 may further include, but is not limited to, one or more non-temporary storage devices 2225 that may include local and / or network-accessible storage and / or be able to communicate with them, and / or may include, but is not limited to, solid-state storage devices such as random-access memory (RAM) and / or read-only memory (ROM), which may be disk drives, drive arrays, optical storage devices, programmable, flash-updatable and / or similar. Such storage devices may be configured to implement any suitable data store, including, but is not limited to, various file systems, database structures and / or similar.

[0137]

[0162] The AR system 2200 may also include a communication subsystem 2219 which may include, but is not limited to, a chipset that includes a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and / or a Bluetooth® device, an 802.11 device, a WiFi device, a WiMAX device, a cellular communication equipment, and / or similar. The communication subsystem 2219 may include one or more input and / or output communication interfaces to enable exchange of data with a network, such as, to give an example, another computer system, a television, and / or any other device described herein. Depending on the desired functionality and / or other implementation concerns, a portable electronic device or similar device may communicate images and / or other information via the communication subsystem 2219. In other embodiments, a portable electronic device, e.g., a first electronic device, may be incorporated into the AR system 2200, e.g., an electronic device as an input device 2215. In some embodiments, the AR system 2200 further includes a working memory 2260 which may include a RAM or ROM device, as described above.

[0138]

[0163] The AR system 2200 may also include software elements, shown as currently located in working memory 2260, including an operating system 2262, device drivers, executable libraries, and / or computer programs provided by various embodiments, and / or other code such as one or more application programs 2264 that may be designed to implement and / or configure the system as described herein, in accordance with the methods provided by other embodiments. As mere examples, one or more procedures described with respect to the methods considered above may be implemented as code and / or instructions executable by a computer and / or a processor within a computer. In one embodiment, such code and / or instructions may be used to configure and / or adapt a general-purpose computer or other device to perform one or more operations in accordance with the described methods.

[0139]

[0164] These instructions and / or sets of code can be stored in a non-temporary computer-readable storage medium such as the storage device(s) 2225 described above. In some cases, the storage medium may be incorporated into a computer system such as the AR system 2200. In other embodiments, the storage medium may be separate from the computer system, and may be a removable medium such as a compact disk, and / or provided in an installation package, so that the storage medium can be used to program, configure, and / or adapt a general-purpose computer with the stored instructions / code. These instructions may take the form of executable code that can be executed by the AR system 2200, and / or in the form of source and / or installable code, which takes the form of executable code when compiled and / or installed on the AR system 2200 using, for example, one of various commonly available compilers, installers, compression / decompression utilities, etc.

[0140]

[0165] It will be apparent to those skilled in the art that substantial modifications can be made according to specific requirements. For example, customized hardware may be used, and / or certain elements may be implemented in hardware, software including portable software such as applets, or both. Furthermore, connections to other computing devices, such as network input / output devices, may be used.

[0141]

[0166] As described above, in one embodiment, several embodiments can be implemented using a computer system such as AR system 2200 to carry out methods according to various embodiments of the Art. According to a set of embodiments, some or all of the steps of such methods are carried out by AR system 2200 in response to a processor 2210 that executes one or more sequences of one or more instructions that may be incorporated into an operating system 2262 and / or other code such as an application program 2264, which are contained in working memory 2260. Such instructions may be read into working memory 2260 from another computer-readable medium, such as one or more of storage devices 2225. As just one example, the execution of a sequence of instructions contained in working memory 2260 can cause the processor 2210 to carry out one or more steps of the methods described herein. Additionally or alternatively, some of the methods described herein may be carried out via dedicated hardware.

[0142]

[0167] As used herein, the terms machine-readable medium and computer-readable medium refer to any medium involved in providing data that causes a machine to operate in a particular way. In embodiments implemented using the AR system 2200, various computer-readable media may be involved in providing instructions / code to the processor(s) 2210 for execution, and / or may be used to store and / or carry such instructions / code. In many implementations, the computer-readable medium is a physical and / or tangible storage medium. Such media may take the form of non-volatile or volatile media. Non-volatile media include, for example, optical and / or magnetic disks such as the storage device(s) 2225. Volatile media include, but are not limited to, dynamic memory such as the working memory 2260.

[0143]

[0168] Common forms of physical and / or tangible computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, or any other magnetic media, CD-ROMs, any other optical media, punch cards, paper tapes, any other physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, any other memory chips or cartridges, or any other media from which a computer can read instructions and / or code.

[0144]

[0169] Various forms of computer-readable media may be involved in transporting one or more sequences of one or more instructions to the processor(s) 2210 for execution. As just one example, the instructions may first be transported onto a magnetic disk and / or optical disk of a remote computer. The remote computer may load the instructions into its dynamic memory and transmit the instructions as signals over a transmission medium to be received and / or executed by the AR system 2200.

[0145]

[0170] The communication subsystem 2219 and / or its components generally receive signals, and the bus 2205 may then transport the signals and / or the data, instructions, etc. carried by the signals to the working memory 2260, from which the processor(s) 2210 retrieves and executes the instructions. Instructions received by the working memory 2260 may optionally be stored in a non-temporary storage device 2225 either before or after execution by the processor(s) 2210.

[0146]

[0171] Various embodiments of the present disclosure are provided below. When used herein, any reference to a set of embodiments should be understood as a disjunctive reference to each of those embodiments (for example, “Embodiments 1-4” should be understood as “Embodiments 1, 2, 3, or 4”).

[0147]

[0172] Example 1 is a method for generating a reprojected image, comprising: receiving motion data; determining, based on the motion data, whether a motion threshold has been exceeded; and, if the motion threshold has been exceeded, generating a depth-based reprojection, or, if the motion threshold has not been exceeded, generating a non-depth-based reprojection.

[0148]

[0173] Example 2 is the method of Example 1, further comprising determining whether a time threshold has been exceeded based on motion data, and generating a non-depth-based reprojection if the time threshold has not been exceeded.

[0149]

[0174] Example 3 is a method of Examples 1-2, further comprising determining whether a time threshold has been exceeded based on motion data, and displaying depth-based reprojection if both the motion threshold and the time threshold have been exceeded.

[0150]

[0175] Example 4 describes the case where the motion threshold and time threshold are exceeded. Storing depth-based reprojection in memory, Searching for depth-based reprojections from memory, Generating non-depth-based reprojections based on depth-based reprojections, Displaying non-depth-based reprojections, The method described in Examples 1 to 3 further includes the above.

[0151]

[0176] Example 5 is a method of Examples 1-4, which includes the use of a color map, to generate a non-depth-based reprojection based on a depth-based reprojection.

[0152]

[0177] Example 6 is a method of Examples 1 to 5, further comprising generating a non-depth-based reprojection after generating a depth-based reprojection.

[0153]

[0178] Example 7 is a method of Examples 1-6 in which generating depth-based reprojection involves the use of depth maps and color maps.

[0154]

[0179] Example 8 is a method of Examples 1-7, which involves the use of a color map to generate a non-depth-based reprojection.

[0155]

[0180] Example 9 is the method according to Examples 1 to 8, further comprising performing depth-based reprojection foveal compression.

[0156]

[0181] Example 10 is the method of Examples 1 to 9, comprising: performing depth-based reprojection foveal compression, determining the user's gaze position, generating a foveal map based on the gaze position, wherein the foveal map includes a first region of depth-based reprojection and a second region of depth-based reprojection, and compressing the first region using a first quality setting and compressing the second region using a second quality setting.

[0157]

[0182] Example 11 is the method according to Examples 1 to 10, wherein determining the gaze position involves the use of an eye-tracking camera of an augmented reality device.

[0158]

[0183] Example 12 is the method according to Examples 1 to 11, wherein the foveal map includes a central region and a peripheral region.

[0159]

[0184] Example 13 is a method of Examples 1 to 12, wherein depth-based reprojection includes virtual content generated by an augmented reality device.

[0160]

[0185] Example 14 is the same as the method described in Examples 1 to 13, wherein the virtual content is included in the virtual content video stream.

[0161]

[0186] Example 15 is a method of Examples 1 to 14, wherein compressing a first region using a first quality setting includes compressing all blocks within the first region using a first quality setting.

[0162]

[0187] Example 16 is the method described in Examples 1 to 15, wherein the first quality setting is higher than the second quality setting.

[0163]

[0188] Example 17 is the method described in Examples 1 to 16, wherein the first quality setting is 100%.

[0164]

[0189] Example 18 is the method according to Examples 1 to 17, further comprising post-processing image content in at least one of the first or second region.

[0165]

[0190] Example 19 is a method according to Examples 1 to 118, further comprising compressing to generate a compressed image and decoding the compressed image using a foveal map.

[0166]

[0191] Example 20 is the method according to Examples 1 to 10, wherein a first region of the image comprises a plurality of first blocks, a second region of the image comprises a plurality of second blocks, compressing the first region of the image comprises compressing each of the plurality of first blocks using a first quality setting, and compressing the second region of the image comprises compressing each of the plurality of second blocks using a second quality setting.

[0167]

[0192] Example 21 is the method according to Examples 1 to 20, further comprising decompressing a first region of an image using a first quality setting, decompressing a second region of an image using a second quality setting, and displaying the image to the user.

[0168]

[0193] Example 22 is the method according to Examples 1 to 21, wherein the second region of the image includes the first region of the image.

[0169]

[0194] Example 23 is a method according to Examples 1 to 22, further comprising: compressing to generate a compressed image; decoding the compressed image using a foveal map to generate a decoded first region and a decoded second region; and reconstructing the image by overlaying the decoded first region on top of the decoded second region.

[0170]

[0195] Embodiment 24 is a system comprising: a motion data unit; a controller coupled to the motion data unit; a memory operable to store depth maps and color maps; a first processor coupled to the memory; a second memory coupled to the first processor and operable to store images to be reprojected; a second processor coupled to the second memory; and a display coupled to the second processor.

[0171]

[0196] Example 25 is the system described in Example 24, wherein the controller includes a central processing unit (CPU).

[0172]

[0197] Example 26 is the system described in Examples 24-25, wherein the first processor includes a graphics processing unit (GPU).

[0173]

[0198] Example 27 is a system described in Examples 24-26, wherein the first processor includes an application-specific integrated circuit (ASIC).

[0174]

[0199] Example 28 is a system described in Examples 24-27, wherein the second processor includes an application-specific integrated circuit (ASIC).

[0175]

[0200] Example 29 is a system described in Examples 24-28, wherein the motion data unit includes an inertial motion unit.

[0176]

[0201] Example 30 is the system described in Examples 24-29, further comprising a foveal compression unit coupled to a first processor and a second memory.

[0177]

[0202] Example 31 is a system described in Examples 4 to 30, wherein the foveal compression unit is configured to perform foveal compression on the reprojected image to form a fovealed image.

[0178]

[0203] Example 32 is a system according to Examples 24-31, wherein the foveal compression includes determining the user's gaze position, generating a foveal map based on the gaze position, wherein the foveal map includes a first region of the reprojected image and a second region of the reprojected image, and compressing the first region using a first quality setting and compressing the second region using a second quality setting.

[0179]

[0204] Example 33 is a system described in Examples 24-32, in which the first quality setting is higher than the second quality setting.

[0180]

[0205] Example 34 is the system described in Examples 24-33, wherein the first quality setting is 100%.

[0181]

[0206] Example 35 is the system described in Examples 24-32, further comprising a decoder configured to decode a fovealized image using a foveal map.

[0182]

[0207] Example 36 is a system described in Examples 24-35, further comprising an eye-tracking camera for the augmented reality device.

[0183]

[0208] Embodiment 37 is a system comprising: a frame; one or more image capture devices coupled to the frame; a set of eye-tracking devices coupled to the frame; a set of displays coupled to the frame; a set of projectors, each of which is optically coupled to one of the displays; a memory; and a processor coupled to the memory, the processor being configured to receive motion data, determine based on the motion data whether a motion threshold has been exceeded, and generate a depth-based reprojection if the motion threshold has been exceeded, or generate a non-depth-based reprojection if the motion threshold has not been exceeded.

[0184]

[0209] Example 38 is the system described in Example 37, wherein the display set includes a right eyepiece waveguide display and a left eyepiece waveguide display.

[0185]

[0210] Embodiment 39 is a non-temporary computer-readable medium comprising program code executable by the processor of a user-wearable device, wherein the program code is executable by the processor to receive motion data, determine based on the motion data whether a motion threshold has been exceeded, and generate a depth-based reprojection if the motion threshold has been exceeded, or generate a non-depth-based reprojection if the motion threshold has not been exceeded.

[0186]

[0211] In the aforementioned specification, this disclosure is described with reference to specific embodiments thereof. However, it will be apparent that various modifications and changes can be made without departing from the broader spirit and scope of this disclosure. Accordingly, this specification and the drawings should be considered illustrative rather than restrictive.

[0187]

[0212] In fact, it will be recognized that each of the systems and methods of this disclosure has several innovative aspects, and that not one alone alone can assume or be required for the desired attributes disclosed herein. The various features and processes described above may be used independently of each other or combined in various ways. All possible combinations and partial combinations are intended to fall within the scope of this disclosure.

[0188]

[0213] Certain features described herein in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented separately or in any preferred partial combination in multiple embodiments. Furthermore, features described above as acting in a particular combination and initially claimed as such, one or more features from a claimed combination may, in some cases, be removed from the combination, and the claimed combination may cover a partial combination or a variation of a partial combination. Not a single feature or group of features is required or essential in every embodiment.

[0189]

[0214] In particular, conditional statements used herein, such as “can,” “could,” “might,” “may,” and “for example,” should generally be understood, unless otherwise specified or understood in the context in which they are used, to convey that a particular embodiment includes certain features, elements, and / or steps, but other embodiments do not. Therefore, such conditional statements are not generally intended to mean that features, elements, and / or steps are required in some way in one or more embodiments, or that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps are included in any particular embodiment, or should be implemented in a particular embodiment, with or without input or instruction from the author. Terms such as “comprising,” “including,” and “having” are synonymous and are used comprehensively and openly, without prejudice to additional elements, features, actions, behaviors, etc. Furthermore, the term "or" is used in its inclusive sense (rather than its exclusive sense), so for example, when used to connect a list of elements, the term "or" means one, some, or all of the elements in the list. In addition, as used in this application and the attached claims, the articles "a," "an," and "the" should be interpreted as meaning "one or more" or "at least one" unless otherwise specified. Similarly, while actions may be depicted in a particular order in the drawings, it should be recognized that, in order to achieve the desired result, such actions do not need to be performed in a particular order or sequential order shown, or that all shown actions will be performed. Moreover, the drawings may schematically illustrate one or more exemplary processes in the form of flowcharts. However, other actions not depicted may be incorporated into the schematicly illustrated exemplary methods and processes. For example, one or more additional actions may be performed before, after, simultaneously with, or in between any of the illustrated actions.Additionally, the operations may be configured or ordered in other embodiments. In certain circumstances, multitasking and parallel processing may be advantageous. Furthermore, the separation of various system components in the embodiments described above should not be understood as requiring such separation in all embodiments, and the described program components and systems may generally be integrated together in a single software product or packaged in multiple software products. Additionally, other embodiments are within the scope of the following claims. In some cases, the operations described in the claims may be performed in a different order and still achieve the desired results.

[0190]

[0215] Accordingly, the claims are not intended to be limited to the embodiments shown herein, but should be given the broadest scope consistent with this disclosure, the principles and novel features disclosed herein. Accordingly, it should be understood that the examples and embodiments described herein are for illustrative purposes only, and various modifications or changes in light thereof should be suggested to those skilled in the art and should be included within the spirit and scope of this application and the appended claims.

Claims

1. A method for generating an image to be reprojected, Receiving motion data and Based on the aforementioned motion data, it is determined whether the motion threshold has been exceeded. If the aforementioned motion threshold is exceeded, generate a depth-based reprojection, or If the aforementioned motion threshold is not exceeded, a non-depth-based reprojection is generated, Methods that include...

2. Based on the aforementioned motion data, determine whether a time threshold has been exceeded, If the aforementioned time threshold is not exceeded, a non-depth-based reprojection is generated, The method according to claim 1, further comprising:

3. Based on the aforementioned motion data, determine whether a time threshold has been exceeded, When the motion threshold is exceeded and the time threshold is exceeded, the depth-based reprojection is displayed. The method according to claim 1, further comprising:

4. When the motion threshold and the time threshold are exceeded, The depth-based reprojection is stored in memory, Retrieving the depth-based reprojection from the memory, To generate a non-depth-based reprojection based on the aforementioned depth-based reprojection, Displaying the aforementioned non-depth-based reprojection, The method according to claim 3, further comprising:

5. The method according to claim 4, wherein generating a non-depth-based reprojection based on the depth-based reprojection includes the use of a color map.

6. The method according to claim 1, further comprising generating a non-depth-based reprojection after generating the depth-based reprojection.

7. The method according to claim 1, wherein generating the depth-based reprojection includes the use of a depth map and a color map.

8. The method according to claim 1, wherein generating the non-depth-based reprojection includes the use of a color map.

9. The method according to claim 1, further comprising performing foveal compression of the depth-based reprojection.

10. Performing the aforementioned depth-based reprojection foveal compression is Determining the user's line of sight, The method involves generating a foveal map based on the line of sight position, wherein the foveal map includes a first region of the depth-based reprojection and a second region of the depth-based reprojection. Compressing the first region using a first quality setting, and compressing the second region using a second quality setting, The method according to claim 9, including the method described in claim 9.

11. The method according to claim 10, wherein determining the gaze position includes using an eye-tracking camera of an augmented reality device.

12. The method according to claim 10, wherein the foveal map includes a central region and a peripheral region.

13. The method according to claim 10, wherein the depth-based reprojection includes virtual content generated by an augmented reality device.

14. The method according to claim 13, wherein the virtual content is included in a virtual content video stream.

15. The method according to claim 10, wherein compressing the first region using the first quality setting includes compressing all blocks within the first region using the first quality setting.

16. The method according to claim 10, wherein the first quality setting is higher than the second quality setting.

17. The method according to claim 16, wherein the first quality setting is 100%.

18. The method according to claim 10, further comprising post-processing image content in at least one of the first region or the second region.

19. The method according to claim 10, wherein the compression generates a compressed image, and the method further comprises decoding the compressed image using the foveal map.

20. The first region includes a plurality of first blocks, The second region includes a plurality of second blocks, Compressing the first region includes compressing each of the plurality of first blocks using the first quality setting, The method according to claim 10, wherein compressing the second region includes compressing each of the plurality of second blocks using the second quality setting.

21. Decompressing the first region using the first quality setting, Decompressing the second region using the second quality setting, The method according to claim 10, further comprising:

22. The method according to claim 10, wherein the second region includes the first region.

23. Compression generates a compressed image, and the method The compressed image is decoded using the foveal map to generate a decoded first region and a decoded second region. The decoded first region is superimposed on the decoded second region, The method according to claim 22, further comprising:

24. It is a system, Motion data unit and, A controller coupled to the motion data unit, Memory capable of storing depth maps and color maps, A first processor coupled to the memory, A second memory coupled to the first processor and capable of storing the image to be reprojected, A second processor coupled to the second memory, A display coupled to the second processor, A system equipped with these features.

25. The system according to claim 24, wherein the controller comprises a central processing unit (CPU).

26. The system according to claim 24, wherein the first processor includes a graphics processing unit (GPU).

27. The system according to claim 24, wherein the first processor includes an application-specific integrated circuit (ASIC).

28. The system according to claim 24, wherein the second processor includes an application-specific integrated circuit (ASIC).

29. The system according to claim 24, wherein the motion data unit includes an inertial motion unit.

30. The system according to claim 24, further comprising a foveal compression unit coupled to the first processor and the second memory.

31. The system according to claim 30, wherein the foveal compression unit is configured to perform foveal compression on the reprojected image to form a fovealed image.

32. The aforementioned foveal compression, Determining the user's line of sight, The method involves generating a foveal map based on the line of sight position, wherein the foveal map includes a first region of the reprojected image and a second region of the reprojected image. Compressing the first region using a first quality setting, and compressing the second region using a second quality setting, The system according to claim 31, including the following:

33. The system according to claim 32, wherein the first quality setting is higher than the second quality setting.

34. The system according to claim 33, wherein the first quality setting is 100%.

35. The system according to claim 32, further comprising a decoder configured to decode the fovealized image using the foveal map.

36. The system according to claim 24, further comprising an eye-tracking camera for an augmented reality device.

37. It is a system, Frame and, One or more image capture devices coupled to the frame, A set of eye-tracking devices coupled to the aforementioned frame, A set of displays coupled to the frame, A set of projectors, wherein each of the projectors in the set is optically coupled to one of the displays in the set, Memory and A processor coupled to the memory, wherein the processor Receive motion data, Based on the aforementioned motion data, it is determined whether the motion threshold has been exceeded. If the aforementioned motion threshold is exceeded, a depth-based reprojection is generated, or If the aforementioned motion threshold is not exceeded, a non-depth-based reprojection is generated. A processor configured in such a way, A system equipped with these features.

38. The system according to claim 37, wherein the set of displays includes a right eyepiece waveguide display and a left eyepiece waveguide display.

39. A non-temporary computer-readable medium containing program code executable by the processor of a user-installable device, wherein the program code is executed by the processor, Receive motion data, Based on the aforementioned motion data, it is determined whether the motion threshold has been exceeded. If the aforementioned motion threshold is exceeded, a depth-based reprojection is generated, or If the aforementioned motion threshold is not exceeded, a non-depth-based reprojection is generated. A non-temporary, computer-readable medium that is executable in such a way.

Images