A method and system for performing gaze-based spatial fovealization.

Abstract

The method includes determining the user's gaze position and generating a spatial fovealization map based on the gaze position. The method also includes receiving an image, forming a spatial fovealization image using the image and the spatial fovealization map, and transmitting the spatial fovealization image to a wearable device. The method further includes defoveating the spatial fovealization image to generate a defovealized image and displaying the defovealized image.

Description

Technical Field

[0001] Cross - reference to related applications

[0001] This application claims priority to U.S. Provisional Patent Application No. 63 / 461,120, "Methods and Systems for Performing Spatial Foveation Based on Gaze," filed on April 21, 2023, the entire content of which is incorporated herein by reference for all purposes.

[0002]

[0002] Modern computing and display technologies have facilitated the development of systems for so - called "virtual reality" or "augmented reality" experiences, where digitally reproduced images or portions thereof are presented to viewers such that they appear or are perceived as real. In a virtual reality (VR) scenario, digital or virtual image information is typically presented in isolation from other actual real - world visual inputs. In an augmented reality (AR) scenario, digital or virtual image information is typically presented as complementary to the visualization of the real world around the viewer.

[0003]

[0003] Referring to FIG. 1, an augmented reality scene 100 is depicted. A user of AR technology sees a setting 106 such as a real - world park with background people, trees, buildings, and a real - world concrete platform 120. The user also "sees" "virtual content" such as a robot statue 110 standing on the real - world concrete platform 120 and a flying cartoon - style avatar character 102 that anthropomorphizes a honeybee. The robot statue 110 and the flying cartoon - style avatar character 102 are "virtual" in the sense that they do not exist in the real world. Due to the complexity of the human visual recognition system, it is difficult to create AR technology that can comfortably, naturally, and richly represent virtual image elements among other virtual or real - world image elements.

[0004]

[0004] Despite these advances in display technology, there is a need in the art for improved methods and systems related to augmented reality systems, particularly display systems. [Overview of the project] [Means for solving the problem]

[0005]

[0005] The present invention generally relates to methods and systems related to projection display systems, including wearable displays. More specifically, embodiments of the present invention provide methods and systems useful for compressing and storing virtual content. As an example, a warp reprojection engine used for head posture correction before displaying virtual content also performs spatial foveal de-centralization processing to conserve system resources. The present invention is applicable to a variety of applications in computer vision and image display systems.

[0006]

[0006] The present invention offers numerous advantages over the prior art. For example, embodiments of the present invention provide methods and systems for reducing memory and processing load. By recognizing and taking into account the current location of the human gaze, embodiments of the present invention can reduce the quality (i.e., bandwidth) of locations in the image that the user is not looking at, i.e., locations in the image that are spatially distant from the gaze position, thereby reducing the image quality in these areas and reducing the overall need to transmit something at a high-quality setting that is indistinguishable to the human eye because the human eye is not currently focused on these non-gaze areas. Accordingly, embodiments of the present invention provide a video compression algorithm that takes into account the human gaze and uses a foveal compression algorithm that depends on the human gaze. The spatially fovealed image is then defoveated by a warp reprojection processor that warps the image for display based on the head pose of the wearable device before it is displayed to the user. These embodiments and other embodiments of the present invention, along with their many advantages and features, will be described in further detail in the following text and accompanying figures. [Brief explanation of the drawing]

[0007] [Figure 1] This shows the user's augmented reality (AR) view through an AR device. [Figure 2A] Each shows a cross-sectional side view of an example of a set of multilayer waveguides including an optical coupling element. [Figure 2B] Figure 2A shows a perspective view of one or more examples of laminated waveguides. [Figure 2C] Figures 2A and 2B show a top-down plan view of one or more examples of laminated waveguides. [Figure 3] This is a simplified diagram of an eyepiece waveguide with a composite pupil dilator according to one embodiment of the present invention. [Figure 4] An example of a wearable display system according to one embodiment of the present invention is shown. [Figure 5] A perspective view of a wearable device according to one embodiment of the present invention is shown. [Figure 6A] This is an example of the original N×M image. [Figure 6B] This is an n×m image of spatial fovealization according to one embodiment of the present invention. [Figure 7] This is a spatial fovealization map according to one embodiment of the present invention. [Figure 8A] This is a simplified calculation flowchart for performing gaze-based spatial fovealization according to one embodiment of the present invention. [Figure 8B] This is a schematic diagram showing a system for performing spatial fovealization based on line of sight according to one embodiment of the present invention. [Figure 9] This is a simplified flowchart illustrating a method for performing spatial fovealization based on gaze position according to one embodiment of the present invention. [Figure 10A] This is the original image before spatial foveal formation. [Figure 10B] This is a spatial foveal image obtained by one embodiment of the present invention.

[0008] [Figure 10C]This is a restored image using one embodiment of the present invention. [Figure 11] This is a simplified block diagram showing the components of an AR system according to one embodiment of the present invention. [Modes for carrying out the invention]

[0009]

[0024] The present invention generally relates to methods and systems related to projection display systems, including wearable displays. More specifically, embodiments of the present invention provide methods and systems that utilize gaze position during spatial foveation and spatial defoveation processes to reduce memory access to memory, internal data movement, and save power to augmented reality (AR) systems. Spatial defoveation is performed using a warp reprojection processor that warps the displayed image based on the head orientation of the wearable device. Embodiments of the present invention are applicable to various applications in computer vision and image display systems, including stereoscopic systems, systems that deliver light beamlets to the user's retina, and light field projection systems.

[0010]

[0025] As described herein, embodiments of the present invention combine the concept of spatial fovealization (i.e., the degradation of image or video quality in areas of an image or video that are not in focus of the human eye) with the concept of warp correction, which is used for other image processing functions including, but not limited to, head posture correction and lens correction, and color correction. As a result, embodiments of the present invention can significantly save power not only when moving display data but also when processing video images. Accordingly, embodiments of the present invention extend late-stage warp processing, which is used to perform image correction based on head posture to reduce image jitter and improve pixel sticking (pixel stability), to include fovealization de-centralization processing to reverse previously performed spatial fovealization based on gaze position.

[0011]

[0026] Embodiments of the present invention are applicable to systems that perform color foveation to reduce the overall computational load while maintaining the total number of pixels, and systems that perform spatial foveation to reduce the overall resolution and number of pixels. As will be described in more detail herein, a first-stage warp occurring remotely performs a prediction of the future head pose position. Subsequently, a final-stage warp corrects this future prediction based on actual corrected head pose measurements. In this final stage, the image warps from the predicted position to the position where it should actually be. According to embodiments of the present invention, both the first-stage and final-stage warps are corrected. The first-stage warp processor is modified to perform color spatial foveation and / or spatial foveation. This spatial foveation compresses the image to a reduced-size image (i.e., an image with a reduced number of pixels). This reduced-size image is transmitted between various system elements, the memory and power used to store this reduced-size image are reduced, and the reduced-size image is restored to, for example, the original image size before being rendered for display. Therefore, in the later warp processor, spatial defovealization and / or chromatic defovealization are performed, and combined with image warping, which performs head pose correction and / or other image correction processes before the image is displayed, the image is restored to, for example, its original image dimensions.

[0012]

[0027] Next, refer to the drawings, where the same reference numbers throughout the drawings refer to the same parts. Unless otherwise noted, the drawings are schematic and not necessarily drawn to scale.

[0013]

[0028] Referring to FIG. 2A, in some embodiments, the light incident on the waveguide may need to be redirected to couple the light into the waveguide. An optical coupling element may be used to redirect and couple the light into the corresponding waveguide. Although referred to herein as an "optical coupling element", the optical coupling element need not be an optical element and may be a non-optical element. FIG. 2A shows a cross-sectional side view of an example of a set of stacked waveguides 200 each including an optical coupling element. Each waveguide may be configured to output light of one or more different wavelengths, or one or more different wavelength ranges. Light from a projector is injected into the set of stacked waveguides 200 and output towards a user, as will be described in more detail below.

[0014]

[0029] The illustrated set of stacked waveguides 200 includes waveguide 202, waveguide 204, and waveguide 206. Each waveguide includes an associated optical coupling element (which may also be referred to as an optical input region on the waveguide). For example, optical coupling element 203 is disposed on the main surface (e.g., the upper main surface) of waveguide 202, optical coupling element 205 is disposed on the main surface (e.g., the upper main surface) of waveguide 204, and optical coupling element 207 is disposed on the main surface (e.g., the upper main surface) of waveguide 206. In some embodiments, one or more optical coupling elements may be disposed on the bottom main surface of their respective waveguides (particularly when one or more of the optical coupling elements are reflective deflection optical elements). As shown, optical coupling element 203, optical coupling element 205, and optical coupling element 207 may be disposed on the upper main surface (or the upper portion of the next lower waveguide), particularly when those optical coupling elements are transmissive deflection optical elements. In some embodiments, optical coupling element 203, optical coupling element 205, and optical coupling element 207 may be disposed within the bodies of waveguide 202, waveguide 204, and waveguide 206, respectively. In some embodiments, as described herein, optical coupling element 203, optical coupling element 205, and optical coupling element 207 are wavelength selective and transmit wavelengths of other light while selectively redirecting one or more wavelengths of light. Although shown on one side or corner of each of waveguide 202, waveguide 204, and waveguide 206, it will be understood that in some embodiments, optical coupling element 203, optical coupling element 205, and optical coupling element 207 may be disposed in other regions of waveguide 202, waveguide 204, and waveguide 206, respectively.

[0015]

[0030] As shown in the figure, the optical couplers 203, 205, and 207 may be offset laterally from each other. In some embodiments, each optical coupler may be offset so that light does not pass through another optical coupler to receive light. For example, each of the optical couplers 203, 205, and 207 may be configured to receive light from different projectors and may be separated from other optical couplers (e.g., laterally spaced) so as not to receive light from the other optical couplers.

[0016]

[0031] Each waveguide also includes associated optical distribution elements, such as optical distribution element 210 located on the main surface of waveguide 202 (e.g., upper main surface), optical distribution element 212 located on the main surface of waveguide 204 (e.g., upper main surface), and optical distribution element 214 located on the main surface of waveguide 206 (e.g., upper main surface). In other embodiments, optical distribution elements 210, 212, and 214 may be located on the bottom main surfaces of the associated waveguides 202, 204, and 206, respectively. In other embodiments, optical distribution elements 210, 212, and 214 may be located on both the upper and lower main surfaces of the associated waveguides 202, 204, and 206, respectively. Alternatively, the optical distribution elements 210, 212, and 214 may be located on different surfaces of the upper and lower main surfaces of different associated waveguides 202, 204, and 206, respectively.

[0017]

[0032] Waveguides 202, 204, and 206 may be spaced apart from each other and separated, for example, by layers of gas, liquid, and / or solid material. For example, as shown in Figure 2A, layer 208 may separate waveguides 202 and 204, and layer 209 may separate waveguides 204 and 206. In some embodiments, layers 208 and 209 are formed of low refractive index material (i.e., material having a lower refractive index than the material forming the most adjacent waveguide among waveguides 202, 204, or 206). Preferably, the refractive index of the material forming layer 208 and / or layer 209 is 0.05 or more, or 0.10 or less, than the refractive index of the material forming waveguide 202, 204, or 206. Advantageously, layers 208 and 209, having lower refractive indices, can function as cladding layers that enhance the total internal reflection (TIR) ​​of light passing through waveguides 202, 204, and 206 (e.g., the TIR between the upper and lower main surfaces of each waveguide). In some embodiments, layers 208 and 209 are formed of air. Although not shown, it will be understood that the upper and lower parts of the illustrated set of laminated waveguides 200 may include the immediately adjacent cladding layers.

[0018]

[0033] Preferably, for ease of manufacture and other considerations, the materials forming waveguides 202, 204, and 206 are similar or identical, and the materials forming layers 208 and 209 are similar or identical. In some embodiments, the materials forming waveguides 202, 204, and 206 may differ between one or more waveguides, and / or the materials forming layers 208 and 209 may differ while maintaining the various refractive index relationships described above.

[0019]

[0034] Continuing to refer to Figure 2A, rays 218, 219, and 220 are incident on the set of stacked waveguides 200. It will be understood that rays 218, 219, and 220 can be injected into waveguides 202, 204, and 206 by one or more projectors (not shown).

[0020]

[0035] In some embodiments, rays 218, 219, and 220 have different characteristics, such as different wavelengths or different wavelength ranges, which may correspond to different colors. Optical couplers 203, 205, and 207 each deflect incident light so that the light propagates through one of waveguides 202, 204, or 206 by TIR. In some embodiments, optical couplers 203, 205, and 207 each selectively deflect one or more specific wavelengths of light, transmitting other wavelengths to the underlying waveguide and associated optical couplers.

[0021]

[0036] For example, the optical coupler 203 may be configured to transmit rays 219 and 220 having different second and third wavelengths or wavelength ranges, respectively, while deflecting a ray 218 having a first wavelength or wavelength range. Ray 219 transmitted through the waveguide 202 collides with an optical coupler 205 configured to deflect light of the second wavelength or wavelength range and is deflected. Ray 220 is deflected by an optical coupler 207 configured to selectively deflect light of the third wavelength or wavelength range.

[0022]

[0037] Continuing to refer to Figure 2A, rays 218, 219, and 220 are deflected to propagate through their respective waveguides 202, 204, and 206. That is, the optical couplers 203, 205, and 207 of each waveguide deflect the light to their respective waveguides 202, 204, or 206, and couple the light to the corresponding waveguide. Rays 218, 219, and 220 are deflected by TIR at an angle such that the light propagates through their respective waveguides 202, 204, and 206. Rays 218, 219, and 220 propagate through waveguides 202, 204, and 206 by TIR and reach the optical distributors corresponding to the waveguides. In other words, the optical distribution elements 210, 212, and 214 are externally coupled to guide light to the outside, providing the guided light ray 216.

[0023]

[0038] Referring now to Figure 2B, a perspective view of an example of the stacked waveguide 200 set shown in Figure 2A is provided. As described above, rays 218, 219, and 220 are coupled and deflected by optical couplers 203, 205, and 207, respectively, and propagate by TIR through waveguides 202, 204, and 206, respectively. Next, rays 218, 219, and 220 reach optical distributors 210, 212, and 214, respectively. Optical distributors 210, 212, and 214 deflect rays 218, 219, and 220 toward optical extraction elements 222, 224, and 226, respectively.

[0024]

[0039] In some embodiments, the optical distribution elements 210, 212, and 214 are orthogonal pupil magnifiers (OPEs). In some embodiments, the OPEs deflect or distribute light to the optical extraction elements 222, 224, and 226, and in some embodiments, they may also magnify the beam or spot size of the light as it propagates to the optical extraction elements. In some embodiments, the optical distribution elements 210, 212, and 214 may be omitted, and optical coupling elements 203, 205, and 207 may be configured to directly deflect light to the optical extraction elements 222, 224, and 226. For example, referring to Figure 2A, the optical distribution elements 210, 212, and 214 may be replaced by optical extraction elements 222, 224, and 226, respectively. In some embodiments, the light extraction optical elements 222, 224, and 226 are exit pupils (EPs) or exit pupil enlargers (EPEs) that guide light to the user's eye. It will be understood that the OPEs may be configured to enlarge the dimensions of the eyebox along at least one axis, and the EPEs may be configured to enlarge the eyebox along an axis intersecting the axis of the OPE, for example, an orthogonal axis. For example, each OPE may be configured such that a portion of the light striking the OPE is redirected to an EPE in the same waveguide, and the remaining portion of the light continues to propagate along the waveguide. Another portion of the remaining light strikes the OPE again and is redirected to an EPE, and the remainder of that portion continues to propagate further down the waveguide, and so on. Similarly, a portion of the incident light that strikes the EPE is guided out of the waveguide to the user, the remainder of that light continues to propagate through the waveguide, and another portion of the incident light that strikes the EPE again is guided out of the waveguide, and so on. As a result, the combined single ray may be "duplicated" each time a portion of its light is redirected by the OPE or EPE, forming a field of duplicated rays. In some embodiments, the OPE and / or EPE may be configured to change the size of the rays.In some embodiments, the functions of the light distribution elements 210, 212, and 214, and the light extraction optical elements 222, 224, and 226 are combined within the composite pupil dilator as described in relation to Figure 3.

[0025]

[0040] Referring to Figures 2A and 2B, in some embodiments, a set of laminated waveguides 200 comprises waveguides 202, 204, and 206, optical couplers 203, 205, and 207, optical distributors 210, 212, and 214 (e.g., OPE), and optical extraction optics 222, 224, and 226 (e.g., EP) for each component color. Waveguides 202, 204, and 206 may be laminated with an air gap / cladding layer in between each. Optical couplers 203, 205, and 207 redirect or deflect incident light (different optical couplers receive light of different wavelengths) into their waveguides. Next, the light propagates through waveguides 202, 204, and 206 at angles that produce TIR, respectively. In the illustrated example, ray 218 (e.g., blue light) is deflected by the optical coupling element 203, then continues to propagate downwards while reflecting off the waveguide, interacting with the optical distribution element 210 (e.g., OPE) as described above, and then interacting with the optical extraction element 222 (e.g., EP). Rays 219 and 220 (e.g., green light and red light, respectively) pass through waveguide 202, and ray 219 collides with the optical coupling element 205 and is deflected. Next, ray 219 is reflected downwards as it propagates through waveguide 204 by TIR, proceeds to the optical distribution element 212 (e.g., OPE), and then proceeds to the optical extraction element 224 (e.g., EP). Finally, the ray 220 (e.g., red light) passes through waveguide 206 and collides with the optical coupling element 207 of waveguide 206. The optical coupling element 207 deflects the ray 220 so that it propagates via TIR to the optical distribution element 214 (e.g., OPE) and then via TIR to the optical extraction element 226 (e.g., EP). The output optical coupling element 226 finally outputs coupled the ray 220 toward the observer, who also receives output coupled light from the other waveguides, namely waveguides 202 and 204.

[0026]

[0041] Figure 2C shows a top-down plan view of an example of the stacked waveguide 200 set shown in Figures 2A and 2B. As shown, waveguides 202, 204, and 206 may be vertically aligned with the optical distribution elements associated with each waveguide, namely optical distribution elements 210, 212, and 214, and the associated optical extraction optical elements, namely optical extraction optical elements 222, 224, and 226. However, as described herein, the optical coupling elements 203, 205, and 207 are not vertically aligned. Rather, it is preferable that the optical coupling elements do not overlap (for example, they are spaced apart laterally when viewed in a top or plan view). As further described herein, this non-overlapping spatial arrangement facilitates the one-to-one injection of light from different resources into different waveguides, making it possible to uniquely couple a particular light source to a particular waveguide. In some embodiments, arrangements including non-overlapping, spatially separated optical coupling elements may be called shift pupil systems. The optical coupling elements in these arrangements may correspond to subpupils.

[0027]

[0042] Figure 3 is a simplified diagram of an eyepiece waveguide equipped with a composite pupil dilator according to one embodiment of the present invention. In the example shown in Figure 3, the eyepiece 310 uses a single-sided configuration with combined OPE / EPE regions. Referring to Figure 3, the eyepiece 310 includes an optical coupling element 322 and a substrate 320 on which a composite OPE / EPE region 324, also called a composite pupil dilator (CPE), is provided. The incident light ray 330 is coupled via the optical coupling element 322 and coupled as an output light ray 332 via the coupled OPE / EPE region 324.

[0028]

[0043] The combined OPE / EPE region 324 includes grids corresponding to both OPE and EPE that spatially overlap in the x and y directions. In some embodiments, the grids corresponding to both OPE and EPE are located on the same side of the substrate 320, with either the OPE grid superimposed on the EPE grid or the EPE grid superimposed on the OPE grid (or both). In other embodiments, the OPE grid is located on the substrate 320 opposite to the EPE grid, and the grids spatially overlap in the x and y directions but are separated from each other in the z direction (i.e., in different planes). Thus, the combined OPE / EPE region 324 can be mounted in either a single-sided or double-sided configuration.

[0029]

[0044] Figure 4 shows an example of a wearable display system 430 in which various waveguides and associated systems disclosed herein may be integrated. Referring to Figure 4, the wearable display system 430 includes a display 432 and various mechanical and electronic modules and systems to support the functionality of the display 432. The display 432 may be coupled to a frame 434 that can be worn by a user 440 (also called the viewer or user of the display system) and is configured to position the display 432 in front of the user 440's eyes. In some embodiments, the display 432 may be conceived as eyeglasses. In some embodiments, a speaker 436 is coupled to the frame 434 and is configured to be positioned adjacent to the user 440's ear canal (in some embodiments, another speaker, not shown, may be optionally positioned adjacent to the user's other ear canal to provide stereo / shapeable sound control). The wearable display system 430 may also include one or more microphones or other devices for detecting sound. In some embodiments, the microphone may be configured to allow the user to provide input or commands to the wearable display system 430 (e.g., selection of voice menu commands, natural language questions, etc.) and / or enable voice communication with others (e.g., other users of similar display systems). The microphone may further be configured as a peripheral sensor to collect voice data (e.g., sounds from the user and / or the environment). In some embodiments, the wearable display system 430 may further include one or more outward-facing environmental sensors configured to detect objects, stimuli, people, animals, places, or other aspects of the world around the user. For example, the environmental sensors may include one or more cameras, which may be positioned to face outward to capture images similar to at least a portion of the user 440's normal field of view. In some embodiments, the wearable display system may also include peripheral sensors that are separate from the frame 434 and can be attached to the user 440's body (e.g., the user 440's head, torso, limbs, etc.).In some embodiments, the peripheral sensor may be configured to acquire data characterizing the physiological state of the user 440. For example, the sensor may be an electrode.

[0030]

[0045] The display 432 is operationally coupled to a local data processing module that can be mounted in various configurations, such as being fixedly attached to a frame 434 or the like by a communication link such as a wired lead or wireless connection, fixedly attached to a helmet or hat worn by the user, embedded in headphones, or detachably attached to the user 440 (e.g., backpack-style configuration, belt-attached configuration). Similarly, sensors can be operationally coupled to the local processor and data module by a communication link such as a wired lead or wireless connection. The local processing and data module consists of a hardware processor and digital memory such as non-volatile memory (e.g., flash memory or hard disk drive), both of which can be used to assist in data processing, caching, and storage. Optionally, the local processor and data module may include one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. The data may include a) data captured from sensors such as an image capture device (such as a camera), a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, a gyroscope, and / or other sensors disclosed herein (which may be operably coupled to frame 434 or otherwise attached to user 440), and / or b) data acquired and / or processed using a remote processing module 452 and / or a remote data repository 454 (which may include data related to virtual content), and which may be transferred to the display 432 after such processing or acquisition. The local processing and data module may be operably coupled to the remote processing and data module 450 by a communication link 438, such as a wired or wireless link, and may include the remote processing module 452, the remote data repository 454, and a battery 460.The remote processing module 452 and the remote data repository 454 can be coupled to the remote processing and data module 450 by communication links 456 and 458, so that these remote modules are operably coupled to each other and become available as resources for the remote processing and data module 450. In some embodiments, the remote processing and data module 450 may include one or more of the following: an image capture device, a microphone, an inertial measurement unit, an accelerometer, a compass, a GPS unit, a wireless device, and / or a gyroscope. In other embodiments, one or more of these sensors may be mounted on the frame 434 and may be standalone structures that communicate with the remote processing and data module 450 by a wired or wireless communication path.

[0031]

[0046] Continuing to refer to Figure 4, in some embodiments, the remote processing and data module 450 may comprise one or more processors configured to analyze and process data and / or image information, including, for example, one or more central processing units (CPUs), graphics processing units (GPUs), dedicated processing hardware, etc. In some embodiments, the remote data repository 454 may comprise digital data storage capabilities accessible via the Internet or other network configurations within a “cloud” resource configuration. In some embodiments, the remote data repository 454 may comprise one or more remote servers that provide information, for example, information for generating augmented reality content, to the local processing and data module and / or the remote processing and data module 450. In some embodiments, all data is stored in the local processing and data module and all calculations are also performed in the same module, enabling fully autonomous use from the remote module. Optionally, an external system (e.g., one or more processors, one or more computer systems) including a CPU, GPU, etc., may perform at least part of the processing (e.g., generating image information, processing data), and may provide and receive information from the illustrated module, for example, via a wireless or wired connection.

[0032]

[0047] Figure 5 shows a perspective view of a wearable device 500 according to one embodiment of the present invention. In some embodiments, the wearable device 500 includes a frame 502 configured to support one or more projectors 504 at various positions along the inner surface of the frame 502, as shown in the figure, in which case the projectors 504 can be mounted near the temples 506. Alternatively, or additionally, another projector may be placed at position 508. Such projectors may comprise, for example, one or more liquid crystal on silicon (LCoS) modules, micro-LED displays, or fiber scanning devices, or operate in conjunction with them. In some embodiments, light from projector 504 or a projector placed at position 508 may be directed to an eyepiece 510 for display to the user's eye. A projector placed at position 512 can be miniaturized because the projector is closer to the waveguide system. The closer distance reduces the amount of light lost as the waveguide system directs the light from the projector to the eyepiece 510. In some embodiments, the projector at position 512 may be used in conjunction with projector 504 or a projector located at position 508. Although not shown, in some embodiments, the projector may also be located below the eyepiece 510. The wearable device 500 is also depicted to include sensors 514 and 516. Sensors 514 and 516 may take the form of forward-facing and lateral-facing optical sensors configured to characterize the real-world environment surrounding the wearable device 500.

[0033]

[0048] Embodiments of the present invention utilize an eye-tracking system to determine the user's gaze position and use that gaze position for image compression processing. Referring to Figure 5, an eye-tracking camera 505 is positioned on frame 502 and may be used to track the gaze position of a user using a wearable device 500. In other embodiments, other eye-tracking systems may be used to determine the gaze position, and the eye-tracking camera 505 shown in Figure 5 is merely illustrative. As described in more detail herein, the image compression process used to compress and decompress virtual content in functions such as storage to memory, internal communication, and display may be modified depending on the gaze position. For example, portions of an image or video stream corresponding to a gaze position may be compressed using a higher quality compression process compared to other portions of the image or video stream located further away from the gaze position. Since these more distant portions of the image or video stream are within the user's peripheral vision, the impact on the user experience resulting from the reduced compression quality may be smaller than the benefits gained in terms of memory and processing efficiency and / or requirements. It will be obvious to those skilled in the art that many variations, modifications, and substitutions are possible.

[0034]

[0049] Figure 6A is an example of the original N×M image. Figure 6B is an n×m image in which a spatial fovea has been formed according to one embodiment of the present invention.

[0035]

[0050] In the application of spatial fovealization, multiple pixels (e.g., 4 pixels) are compressed into 1 pixel. As shown in Figures 6A and 6B, the 16 pixels in the upper left corner and the 16 pixels in the lower right corner of the image shown in Figure 6A are compressed into the 4 pixels in the upper left corner and the 4 pixels in the lower right corner of the image shown in Figure 6B. Referring to the upper left corner of Figures 6A and 6B, pixels R1C1, R1C2, R2C1, and R2C2 are compressed into pixel R1'C1'. Pixels R1C3, R1C4, R2C3, and R2C4 are compressed into pixel R1'C2'. Pixels R3C1, R3C2, R4C1, and R4C2 are compressed into pixel R2'C1'. Pixels R3C3, R3C4, R4C3, and R4C4 are compressed into pixel R2'C2'. Similarly, referring to the lower right corner of Figures 6A and 6B, pixels RN-3CM-3, RN-3CM-2, RN-2CM-3, and RN-2CM-2 are compressed into pixel Rn-1Cm-1. Pixels RN-3CM-1, RN-3CM, RN-2CM-1, and RN-2CM are compressed into pixel Rn-1Cm. Pixels RN-1CM-3, RN-1CM-2, RNCM-3, and RNCM-2 are compressed into pixel RnCm-1. Pixels RN-1CM-1, RN-1CM, RNCM-1, and RNCM are compressed into pixel RnCm.

[0036]

[0051] Thus, spatial foveation reduces the number of pixels, alleviating processing and memory requirements at the expense of reduced resolution. Applying spatial foveation reduces the size of the image as a result. In the embodiments described herein, spatial foveation is applied non-uniformly as a function of image position, with reduced or no foveation at positions corresponding to the line of sight (i.e., the line of sight position) and stronger spatial foveation at positions further from the line of sight position. Defoveation is performed in the reverse manner, generating multiple pixels based on one pixel. Defoveation is also applied non-uniformly across the image. As will be apparent to those skilled in the art, many modifications, alterations, and substitutions are possible.

[0037]

[0052] Referring to Figures 6A and 6B, the illustrated spatial fovealization process combines multiple pixels (e.g., 4 pixels) into one pixel. This spatial fovealization process is performed based on the gaze position, with higher compression ratios in areas of the image farther from the gaze position. Thus, at the corner of the N×M image shown in Figure 6A, the original 16 pixels are compressed into 4 pixels. During the spatial fovealization de-fovealization process, the process is reversed, using one pixel to generate multiple pixels (e.g., 4 pixels). As will be described in more detail herein, the spatial fovealization de-fovealization process is performed by a warp processor, which also performs head pose correction and other image correction processes before the image is displayed.

[0038]

[0053] In some embodiments, color space foveation is performed in addition to, or instead of, spatial foveation. Color space foveation, which can be used to reduce computational requirements, assigns the color of one pixel among multiple pixels to the pixels that make up the multiple pixels. This is shown by the color of one pixel (e.g., pixel (1,1)) being assigned to the other pixels in the multiple pixels (e.g., pixel (1,2), pixel (2,1), and pixel (2,2)). In this example, as a result of color space foveation, the original single pixel's color is assigned to all four pixels. Thus, the number of pixels remains the same, and as a result, the color space foveated image retains the size of the original image, but the color content is reduced. In some embodiments, color space foveation is used to assign one color to multiple pixels, and in combination with this, spatial foveation is used to reduce the number of pixels in the color space / spatial foveated image. In the restoration stage, both color space foveation and spatial foveation are performed in reverse. As will be apparent to those skilled in the art, many variations, modifications, and substitutions are possible.

[0039]

[0054] Figure 7 shows a spatial fovealization map according to one embodiment of the present invention. The spatial fovealization map 700 shown in Figure 7 can be created based on gaze information, that is, by actively determining where the human eye is currently focused or looking.

[0040]

[0055] As described herein, the length of the spatial fovealization vector represented by the spatial fovealization map 700 is a function of the gaze position. For the spatial fovealization map 700 shown in Figure 7, the gaze position is the center of the image (i.e., pixel (900,900)), which can also be called the central region. Pixels near the gaze position undergo reduced spatial fovealization or no spatial fovealization at all. Therefore, after spatial fovealization, pixel (900,900) remains unchanged. Pixels near pixel (900,900) correspond to short fovealization vectors. Therefore, the magnitude of spatial fovealization for pixels close to the central pixel is small, and after spatial fovealization processing, a small number of pixels are represented by a single pixel.

[0041]

[0056] As the distance between a particular pixel and the line of sight increases, the magnitude of the spatial foveation vector increases, corresponding to an increase in the magnitude of spatial foveation, as shown in Figure 7. Therefore, the magnitude of the spatial foveation vector and the corresponding magnitude of spatial foveation are maximized at the corners of adjacent pixels (0,0), (1800,0), (0,1800), and (1800,1800) in the image, which can also be called peripheral regions. Thus, for these pixels in peripheral regions, spatial foveation is performed by grouping more pixels (i.e., a number of pixels corresponding to the length of the foveation vector) to form a single pixel. As a result, the number of pixels decreases with distance from the line of sight, as shown in Figure 10B later.

[0042]

[0057] The spatial fovealization map 700 is dynamic and changes according to the gaze position. For example, if the gaze position moves to the left side of the image, spatial fovealization decreases, or areas where it does not exist move to the left side of the image, and spatial fovealization performed on pixels on the right side of the image is enhanced. Thus, as the gaze position moves to a different location in the image, the spatial fovealization map 700 is updated in real time to track the gaze position. As a result, the spatial fovealization map 700 shown in Figure 7 is merely illustrative and is associated with the gaze position being located at the center of the image. Thus, during operation, the spatial fovealization map 700 can be dynamically updated based on the gaze position. During spatial fovealization decentration processing, where multiple pixels are generated based on a single pixel, a spatial fovealization decentration processing map, which is the inverse of the spatial fovealization map, is used. As a result, the spatial fovealization decentration processing map based on the gaze position can also be dynamically updated. As those skilled in the art will see that many variations, modifications, and substitutions are possible.

[0043]

[0058] FIG. 8A is a simplified computational flowchart for performing spatial foveation based on line of sight according to an embodiment of the present invention. The system 800 shown in FIG. 8A includes a remote computing device 810 and a local wearable device 820 that can be utilized to present virtual content to a user. The remote computing device 810 and the local wearable device 820 can be elements of an augmented reality (AR) system. In the line of sight position 812, the spatial foveation map 816, and the N×M input image 814 (e.g., virtual content), a graphics processing unit (GPU) 818 performs spatial foveation based on the line of sight position 812 and the spatial foveation map 816. By this spatial foveation process, an n×m spatial foveation image 822 is generated and transmitted from the remote computing device 810 to the local wearable device 820. In an embodiment of the present invention, n < N and m < M. For example, the original image is an image of N = 640×M = 480, and the spatial foveation image is an image of n = 480×m = 360. As a result, the number of pixels in the spatial foveation image 822 is less than the number of pixels in the N×M input image 814.

[0044]

[0059] As shown in FIG. 8A, the line of sight position 812 is provided to the remote computing device 810. The n×m spatial foveation image 822 is compressed non-uniformly, and the spatial compression is reduced or does not occur in the vicinity of the line of sight position 812, and the spatial compression becomes larger for pixels at a longer distance from the line of sight position 812. The reduction in the size of the image is beneficial in several respects, such as reducing the data transfer bandwidth and reducing the processing load.

[0045]

[0060] To provide the user with an image for display, the spatial fovealization performed by the GPU818 is reverse-transformed using a spatial fovealization defoveation process (also called spatial fovealization correction). In addition to spatial fovealization defoveation, other processes can be performed to generate an image suitable for display to the user. For example, the optical elements used in the display may be characterized by distortion. This distortion can be removed by performing a lens correction process. Furthermore, since the user's head may move and / or rotate between the time the virtual content (e.g., an N×M input image) is generated and the image is displayed to the user through one or more external displays 830, a head pose correction process can be performed.

[0046]

[0061] Furthermore, in embodiments using an eyepiece waveguide display, the eyepiece waveguide may be characterized by a bending value, which can be corrected using a bending correction process. Bending correction can be used, for example, to correct mechanical deformation of the eyepiece lens from a planar shape to a non-planar shape. In some embodiments, a planar-facing eyepiece waveguide is used. The eyepiece waveguide may have a flexible frame rather than a rigid mechanical structure. One or more factors may cause the shape of the eyepiece waveguide to deviate from a planar shape and become a non-planar shape. In other embodiments, the desired shape of the eyepiece waveguide is a non-planar shape, such as a uniform curvature, but deformation of the eyepiece waveguide may result in a shape with non-uniform curvature, such as a plane. The deformation from the desired shape may exist in real time (i.e., instantaneously) or may occur over time. In either case, unwanted curvature present in the eyepiece waveguide can be measured and stored in memory, and the image can be warped using a bending correction process in a similar manner to how optical distortion is removed using a lens correction process. Therefore, the bending correction described here can compensate for changes in the shape of the eyepiece, relative movement between eyepieces, and / or movement of the eyepiece relative to the user's eye position. Thus, as shown in Figure 8A, bending correction is one of the corrections that can be performed by the warp reprojection processor 826 (such as the post-reprojection processor), in addition to other possible corrections (lens, fovea, color, etc.).

[0047]

[0062] As shown in Figure 8A, the gaze position 812 is also transmitted to the local wearable device 820. In the n×m spatial foveal image 822, correction input 824 including lens correction, head posture correction, bending correction, spatial foveal correction, etc., and spatial foveal map 816, the warp reprojection processor 826 is used to generate an image (i.e., a defovealed image including warp correction and / or other corrections) that is displayed using one or more external displays 830. For example, one or more external displays can be optically coupled to an eyepiece waveguide located within the head-mounted display (HMD), and one or more external displays 830 can display stereo video content. These correction processes can be performed simultaneously, in parallel, sequentially, or in combination thereof.

[0048]

[0063] Therefore, the spatial fovealization performed by the GPU 818 is inversely transformed by the warp reprojection processor 826 during image warping, which is used to perform head pose correction and other operations. As explained in relation to the spatial fovealization map 700 shown in Figure 7, during spatial fovealization de-centralization, multiple pixels at a distance from the line of sight are generated based on a single pixel, and the resolution decreases as a function of the distance from the line of sight. Since the same spatial fovealization map (or spatial fovealization map and inverse spatial fovealization de-centralization map) can be used for both fovealization and de-centralization, the image compression performed during spatial fovealization can be inversely transformed during spatial fovealization de-centralization. After spatial fovealization correction (i.e., spatial fovealization de-centralization) by the warp reprojection processor 826, the output image is again N×M pixels, but the spatial resolution decreases at positions far from the line of sight.

[0049]

[0064] Figure 8B is a schematic diagram showing a system that performs line-of-sight-based spatial fovealing according to one embodiment of the present invention. This schematic diagram shares elements with the simplified computational flow diagram shown in Figure 8A, and the explanations provided in relation to Figure 8A can be applied to Figure 8B as needed.

[0050]

[0065] Referring to FIG. 8B, the system 850 includes a remote computing device 860 and a local wearable device 870 that can be used to present virtual content to a user. The remote computing device 860 and the local wearable device 870 can be elements of an augmented reality (AR) system. A gaze tracking system 871 generates a gaze position provided to a CPU 861, a GPU 868, and a warp reprojection processor 876. The CPU 861 is used to generate virtual content represented by an N×M input image 864 and a spatial foveation map 866. The GPU 868 performs spatial foveation based on the gaze position provided by the gaze tracking system 871 and the spatial foveation map 866. A communication system 869 is used to transmit an n×m spatially foveated image 872 generated by the spatial foveation process from the remote computing device 860 to a local wearable device 870 that includes a communication device 875 communicating with the communication system 869, together with the spatial foveation map 866. In an embodiment of the present invention, n < N and m < M. For example, the original image is an image of N = 640×M = 480, and the spatially foveated image is an image of n = 480×m = 360. As a result, the number of pixels of the spatially foveated image 872 is less than the number of pixels of the N×M input image 814.

[0051]

[0066] In addition to the CPU 861 and the GPU 868, the gaze position generated by the gaze tracking system 871 is provided to the warp reprojection processor 876 of the local wearable device 870. The n×m spatially foveated image 872 is unevenly compressed, with reduced spatial compression or no spatial compression in the vicinity of the gaze position, and increased spatial compression for pixels far from the gaze position. This reduction in image size is beneficial in several respects, such as reducing data transfer bandwidth and processing load.

[0052]

[0067] To provide the user with an image for display, spatial fovealization performed by the GPU868 is reversed using spatial fovealization de-centralization, also known as spatial fovealization correction. In addition to spatial fovealization de-centralization, other processes can be performed to generate an image suitable for display to the user. For example, the optical elements used for display may be characterized by distortion. This distortion can be removed by performing lens correction. Furthermore, since the user's head may move and / or rotate between the time the virtual content (e.g., an N×M input image) is generated and the image is displayed to the user through one or more external displays 880, head pose correction can be performed.

[0053]

[0068] Furthermore, in embodiments using an eyepiece waveguide display, the eyepiece waveguide is characterized by a bending value, which can be corrected using bending correction processing as described above. Thus, as shown in Figure 8B, bending correction is one of the corrections that can be performed by the warp reprojection processor 876 (or a subsequent reprojection processor), in addition to other possible corrections (lens, foveal formation, color, etc.).

[0054]

[0069] A warp reprojection processor 876 is used to generate an image to be displayed using one or more external displays 880, using the gaze position, an n×m spatial foveal image 872, correction inputs 874 including lens correction, head posture correction, bending correction, spatial foveal correction, etc., and a spatial foveal map 866. For example, one or more external displays can be optically coupled to an eyepiece waveguide located within the head-mounted display (HMD), and one or more external displays 880 can display stereo video content. These correction processes can be performed simultaneously, in parallel, sequentially, or in combination thereof.

[0055]

[0070] Therefore, the spatial fovealization performed by GPU868 is inversely transformed by warp reprojection processor 876 during image warp processing used to perform head pose correction, etc. After spatial fovealization correction (i.e., spatial fovealization de-processing) by warp reprojection processor 876, the output image is again N×M pixels, but the spatial resolution decreases at positions far from the line of sight. It will be obvious to those skilled in the art that many deformations, modifications, and substitutions are possible.

[0056]

[0071] Figure 9 is a simplified flowchart illustrating a method for performing gaze-based spatial fovealing according to one embodiment of the present invention. Method 900 includes performing spatial and / or color-space fovealing on a remote device based on gaze position (910). The remote device may be a component of an AR system or a remote server (such as a cloud server). Spatial fovealing is performed using a spatial fovealing map. In addition to spatial fovealing, an initial warp based on expected head pose can be performed in combination with spatial fovealing. The gaze position is transmitted to a local device (912).

[0057]

[0072] On the local device, the method includes receiving a spatially warped image (920) and creating an inverse foveal vector map (922) based on the line of sight. In some embodiments, the inverse foveal vector map, also called an inverse spatial fovealization map or spatial fovealization de-processing map, can be created on a remote device and transmitted to the local device. The method also includes combining multiple vector map corrections, including but not limited to warp, lens, bend, and / or spatial fovealization corrections (924). Using the combined vector map, the combined corrections are applied (926) to form and output a corrected display image (928).

[0058]

[0073] It should be understood that the specific steps shown in Figure 9 provide a particular method for performing spatial fovealization based on line of sight according to one embodiment of the present invention. Other sets of steps may also be performed according to alternative embodiments. For example, alternative embodiments of the present invention may perform the steps outlined above in a different order. Furthermore, the individual steps shown in Figure 9 may include multiple substeps that can be performed in various orders depending on the individual step. Additionally, additional steps may be added or removed depending on the specific application. It will be obvious to those skilled in the art that many variations, modifications, and substitutions are possible.

[0059]

[0074] Figure 10A is the original image before spatial fovealization. The image shown in Figure 10A is an N×M image, which is a virtual image that can be displayed using, for example, the AR system described herein.

[0060]

[0075] Figure 10B is a spatially foveated image according to one embodiment of the present invention. In the N×M input image shown in Figure 10A, the spatial foveation process described herein is used to spatially foveate the N×M input image and generate the n×m image shown in Figure 10B. In the spatial foveation process shown in Figure 10B, the line of sight is at the center of the image. As shown in Figure 10B, the portion of the image furthest from the line of sight exhibits the greatest foveal formation, with pixels at the corners and sides of the image being removed. Because the number of pixels associated with the spatially foveated image shown in Figure 10B is reduced, the memory and processing requirements are reduced in this embodiment of the present invention. It should be understood that the foveated image does not need to be rectangular, as the number of pixels combined during the spatial foveation process increases with distance from the line of sight. Therefore, the foveated image shown in Figure 10B has fewer pixels in the first and last rows than in the center of the image. It will be obvious to those skilled in the art that many variations, modifications, and substitutions are possible. Therefore, when referring to the spatially foveated image as an n×m image, the spatially foveated image does not need to be rectangular.

[0061]

[0076] Figure 10C is a reconstructed image according to one embodiment of the present invention. In the spatial fovealization image shown in Figure 10B and the spatial fovealization de-processing map, such as the spatial fovealization map 700 shown in Figure 7, the spatial fovealization image is reconstructed (i.e., the n×m image shown in Figure 10B) and converted back to the original N×M image size using the warp reprojection processor 826 shown in Figure 8A / warp reprojection processor 876 shown in Figure 8B. The spatial fovealization de-processing map can be generated by the warp reprojection processor 826 shown in Figure 8A / warp reprojection processor 876 shown in Figure 8B based on the spatial fovealization map 816 / spatial fovealization map 866, or it may be provided to the warp reprojection processor 826 shown in Figure 8A / warp reprojection processor 876 shown in Figure 8B. Therefore, in some embodiments, instead of sending the spatial fovealization map to the local computing device, the spatial fovealization de-processing map is generated on a remote computing device and sent to the local computing device. Those skilled in the art will see that many variations, modifications, and substitutions are possible. As shown in Figure 10C, the resolution at the corners and sides of the image is reduced compared to the N×M input image, but since the areas with reduced resolution are within the user's peripheral vision, the impact of these changes on the user experience is limited.

[0062]

[0077] In the examples shown in Figures 10A-10C, the warp reprojection processor is used to enlarge the n×m foveated image shown in Figure 10B to the original N×M image size shown in Figures 10A and 10. However, embodiments of the present invention are not limited to enlarging the foveated image to the original image size. In some cases, the warp reprojection processor may be used to enlarge the restored image to a size larger than the original size, i.e., N'×M' (N'>N, M'>M). For example, this enlargement to a restored image size larger than the original image size can be implemented to display the restored image on a larger external display while maintaining the bandwidth corresponding to the original image. Thus, embodiments of the present invention can implement a “zoom” function by applying an enlargement of the image size using a warp reprojection processor, in addition to other corrections described herein. It will be apparent to those skilled in the art that many variations, modifications, and substitutions are possible.

[0063]

[0078] Figure 11 is a simplified block diagram showing components of an AR system according to one embodiment of the present invention. The AR system 1100 shown in Figure 11 may be incorporated into an AR device as described herein. Figure 11 shows a schematic diagram of one embodiment of the AR system 1100 that can perform some or all of the steps of the method provided in various embodiments. Note that Figure 11 is for general illustration purposes only of the various components, and some or all of them may be used as needed. Thus, Figure 11 broadly shows how individual system elements may be implemented in a relatively isolated or relatively integrated manner.

[0064]

[0079] The AR system 1100 includes hardware elements that can be electrically connected via bus 1105, or communicate as needed. The hardware elements may include one or more processors 1110, including but not limited to one or more general-purpose processors and / or one or more dedicated processors such as digital signal processing chips, graphics acceleration processors, etc., one or more input devices 1130, including but not limited to a mouse, keyboard, camera, etc., and one or more output devices 1140, including but not limited to a display device, printer, etc. Furthermore, the AR system 1100 includes an eye-tracking system 1170 that can provide the AR system with the user's gaze position. Using one or more processors 1110, the foveal image compression technique described herein can be implemented. To provide the user's head pose, the AR system 110 includes a head pose tracking system 1172 that provides the AR system with a head pose corresponding to the position and orientation of the wearable device. As shown in Figure 8B, the head pose tracking system 873 corresponds to the head pose tracking system 1172 and provides the head pose to the warp reprojection processor 876.

[0065]

[0080] The AR system 1100 may further include and / or communicate with a storage device 1120 (e.g., one or more non-temporary storage devices), the storage device 1120 may include, but not be limited to, a local and / or network-accessible storage device, and / or a solid-state storage device such as a disk drive, drive array, optical storage device, random access memory (RAM), and / or a read-only memory (ROM) which may be programmable, flash-updatable, and / or similar. Such a storage device may be configured to implement any suitable data storage, including, but not limited to, various file systems, database structures, and the like.

[0066]

[0081] The AR system 1100 includes a modem, a network card (wireless or wired), an infrared communication device, a wireless communication device, and / or Bluetooth. TM The communication subsystem 1150 may include, but is not limited to, chipsets such as devices, 802.11 devices, WiFi devices, WiMax devices, and cellular communication equipment. The communication subsystem 1150 may include one or more input and / or output communication interfaces for exchanging data with networks such as the networks described below, other computer systems, televisions, and / or other devices described herein. Depending on the desired functionality and / or other implementation concerns, a portable electronic device or similar device may communicate images and / or other information through the communication subsystem 1150. In other embodiments, a portable electronic device (e.g., the first electronic device) may be incorporated into the AR system 1100 (e.g., the electronic device as an input device 1130). In some embodiments, the AR system 1100 further comprises a working memory 1160 which may include a RAM or ROM device as described above.

[0067]

[0082] The AR system 1100 also includes software elements, which are currently located in working memory 1160, and includes other code such as an operating system 1162, device drivers, executable libraries, and / or one or more application programs 1164, and may consist of computer programs provided by various embodiments as described herein, and / or may be designed to implement methods provided by other embodiments, and / or may constitute a system. As mere examples, one or more procedures described with respect to the methods described above may be implemented as code and / or instructions executable by a computer and / or a processor in a computer, and in some embodiments such code and / or instructions can be used to configure and / or adapt a general-purpose computer or other device to perform one or more operations by the methods described.

[0068]

[0083] These instructions and / or sets of code may be stored in a non-temporary computer-readable storage medium, such as the storage device 1120 described above. In some cases, the storage medium may be incorporated into a computer system, such as the AR system 1100. In other embodiments, the storage medium may be separate from the computer system and be a removable medium, such as a compact disk, and / or provided in an installation package, and the storage medium can be used to program, configure, and / or adapt a general-purpose computer using the instructions / code stored therein. These instructions may take the form of executable code that can be executed by the AR system 1100, and / or in the form of source code and / or installable code. These codes become executable code when compiled and / or installed on the AR system 1100 using various commonly available compilers, installation programs, compression / decompression utilities, etc.

[0069]

[0084] It will be apparent to those skilled in the art that significant modifications may be made depending on specific requirements. For example, customized hardware may be used, and / or certain elements may be implemented in hardware, software including mobile software such as applets, or both. Furthermore, connectivity to other computing devices, such as network input / output devices, may also be employed.

[0070]

[0085] As described above, in one embodiment, several embodiments may employ a computer system such as an AR system 1100 to perform the methods according to various embodiments of the Art. According to a series of embodiments, some or all of the steps of such a method are performed by the AR system 1100 in response to one or more processors 1110 executing one or more sequences of one or more instructions. These instructions may be incorporated into other code, such as an application program 1164 contained in the operating system 1162 and / or working memory 1160. Such instructions may be read into working memory 1160 from another computer-readable medium, such as one or more storage devices 1120. As just one example, executing a sequence of instructions contained in working memory 1160 may cause one or more processors 1110 to perform one or more steps of the method described herein. Additionally or alternatively, some of the methods described herein may be performed through dedicated hardware.

[0071]

[0086] As used herein, the terms machine-readable media and computer-readable media refer to any media involved in providing data that enables a machine to operate in a particular way. In embodiments implemented using the AR system 1100, various computer-readable media may be involved in providing instructions / code to one or more processors 1110 for execution, or may be used to store and / or execute such instructions / code. In many implementations, computer-readable media are physical and / or tangible storage media. Such media may take the form of non-volatile or volatile media. Non-volatile media include, for example, optical disks and / or magnetic disks such as storage device 1120. Volatile media include, but are not limited to, dynamic memory such as working memory 1160.

[0072]

[0087] Common forms of physical and / or tangible computer-readable media include, for example, floppy disks, flexible disks, hard disks, magnetic tapes, or other magnetic media, CD-ROMs, other optical media, punch cards, paper tapes, other physical media having a pattern of holes, RAM, PROMs, EPROMs, FLASH-EPROMs, other memory chips or cartridges, or other media from which a computer can read instructions and / or code.

[0073]

[0088] When transporting one or more sequences of one or more instructions to one or more processors 1110 for execution, various forms of computer-readable media may be used. For example, the instructions may initially be stored on a magnetic disk and / or optical disk of a remote computer. The remote computer may load the instructions into dynamic memory, and the instructions may be transmitted as signals over a transmission medium and received and / or executed by the AR system 1100.

[0074]

[0089] The communication subsystem 1150 and / or its components typically receive signals, and the bus 1105 carries the signals and / or the data, instructions, etc. carried by the signals to the working memory 1160, from which one or more processors 1110 retrieve and execute the instructions. Instructions received by the working memory 1160 may optionally be stored in a storage device 1120 (e.g., a non-temporary storage device) before or after execution by one or more processors 1110.

[0075]

[0090] Various embodiments of the present disclosure are provided below. References to a series of examples used herein are understood to refer to each example separately (for example, “Examples 1-4” are understood to mean “Examples 1, 2, 3, or 4”).

[0076]

[0091] Example 1 is a method comprising determining the user's gaze position, generating a spatial fovealization map based on the gaze position, receiving an image, forming a spatial fovealized image using the image and the spatial fovealization map, transmitting the spatial fovealized image to a wearable device, de-fovealizing the spatial fovealized image and generating a de-fovealized image, and displaying the de-fovealized image.

[0077]

[0092] Example 2 is the method of Example 1, further comprising the steps of performing image warp correction based on the head posture of the wearable device, and spatially defoveating the spatially foveated image and generating the spatially defoveated image.

[0078]

[0093] Example 3 is the method of Examples 1 and 2, wherein the spatial defovealization process of the spatially fovealized image includes warping the image based on the head posture of the wearable device.

[0079]

[0094] Example 4 is the method of Examples 1 to 3, wherein the determination of the gaze position includes the use of an eye-tracking camera of an augmented reality device.

[0080]

[0095] Example 5 is the same as the method of Examples 1-4, and the spatial fovealization map includes vectors of different magnitudes as a function of distance from the line of sight position.

[0081]

[0096] Example 6 is the method of Examples 1 to 5, wherein the spatial fovealization map includes the central region and the peripheral region.

[0082]

[0097] Example 7 is the method of Examples 1 to 6, wherein the image is virtual content generated by an augmented reality device.

[0083]

[0098] Example 8 is the method of Examples 1 to 7, wherein the wearable device includes one or more displays.

[0084]

[0099] Example 9 is the method of Examples 1-8, wherein the formation of a spatially fovealed image is performed on a remote computing device, and the wearable device and the remote computing device are components of an augmented reality system.

[0085]

[0100] Example 10 is the same as the method of Examples 1 to 9, in which the formation of the spatial foveal image is performed by a cloud server.

[0086]

[0101] Example 11 is the method of Examples 1 to 10, further comprising generating an inverse spatial fovealization map before defovealization processing of the spatially fovealized image.

[0087]

[0102] Example 12 is the method of Example 11, wherein the inverse spatial fovealization map includes at least one of image warp correction, lens correction, or bending correction based on the head posture of the wearable device.

[0088]

[0103] Example 13 includes determining the user's gaze position, generating a spatial fovealization map based on the gaze position in a remote computing device, receiving an image, forming a spatial fovealization image using the image and the spatial fovealization map, transmitting the spatial fovealization image to a wearable computing device, receiving head posture correction from the wearable computing device, warping the spatial fovealization image, spatially fovealizing it to generate a display image, and displaying the display image on the wearable computing device.

[0089]

[0104] Example 14 is the method of Example 13, wherein warping a spatially fovealed image and performing spatial foveal de-fovealization processing includes forming a de-fovealized image by processing a spatially fovealed image to de-fovealize it, and then warping the de-fovealized image.

[0090]

[0105] Example 15 is a method of Examples 13-14, which includes warping a spatially foveated image and performing a spatial foveal de-fovealization process, generating a warped image by warping the spatially foveated image, and then performing a spatial foveal de-fovealization process on the warped image.

[0091]

[0106] Example 16 is the method of Examples 13-15, wherein determining the gaze position includes the use of an eye-tracking camera in an augmented reality device.

[0092]

[0107] Example 17 is a method of Examples 13-16, wherein the spatial fovealization map includes vectors of different magnitudes as a function of distance from the line of sight.

[0093]

[0108] Example 18 is the method of Examples 13-17, wherein the spatial fovealization map includes the central region and the peripheral region.

[0094]

[0109] Example 19 is the method of Examples 13 to 18, wherein the image is virtual content generated by an augmented reality device.

[0095]

[0110] Example 20 is the method of Examples 13 to 19, wherein the wearable computing device includes one or more displays.

[0096]

[0111] Example 21 is a method of Examples 13-20, in which the formation of a spatially fovealed image is performed on a remote computing device, and the wearable computing device and the remote computing device are components of an augmented reality system.

[0097]

[0112] Example 22 is the same as the method of Examples 13-21, in which the formation of the spatial foveal image is performed by a cloud server.

[0098]

[0113] Example 23 is the method of Examples 13 to 22, further comprising warping the spatially foveated image and generating an inverse spatially foveated map before spatially defoveated processing.

[0099]

[0114] Example 24 is the method of Examples 13-23, further comprising warping the spatially foveated image and performing a spatial foveal de-foveation process that performs lens correction or bending correction.

[0100]

[0115] Example 25 is an augmented reality (AR) system in which a remote computing device includes a processor, memory, and a communication system coupled to the processor, and a wearable device includes a frame, a communication device coupled to the frame, a projector coupled to the frame, a display optically coupled to the projector, an eye-tracking system, and a reprojection processor coupled to the frame.

[0101]

[0116] Example 26 is an AR system of Example 25, wherein the processor comprises a central processing unit and a graphics processing unit.

[0102]

[0117] Example 27 is an AR system of Examples 25-26, wherein the processor is configured to generate an image, a spatial fovealization map based on the gaze position, and a spatial fovealization image.

[0103]

[0118] Example 28 is an AR system of Examples 25-27, comprising virtual content in which images are generated by a processor.

[0104]

[0119] Example 29 is an AR system of Examples 25-27, wherein the communication device is configured to receive images and spatial foveal maps.

[0105]

[0120] Example 30 is an AR system of Examples 27-29, wherein the reprojection processor receives a spatial foveal image and a spatial foveal map, generates a spatial foveal de-scaling processing map based on the gaze position or spatial foveal map, and generates a spatial foveal de-scaling image based on the spatial foveal image and the spatial foveal de-scaling processing map.

[0106]

[0121] Example 31 is an AR system of Examples 27-30, wherein the spatial fovealization map includes vectors of different magnitudes as a function of distance from the line of sight position.

[0107]

[0122] Example 32 is an AR system of Examples 27-31, in which the spatial fovealization map includes a central region and a peripheral region.

[0108]

[0123] Example 33 is an AR system of Examples 25-32, further comprising a head posture tracking system.

[0109]

[0124] Example 34 is an AR system of Example 33, wherein the reprojection processor is further configured to warp images based on the head pose of the wearable device.

[0110]

[0125] Example 35 is an AR system of Examples 25-34, wherein the remote computing device includes a cloud server.

[0111]

[0126] Embodiment 36 is an augmented reality (AR) system comprising a wearable device including a frame, one or more image capture devices coupled to the frame, a set of projectors coupled to the frame, a set of displays coupled to the frame, each of the displays optically coupled to one of the set of projectors, and a set of eye-tracking devices coupled to the frame; a memory; and a processor coupled to the memory, the processor configured to determine the user's gaze position, generate a spatial fovealization map based on the gaze position, receive an image, form a spatial fovealized image using the image and the spatial fovealization map, transmit the spatial fovealized image to the wearable device, defovealize the spatial fovealized image, generate a defovealized processed image, and display the defovealized processed image.

[0112]

[0127] Example 37 is an AR device of Example 36, in which the processor is further configured to perform image warp correction based on the head pose of the wearable device and to perform spatial fovealization defovealization processing on spatially fovealized images.

[0113]

[0128] Example 38 is an AR device of Examples 36-37, wherein the processor is further configured to perform lens correction or bending correction, and to perform spatial fovealization defovealization processing on spatially fovealized images.

[0114]

[0129] Example 39 is an AR device of Examples 36-38, wherein the eye-tracking device set comprises multiple eye-tracking cameras.

[0115]

[0130] Example 40 is an AR device of Examples 36-39, further configured so that the processor generates virtual content.

[0116]

[0131] Example 41 is an AR device of Examples 36-40, further configured so that the processor generates an inverse spatial fovealization map before processing the spatially fovealized image to defovealize it.

[0117]

[0132] Example 42 is a non-temporary computer-readable medium containing program code executable by the processor of a user-wearable device, the program code being executable by the processor, determining the user's gaze position, generating a spatial fovealization map based on the gaze position, receiving an image, forming a spatial fovealized image using the image and the spatial fovealization map, transmitting the spatial fovealized image to a wearable device, de-fovealizing the spatial fovealized image, creating a de-fovealized image, and displaying the de-fovealized image.

[0118]

[0133] Example 43 is a non-temporary computer-readable medium of Example 42, further comprising program code executable by a processor that performs image warp correction based on the head orientation of a device and spatially defoveated images to create spatially defoveated processed images.

[0119]

[0134] Example 44 is a non-temporary computer-readable medium of Examples 42-43, further comprising program code executable by a processor that performs lens correction or bending correction, and spatially defoveated images.

[0120]

[0135] Example 45 is a non-temporary computer-readable medium of Examples 42-44, further comprising program code that is executable by a processor and generates virtual content.

[0121]

[0136] Example 46 is a non-temporary computer-readable medium of Examples 42-45, in which the spatial fovealization map includes vectors of different magnitudes as a function of distance from the line of sight.

[0122]

[0137] Example 47 is a non-temporary computer-readable medium of Examples 42-46, in which the spatial fovealization map includes central and peripheral regions.

[0123]

[0138] Example 48 is a non-transient computer-readable medium of Examples 42-47, wherein the processor is further configured to generate an inverse spatial fovealization map before spatial fovealization de-processing of the spatial fovealization image.

[0124]

[0139] In the aforementioned specification, the present invention has been described with reference to specific embodiments. However, it is clear that various modifications and changes can be made without departing from the broad spirit and scope of this disclosure. Therefore, the specification and drawings should be considered illustrative rather than restrictive.

[0125]

[0140] In fact, each of the systems and methods disclosed herein has multiple innovative aspects, and it will be understood that none of these aspects alone are solely involved in or required for the desired attributes disclosed herein. The various functions and processes described above may be used independently of each other or combined in various ways. The scope of the disclosure herein is intended to include all possible combinations and partial combinations.

[0126]

[0141] Certain features described in this specification in the context of separate embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable partial combination in multiple embodiments. Furthermore, even if features are described above as acting in a particular combination and initially claimed as such, one or more features from the claimed combination may be removed from the combination in some cases, and the claimed combination may be directed towards a partial combination or a variation of a partial combination. A single function or group of functions is not necessarily required or essential in all embodiments.

[0127]

[0142] The conditional language used herein, particularly “can,” “may,” “possibly,” “may,” and “for example,” will generally be understood to indicate that certain embodiments include certain features, elements, and / or steps, and that other embodiments do not, unless otherwise specifically stated or understood in the context in which they are used. Therefore, such conditional language is not generally intended to imply that features, elements, and / or steps are required in any way in one or more embodiments, nor is it intended that one or more embodiments necessarily include logic for determining whether these features, elements, and / or steps are included in or performed in a particular embodiment, with or without author input or prompting. Terms such as “equip,” “include,” and “have” are synonymous and are used inclusively and without limitation, not excluding additional elements, features, actions, operations, etc. Furthermore, the term “or” is used in an inclusive rather than exclusive sense; for example, when used to connect a list of elements, “or” means one, some, or all of the elements in the list. Furthermore, the articles “a,” “an,” and “the” used in this application and the attached claims shall be interpreted as meaning “one or more” or “at least one” unless otherwise specified. Similarly, while operations may be depicted in a particular order in the drawings, it should be recognized that such operations do not need to be performed in a particular order or sequence as shown, or that not all illustrated operations need to be performed, in order to obtain the desired result. Furthermore, drawings may schematically illustrate one or more exemplary processes in flowchart form. However, other operations not illustrated may be incorporated into the exemplary methods and processes schematically shown. For example, one or more additional operations may be performed before, after, simultaneously with, or between illustrated operations. Furthermore, in other embodiments, operations may be rearranged or reordered. In some situations, multitasking and parallel processing may be advantageous.Furthermore, the separation of various system components in the embodiments described above should not be understood as necessary in all embodiments, and the program components and systems described may generally be integrated into a single software product or packaged into multiple software products. Other embodiments are also within the scope of the following claims. In some cases, the desired results may be obtained by performing the operations described in the claims in a different order.

[0128]

[0143] Accordingly, the claims of the present invention are not intended to be construed as being limited to the embodiments shown herein, but should be construed to the maximum extent consistent with the present disclosure, its principles, and the novel features disclosed herein. Accordingly, the examples and embodiments described herein are for illustrative purposes only, and it will be understood that, in consideration of them, various modifications or changes are suggested and included within the spirit and scope of this application and the appended claims.

Claims

1. Determining the user's line of sight, To generate a spatial foveal map based on the aforementioned gaze position, Receiving images and, Using the aforementioned image and the spatial fovealization map, a spatial fovealization image is formed. Transmitting the aforementioned spatial foveal image to a wearable device, The spatially fovealized image is subjected to a spatial fovealization de-processing to generate a spatially fovealization de-processed image, Displaying the spatially foveal-decompressed image, A method that includes this.

2. The method according to claim 1, further comprising performing image warp correction based on the head posture of the wearable device, and defoveating the spatially fovealed image to generate the spatially defovealed image.

3. The method according to claim 1, wherein the spatial fovealization de-scaling of the spatially fovealized image includes performing image warping based on the head posture of the wearable device.

4. The method according to claim 1, wherein determining the gaze position includes using an eye-tracking camera of an augmented reality device.

5. The method according to claim 1, wherein the spatial fovealization map includes vectors of different magnitudes as a function of distance from the line of sight position.

6. The method according to claim 1, wherein the spatial fovealization map includes a central region and a peripheral region.

7. The method according to claim 1, wherein the image includes virtual content generated by an augmented reality device.

8. The method according to claim 1, wherein the wearable device includes one or more displays.

9. The method according to claim 1, wherein the formation of the spatially fovealed image is performed by a remote computing device, and the wearable device and the remote computing device are components of an augmented reality system.

10. The method according to claim 1, wherein the formation of the spatially fovealed image is performed by a cloud server.

11. The method according to claim 1, further comprising generating an inverse spatial foveal map before processing the spatial fovealization image to remove spatial fovealization.

12. The method according to claim 11, wherein the inverse spatial fovealization map includes at least one of image warp correction, lens correction, or bending correction based on the head posture of the wearable device.

13. Determining the user's line of sight, In a remote computing device, a spatial fovealization map is generated based on the gaze position, Receiving images and, Using the aforementioned image and the spatial fovealization map, a spatial fovealization image is formed. Transmitting the aforementioned spatial foveal image to a wearable computing device, In the wearable computing device, the wearable computing device receives head posture correction, In order to generate the display image, the spatially fovealed image is warped and the spatial fovealization is removed. A method comprising displaying the display image on the wearable computing device.

14. Warping the aforementioned spatially fovealed image and performing a spatial foveal de-scaling process is, In order to form a defovealization processed image, the spatially defovealized image is subjected to a spatial defovealization process, Warping the aforementioned foveal deformation processing image The method according to claim 13, including the method described in claim 13.

15. Warping the aforementioned spatially fovealed image and performing a spatial foveal de-serumization process is, Warping the spatially fovealed image in order to form a warped image, The method according to claim 13, further comprising processing the warped image to remove spatial foveal deformation.

16. The method according to claim 13, wherein determining the gaze position includes using an eye-tracking camera of an augmented reality device.

17. The method according to claim 13, wherein the spatial fovealization map includes vectors of different magnitudes as a function of distance from the line of sight position.

18. The method according to claim 13, wherein the spatial fovealization map includes a central region and a peripheral region.

19. The method according to claim 13, wherein the aforementioned image includes virtual content generated by an augmented reality device.

20. The method according to claim 13, wherein the wearable computing device includes one or more displays.

21. The method according to claim 13, wherein forming the spatially fovealed image is performed by a remote computing device, and the wearable computing device and the remote computing device are components of an augmented reality system.

22. The method according to claim 13, wherein the formation of the spatially fovealed image is performed by a cloud server.

23. The method according to claim 13, further comprising warping the spatial fovealization image and generating an inverse spatial fovealization map before spatial fovealization de-processing.

24. The method according to claim 13, further comprising warping the spatially foveated image and performing a spatial foveal de-fovealing process, which further includes performing lens correction or bending correction.

25. It is an augmented reality (AR) system, Processor and Memory and Includes a communication system coupled to the aforementioned processor, Remote computing devices and Frame and, A communication device coupled to the frame, A projector coupled to the aforementioned frame, A display optically coupled to the aforementioned projector, Eye-tracking system and, Includes a reprojection processor coupled to the frame, Wearable devices and, An augmented reality (AR) system equipped with [the following features].

26. The AR system according to claim 25, wherein the processor comprises a central processing unit and a graphics processing unit.

27. The AR system according to claim 25, wherein the processor is configured to generate an image, a spatial fovealization map based on the gaze position, and a spatial fovealization image.

28. The AR system according to claim 27, wherein the image includes virtual content generated by the processor.

29. The AR system according to claim 27, wherein the communication device is configured to receive the image and the spatial foveal map.

30. The aforementioned reprojection processor, The spatial fovealization image and the spatial fovealization map are received. A spatial foveal deactivation processing map is generated based on the aforementioned gaze position or the spatial fovealization map. The AR system according to claim 27, configured to generate a spatial foveal deactivation processing image based on the spatial fovealization image and the spatial foveal deactivation processing map.

31. The AR system according to claim 27, wherein the spatial fovealization map includes vectors of different magnitudes as a function of distance from the line of sight position.

32. The AR system according to claim 27, wherein the spatial fovealization map includes a central region and a peripheral region.

33. The AR system according to claim 25, further comprising a head posture tracking system.

34. The AR system according to claim 33, wherein the reprojection processor is further configured to perform image warping based on the head pose of the wearable device.

35. The AR system according to claim 25, wherein the remote computing device includes a cloud server.

36. It is an augmented reality (AR) system, It is a wearable device, Frame and, One or more image capture devices coupled to the frame, A set of projectors coupled to the aforementioned frame, A set of displays coupled to the frame, wherein each of the sets of displays is optically coupled to one of the sets of projectors, A wearable device including a set of eye-tracking devices coupled to the frame, Memory and A processor coupled to the memory, wherein the processor Determine the user's line of sight, Based on the aforementioned gaze position, a spatial fovealization map is generated. Image received, Using the aforementioned image and the spatial fovealization map, a spatial fovealization image is formed. The spatial foveal image is transmitted to a wearable device. In order to generate a spatially defoveated image, the spatially defoveated image is subjected to a spatial defoveated process. A processor configured to display the spatial foveal de-processing image, An augmented reality (AR) system equipped with [the following features].

37. The AR device according to claim 36, wherein the processor is further configured to perform image warp correction based on the head posture of the wearable device and to perform spatial fovealization de-fovealization processing on the spatially fovealized image.

38. The AR device according to claim 37, wherein the processor is further configured to perform lens correction or bending correction and to perform spatial fovealization defovealization processing on the spatially fovealized image.

39. The AR device according to claim 36, wherein the set of eye-tracking devices comprises a plurality of eye-tracking cameras.

40. The AR device according to claim 36, wherein the processor is further configured to generate virtual content.

41. The AR device according to claim 36, wherein the processor is further configured to generate an inverse spatial fovealization map before processing the spatial fovealization image to defovealize it.

42. A non-temporary computer-readable medium containing program code executable by the processor of a user-worn device, wherein the program code is: Determine the user's line of sight, Based on the aforementioned gaze position, a spatial fovealization map is generated. Image received, Using the aforementioned image and the spatial fovealization map, a spatial fovealization image is formed. The spatial foveal image is transmitted to a wearable device. The spatially fovealed image is subjected to a spatial foveal de-fovealization process to generate a spatially foveal de-fovealized image. A non-temporary, computer-readable medium executable by the processor for displaying the spatial foveal de-processing image.

43. A non-temporary computer-readable medium according to claim 42, further comprising program code executable by the processor for performing image warp correction based on the head posture of the device, and for spatially defoveating the spatially foveated image and generating the spatially defoveated processed image.

44. A non-temporary computer-readable medium according to claim 42, further comprising program code executable by the processor for performing lens correction or bending correction and for spatially defoveating the spatially foveated image.

45. A non-temporary computer-readable medium according to claim 42, further comprising program code executable by the processor for generating virtual content.

46. The non-transient computer-readable medium according to claim 42, wherein the spatial fovealization map includes vectors of different magnitudes as a function of distance from the line of sight position.

47. The non-temporary computer-readable medium according to claim 42, wherein the spatial fovealization map includes a central region and a peripheral region.

48. The non-temporary computer-readable medium according to claim 42, wherein the processor is further configured to generate an inverse spatial fovealization map before spatial fovealization defovealization processing of the spatial fovealization image.

Images