Dynamic fovea fovea focusing pipeline

Through dynamic fovea focusing technology, the image resolution is dynamically adjusted according to the user's gaze point and field of view, solving the problem of high image rendering computing burden in the prior art, and achieving efficient resource utilization and performance improvement.

CN118942372APending Publication Date: 2024-11-12APPLE INC
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411307283.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Priority Date
2018-05-07
Filing Date
2019-05-06
Publication Date
2024-11-12

AI Technical Summary

Technical Problem

The prior art has a high computational burden during image rendering in simulated reality (SR) systems, especially when different parts of an image are rendered at different resolutions on the display panel, resulting in waste of resources and degradation of performance.

Method used

Using dynamic fovea focusing technology, by dynamically adjusting resolution according to the user's gaze point and field of view in different parts of the image, distorted images are generated and corresponding scaling factors are transmitted to reduce the size of the rendered image without reducing the user experience. and handle load.

Benefits of technology

Effectively reduces the size and processing load of rendered images, reduces resource consumption, and improves the performance and efficiency of SR systems, especially when processing images in distorted space.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118942372A_ABST
    Figure CN118942372A_ABST
Patent Text Reader

Abstract

The invention relates to a dynamic fovea fovea focusing pipeline. In one implementation, a method includes receiving a warped image representing (e.g., to be displayed in a display space) simulated reality (SR) content, the warped image having a plurality of pixels at respective locations uniformly spaced in a mesh pattern in a warped space, wherein the plurality of pixels are respectively associated with a plurality of respective pixel values and a plurality of respective scaling factors that indicate a plurality of respective resolutions at a plurality of respective positions of the SR content (e.g., in the display space). The method includes processing the warped image in the warped space based on the plurality of respective scaling factors to generate a processed warped image and transmitting the processed warped image.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] This application is a divisional application of the PCT application entering the Chinese national phase, with an international filing date of May 6, 2019, national application number 201980030981.5, and invention name “Dynamic Foveal Focusing Pipeline”. Technical Field

[0002] The present disclosure relates generally to image rendering, and in particular to systems, methods, and apparatus for rendering images that simulate reality with varying amounts of detail. Background Art

[0003] A physical set refers to a world that an individual can perceive and / or interact with without the aid of an electronic system. A physical set (e.g., a physical forest) includes physical elements (e.g., physical trees, physical structures, and physical animals). An individual can directly interact with and / or perceive a physical set, such as by touch, sight, smell, hearing, and taste.

[0004] In contrast, a simulated reality (SR) set refers to a set that is fully or partially computer-created that an individual can perceive and / or that the individual can interact with via an electronic system. In SR, a subset of the individual's movements is monitored, and in response to the subset, one or more properties of one or more virtual objects in the SR set are changed in a manner that complies with one or more laws of physics. For example, the SR system may detect that the individual has taken a few steps forward, and in response thereto, adjust the graphics and audio presented to the individual in a manner similar to how such scenes and sounds would change in a physical set. Modifications to one or more properties of one or more virtual objects in the SR set may also be made in response to a representation of movement (e.g., an audio instruction).

[0005] The individual may interact with and / or perceive SR objects using any of his senses, including touch, smell, sight, taste, and sound. For example, the individual may interact with and / or perceive auditory objects that create a multi-dimensional (e.g., three-dimensional) or spatial auditory scenery and / or implement auditory transparency. The multi-dimensional or spatial auditory scenery provides the individual with the perception of discrete auditory sources in a multi-dimensional space. Auditory transparency selectively combines sounds from the physical scenery with or without computer-created audio. In some SR scenery, the individual may interact with and / or perceive only auditory objects.

[0006] An example of SR is virtual reality (VR). A VR set refers to a simulated set designed to include only computer-created sensory input for at least one sense. The VR set includes a plurality of virtual objects that an individual can interact with and / or perceive. The individual can interact with and / or perceive virtual objects in the VR set by simulating a subset of the individual's actions within the computer-created set and / or by simulating the individual or his / her presence within the computer-created set.

[0007] Another example of SR is mixed reality (MR). An MR set refers to a simulated set designed to integrate computer-created sensory input (e.g., virtual objects) with sensory input from a physical set, or representation thereof. On the reality spectrum, a mixed reality set is between VR sets at one end and fully physical sets at the other end, and does not include these sets.

[0008] In some MR sets, computer-created sensory input can adapt to changes in sensory input from the physical set. In addition, some electronic systems used to render MR sets can monitor orientation and / or position relative to the physical set to enable virtual objects to interact with real objects (i.e., physical elements from the physical set or representations thereof). For example, the system can monitor motion so that virtual plants appear stationary relative to physical buildings.

[0009] An example of mixed reality is augmented reality (AR). AR scenery refers to a simulated scenery in which at least one virtual object is superimposed on a physical scenery or a representation thereof. For example, an electronic system may have an opaque display and at least one imaging sensor, the imaging sensor being used to capture images or videos of a physical scenery, which are representations of the physical scenery. The system combines the image or video with the virtual object and displays the combination on the opaque display. An individual uses the system to indirectly view the physical scenery via an image or video of the physical scenery and observes the virtual object superimposed on the physical scenery. When the system uses one or more image sensors to capture images of the physical scenery and uses those images to present the AR scenery on an opaque display, the displayed image is referred to as video transmission. Alternatively, an electronic system for displaying an AR scenery may have a transparent or translucent display through which an individual can directly view the physical scenery. The system may display virtual objects on a transparent or translucent display so that an individual uses the system to observe virtual objects superimposed on the physical scenery. As another example, the system may include a projection system that projects virtual objects into a physical scenery. Virtual objects may be projected, for example, on a physical surface or as a hologram, such that an individual using the system observes the virtual objects superimposed on the physical scenery.

[0010] An augmented reality set may also refer to a simulated set in which a representation of a physical set is altered by computer-created sensory information. For example, a portion of the representation of a physical set may be graphically altered (e.g., enlarged) such that the altered portion may still represent one or more originally captured images but is not a faithfully reproduced version. For another example, when providing video pass-through, the system may alter at least one of the sensor images to impose a particular viewpoint that is different from the viewpoint captured by one or more image sensors. For another example, the representation of a physical set may be altered by graphically blurring or eliminating portions thereof.

[0011] Another example of mixed reality is augmented virtuality (AV). An AV set refers to a simulated set in which a computer-created set or a virtual set incorporates at least one sensory input from a physical set. The one or more sensory inputs from the physical set may be a representation of at least one feature of the physical set. For example, a virtual object may present the color of a physical element captured by one or more imaging sensors. As another example, a virtual object may present features consistent with actual weather conditions in a physical set, as identified via weather-related imaging sensors and / or online weather data. In another example, an augmented reality forest may have virtual trees and structures, but animals may have features accurately reproduced from images taken of physical animals.

[0012] Many electronic systems enable individuals to interact with and / or perceive various SR scenery. One example includes a head-mounted system. The head-mounted system may have an opaque display and one or more speakers. Alternatively, the head-mounted system may be designed to receive an external display (e.g., a smart phone). The head-mounted system may have one or more imaging sensors and / or microphones for taking images / video of the physical scenery and / or capturing audio of the physical scenery, respectively. The head-mounted system may also have a transparent or translucent display. The transparent or translucent display may be combined with a substrate through which light representing the image is directed to the individual's eyes. The display may be combined with LEDs, OLEDs, digital light projectors, laser scanning light sources, liquid crystal on silicon, or any combination of these technologies. The substrate that transmits light may be an optical waveguide, an optical combiner, an optical reflector, a holographic substrate, or any combination of these substrates. In one embodiment, the transparent or translucent display may be selectively switched between an opaque state and a transparent or translucent state. As another example, the electronic system may be a projection-based system. A projection-based system may project an image onto the retina of an individual using retinal projection. Alternatively, the projection system may also project the virtual object into a physical setting (e.g., onto a physical surface or as a hologram). Other examples of SR systems include a head-up display, a car windshield capable of displaying graphics, a window capable of displaying graphics, a lens capable of displaying graphics, headphones or earbuds, a speaker arrangement, an input mechanism (e.g., a controller with or without tactile feedback), a tablet, a smartphone, and a desktop or laptop computer.

[0013] Rendering an image to achieve an SR experience can be computationally expensive. Therefore, to reduce this computational burden, the fact that humans typically have relatively weak peripheral vision is exploited. Therefore, different portions of the image are rendered at different resolutions on the display panel. For example, in various implementations, portions corresponding to the user's focused field of view are rendered at a higher resolution than portions corresponding to the user's periphery. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] So that the present disclosure may be understood by those of ordinary skill in the art, a more detailed description may be obtained with reference to aspects of some exemplary implementations, some of which are illustrated in the accompanying drawings.

[0015] Figure 1 is a block diagram of an exemplary operating environment according to some implementations.

[0016] Figure 2 An SR pipeline is shown that receives SR content and displays an image on a display panel based on the SR content according to some implementations.

[0017] FIG. 3A to FIG. 3DVarious rendering resolution functions in the first dimension are shown according to various implementations.

[0018] FIG. 4A to FIG. 4D Various two-dimensional rendering resolution functions are shown according to various implementations.

[0019] Figure 5A An exemplary rendering resolution function that characterizes resolution in display space as a function of angle in warp space is shown according to some implementations.

[0020] Figure 5B According to some specific implementations Figure 5A An example rendering of the integral of the resolution function.

[0021] Figure 5C According to some specific implementations Figure 5A An exemplary rendering resolution function is the tangent of the integral inverse.

[0022] Fig. 6A An exemplary rendering resolution function for performing static foveation according to some implementations is shown.

[0023] Figure 6B An exemplary rendering resolution function for performing dynamic foveation according to some implementations is shown.

[0024] Figure 7 is a flowchart representation of a method for rendering an image based on a rendering resolution function according to some specific implementations.

[0025] Fig. 8A An exemplary image representation of SR content to be rendered in a display space is shown according to some implementations.

[0026] Figure 8B According to some specific implementations Fig. 8A The distorted image of the SR content.

[0027] Fig. 9 is a flowchart representation of a method of rendering an image in one of multiple foveal focus modes according to some specific implementations.

[0028] FIG. 10A to FIG. 10C Various constrained rendering resolution functions are shown according to various implementations.

[0029] Fig.11 is a flowchart representation of a method for rendering an image using a constrained rendering resolution function according to some specific implementations.

[0030] Fig.12 is a flowchart representation of a method for rendering an image based on eye tracking metadata according to some specific implementations.

[0031] FIG. 13A to FIG. 13B Various confidence-based rendering resolution functions are shown according to various implementations.

[0032] Fig.14 is a flowchart representation of a method for rendering an image based on SR content according to some specific implementations.

[0033] Fig.15 is a flowchart representation of a method of transmitting an image according to some specific implementations.

[0034] FIG. 16A to FIG. 16B An exemplary image and a wavelet image generated by a single-layer wavelet transform of the exemplary image are shown.

[0035] Fig.17 is a flowchart representation of a method for receiving an image using a constrained rendering resolution function according to some specific implementations.

[0036] FIG. 18A to FIG. 18B Example undistorted and distorted images are shown according to some implementations.

[0037] FIG. 19A to FIG. 19B Exemplary pre-distorted images and distorted pre-distorted images (or undistorted images) according to some implementations are shown.

[0038] Fig. 20 is a flowchart representation of a method of pre-distorting an image according to some specific implementations.

[0039] FIG. 21A to FIG. 21B A representation of panel data indicating multiple resolution zones is shown according to some implementations.

[0040] Fig. 22 A graph showing an exemplary resolution function defined by a scaling factor and a resolution region according to some implementations is shown.

[0041] Fig.23 Another graph illustrating an exemplary resolution function defined by a scaling factor and a resolution region according to some implementations is shown.

[0042] Fig.24 is a flowchart representation of a method for generating panel data according to some specific implementations.

[0043] Fig.25 is a flowchart representation of a method for processing a warped image in a warped space according to some specific implementations.

[0044] Fig.26is a flowchart representation of a method for processing a distorted image based on multiple scaling factors according to some specific implementations.

[0045] As is common practice, the various features shown in the drawings may not be drawn to scale. Therefore, the sizes of the various features may be arbitrarily expanded or reduced for clarity. In addition, some drawings may not depict all components of a given system, method, or device. Finally, throughout the specification and drawings, similar reference numerals may be used to represent similar features. Summary of the invention

[0046] Various specific implementations disclosed herein include devices, systems, and methods for processing images in a warped space. The method includes receiving a warped image representing simulated reality (SR) content to be displayed in a display space, the warped image having a plurality of pixels at corresponding positions evenly spaced in a grid pattern in the warped space, wherein the plurality of pixels are respectively associated with a plurality of corresponding pixel values ​​and a plurality of corresponding scaling factors, the plurality of corresponding scaling factors indicating a plurality of corresponding resolutions at a plurality of corresponding positions in the display space. The method includes processing the warped image in the warped space based on the plurality of corresponding scaling factors to generate a processed warped image and transmitting the processed warped image.

[0047] Various implementations disclosed herein include devices, systems, and methods for processing an image based on multiple scaling factors. In one implementation, the method includes receiving a warped image representing simulated reality (SR) content, the warped image having a plurality of pixels at corresponding locations evenly spaced in a grid pattern in a warped space, wherein the plurality of pixels are respectively associated with a plurality of corresponding pixel values ​​and a plurality of corresponding scaling factors, the plurality of corresponding scaling factors indicating a plurality of corresponding resolutions at a plurality of corresponding locations of the SR content. The method includes processing the warped image based on the plurality of corresponding scaling factors to generate a processed warped image and transmitting the processed warped image.

[0048] According to some specific implementations, a device includes one or more processors, non-volatile memory, and one or more programs; the one or more programs are stored in the non-volatile memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing or causing the execution of any of the methods described herein. According to some specific implementations, a non-volatile computer-readable storage medium stores instructions that, when executed by one or more processors of the device, cause the device to perform or cause the execution of any of the methods described herein. According to some specific implementations, a device includes: one or more processors, non-volatile memory, and means for performing or causing the execution of any of the methods described herein. DETAILED DESCRIPTION

[0049] Many details are described in order to provide a thorough understanding of the example implementations shown in the accompanying drawings. However, the accompanying drawings only illustrate some example aspects of the present disclosure and should not be considered limiting. One of ordinary skill in the art will appreciate that other effective aspects and / or variations do not include all of the specific details described herein. In addition, well-known systems, methods, components, devices, and circuits are not described in detail in order to avoid obscuring more relevant aspects of the exemplary implementations described herein.

[0050] As described above, in various implementations, different portions of an image are rendered at different resolutions on a display panel. Various methods for determining the resolution of different portions of an image based on a plurality of factors will be described below.

[0051] Figure 1 1 is a block diagram of an exemplary operating environment 100 according to some implementations. Although relevant features are shown, one of ordinary skill in the art will recognize from this disclosure that various other features are not shown for the sake of brevity and so as not to obscure more relevant aspects of the exemplary implementations disclosed herein. To this end, as a non-limiting example, the operating environment 100 includes a controller 110 and a head mounted device (HMD) 120.

[0052] In some implementations, the controller 110 is configured to manage and coordinate the user's simulated reality (SR) experience. In some implementations, the controller 110 includes a suitable combination of software, firmware, and / or hardware. In some implementations, the controller 110 is a computing device that is locally or remotely located relative to the scene 105. For example, the controller 110 is a local server located within the scene 105. As another example, the controller 110 is a remote server (e.g., a cloud server, a central server, etc.) located outside the scene 105. In some implementations, the controller 110 is communicatively coupled to the HMD 120 via one or more wired or wireless communication channels 144 (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE802.3x, etc.). As another example, the controller 110 is contained within the housing of the HMD 120.

[0053] In some implementations, HMD 120 is configured to present a SR experience to a user. In some implementations, HMD 120 includes a suitable combination of software, firmware, and / or hardware. In some implementations, the functionality of controller 110 is provided by and / or combined with HMD 120.

[0054] According to some implementations, the HMD 120 provides a SR experience to the user while the user is virtually and / or physically present within the scene 105. In some implementations, when presenting an AR experience, the HMD 120 is configured to present AR content (e.g., one or more virtual objects) and to enable an optical see-through of the scene 105. In some implementations, when presenting an AR experience, the HMD 120 is configured to present AR content (e.g., one or more virtual objects) that is superimposed or otherwise combined with an image or portion thereof captured by a scene camera of the HMD 120. In some implementations, when presenting AV content, the HMD 120 is configured to present real-world elements or representations thereof that are combined with or superimposed on a user's view of a computer-simulated environment. In some implementations, when presenting a VR experience, the HMD 120 is configured to present VR content.

[0055] In some implementations, the user wears the HMD 120 on the head. Thus, the HMD 120 includes one or more SR displays provided for displaying SR content (optionally through an eyepiece or other optical lens system). For example, in various implementations, the HMD 120 surrounds the user's field of view. In some implementations, a handheld device (such as a smartphone or tablet) configured to present SR content is used in place of the HMD 120, wherein the user does not wear the HMD 120 but instead holds the device with the display facing the user's field of view and the camera facing the scene 105. In some implementations, the handheld device may be placed in a housing that can be worn on the user's head. In some implementations, an SR cabin, housing, or chamber configured to present SR content is used in place of the HMD 120, wherein the user does not wear or hold the HMD 120.

[0056] In various implementations, HMD 120 includes a SR pipeline that renders SR content. Figure 2 An SR pipeline 200 is shown that receives SR content and displays an image on a display panel 240 based on the SR content.

[0057] The SR pipeline 200 includes a rendering module 210 that receives SR content (and eye tracking data from an eye tracker 260) and renders an image based on the SR content. In various implementations, the SR content includes definitions of the geometry of virtual objects, colors and / or textures of virtual objects, images (such as perspective images of a scene), and other information describing the content to be represented in the rendered image.

[0058] The image includes a pixel matrix, each pixel having a corresponding pixel value and a corresponding pixel position. In various specific implementations, the pixel value ranges from 0 to 255. In various specific implementations, each pixel value is a color triplet, which includes three values ​​corresponding to three color channels. For example, in one specific implementation, the image is an RGB image, and each pixel value includes a red value, a green value, and a blue value. For example, in one specific implementation, the image is a YUV image, and each pixel value includes a brightness value and two chromaticity values. In various specific implementations, the image is a YUV444 image, wherein each chromaticity value is associated with a pixel. In various specific implementations, the image is a YUV420 image, wherein each chromaticity value is associated with a 2×2 pixel block (e.g., the chromaticity value is downsampled). In some specific implementations, the image includes a tile matrix, each tile has a corresponding tile position and includes a pixel block with a corresponding pixel value. In some specific implementations, each tile is a 32×32 pixel block. Although specific pixel values, image formats, and tile sizes are provided, it should be understood that other values, formats, and tile sizes may be used.

[0059] The image rendered by the rendering module 210 (e.g., the rendered image) is provided to the transmission module 220, which couples the rendering module 210 to the display module 230. The transmission module 220 includes a compression module 222 that compresses the rendered image (thereby generating a compressed image), a communication channel 224 that transmits the compressed image, and a decompression module 226 that decompresses the compressed image (thereby generating a decompressed image).

[0060] The decompressed image is provided to the display module 230, which converts the decompressed image into panel data. The panel data is provided to the display panel 240, which displays the displayed image as described by the panel data (e.g., according to the panel data). The display module 230 includes a lens compensation module 232, which compensates for the distortion caused by the eyepiece 242 of the HMD. For example, in various specific implementations, the lens compensation module 232 pre-distorts the decompressed image with an inverse relationship to the distortion caused by the eyepiece 242, so that when the user 250 observes through the eyepiece 242, the displayed image appears to be distortion-free. The display module 230 also includes a panel compensation module 234, which converts the image data into panel data for reading by the display panel 240.

[0061] The eyepiece 242 limits the resolution that can be perceived by the user 250. In various implementations, the maximum resolution that the eyepiece 242 can support is expressed as an eyepiece resolution function that varies with the distance from the origin of the display space. In various implementations, the maximum resolution that the eyepiece 242 can support is expressed as an eyepiece resolution function that varies with the angle between the viewing axis of the user 250 and the viewing axis when the user 250 is looking at the center of the eyepiece 242. In various implementations, the maximum resolution that the eyepiece 242 can support is expressed as an eyepiece resolution function that varies with the angle between the viewing axis of the user 250 and the viewing axis when the user 250 is looking at the center of the display panel 240.

[0062] The display panel 240 includes a matrix of M×N pixels located at corresponding positions in a display space. The display panel 240 displays a displayed image as described by (eg, according to) the panel data by emitting light from each pixel.

[0063] In various implementations, the SR pipeline 200 includes an eye tracker 260 that generates eye tracking data indicating a gaze of the user 250. In various implementations, the eye tracking data includes data indicating a gaze point of the user 250 on the display panel 240. In various implementations, the eye tracking data includes data indicating a gaze angle of the user 250, such as an angle between a current viewing axis of the user 250 and a viewing axis when the user 250 is looking at the center of the display panel 240.

[0064] In one specific implementation, to render an image to be displayed on the display panel 240, the rendering module 210 generates M×N pixel values ​​for each pixel of the M×N images. Therefore, each pixel of the rendered image corresponds to a pixel of the display panel 240 having a corresponding position in the display space. Therefore, the rendering module 210 generates pixel values ​​for the M×N pixel positions evenly spaced in a grid pattern in the display space. Equivalently, the rendering module 210 generates a tile of T×T pixels at the M / T×N / T tile positions evenly spaced in the grid pattern in the display space, each pixel having a corresponding pixel value.

[0065] Rendering M×N pixel values ​​may be computationally expensive. In addition, as the size of the rendered image increases, the amount of processing required to compress the image at compression module 222, the amount of bandwidth required to transmit the compressed image across communication channel 224, and the amount of processing required to decompress the compressed image at decompression module 226 also increases.

[0066] In various embodiments, in order to reduce the size of the rendered image without reducing the user experience, foveal focus (e.g., foveal focused imaging) is used. Foveal focus is a digital image processing technique in which the image resolution or amount of detail varies across the image. Therefore, the foveal focused image has different resolutions at different parts of the image. Humans generally have relatively weak peripheral vision. According to one model, the user's resolvable resolution reaches a maximum value within the gaze field of view (e.g., where the user is staring) and decreases in an inverse linear manner. Therefore, in one embodiment, the displayed image displayed by the display panel 240 is a foveal focused image having a maximum resolution at the focused field of view and a resolution that decreases in an inverse linear manner proportional to the distance from the focused field of view.

[0067] Because some portions of the image have a lower resolution, the M×N foveal focused image includes less information than the M×N non-foveal focused image. Therefore, in various embodiments, the rendering module 210 generates the foveal focused image in the form of a rendered image. The rendering module 210 can generate the M×N foveal focused image more quickly and with less processing power (and battery power) than the rendering module 210 can generate the M×N non-foveal focused image. In addition, the M×N foveal focused image can be represented using less data than the M×N non-foveal focused image. In other words, the size of the M×N foveal focused image file is smaller than the M×N non-foveal focused image file. In various embodiments, compressing the M×N foveal focused image using various compression techniques produces fewer bits than compressing the M×N non-foveal focused image.

[0068] The foveal focus ratio R may be defined as the amount of information in the M×N non-foveal focused images divided by the amount of information in the M×N foveal focused images. In various implementations, the foveal focus ratio is between 1.5 and 10. For example, in some implementations, the foveal focus ratio is 2. In some implementations, the foveal focus ratio is 3 or 4. In some implementations, the foveal focus ratio is constant across images. In some implementations, the foveal focus ratio is selected based on the image to be rendered.

[0069] In some implementations, to render an image to be displayed on the display panel 240, the rendering module 210 generates M / R×N / R pixel values ​​for each pixel of the M / R×N / R warped images. Each pixel of the warped image corresponds to a larger area than the pixel of the display panel 240 at the corresponding position in the display space. Therefore, the rendering module 210 generates a pixel value for each of the M / R×N / R positions in the display space that are not evenly distributed in the grid pattern. Similarly, in some implementations, the rendering module 210 generates a tile of T×T pixels at each of the M / (RT)×N / (RT) positions in the display space that are not evenly distributed in the grid pattern, each pixel having a corresponding pixel value. The corresponding area in the display space corresponding to each pixel value (or each tile) is defined by the corresponding position in the display space (rendering position) and the scaling factor (or a set of horizontal scaling factors and vertical scaling factors).

[0070] In various implementations, the rendering module 210 generates the warped image in the form of a rendered image. In various implementations, the warped image includes a matrix of M / R×N / R pixel values ​​at M / R×N / R locations that are evenly spaced in a grid pattern in a warp space that is different from the display space. Specifically, the warped image includes a matrix of M / R×N / R pixel values ​​at M / R×N / R locations in the display space that are not evenly distributed in the grid pattern. Therefore, although the resolution of the warped image is uniform in the warp space, the resolution varies in the display space. Fig. 8A and Figure 8B Describe this in more detail.

[0071] The rendering module 210 determines a rendering position and a corresponding scaling factor based on a rendering resolution function, which generally characterizes a resolution of a rendered image in a display space.

[0072] In one implementation, the rendering resolution function S(x) varies with the distance from the origin of the display space (which may correspond to the center of the display panel 240). In another implementation, the rendering resolution function S(θ) varies with the angle between the viewing axis of the user 250 and the viewing axis when the user 250 is looking at the center of the display panel 240. Therefore, in one implementation, the rendering resolution function S(θ) is expressed in pixels per degree (PPD).

[0073] Humans typically have relatively poor peripheral vision. According to one model, the resolution that a user can discern is maximum within the focused field of view (where the user is gazing) and decreases in an inverse linear manner as the angle increases from the visual axis. Thus, in one specific implementation, the rendering resolution function (in the first dimension) is defined as:

[0074]

[0075] Where S max is the maximum value of the rendering resolution function (e.g., about 60 PPD), S min is the asymptote of the rendering resolution function, θ fof represents the size of the focused field of view, and w represents the width of the rendering resolution function.

[0076] Figure 3A A rendering resolution function 310 (in the first dimension) is shown decreasing from the focused field of view in an inverse linear fashion. Figure 3B A rendering resolution function 320 (in a first dimension) is shown decreasing in a linear fashion from the focused field of view. Figure 3C A rendered resolution function 330 (in a first dimension) that approximates a Gaussian function is shown. Figure 3D A rendering resolution function 340 is shown decreasing in a circular step fashion (in a first dimension).

[0077] FIG. 3A to FIG. 3D Each of the rendering resolution functions 310-340 is in the form of a peak including a peak height (e.g., a maximum value) and a peak width. The peak width can be defined in a variety of ways. In one embodiment, the peak width is defined as the size of the focused field of view (e.g., Figure 3A The width is 311 and Figure 3B In one embodiment, the peak width is defined as the full width at half maximum (eg Figure 3C In one embodiment, the peak width is defined as the distance between the two inflection points closest to the origin (e.g. Figure 3D The width is shown as 341).

[0078] Although FIG. 3A to FIG. 3D The rendering resolution function is shown in a single dimension, but it should be understood that the rendering resolution function used by the rendering module 210 may be a two-dimensional function. Figure 4A A two-dimensional rendering resolution function 410 is shown, where the rendering resolution function 410 has two dimensions (θ) in the horizontal dimension and the vertical dimension. is independent. Figure 4B A two-dimensional rendering resolution function 420 is shown, where the rendering resolution function 420 is a function of a single variable (e.g., ). Figure 4C A two-dimensional rendering resolution function 430 is shown, where the rendering resolution function 430 has two dimensions (θ) in the horizontal dimension and the vertical dimension. is different. Figure 4D A two-dimensional rendering resolution function 440 based on a human vision model is shown.

[0079] As described in detail below, the rendering module 210 generates a rendering resolution function based on multiple factors, including biological information related to human vision, eye tracking data, eye tracking metadata, SR content, and various constraints (such as constraints imposed by the hardware of the HMD).

[0080] Figure 5A An exemplary rendering resolution function 510 (denoted as S(θ)) is shown, which characterizes the resolution in display space as a function of angle in warp space. The rendering resolution function 510 is within the focus field of view (between -θ fof With +θ fof between) is a constant (for example, S max ) and decreases in an inverse linear manner outside this window.

[0081] Figure 5B The field of view (e.g., -θ fov To +θ fov )of Figure 5A The integral 520 of the rendering resolution function 510 (denoted as U(θ)). Therefore, The range of integral 520 is -θ fov 0 to +θ fov The maximum value under (denoted as U max ).

[0082] Figure 5C Shows Figure 5A The inverse tangent 530 of the integral 520 of the rendering resolution 510 (denoted as V(x R )). Therefore, V(x R )=tan(U -1 (x R )). Tangent 530 shows the R Rendering space in x D According to the foveal focus indicated by the rendering resolution function 510, uniform sampling points in the warped space (along x R axis) corresponds to non-uniform sampling points in display space (along the x D The scaling factor can be determined by the distance between the non-uniform sampling points in display space.

[0083] When performing static foveal focusing, the rendering module 210 uses a rendering resolution function that is not dependent on the user's gaze. However, when performing dynamic foveal focusing, the rendering module 210 uses a rendering resolution function that is dependent on the user's gaze. Specifically, when performing dynamic foveal focusing, the rendering module 210 uses a rendering resolution function that has a peak height at a location corresponding to the location in the display space that the user is looking at (e.g., the gaze point as determined by the eye tracker 260).

[0084] Fig. 6A 6 shows a rendering resolution function 610 that the rendering module 210 can use when performing static foveal focusing. The rendering module 210 can also use when performing dynamic foveal focusing and the user is looking at the center of the display panel 240. Fig. 6A The rendering resolution function 610 is shown. Figure 6B The following diagram shows a situation in which dynamic foveal focusing is performed and the user is moving at an angle (θ) away from the center of the display panel 240. g ) is viewed using a rendering resolution function 620 that can be used by a rendering module.

[0085] Figure 7 700 is a flowchart representation of a method 700 for rendering an image according to some implementations. In some implementations (and as described in detail below as an example), the method 700 is performed by a rendering module (such as Figure 2 In various implementations, the method 700 is performed by an HMD (such as a Figure 1 HMD 100) or a portion thereof (such as Figure 2 SR pipeline 200). In various implementations, method 700 is performed by a device having one or more processors, non-transitory memory, and one or more SR displays. In some implementations, method 700 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 700 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0086] The method 700 begins at block 710, when the rendering module obtains SR content to be rendered into the display space. In various implementations, the SR content may include a definition of the geometry of a virtual object, the color and / or texture of a virtual object, an image (such as a perspective image of a scene), or other information describing the content to be represented in the rendered image.

[0087] The method 700 continues at block 720 when the rendering module obtains a rendering resolution function that defines a mapping between display space and warp space. FIG. 3A to FIG. 3D and FIG. 4A to FIG. 4DVarious rendering resolution functions are shown in . Various methods of generating rendering resolution functions are further described below.

[0088] In various implementations, the rendering resolution function generally characterizes the resolution of the rendered image in the display space. Thus, the integral of the rendering resolution function provides a mapping between the display space and the warp space (e.g., FIG. 5A to FIG. 5C ). In one implementation, the rendering resolution function S(x) varies with the distance from the origin of the display space. In another implementation, the rendering resolution function S(θ) varies with the angle between the user's visual axis and the user's visual axis when the user is looking at the center of the display panel. Therefore, the rendering resolution function characterizes the resolution in the display space that varies with the angle (in the display space). Therefore, in one implementation, the rendering resolution function S(θ) is expressed in pixels per degree (PPD).

[0089] In various implementations, the rendering module performs dynamic foveal focusing and the rendering resolution function depends on the user's gaze. Thus, in some implementations, obtaining the rendering resolution function includes, for example, obtaining Figure 2 The eye tracker 260 of the embodiment of the present invention obtains eye tracking data indicating a gaze of the user, and generates a rendering resolution function based on the eye tracking data. In various specific implementations, the eye tracking data includes at least one of data indicating a gaze angle of the user or data indicating a gaze point of the user. Specifically, in various specific implementations, generating the rendering resolution function based on the eye tracking data includes generating a rendering resolution function having a peak height at a position where the user is looking as indicated by the eye tracking data.

[0090] The method 700 continues at block 730, where the rendering module generates a rendered image based on the SR content and the rendering resolution function. The rendered image includes a warped image having a plurality of pixels at corresponding locations evenly spaced in a grid pattern in the warp space. The plurality of pixels are respectively associated with a plurality of corresponding pixel values ​​based on the SR content. The plurality of pixels are respectively associated with a plurality of corresponding scaling factors that define an area in the display space based on the rendering resolution function.

[0091] An image that is referred to as being in display space has evenly spaced regions (e.g., pixels or groups of pixels) that map to evenly spaced regions (e.g., pixels or groups of pixels) of a display. An image that is referred to as being in warp space has evenly spaced regions (e.g., pixels or groups of pixels) that map to non-evenly spaced regions (e.g., pixels or groups of pixels) in display space. The relationship between evenly spaced regions in warp space and non-evenly spaced regions in display space is defined, at least in part, by a scaling factor. Thus, the plurality of corresponding scaling factors (as with the rendering resolution function) define a mapping between warp space and display space.

[0092] In various implementations, the warped image includes a plurality of tiles at corresponding positions evenly spaced in a grid pattern in the warped space, and each tile of the plurality of tiles is associated with a corresponding one or more scaling factors. For example, in some implementations, each tile (comprising a plurality of pixels) is associated with a single horizontal scaling factor and a single vertical scaling factor. In some implementations, each tile is associated with a single scaling factor for both horizontal and vertical scaling. In various implementations, each tile is a 32×32 pixel matrix.

[0093] In various implementations, the rendering module transmits the warped image including the plurality of pixel values ​​associated with the plurality of respective scaling factors. Thus, the warped image and scaling factors are propagated through the pipeline, rather than a foveated focused image that can be generated using this information.

[0094] Specifically, relative to Figure 2 In various embodiments, the rendering module 210 generates a warped image and a plurality of corresponding scaling factors that are transmitted by the rendering module 210. At various stages in the pipeline 200, the warped image (or a processed version of the warped image) and the plurality of corresponding scaling factors are received by the transmission module 220 (and its compression module 222 and decompression module 226) (and used to process the warped image), as further described below. At various stages in the pipeline 200, the warped image (or a processed version of the warped image) and the plurality of corresponding scaling factors are received by the display module 230 (and its lens compensation module 232 and panel compensation module 234) (and used to process the warped image), as further described below.

[0095] In various implementations, the rendering module generates a scaling factor based on a rendering resolution function. For example, in some implementations, based on the above relative FIG. 5A to FIG. 5C In various embodiments, generating the scaling factor comprises determining an integral of the rendering resolution function. In various embodiments, generating the scaling factor comprises determining a tangent of an inverse of the integral of the rendering resolution function. In various embodiments, generating the scaling factor comprises determining a corresponding scaling factor for each of the corresponding positions evenly spaced in the grid pattern in the warped space based on the tangent of the inverse of the integral of the rendering resolution function. Thus, for a plurality of evenly spaced positions in the warped space, a plurality of non-evenly spaced positions in the display space are represented by the scaling factor.

[0096] Fig. 8A A graphic representation of SR content 810 to be rendered in a display space is shown. Figure 8B Shown according to Figure 7The warped image 820 is generated by the method 700. Different portions of the SR content 810 corresponding to non-uniformly spaced regions (e.g., regions of different amounts) in the display space are rendered into evenly spaced regions (e.g., regions of the same amount) in the warped image 820 according to the rendering resolution function.

[0097] For example, a graph comprising K pixels (and K pixel values) Figure 8B The region in the distorted image 820 is represented by Fig. 8A The area at the center of the image representation of the SR content 810. Similarly, the image representation of the SR content 810 is composed of K pixels (and K pixel values). Figure 8B The region in the distorted image 820 is represented by Fig. 8A The image of the SR content 810 shows the area on the corner ( Fig. 8A The area at the center is larger).

[0098] As described above, the rendering module 210 may perform static foveal focusing or dynamic foveal focusing. In various specific implementations, the rendering module 210 determines a foveal focusing mode to be applied to render the SR content, and performs static foveal focusing or dynamic foveal focusing according to the determined foveal focusing mode. In the static foveal focusing mode, the SR content is rendered independently of the eye tracking data. In the non-foveal focusing mode, the rendered image is characterized by a fixed resolution per display region (e.g., a constant number of pixels per tile). In the dynamic foveal focusing mode, the resolution of the rendered image depends on the user's gaze.

[0099] Fig. 9 900 is a flowchart representation of a method 900 for rendering an image according to some implementations. In some implementations (and as described in detail below as an example), the method 900 is performed by a rendering module (such as Figure 2 In various implementations, the method 900 is performed by an HMD (such as a Figure 1 HMD 100) or a portion thereof (such as Figure 2 SR pipeline 200). In various implementations, method 900 is performed by a device having one or more processors, non-transitory memory, and one or more SR displays. In some implementations, method 900 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 900 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0100] Method 900 begins in block 910 when a rendering module obtains eye tracking data indicating a user's gaze (e.g., where the user is looking, such as the user's gaze direction or gaze point). In various implementations, the eye tracking data includes at least one of data indicating a user's gaze angle or data indicating a user's gaze point.

[0101] The method 900 continues in block 920, where the rendering module obtains SR content to be rendered. In various implementations, the SR content may include a definition of the geometry of a virtual object, the color and / or texture of a virtual object, an image (such as a perspective image of a scene), or other information describing the content to be represented in a rendered image.

[0102] Method 900 continues in box 930, when the rendering module determines the fovea focus mode to be applied for rendering the SR content. In various embodiments, the rendering module determines the fovea focus mode based on various factors. In some embodiments, the rendering module determines the fovea focus mode based on rendering processor characteristics. For example, in some embodiments, the rendering module determines the fovea focus mode based on available processing power, processing speed, or processor type of the rendering processor of the rendering module. When the rendering module has large available processing power (due to large processing capacity or low usage of the processing capacity), the rendering module selects the dynamic fovea focus mode, and when the rendering module has small available processing power (due to small processing capacity or high usage of the processing capacity), the rendering module selects the static fovea focus mode or the non-fovea focus mode. Reference Figure 1 , when rendering is performed by the controller 110 (e.g., the rendering processor is at the controller), the rendering module selects the dynamic foveal focus mode, and when rendering is performed by the HMD 120 (e.g., the rendering processor is at the HMD), the rendering module selects the static foveal focus mode or the non-foveal focus mode. In various implementations, switching between the static foveal focus mode and the dynamic foveal focus mode is performed based on characteristics of the HMD 120 (such as the processing power of the HMD 120 relative to the processing power of the controller 110).

[0103] In some implementations, when eye tracking performance (e.g., reliability) sufficiently degrades, the rendering module selects a static foveal focus mode or a non-foveal focus mode. For example, in some implementations, when eye tracking is lost, a static foveal focus mode or a non-foveal focus mode is selected. For another example, in some implementations, when eye tracking performance exceeds a threshold, such as when eye tracking accuracy drops too low (e.g., due to large gaps in eye tracking data) and / or a delay associated with eye tracking exceeds a certain value, a static foveal focus mode or a non-foveal focus mode is selected. In some implementations, the rendering module moves the focus to the center of the HMD 120, and when it is suspected that eye tracking performance has weakened during dynamic foveal focus (e.g., after a timeout, as indicated by a low prediction confidence), a static foveal focus is used to gradually increase the gaze field of view (FoF).

[0104] In various implementations, the rendering module selects the static foveal focus mode or the non-foveal focus mode to account for other considerations. For example, in some implementations, the rendering module selects the static foveal focus mode or the non-foveal focus mode when superior eye tracking sensor performance is desired. For another example, in some implementations, the rendering module selects the static foveal focus mode or the non-foveal focus mode when the user wearing the HMD 120 has a medical condition that prevents or substantially ineffective eye tracking.

[0105] In various implementations, a static foveal focus mode or a non-foveal focus mode is selected because it provides better performance in various aspects of the rendering imaging system. For example, in some implementations, a static foveal focus mode or a non-foveal focus mode provides better rate control. As another example, in some implementations, a static foveal focus mode or a non-foveal focus mode provides better hiding of mixed foveal focus areas and non-foveal focus areas (e.g., by making the lines defining these areas lighter). As another example, in some implementations, a static foveal focus mode or a non-foveal focus mode provides better display panel consumption bandwidth, for example, by using static grouping compensation data to maintain similar power and / or bandwidth. As another example, in some implementations, a static foveal focus mode or a non-foveal focus mode reduces the risk of rendering undesirable visual aspects such as flicker and / or artifacts (e.g., grouped rolling emission shear artifacts).

[0106] The method 900 continues at decision block 935. Based on the determination that the foveal focus mode is a dynamic foveal focus mode, the method 900 continues at block 940 where the rendering module renders the SR content based on the eye tracking data according to the dynamic foveal focus (e.g., as described above with respect to FIG. Figure 7 Based on the determination that the foveal focus mode is a static foveal focus mode, method 900 continues at block 942 where the rendering module renders the SR content based on the static foveal focus independently of the eye tracking data (e.g., as described above with respect to Figure 7 Based on the determination that the foveal focus mode is a non-foveal focus mode, the method 900 continues in block 944 where the rendering module renders the SR content without foveal focus.

[0107] In various implementations, method 900 returns to block 920 where additional SR content is received. In various implementations, the rendering module renders different SR content in different foveal focus modes based on the changing conditions. Although shown in a particular order, it should be understood that the blocks of method 900 may be performed in a different order or simultaneously. For example, eye tracking data may be obtained throughout the execution of method 900 (e.g., as in block 910), and blocks that rely on this data may use any previously obtained (e.g., most recently obtained) eye tracking data or variations thereof (e.g., windowed averages, etc.).

[0108] Fig. 10A An eyepiece resolution function 1020, E(θ), is shown as a function of angle. The eyepiece resolution function 1020 has a maximum at the center of the eyepiece 242 and decreases toward the edge. In various implementations, the eyepiece resolution function 1020 comprises a portion of a circle, an ellipse, a parabola, or a hyperbola.

[0109] Fig. 10A Also shown are the g ) is the peak centered at the unconstrained rendering resolution function 1010, S u (θ). Around this peak, the unconstrained rendering resolution function 1010 is greater than the eyepiece resolution function 1020. Therefore, if the rendering module 210 were to render an image having the resolution indicated by the unconstrained rendering resolution function 1010, the eyepiece 242 would strip away details at these angles. Therefore, in various implementations, in order to avoid computational overhead and delay when rendering these details, the rendering module 210 generates an upper bounded rendering resolution function 1030 (in bold), S c (θ), which is equal to the smaller of the eyepiece resolution function 1010 and the unconstrained rendering resolution function. Therefore, S c (θ)=min(E(θ),S u (θ)).

[0110] In various implementations, the amount of computational overhead and latency associated with rendering the rendered image by the rendering module 210 remains relatively constant (e.g., normalized) regardless of the gaze angle of the user 250. Thus, in various implementations, the rendering module 210 renders the rendered image using a rendering resolution function having a fixed summation value that indicates the total amount of detail in the rendered image. In various implementations, the summation value is generally equal to the integral of the rendering resolution function within the field of view. In other words, the summation value corresponds to the area under the rendering resolution function within the field of view. In various implementations, the summation value corresponds to the number of pixels, tiles, and / or (x, y) positions in the rendered image.

[0111] The summed value of the upper bound rendering resolution function 1030 is less than the summed value of the unconstrained rendering resolution function 1010. To generate a rendering resolution function having a larger summed value (e.g., equal to the fixed summed value), the rendering module increases the values ​​of the upper bound rendering resolution function 1030 that are not decreased compared to the unconstrained rendering resolution function 1010. For example, Fig. 10B A first constrained rendering resolution function 1032 is shown, wherein the descending portion of the rendering resolution function is increased compared to the descending portion of the upper-bounded rendering resolution function 1030. Fig. 10C A second constrained rendering resolution function 1034 is shown, wherein the peak width of the rendering resolution function is increased compared to the peak width of the capped rendering resolution function 1030 .

[0112] Fig.11 1 is a flowchart representation of a method 1100 for rendering an image according to some implementations. In some implementations (and as described in detail below as an example), the method 1100 is performed by a rendering module (such as Figure 2 In various implementations, the method 1100 is performed by an HMD (such as a Figure 1 HMD 100) or a portion thereof (such as Figure 2 100 is performed by a device having one or more processors, non-transitory memory, and one or more SR displays. In some implementations, the method 1100 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 1100 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0113] The method 1100 begins in block 1110 when a rendering module obtains eye tracking data indicating a user's gaze (e.g., where the user is looking, such as the user's gaze direction and / or gaze point). In various implementations, the rendering module receives data indicating at least a performance characteristic of the eyepiece when the user is gazing. In various implementations, the performance characteristic of the eyepiece when the user is gazing may be determined from the eye tracking data.

[0114] The method 1110 continues in block 1120 when the rendering module generates a rendering resolution function based on the eye tracking data, the rendering resolution function having a maximum value that is dependent on the eye tracking data and a summed value that is independent of the eye tracking data.

[0115] In various implementations, generating the rendering resolution function includes generating an unconstrained rendering resolution function (such as Fig. 10A unconstrained rendering resolution function 1010); based on eye tracking data (and optionally, an eyepiece resolution function such as Fig. 10A eyepiece resolution function 1020) to determine the maximum value (of the rendering resolution function after the constraint); reducing the values ​​of the unconstrained rendering resolution function above the maximum value to the maximum value in order to generate a rendering resolution function with an upper limit (such as Fig. 10A The method of claim 1030 is to increase the non-decreased value of the capped rendering resolution function to generate the rendering resolution function. In various implementations, increasing the non-decreased value of the capped rendering resolution function includes increasing a decreasing portion of the capped rendering resolution function. In some implementations, the peripheral portion of the rendering resolution function decreases in an inverse linear manner (e.g., hyperbolically). In various implementations, increasing the non-decreased value of the capped rendering resolution function includes increasing a peak width of the capped rendering resolution function, such as increasing a size of a focused field of view.

[0116] In various implementations, the maximum value is based on a mapping between the user's gaze and a lens performance characteristic. In some implementations, the lens performance characteristic is represented by an eyepiece resolution function or a modulation transfer function (MTF). In some implementations, the lens performance characteristic is determined by surface lens modeling.

[0117] In various implementations, the maximum value is determined as a function of gaze direction (because the eyepiece resolution function changes with gaze direction). In various implementations, the maximum value is based on changes in the user's gaze, such as gaze movement (e.g., changing gaze position). For example, in some implementations, the maximum value of the rendered resolution function decreases as the user looks around (because resolution perception decreases during eye movement). For another example, in some implementations, the maximum value of the rendered resolution function decreases when the user blinks (because resolution perception [and eye tracking confidence] decreases when the user blinks).

[0118] In various implementations, the maximum value is affected by lens performance characteristics. For example, in some implementations, when the lens performance characteristics indicate that the lens cannot support higher resolution, the maximum value is reduced. In some implementations, the lens performance characteristics include distortion introduced by the lens.

[0119] The method 1100 continues at block 1130, where the rendering module generates a rendered image based on the SR content and the rendering resolution function (e.g., as described above with respect to Figure 7 In various implementations, the rendered image is a fovea-focused image, such as an image with lower resolution outside the user's field of view (FoF). In various implementations, the rendered image is a warped image, such as an image transformed into a non-uniform space compared to the SR content.

[0120] Fig.12 1 is a flowchart representation of a method 1200 for rendering an image according to some implementations. In some implementations (and as described in detail below as an example), the method 1200 is performed by a rendering module (such as Figure 2 In various implementations, the method 1200 is performed by an HMD (such as a Figure 1 HMD 100) or a portion thereof (such as Figure 2 SR pipeline 200). In various implementations, method 1200 is performed by a device having one or more processors, non-transitory memory, and one or more SR displays. In some implementations, method 1200 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 1200 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0121] Method 1200 begins at block 1210, when a rendering module obtains eye tracking data indicating a user's gaze (e.g., where the user is looking, such as the user's gaze direction or gaze point). In various implementations, the eye tracking data includes at least one of data indicating a user's gaze angle or data indicating a user's gaze point.

[0122] The method 1200 continues at block 1220 when the rendering module obtains eye tracking metadata indicating characteristics of the eye tracking data. In various implementations, the eye tracking metadata associated with the corresponding eye tracking data is obtained. In various implementations, the eye tracking metadata is obtained from an eye tracker (such as Figure 2 The eye tracker 260) receives the eye tracking data and associated eye tracking metadata.

[0123] In various implementations, the eye tracking metadata includes data indicating a confidence level of the eye tracking data. For example, in various implementations, the eye tracking metadata provides a measure of the confidence level that the eye tracking data correctly indicates a user's gaze.

[0124] In various implementations, the data indicating the confidence of the eye tracking data includes data indicating the accuracy of the eye tracking data. In various implementations, the rendering module generates the data indicating the accuracy of the eye tracking data based on a series of recently captured images of the user's eyes, recent measurements of the user's gaze, user biometric information, and / or other obtained data.

[0125] In various implementations, the data indicating the confidence of the eye tracking data includes data indicating a delay of the eye tracking data (e.g., a difference between a time when the eye tracking data is generated and a time when the eye tracking data is received by the rendering module). In various implementations, the rendering module generates the data indicating the delay of the eye tracking data based on a timestamp of the eye tracking data. In various implementations, the confidence of the eye tracking data is higher when the delay is smaller than when the delay is larger.

[0126] In various specific implementations, the eye tracking data includes data indicating the prediction of the user's gaze, and the data indicating the confidence of the eye tracking data includes data indicating the confidence of the prediction. In various specific implementations, the data indicating the prediction of the user's gaze is based on past measurements of the user's gaze based on images captured in the past. In various specific implementations, the prediction of the user's gaze is based on the past movement of the user's gaze being classified as continuous fixation, smooth follow or glance. In various specific implementations, the confidence of the prediction is based on the classification. Specifically, in various specific implementations, the confidence of the prediction is higher when the past movement is classified as continuous fixation or smooth follow than when the past movement is classified as glance.

[0127] In various implementations, the eye tracking metadata includes data indicative of one or more biometric information of the user, particularly biometric information that affects the eye tracking metadata or its confidence. Specifically, in various implementations, the user's biometric information includes one or more of eye anatomy, ethnicity / appearance, eye color, age, visual aids (e.g., corrective lenses), makeup (e.g., eyeliner or mascara), medical conditions, historical gaze changes, input preferences or calibration, headset position / orientation, pupil dilation / center displacement, and / or eyelid position.

[0128] In various implementations, the eye tracking metadata includes data indicating one or more environmental conditions of the environment of the user generating the eye tracking data. Specifically, in various implementations, the environmental conditions include one or more of vibration, ambient temperature, IR directional light, or IR light intensity.

[0129] The method 1200 continues at block 1230, where the rendering module generates a rendering resolution function based on the eye tracking data and the eye tracking metadata. In various implementations, the rendering module generates the rendering resolution function with a peak maximum value based on the eye tracking data (e.g., the resolution is highest where the user is looking). In various implementations, the rendering module generates the rendering resolution function with a peak width based on the eye tracking metadata (e.g., the peak is wider when the eye tracking metadata indicates a lower confidence level in the correctness of the eye tracking data).

[0130] Method 1200 continues at block 1240, where the rendering module generates a rendered image based on the SR content and the rendering resolution function (e.g., as described above with respect to Figure 7 In various implementations, the rendered image is a fovea-focused image, such as an image with lower resolution outside the user's field of view (FoF). In various implementations, the rendered image is a warped image, such as an image transformed into a non-uniform space compared to the SR content.

[0131] Fig.13A Rendering resolution function 1310 is shown, when dynamic foveal focusing is performed, when eye tracking data indicates that the user is looking away from the center of the display panel at an angle (θ g ) is viewed, and when the eye tracking metadata indicates a first confidence level resulting in a first peak width 1311, the rendering module may use the rendering resolution function. Fig. 13B Rendering resolution function 1320 is shown, when dynamic foveal focusing is performed, when eye tracking data indicates that the user is looking away from the center of the display panel at an angle (θ g) is viewed, and when the eye tracking metadata indicates a second confidence level that is less than the first confidence level and produces a second peak width 1321 that is greater than the first peak width 1311, the rendering module may use the rendering resolution function.

[0132] In various implementations, the rendering module detects a loss of an eye tracking stream (including eye tracking metadata and eye tracking data). In response, the rendering module generates a second rendering resolution function based on detecting the loss of the eye tracking stream, and generates a second rendered image based on the SR content and the second rendering resolution function.

[0133] In various implementations, detecting a loss of the eye tracking stream includes determining that the user's gaze is static when the eye tracking stream is lost. Thus, in various implementations, generating a second rendering resolution function includes generating a second rendering resolution function having a peak maximum at the same location as a peak maximum of the rendering resolution function and having a peak width greater than a peak width of the rendering resolution function. Thus, in various implementations, in response to detecting a loss of the eye tracking stream, the rendering resolution function remains at the same location, but the size of the fixation field of view increases.

[0134] In various implementations, detecting a loss of the eye tracking stream includes determining that a gaze of the user is moving when the eye tracking stream is lost. Thus, in various implementations, generating a second rendering resolution function includes generating a second rendering resolution function having a peak maximum at a location displaced toward the center compared to a peak maximum of the rendering resolution function and having a peak width greater than a peak width of the rendering resolution function. Thus, in various implementations, in response to detecting a loss of the eye tracking stream, the rendering resolution function moves to the center of the display panel and the size of the fixation field of view increases.

[0135] In various implementations, detecting a loss of the eye tracking stream includes determining that a gaze of the user is moving in a certain direction when the eye tracking stream is lost. Thus, in various implementations, generating a second rendering resolution function includes generating a second rendering resolution function having a peak maximum at a position displaced in the direction compared to a peak maximum of the rendering resolution function and having a peak width greater than a peak width of the rendering resolution function. Thus, in various implementations, in response to detecting a loss of the eye tracking stream, the rendering resolution function moves to a predicted position and a size of the fixation field of view increases.

[0136] Fig.14 1 is a flowchart representation of a method 1400 for rendering an image according to some implementations. In some implementations (and as described in detail below as an example), the method 1400 is performed by a rendering module (such as Figure 2 In various implementations, the method 1400 is performed by an HMD (such as a Figure 1HMD 100) or a portion thereof (such as Figure 2 SR pipeline 200). In various implementations, method 1400 is performed by a device having one or more processors, non-transitory memory, and one or more SR displays. In some implementations, method 1400 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 1400 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0137] Method 1400 begins at block 1410, when a rendering module obtains eye tracking data indicating a user's gaze (e.g., where the user is looking, such as the user's gaze direction or gaze point). In various implementations, the eye tracking data includes at least one of data indicating a gaze angle of the user or data indicating a gaze point of the user.

[0138] The method 1400 continues at block 1420, where the rendering module obtains SR content to be rendered. In various implementations, the SR content may include a definition of the geometry of a virtual object, the color and / or texture of a virtual object, an image (such as a perspective image of a scene), or other information describing the content to be represented in a rendered image.

[0139] Method 1400 continues at box 1430, where the rendering module generates a rendering resolution function based on the eye tracking data and the SR content. In various embodiments, the rendering module generates a rendering resolution function with a peak maximum value based on the eye tracking data (e.g., the resolution is highest where the user is looking). In various embodiments, the rendering module generates a rendering resolution function based on the eye tracking data and adjusts the rendering resolution function based on the SR content. For example, in some embodiments, the rendering module increases the rendering resolution function in one or more lead-in areas such as game objects or content that humans are particularly suitable for distinguishing (e.g., content that humans may pay attention to), such as faces or high-resolution objects. For another example, in some embodiments, the rendering module increases the rendering resolution function in one or more motion areas (e.g., objects of SR content).

[0140] In various implementations, the rendering module generates a rendering resolution function based on the brightness of the SR content. For example, in some implementations, because peripheral vision is more sensitive to light than central vision, peripheral resolution is increased in darker conditions (compared to brighter conditions). In various implementations, increasing the peripheral resolution includes increasing the peak width of the rendering resolution function and / or increasing the falling portion of the rendering resolution function.

[0141] In various implementations, the rendering module generates a rendering resolution function based on the color of the SR content. For example, in some implementations, because sensitivity to red-green color changes drops more steeply toward the periphery than sensitivity to brightness or blue-yellow color, peripheral resolution is reduced when the SR content is primarily red-green rather than blue-yellow. In various implementations, reducing the peripheral resolution includes reducing the peak width of the rendering resolution function and / or reducing a falling portion of the rendering resolution function.

[0142] In various implementations, generating a rendering resolution function based on the SR content (e.g., the color of the SR content) includes generating different rendering resolution functions for different color channels (e.g., three different rendering resolution functions for three different color channels such as red, green, and blue). Specifically, in various implementations, the rendering module generates a first rendering resolution function for a first color channel and a second rendering resolution function for a second color channel that is different from the first rendering resolution function for the first color channel. Furthermore, in generating a rendered image (described below), the rendering module generates a first color channel image of the rendered image based on the first rendering resolution function, and generates a second color channel image of the rendered image based on the second rendering resolution function.

[0143] In various implementations, the rendering module generates a rendering resolution function based on the complexity of the SR content. For example, in various implementations, the rendering module increases the rendering resolution function in areas with high-resolution spatial variation and / or rapid spatial variation.

[0144] Method 1400 continues at block 1440, where the rendering module generates a rendered image based on the SR content and the rendering resolution function (e.g., as described above with respect to Figure 7 In various implementations, the rendered image is a fovea-focused image, such as an image with lower resolution outside the user's field of view (FoF). In various implementations, the rendered image is a warped image, such as an image transformed into a non-uniform space compared to the SR content.

[0145] As above relative to Figure 2 As described above, the rendering module 210 provides the rendered image to the transmission module 220. The transmission module 220 includes a compression module 222 for compressing the rendered image (to generate a compressed image), a communication channel 224 for transmitting the compressed image, and a decompression module 226 for decompressing the compressed image (to generate a decompressed image).

[0146] In various implementations, the communication channel 224 is a wired or wireless communication channel (e.g., Bluetooth, IEEE 802.11x, IEEE 802.16x, IEEE 802.3x, etc.). For example, in various implementations, the communication channel 224 includes a first device (e.g., Figure 1 110) is coupled to a second device (eg, Figure 1 In various implementations, the communication channel couples two processing units of a single device, for example, a first processing unit including a compression module 222 and a second processing unit including a decompression module 226. In various implementations, the communication channel couples two processing modules of a single processing unit, for example, the compression module 222 and the decompression module 226.

[0147] In various implementations, the compression module 222 receives the foveated focused image from the rendering module 210, with different resolutions at different portions of the image. In various implementations, compressing the M×N foveated focused image using various compression techniques may produce fewer bits than compressing the M×N non-foveated focused image.

[0148] In various implementations, the compression module 222 receives a warped image from the rendering module 210, the warped image having a plurality of pixels at corresponding locations evenly spaced in a grid pattern in the warped space, the plurality of pixels being respectively associated with a plurality of corresponding pixel values ​​and a plurality of corresponding scaling factors, the plurality of corresponding scaling factors indicating a plurality of corresponding resolutions at a plurality of corresponding locations in the display space.

[0149] In various implementations, the compression module 222 receives the scaling factor from the rendering module 210. In various implementations, the compression module 222 receives a single scaling factor for each pixel. In various implementations, the compression module 222 receives a horizontal scaling factor and a vertical scaling factor for each pixel. In various implementations, the compression module 222 receives a single scaling factor for each pixel block (e.g., each 32×32 pixel block). In various implementations, the compression module 222 receives a horizontal scaling factor and a vertical scaling factor for each pixel block (e.g., each 32×32 pixel block).

[0150] Since the warped image is an image (e.g., a matrix of pixel values), various conventional compression techniques can be applied to the image. However, in various implementations, the compression module 222 uses a scaling factor during compression and / or transmission of the warped image to further reduce the bandwidth of the communication channel 224 used to transmit the warped image.

[0151] Fig.151 is a flowchart representation of a method 1500 for transmitting an image according to some implementations. In some implementations (and as described in detail below as an example), the method 1500 is performed by a transmission module (or a portion thereof) such as Figure 2 In various implementations, the method 1500 is performed by an HMD (such as Figure 1 HMD 100) or a portion thereof (such as Figure 2 SR pipeline 200). In various implementations, method 1500 is performed by a device having one or more processors, non-transitory memory, and one or more SR displays. In some implementations, method 1500 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 1500 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0152] Method 1500 begins at box 1510, when the transmission module receives a distorted image representing simulated reality (SR) content to be displayed in a display space, the distorted image having a plurality of pixels at corresponding positions evenly spaced in a grid pattern in the distorted space, wherein the plurality of pixels are respectively associated with a plurality of corresponding pixel values ​​and a plurality of corresponding scaling factors, the plurality of corresponding scaling factors indicating a plurality of corresponding resolutions at a plurality of corresponding positions in the display space.

[0153] In various implementations, the plurality of respective scaling factors define a mapping between the warp space and the display space. For example, in various implementations, different portions of the SR content corresponding to non-uniformly spaced regions in the display space are represented by uniformly spaced regions in the warp space.

[0154] In various implementations, each pixel of the plurality of pixels is respectively associated with a separately received pixel value. In various implementations, each pixel of the plurality of pixels is respectively associated with a separately received scaling factor (or a set of horizontal scaling factors and vertical scaling factors). In various implementations, each tile of the plurality of tiles of the plurality of pixels is respectively associated with a separately received scaling factor (or a set of horizontal scaling factors and vertical scaling factors). Thus, a plurality of pixels (e.g., those pixels of a single tile) are associated with a single received scaling factor (or a set of horizontal scaling factors and vertical scaling factors).

[0155] Thus, in various implementations, the warped image includes a plurality of tiles at corresponding positions evenly spaced in a grid pattern in the warped space, wherein each tile in the plurality of tiles is associated with a corresponding one or more scaling factors. Additionally, in various implementations, one or more scaling factors in the plurality of corresponding scaling factors include a horizontal scaling factor and a vertical scaling factor.

[0156] The method 1500 continues at box 1520, when the transmission module transmits the warped image through one or more channels, so that at least one bandwidth of the one or more channels used for transmission of the corresponding portion of the warped image is based on one or more scaling factors of the multiple scaling factors corresponding to the corresponding portion of the warped image.

[0157] In various implementations, at least one bandwidth of one or more channels used for transmission of the distorted image is changed by compressing the portion of the distorted image based on the scaling factor. The more compressed portion uses less bandwidth of the channel than the less compressed portion.

[0158] Specifically, in various implementations, the transmission module compresses the corresponding portion of the warped image based on one or more of the plurality of scaling factors corresponding to the corresponding portion of the warped image. For example, in various implementations, the transmission module compresses the low-resolution portion to a greater extent than the high-resolution portion (e.g., by allocating fewer bits at the MAC layer and / or setting a quantization parameter to a lower value [e.g., more quantization]).

[0159] In various implementations, at least one bandwidth of one or more channels used for transmission of a portion of the distorted image is changed by error correction encoding the portion based on the scaling factor, so that the portion that has been error correction encoded with a weaker error correction code uses less bandwidth than the portion that has been error correction encoded with a stronger error correction code.

[0160] Specifically, in various implementations, the transmission module performs error correction encoding on the corresponding portion of the distorted image based on one or more of the multiple scaling factors corresponding to the corresponding portion of the distorted image. For example, in various implementations, the transmission module uses a stronger error correction code (e.g., adding more redundancy at the PHY layer) for the high-resolution portion compared to the low-resolution portion.

[0161] In various specific implementations, at least one bandwidth of one or more channels used for transmission of the corresponding portion is changed by changing the probability of transmission (or retransmission) of the portion of the distorted image based on the scaling factor. When the probability of transmission (or retransmission) of a portion is smaller, the bandwidth used for transmission of the portion is less.

[0162] Specifically, in various implementations, the probability of transmission of the corresponding portion of the distorted image is based on one or more of the plurality of scaling factors corresponding to the corresponding portion of the distorted image. In various implementations, the probability of transmission is a probability of retransmission in the event of packet loss. For example, if a data packet corresponding to a portion of the distorted image is lost on a channel, the probability of retransmitting the packet may be based on the scaling factor such that a data packet corresponding to a higher resolution portion is more likely to be retransmitted than a data packet corresponding to a lower resolution portion.

[0163] In various implementations, the probability of transmission is based on the selected subchannel. For example, by selecting a subchannel with a higher signal-to-noise ratio (e.g., at the PHY layer), the probability of retransmission due to bit errors, packet errors, or packet loss is reduced. Therefore, in various implementations, the transmission module selects a subchannel for transmission of the corresponding portion based on the corresponding scaling factor. In various implementations, the transmission module selects a subchannel with a higher signal-to-noise ratio for the higher resolution portion compared to the lower resolution portion.

[0164] In various implementations, the probability of transmission is based on a buffer (e.g., Figure 2 For example, in various implementations, when the buffer is approaching or has approached overflow, the probability of transmitting a portion of the distorted image is reduced, and the probability of transmitting a lower resolution portion of the distorted image is reduced more than the higher resolution portion of the distorted image. In various implementations, when the buffer is approaching or has approached overflow, the receiver (e.g., the decompression module) discards the received portion of the distorted image based on the scaling factor. For example, in various implementations, the receiver is more likely to discard the lower resolution portion than the higher resolution portion.

[0165] In various implementations, transmitting the warped image includes transforming the warped image. For example, in various implementations, the transmission module generates a wavelet warped image by performing a wavelet transform on the warped image. The wavelet warped image includes a plurality of portions of the wavelet warped image corresponding to different frequency bands for each corresponding portion of the warped image.

[0166] Fig.16A An exemplary image 1610 is shown. Fig. 16BA wavelet image 1620 generated by a single-layer wavelet transform of an exemplary image is shown. Each portion of the exemplary image 1610 (e.g., upper middle block 1611) is represented by a plurality of portions of the wavelet image 1620 corresponding to different frequency bands (e.g., blocks 1621a-1621d). Specifically, block 1621a corresponds to low frequencies in the horizontal and vertical directions, block 1621b corresponds to high frequencies in the horizontal direction and low frequencies in the vertical direction, block 1621c corresponds to low frequencies in the horizontal direction and high frequencies in the vertical direction, and block 1021d corresponds to high frequencies in the horizontal and vertical directions.

[0167] In various implementations, the wavelet-distorted image is generated using a single-layer wavelet function ( FIG. 16A to FIG. 16B In various implementations, the wavelet-warped image is generated using a two-layer wavelet function.

[0168] In various implementations, at least one bandwidth of one or more channels used for transmission of corresponding portions of the wavelet distorted image is based on different frequency bands. Specifically, in various implementations, corresponding portions of the wavelet distorted image associated with lower frequency bands are allocated more bandwidth than portions of the wavelet distorted image associated with higher frequency bands.

[0169] In various implementations, the transmission module filters the distorted image based on the plurality of corresponding scaling factors. For example, in various implementations, the transmission module performs wavelet thresholding on the distorted image. In various cases, this may reduce noise on a frequency band basis to make it more understandable and / or remove some details to provide a continuous stream.

[0170] In various specific implementations, at least one bandwidth of the one or more channels used for transmission of the corresponding portion of the warped image is further based on at least one available bandwidth of the one or more channels. Specifically, when the available bandwidth is greater, the bandwidth of the channel used is also greater. In various specific implementations, the transmission module includes a high efficiency video coder (HEVC).

[0171] As above relative to Figure 2 As described above, in various specific implementations, the transmission module 220 receives the plurality of corresponding scaling factors from the rendering module 210. In various specific implementations, the transmission module 220 provides the corresponding scaling factors to the display module 230. Therefore, the transmission module 220 transmits the corresponding scaling factors through the communication channel 224.

[0172] Therefore, in various specific implementations, Fig.15Method 1500 also includes receiving the multiple corresponding scaling factors (e.g., from a rendering module), compressing the multiple corresponding scaling factors, transmitting the multiple corresponding scaling factors via a channel, decompressing the multiple corresponding scaling factors, and / or providing the multiple corresponding scaling factors (e.g., to a display module).

[0173] refer to Figure 2 , the decompression module 226 receives the distorted image via the communication channel 224. In various specific implementations, the distorted image is compressed, encoded and / or transformed. Therefore, in various specific implementations, the decompression module 226 decompresses, decodes and / or de-transforms the distorted image. In addition, in various specific implementations, the decompression module 226 receives the multiple corresponding scaling factors via the communication channel 224. In various specific implementations, the multiple corresponding scaling factors are compressed. Therefore, in various specific implementations, the decompression module 226 decompresses the multiple corresponding scaling factors.

[0174] In some cases, the decompression module 226 fails to correctly receive a data packet associated with a portion of the distorted image associated with the corresponding scaling factor. For example, the decompression module 226 may receive the corresponding scaling factor but not the corresponding data packet. As another example, the decompression module 226 may receive a damaged data packet associated with the corresponding scaling factor (e.g., as indicated by a parity bit, a cyclic redundancy check, or other indicator indicating data packet corruption). As another example, the decompression module 226 may discard a packet because a buffer of the decompression module 226 is full or nearly full.

[0175] In various implementations, the decompression module 226 sends a request to the compression module 222 to retransmit the data packet that was not correctly received. In various implementations, the decompression module 226 determines whether to send such a retransmission request based on the corresponding scaling factor. For example, in various implementations, the decompression module 226 is more likely to send a retransmission request when the corresponding scaling factor indicates that the corresponding portion of the image has a higher resolution than when the corresponding scaling factor indicates that the corresponding portion of the image has a lower resolution.

[0176] As described above, in various specific implementations, the decompression module 226 includes a buffer for storing received data packets. In some cases, the buffer may become full, resulting in packet loss. In various specific implementations, when the buffer is close to being full (e.g., exceeding a threshold percentage of fullness), the decompression module 226 determines whether to store the received packet or discard the received packet based on the corresponding scaling factor. For example, in various specific implementations, the decompression module 226 is more likely to discard packets when the corresponding scaling factor indicates that the resolution of the corresponding portion of the image is lower than when the corresponding scaling factor indicates that the resolution of the corresponding portion of the image is higher. In various specific implementations, the decompression module 226 determines whether to store the received packet or discard the received packet based on a continuous function of the buffer fullness (e.g., how full the buffer is) and the corresponding scaling factor. The function may make the decompression module 226 more likely to discard packets when the buffer fullness increases and / or the corresponding scaling factor indicates that the resolution of the corresponding portion of the image is lower.

[0177] Fig.17 1 is a flowchart representation of a method 1700 for receiving an image according to some implementations. In some implementations (and as described in detail below as an example), the method 1700 is performed by a transmission module (or a portion thereof) such as Figure 2 In various implementations, the method 1700 is performed by an HMD (such as Figure 1 HMD 100) or a portion thereof (such as Figure 2 SR pipeline 200). In various implementations, method 1700 is performed by a device having one or more processors, non-transitory memory, and one or more SR displays. In some implementations, method 1700 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 1700 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0178] Method 1700 begins at box 1710, when a transmission module receives at least a subset of a plurality of data packets corresponding to a portion of a warped image representing simulated reality (SR) content to be displayed in a display space, the warped image having a plurality of pixels at corresponding locations evenly spaced in a grid pattern in the warped space.

[0179] In various implementations, the plurality of respective scaling factors define a mapping between the warp space and the display space. For example, in various implementations, different portions of the SR content corresponding to non-uniformly spaced regions in the display space are represented by uniformly spaced regions in the warp space.

[0180] The method 1700 continues at block 1720 when the transmission module receives a plurality of scaling factors indicating respective resolutions of portions of the warped image at a plurality of respective locations in the display space.

[0181] The method 1700 continues at block 1730 when the transport module detects an error condition for a particular one of the plurality of data packets. In various implementations, detecting the error condition includes receiving a particular one of the plurality of scaling factors without receiving a particular one of the plurality of data packets. This may indicate packet loss. In various implementations, detecting the error condition includes determining that a particular one of the plurality of data packets is damaged. For example, in various implementations, a parity bit, a cyclic redundancy check, or other indicator indicates that the data packet is damaged. In various implementations, detecting the error condition includes receiving a particular one of the plurality of data packets when a buffer is full or nearly full. In various implementations, detecting the error condition includes determining that a buffer is storing at least a threshold percentage of data.

[0182] The method 1700 continues at block 1740, where the transmission module resolves the error condition based on a particular one of the plurality of scaling factors corresponding to a particular one of the plurality of data packets. In various implementations, resolving the error condition includes determining whether to send a retransmission request for a particular one of the plurality of data packets based on the particular one of the plurality of scaling factors. In various implementations, the transmission module is more likely to send a retransmission request when the particular one of the plurality of scaling factors indicates that the resolution of the corresponding portion of the distorted image is higher than when the particular one of the plurality of scaling factors indicates that the resolution of the corresponding portion of the distorted image is lower. Thus, in various implementations, resolving the error condition includes determining that a retransmission request is to be sent and sending the retransmission request (or determining not to send a retransmission request).

[0183] In various implementations, resolving the error condition includes determining whether to discard or store a particular one of the plurality of data packets based on a particular one of the plurality of scaling factors. In various implementations, the transport module is more likely to store the data packet (in a buffer) when the particular one of the plurality of scaling factors indicates that the resolution of the corresponding portion of the distorted image is higher than when the particular one of the plurality of scaling factors indicates that the resolution of the corresponding portion of the distorted image is lower. Thus, in various implementations, resolving the error condition includes determining that the data packet is to be stored and storing the data packet (or determining that the data packet is to be discarded and not stored).

[0184] In various implementations, method 1700 includes decompressing, decoding, and / or detransforming the distorted image.

[0185] As above relative to Figure 2 As described above, the rendering module 210 provides the rendered image to the transmission module 220. As described above, the transmission module 220 includes a compression module 222 that compresses the rendered image (thereby generating a compressed image), a communication channel 224 that transmits the compressed image, and a decompression module 226 that decompresses the compressed image (thereby generating a decompressed image) and provides the decompressed image to the display module 230. Therefore, the display module 230 receives the rendered image provided by the rendering module 210 due to distortion caused by compression, transmission, reception and / or decompression. In various specific implementations, the transmission module 220 does not exist and the rendering module 210 directly provides the rendered image to the display module 230.

[0186] In various implementations, the display module 230 receives a foveal focused image from the transmission module 220 or the rendering module 210, and different portions of the image have different resolutions. In various implementations, the display module 230 receives a warped image from the transmission module 220 or the rendering module 230, and the warped image has a plurality of pixels at corresponding positions evenly spaced in a grid pattern in the warped space. The plurality of pixels are respectively associated with a plurality of corresponding pixel values ​​and a plurality of corresponding scaling factors, and the plurality of corresponding scaling factors indicate a plurality of corresponding resolutions at a plurality of corresponding positions in the display space.

[0187] In various implementations, the display module 230 receives the scaling factor from the transmission module 220 or the rendering module 210. In various implementations, the display module 230 receives a single scaling factor for each pixel. In various implementations, the display module 230 receives a horizontal scaling factor and a vertical scaling factor for each pixel. In various implementations, the display module 230 receives a single scaling factor for each pixel block (e.g., each 32×32 pixel block). In various implementations, the display module 230 receives a horizontal scaling factor and a vertical scaling factor for each pixel block (e.g., each 32×32 pixel block).

[0188] The display module 230 includes a lens compensation module 232 that distorts the decompressed image in an inverse relationship to the distortion caused by the eyepiece 242 of the HMD so that the displayed image appears distortion-free when viewed by the user 250 through the eyepiece 242. The distortion caused by the eyepiece 242 may be described by a homography distortion function that takes as input the pixel positions of pixels of an input image and provides as output the pixel positions of pixels in an output image.

[0189] Therefore, in the case where F is the homography distortion function, (x 1 ,y 1 ) is the pixel position of the pixel in the input image, and (x2 ,y 2 ) is the pixel position of the pixel in the output image.

[0190] F(x 1 ,y 1 )=(x 2 ,y 2 ).

[0191] Fig.18A An undistorted image 1810 is shown that includes a pixel 1811 having a first pixel value at a first pixel position. Fig.18B A distorted image is shown, which includes a pixel 1821 having a first pixel value at a second pixel position. FIG. 18A to FIG. 18B , the first pixel position is (2,1) and the second pixel position is (2.2,1.1).

[0192] Thus, a pixel (having a pixel value) at a first pixel position of the input image appears to have the same pixel value at a second pixel position of the output image when projected through the eyepiece. This relationship (where I(x,y) is the pixel value of the input image at position (x,y) and O(x,y) is the pixel value of the output image at position (x,y)) can be expressed as:

[0193] I(x 1 ,y 1 )=O(x 2 ,y 2 ).

[0194] therefore:

[0195] I(x 1 ,y 1 )=O(F(x 1 ,y 1 )).

[0196] To account for this distortion, the lens compensation module 232 generates a pre-distorted image so that the output image distorted by the eyepiece 242 looks like the input image. Thus, where P(x,y) is the pixel value of the pre-distorted image at position (x,y),

[0197] P(x 1 ,y 1 )=I(x 2 ,y 2 ).

[0198] therefore:

[0199] P(x 1 ,y 1 )=I(F(x 1 ,y 1)).

[0200] When the pre-distorted image is distorted by the eyepiece,

[0201] P(x 1 ,y 1 )=I(x 2 ,y 2 )=O(x 2 ,y 2 ).

[0202] therefore:

[0203] P(x 1 ,y 1 )=I(F(x 1 ,y 1 ))=O(F(x 1 ,y 1 )).

[0204] Fig.19A A pre-distorted image 1910 is shown, comprising a pixel 1911 having a pixel value at a first pixel position. Fig.19B A distorted pre-distorted image (or undistorted image) 1920 is shown, the distorted pre-distorted image comprising a pixel 1921 having a first pixel value at a second pixel position. FIG. 19A to FIG. 19B , the first pixel position is (2,1) and the second pixel position is (2.2,1.1).

[0205] To determine the first position (x 1 ,y 1 ), the lens compensation module 232 determines the pixel value of the pixel of the predistorted image at the second position (x 2 ,y 2 ). However, in some cases, the second position (x 2 ,y 2 ) falls between the pixel locations of the input image of known pixel values.

[0206] For example, in Fig.19A In the example, the pixel value of pixel 1911 is not consistent with Fig.18A The pixel value of pixel 1911 is the same as the pixel value of the pixel of the undistorted image 1810 at (2.2, 1.1) if such a pixel exists. Conversely, the pixel value of pixel 1911 is the same as the pixel value of the pixel of the undistorted image 1810 at (2.2, 1.1) if such a pixel exists.

[0207] Therefore, in various implementations, in order to determine the first position (x 1 ,y 1 ), the lens compensation module 232 uses the pixel value of the pixel of the predistorted image at the second position (x 2 ,y2 ) is determined by using the known pixel values ​​of pixels in the neighborhood around (e.g., using interpolation) to determine the second position (x 2 ,y 2 ) is the pixel value of the pixel of the input image at ).

[0208] If the distance of a pixel from the second position is less than a predetermined distance, then the pixel is in the neighborhood of the second position. In various embodiments, the distance is L 2 In various embodiments, the distance is L 1 Distance. In various implementations, the predetermined distance is 1 divided by the resolution of the warped image (eg, the distance between adjacent pixels). In various implementations, the predetermined distance is 2 divided by the resolution of the warped image.

[0209] For example, in Fig.19A In order to determine the pixel value of pixel 1911, the pixel values ​​of the pixels of the undistorted image 1810 at (2,1), (2,2), (3,1) and (3,2) are used.

[0210] In various implementations, the pixel value of the pixel of the predistorted image at the first position is a weighted sum of the pixel values ​​of the pixels of the undistorted image in the neighborhood around the second position. Therefore, in various implementations, the lens compensation module 232 performs linear interpolation to determine the pixel values ​​of the pixels of the predistorted image. In various implementations, the lens compensation module 232 performs various types of interpolation to determine the pixel values ​​of the pixels of the predistorted image, such as nearest neighbor interpolation, polynomial interpolation, or spline interpolation.

[0211] Therefore, N(x 2 ,y 2 ) is the neighborhood of pixels around the second position and w i is the weight,

[0212]

[0213] In various implementations, based on the distance between the pixel position and the second position (eg, L 2 distance) to determine the weight.

[0214] Thus, in some implementations, where M is a normalization factor,

[0215]

[0216] In various implementations, the lens compensation module 232 receives a warped image having a plurality of pixels at corresponding positions evenly spaced in a grid pattern in a warped space, wherein the plurality of pixels are respectively associated with a plurality of corresponding pixel values ​​and a plurality of corresponding scaling factors, the plurality of corresponding scaling factors indicating a plurality of corresponding resolutions at the plurality of corresponding positions in the display space. Because the display space and the warped space are different and because the output image is distorted in the display space (not the warped space) due to the eyepiece, the lens compensation module 232 uses the scaling factors in addition to the pixel values ​​of neighboring pixels to generate pixel values ​​for the pixels of the pre-distorted image. For a particular pixel, the pixel position and scaling factor in the warped image generally describe the position in the display space. Different scaling factors for different pixels in a neighborhood indicate different distances than the pixel position would indicate alone. These different distances can be used to perform interpolation (e.g., determine interpolation weights or determine nearest neighbor points).

[0217] For example, in various implementations, where a x (x,y) and a y (x,y) is the scaling factor associated with (or proportional to or inversely proportional to) the pixel at pixel location (x,y), and the predistorted image is calculated as follows:

[0218]

[0219] Fig. 20 2 is a flowchart representation of a method 2000 for pre-distorting an image according to some implementations. In some implementations (and as described in detail below as an example), the method 2000 is performed by a display module (or a portion thereof) such as Figure 2 In various implementations, the method 2000 is performed by a display module 230 or a lens compensation module 232 of an HMD (such as Figure 1 HMD100) or a portion thereof (such as Figure 2 In various implementations, the method 2000 is performed by a device having one or more processors, non-transitory memory, and one or more SR displays. In some implementations, the method 2000 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 2000 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0220] Method 2000 begins at box 2010, when the display module receives a distorted image representing simulated reality (SR) content to be displayed in a display space, the distorted image having a plurality of pixels at corresponding positions evenly spaced in a grid pattern in the distorted space, wherein the plurality of pixels are respectively associated with a plurality of corresponding pixel values ​​and a plurality of corresponding scaling factors, the plurality of corresponding scaling factors indicating a plurality of corresponding resolutions at a plurality of corresponding positions in the display space.

[0221] In various implementations, the plurality of respective scaling factors define a mapping between the warp space and the display space. For example, in various implementations, different portions of the SR content corresponding to non-uniformly spaced regions in the display space are represented by uniformly spaced regions in the warp space.

[0222] In various implementations, each pixel of the plurality of pixels is respectively associated with a separately received pixel value. In various implementations, each pixel of the plurality of pixels is respectively associated with a separately received scaling factor (or a set of horizontal scaling factors and vertical scaling factors). In various implementations, each tile of the plurality of tiles of the plurality of pixels is respectively associated with a separately received scaling factor (or a set of horizontal scaling factors and vertical scaling factors). Thus, a plurality of pixels (e.g., those pixels of a single tile) are associated with a single received scaling factor (or a set of horizontal scaling factors and vertical scaling factors).

[0223] Thus, in various implementations, the warped image includes a plurality of tiles at corresponding positions evenly spaced in a grid pattern in the warped space, wherein each tile in the plurality of tiles is associated with a corresponding one or more scaling factors. Additionally, in various implementations, one or more scaling factors in the plurality of corresponding scaling factors include a horizontal scaling factor and a vertical scaling factor.

[0224] The method 2000 continues at box 2020, when the display module generates a lens-compensated distorted image based on the distorted image (particularly the plurality of corresponding pixel values ​​thereof), the plurality of corresponding scaling factors, and a distortion function describing the distortion caused by the eyepiece, the lens-compensated distorted image being a version of the distorted image distorted in an inverse relationship to the distortion caused by the eyepiece. In various specific implementations, the lens-compensated distorted image appears undistorted when viewed by a user on a display panel through the eyepiece (which may be undistorted after lens compensation). In various specific implementations, the display module generates the lens-compensated distorted image without generating an intermediate image in the display space. Specifically, the display module generates the lens-compensated distorted image without undistorting the distorted image. Thus, as described above, the display module generates the lens-compensated distorted image based on the plurality of corresponding scaling factors.

[0225] In various embodiments, generating a lens-compensated distorted image includes determining pixel values ​​of a plurality of pixels (e.g., each pixel) of the lens-compensated distorted image. Thus, in various embodiments, the display module generates the lens-compensated distorted image by determining the pixel value of a particular pixel at a first position of the lens-compensated distorted image. In various embodiments, the display module determines the pixel of the particular pixel at the first position of the lens-compensated distorted image based on the plurality of corresponding pixel values ​​of the pixels of the distorted image in a neighborhood of the second position. In various embodiments, the neighborhood is a 4×4 pixel block closest to the second position. In various embodiments, the neighborhood is a 9×9 pixel block. In various embodiments, the display module determines the second position based on the first position and a distortion function of the eyepiece. In various embodiments, the distortion function is a homography distortion function that takes as input the pixel positions of pixels of an input image and provides as output the pixel positions of pixels in an output image.

[0226] In various implementations, determining a pixel value for a particular pixel of the lens-compensated distorted image includes determining a corrected distance between the second position and pixel positions of the plurality of pixels in the neighborhood of the corresponding pixel of the distorted image based on respective scaling factors of pixels in the neighborhood of the corresponding pixel of the distorted image.

[0227] For example, in the absence of a scaling factor, the distance between a second position at (0,2) and a first pixel at (0,1) may be determined to be the same as the distance between the second position and a second pixel at (0,3), e.g., 1. However, if the first pixel is associated with a scaling factor of 1 and the second pixel is associated with a scaling factor of 4, then the corrected distance between the second position and the first pixel (e.g., 1) is less than the corrected distance between the second position and the second pixel (e.g., 4).

[0228] In various implementations, determining a pixel value for a particular pixel of the lens-compensated distorted image includes selecting one of the plurality of corresponding pixel values ​​of the distorted image in the neighborhood of the second location based on corresponding scaling factors of the pixels of the distorted image in the neighborhood of the second location. Thus, in various implementations, the display module performs nearest neighbor interpolation, wherein the nearest neighbor point is determined based on a corrected distance based on the scaling factor.

[0229] In various implementations, determining a pixel value of a particular pixel of the lens-compensated distorted image includes averaging (e.g., determining a weighted average of) the plurality of corresponding pixel values ​​of the pixels of the distorted image in the neighborhood of the second location based on corresponding scaling factors of the pixels of the distorted image in the neighborhood of the second location. Thus, in various implementations, the display module performs polynomial interpolation (linear interpolation is an example of which) where weights are determined based on a corrected distance based on the scaling factor. Thus, in various implementations, the pixel value of the particular pixel of the lens-compensated distorted image is a weighted average of the pixel values ​​of the pixels weighted by the corrected distance between the second location and the pixel.

[0230] As above relative to Figure 2 As described, the lens compensation module 232 provides the lens-compensated distorted image (and the corresponding scaling factor) to the panel compensation module 234. The panel compensation module 234 converts the lens-compensated distorted image into panel data for reading by the display panel 240. Therefore, in various specific implementations, the panel compensation module 234 receives a distorted image (e.g., a lens-compensated distorted image) having a plurality of pixels at corresponding positions evenly spaced in a grid pattern in a distorted space, wherein the plurality of pixels are respectively associated with a plurality of corresponding pixel values ​​and a plurality of corresponding scaling factors, and the plurality of corresponding scaling factors indicate a plurality of corresponding resolutions at a plurality of corresponding positions in the display space.

[0231] The display panel 240 includes a matrix of M×N pixels located at corresponding positions in the display space. The display panel 240 displays the displayed image as described by the panel data by emitting light from each pixel. In various specific implementations, the panel data includes a pixel value for each pixel of the display panel 240. For example, the panel data may include a pixel value for a specific pixel of the display panel 240, and the specific pixel emits light according to the pixel value. In various specific implementations, the panel data includes a pixel value for each pixel block in a plurality of pixel blocks of the display panel 240. For example, the panel data may include a pixel value for a 2×2 pixel block of the display panel 240, and each pixel of the block emits light according to the pixel value.

[0232] In various implementations, the panel data includes data associated with a plurality of resolution zones. Each resolution zone is defined by boundaries and one or more resolution reduction factors (eg, a single resolution reduction factor, or both a horizontal resolution reduction factor and a vertical resolution reduction factor) that are integers.

[0233] Fig.21AA representation of panel data 2110 indicating a plurality of resolution zones 2111A-2111C is shown. The plurality of resolution zones 2111A-2111C includes a first resolution zone 2111A having a first boundary and a resolution reduction factor of 1. In the case where the resolution reduction factor is 1, the resolution is not reduced from the maximum resolution of the display panel 240, and each pixel value received is displayed by a single pixel of the display panel 240. The plurality of resolution zones 2111A-2111C includes a second resolution zone 2111B having a second boundary and a resolution reduction factor of 2. The second boundary of the second resolution zone 2111B includes two portions, an interior at the same location as the first boundary of the first resolution zone 2111A and an exterior surrounding the second resolution zone 2111B. In the case where the resolution reduction factor is 2, the resolution is reduced to half of the maximum resolution of the display panel 240, and each pixel value received by the display panel 240 is displayed by a 2×2 pixel block of the display panel 240. The plurality of resolution zones 2111A-2111C include a third resolution zone 2111C having a third boundary and a resolution reduction factor of 4. The third boundary of the third resolution zone 2111C includes two portions, an inner portion at the same position as the outer portion of the second boundary of the second resolution zone 2111B and an outer portion at an edge of the display panel 240. In the case where the resolution reduction factor is 4, the resolution is reduced to one quarter of the maximum resolution of the display panel 240, and each pixel value received by the display panel 240 is displayed by a 4×4 pixel block of the display panel 240.

[0234] In various implementations, the panel data indicating the plurality of resolution zones takes various forms. For example, in various implementations, the plurality of resolution zones 2111A-2111C are transmitted as the position of the first resolution zone 2111A (e.g., the position of the upper left pixel), the size of the first resolution zone 2111A, the position of the second resolution zone 2111B (e.g., the position of the upper left pixel), and the size of the second resolution zone 2111B (if it includes the first resolution zone 2111A). The boundary of the third resolution zone 2111C may be determined by inference to occupy the remaining portion of the display panel 240. Similarly, the resolution reduction factor may be determined by inference to be 1, 2, and 4.

[0235] Fig.21BA representation of panel data 2120 indicating a plurality of resolution zones 2121A-2121Y is shown. The plurality of resolution zones 2121A-2121Y include a first resolution zone 2121M having a first boundary, a horizontal resolution reduction factor of 1, and a vertical resolution reduction factor of 1. In the case where both the horizontal resolution reduction factor and the vertical resolution reduction factor are 1, the resolution is not reduced from the maximum resolution of the display panel 240, and each pixel value received is displayed by a single pixel of the display panel 240. Thus, in various implementations, the plurality of resolution zones include a resolution zone where the resolution is not reduced.

[0236] The plurality of resolution zones 2121A-2121Y include a second resolution zone 2121N having a second boundary, a horizontal resolution reduction factor of 2, and a vertical resolution reduction factor of 1. In the case where the horizontal resolution reduction factor is 2 and the vertical resolution reduction factor is 1, the horizontal resolution is half of the maximum horizontal resolution of the display panel 240, the vertical resolution is not reduced from the maximum vertical resolution of the display panel 240, and each pixel value received is displayed by a 2×1 pixel block of the display panel 240. Thus, in various implementations, the plurality of resolution zones include resolution zones where the resolution is reduced in one dimension but not in another dimension.

[0237] The plurality of resolution zones 2121A-2121Y include a third resolution zone 1221S having a third boundary, a horizontal resolution reduction factor of 2, and a vertical resolution reduction factor of 2. When both the horizontal resolution reduction factor and the vertical resolution reduction factor are 2, the resolution is half the maximum resolution of the display panel 240, and each received pixel value is displayed by a 2×2 pixel block of the display panel 240. Thus, in various implementations, the plurality of resolution zones include resolution zones where the resolution is reduced in both dimensions.

[0238] The plurality of resolution zones 2121A-2121Y include a fourth resolution zone 1221T having a horizontal resolution reduction factor of 4 and a vertical resolution reduction factor of 2. With the horizontal resolution reduction factor of 4 and the vertical resolution reduction factor of 2, the horizontal resolution is one-fourth of the maximum horizontal resolution of the display panel 240, the vertical resolution is half of the maximum vertical resolution of the display panel 240, and each received pixel value is displayed by a 4×2 pixel block of the display panel 240. Thus, in various implementations, the plurality of resolution zones include resolution zones where the resolution is reduced by a first amount in one dimension and by a second amount (different from the first amount) in another dimension.

[0239] As described above, in various implementations, the panel data indicating the plurality of resolution zones takes various forms. For example, in various implementations, the plurality of resolution zones 2121A-2121Y are transmitted as the position of each resolution zone, the size of each resolution zone, and the corresponding horizontal resolution reduction factor and vertical resolution reduction factor. In various implementations, the position of the resolution zones can be inferred from the size (e.g., the first resolution zone is located at the top left, the second resolution zone is located adjacent to and to the right of the first resolution zone, the third resolution zone is located adjacent to and to the right of the second resolution zone, the fourth resolution zone is located adjacent to and to the right of the third resolution zone, the fifth resolution zone is located adjacent to and to the right of the fourth resolution zone, the sixth resolution zone is located adjacent to and below the first resolution zone, etc.).

[0240] Reference again Figure 2 , the panel compensation module 234 generates panel data based on the distorted image and the scaling factor. As described above, in various implementations, the scaling factor is generated (by the rendering module 210) based on the rendering resolution function. Thus, the scaling factor defines a resolution function that generally describes the resolution of a displayed image as a function of position in the display space. Similarly, the resolution region (e.g., its position, size, and resolution reduction factor) defines a resolution function that describes the resolution of a displayed image as a function of position in the display space.

[0241] Fig. 22 A graph 2200 of an exemplary resolution function defined by scaling factors and resolution zones is shown. Graph 2200 includes a first resolution function 2210 defined by scaling factors associated with a warped image. Graph 2200 includes a second resolution function 2220 defined by a plurality of resolution zones having corresponding resolution reduction factors.

[0242] Fig.23 Another graph 2300 of exemplary resolution functions defined by scaling factors and resolution zones is shown. Graph 2300 includes a first resolution function 2310 defined by scaling factors associated with a warped image. Graph 2300 includes a second resolution function 2320 defined by a plurality of resolution zones having corresponding resolution reduction factors.

[0243] In various specific implementations, the panel compensation module 234 determines the boundaries of the resolution zones and the corresponding resolution reduction factors of the resolution zones based on the scaling factors, so that the resolution function defined by the resolution zones is greater than the resolution function defined by the scaling factors at all locations, thereby minimizing artifacts at the edges of the resolution zones caused by resolution changes.

[0244] In various implementations, the panel compensation module 234 determines the boundaries (e.g., size and location) of a resolution region of full resolution where the resolution reduction factor is 1 (or both the horizontal resolution reduction factor and the vertical resolution reduction factor are equal to 1). In various implementations, the center of the resolution region of full resolution is located at the center of the maximum value of the resolution function indicated by the zoom factor (e.g., where the user is looking).

[0245] In various implementations, the size of the resolution zone at full resolution is determined based on various factors. For example, in various implementations, the size of the resolution zone at full resolution is determined based on the peak width of the resolution function indicated by the scaling factor. In various implementations, the size of the resolution zone at full resolution is determined based on eye tracking data (e.g., how much the user's gaze moves). In various implementations, the size of the resolution zone at full resolution is determined based on eye tracking metadata that indicates characteristics of the eye tracking data.

[0246] In various implementations, the eye tracking metadata includes data indicating the confidence of the eye tracking data. For example, in various implementations, the eye tracking metadata provides a measure of the confidence that the eye tracking data correctly indicates the gaze of the user. In various implementations, the data indicating the confidence of the eye tracking data includes data indicating the accuracy of the eye tracking data. In various implementations, the data indicating the confidence of the eye tracking data includes data indicating the delay of the eye tracking data (e.g., the difference between the time when the eye tracking data is generated and the time when the eye tracking data is received). In various implementations, the eye tracking data includes data indicating a prediction of the user's gaze, and the data indicating the confidence of the eye tracking data includes data indicating the confidence of the prediction. In various implementations, the data indicating the prediction of the user's gaze is based on past measurements of the user's gaze derived relative to past captured images. In various implementations, the prediction of the user's gaze is based on classifying the past movement of the user's gaze into sustained gaze, smooth follow-through, saccade, etc. In various implementations, the confidence of the prediction is based on the classification. Specifically, in various implementations, the confidence of the prediction is higher when the past motion is classified as sustained fixation or smooth follow-up than when the past motion is classified as a saccade. Therefore, in various implementations, the size of the resolution zone of the full resolution is determined based on data indicating the confidence of the eye tracking data. For example, in various implementations, the size of the resolution zone of the full resolution is larger when the confidence is smaller.

[0247] In various implementations, the eye tracking metadata includes data indicating one or more biometric information of the user, in particular biometric information that affects the eye tracking metadata or its confidence. Specifically, in various implementations, the user's biometric information includes one or more of eye anatomy, ethnicity / appearance, eye color, age, visual aids (e.g., corrective lenses), makeup (e.g., mascara), medical conditions, historical gaze changes, input preferences or calibration, headset position / orientation, pupil dilation / center displacement, and / or eyelid position. Therefore, in various implementations, the size of the resolution zone of the full resolution is determined based on the one or more biometric information of the user.

[0248] In various implementations, the eye tracking metadata includes data indicating one or more conditions of the environment of the user generating the eye tracking data. Specifically, in various implementations, the environmental conditions include one or more of vibration, ambient temperature, IR directional light, or IR light intensity. Accordingly, in various implementations, the size of the resolution zone at full resolution is determined based on the one or more conditions of the user's environment.

[0249] In various implementations, after determining the size of the resolution region of full resolution (e.g., Fig. 22 In the case of a width of the top of the resolution function 2220 of the resolution function 2220, the boundaries of the remaining resolution region are determined so that the resolution function indicated by the resolution region is minimized while still being larger than the resolution function indicated by the scaling factor. In various implementations, the minimization is limited by the fact that the resolution reduction factor is an integer (or a subset thereof, such as only 1, 2, or 4). In various implementations, the minimization is limited by the fact that the size of the resolution region must be a multiple of the resolution reduction factor (e.g., a resolution region with a resolution reduction factor of 2 must be 2U×2V, where U and V are integers). In various implementations, the minimization is limited by the fact that the total size of the resolution region must be equal to the size of the display panel.

[0250] In various implementations, the size of the other resolution regions is based on the shape of the resolution function indicated by the scaling factor. Fig.23 In the figure, the resolution function 2310 indicated by the scaling factor decreases with the distance from the focusing window according to a curved step function having steps at a set of distances, and the resolution function 2320 indicated by the resolution region decreases with the distance from the focusing window according to a step function having steps at the set of distances.

[0251] Fig.24 2400 is a flowchart representation of a method 2400 for generating panel data according to some specific implementations. In some specific implementations (and as described in detail below as an example), the method 2400 is performed by a display module (or a portion thereof) such as Figure 2 In various implementations, the method 2400 is performed by a HMD (such as Figure 1 HMD100) or a portion thereof (such as Figure 2 SR pipeline 200). In various implementations, method 2400 is performed by a device having one or more processors, non-transitory memory, and one or more SR displays. In some implementations, method 2400 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, method 2400 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0252] Method 2400 begins at box 2410, when the display module receives a distorted image (e.g., a lens-compensated distorted image) representing simulated reality (SR) content to be displayed in a display space, the distorted image having a plurality of pixels at corresponding positions evenly spaced in a grid pattern in the distorted space, wherein the plurality of pixels are respectively associated with a plurality of corresponding pixel values ​​and a plurality of corresponding scaling factors indicating a plurality of corresponding resolutions at a plurality of corresponding positions in the display space.

[0253] In various implementations, the plurality of respective scaling factors define a mapping between the warp space and the display space. For example, in various implementations, different portions of the SR content corresponding to non-uniformly spaced regions in the display space are represented by uniformly spaced regions in the warp space.

[0254] In various implementations, each pixel of the plurality of pixels is respectively associated with a separately received pixel value. In various implementations, each pixel of the plurality of pixels is respectively associated with a separately received scaling factor (or a set of scaling factors including a horizontal scaling factor and a vertical scaling factor). In various implementations, each tile of the plurality of tiles of the plurality of pixels is respectively associated with a separately received scaling factor (or a set of scaling factors including a horizontal scaling factor and a vertical scaling factor). Thus, a plurality of pixels (e.g., those pixels of a single tile) are associated with a single received scaling factor (or a set of scaling factors including a single horizontal scaling factor and a single vertical scaling factor).

[0255] Thus, in various implementations, the warped image includes a plurality of tiles at corresponding positions evenly spaced in a grid pattern in the warped space, wherein each tile in the plurality of tiles is associated with a corresponding one or more scaling factors. Additionally, in various implementations, one or more scaling factors in the plurality of corresponding scaling factors include a horizontal scaling factor and a vertical scaling factor.

[0256] In various implementations, at least one of the plurality of scaling factors is not an integer. For example, in a process of generating the scaling factor as a difference between two tangent functions, most of the plurality of scaling factors will not be integers.

[0257] Method 2400 continues at box 2420, where the display module determines multiple corresponding boundaries and multiple corresponding resolution reduction factors in the display space for multiple resolution zones based on multiple corresponding scaling parameters, wherein the multiple corresponding resolution reduction factors are integers. In various specific implementations, the multiple corresponding resolution reduction factors include a single resolution reduction factor for a specific resolution zone. In various specific implementations, the multiple corresponding resolution reduction factors include a horizontal resolution reduction factor and a vertical resolution reduction factor for a specific resolution zone. In various specific implementations, the multiple corresponding resolution reduction factors include a first resolution reduction factor of 1, a second resolution reduction factor of 2, and a third resolution reduction factor of 4.

[0258] The plurality of resolution reduction factors indicate a resolution (generally describing a resolution function) at each of the plurality of corresponding positions in the display space. Similarly, the plurality of corresponding scaling factors indicate a resolution (generally describing a resolution function) at each of the plurality of corresponding positions in the display space. In various implementations, the plurality of corresponding resolution reduction factors indicate a resolution at each of the plurality of corresponding positions in the display space that is greater than a corresponding resolution at the plurality of corresponding positions in the display space indicated by the plurality of corresponding scaling factors.

[0259] For example, in various implementations, the respective resolutions at the respective positions in the display space indicated by the respective scaling factors are reduced according to a strictly decreasing function (e.g., Fig. 22 The resolution function 2210 of FIG. 2210 (shown in FIG. 2211 ) decreases with the distance from the focus window, and the resolution at each corresponding position of the plurality of corresponding positions in the display space indicated by the corresponding resolution reduction factor decreases according to a step function (e.g., as shown in FIG. 2211 ). Fig. 22 The resolution function 2220 decreases with the distance from the focusing window.

[0260] For example, in various specific implementations, the respective resolutions at the respective positions in the display space indicated by the respective scaling factors are calculated according to a curved step function having steps at a set of distances (e.g., Fig.23 The resolution function 2310 of FIG. 2310 (shown in FIG. 2311 ) decreases with the distance from the focus window, and the resolution at each corresponding position of the plurality of corresponding positions in the display space indicated by the corresponding resolution reduction factor is calculated according to a step function having steps at the set of distances (e.g., as shown in FIG. 2311 ). Fig.23 The resolution function 2320 decreases with the distance from the focusing window.

[0261] In various specific implementations, the size of one or more of the plurality of resolution zones, in particular the resolution zone of full resolution, is based on eye tracking data. In various specific implementations, the size of one or more of the plurality of resolution zones, in particular the resolution zone of full resolution, is based on eye tracking metadata, such as confidence in the eye tracking data.

[0262] The method 2400 continues at block 2430 when the display module transmits data indicating the plurality of corresponding boundaries in the display space, the plurality of corresponding resolution reduction factors, and corresponding pixel values ​​based on the plurality of corresponding pixel values. In various implementations, the transmitted data is panel data. In various implementations, the transmitted data is transmitted to a display panel.

[0263] Fig.25 25 is a flowchart representation of a method 2500 for processing a warped image in a warped space according to some implementations. In some implementations (and as described in detail below as an example), the method 2500 is composed of modules (or a portion thereof) such as Figure 2 In various implementations, the method 2500 is performed by the HMD (such as Figure 1 HMD 100) or a portion thereof (such as Figure 2 In various implementations, the method 2500 is performed by a device having one or more processors, non-transitory memory, and one or more SR displays. In some implementations, the method 2500 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 2500 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0264] Method 2500 begins at box 2510, when the module receives a distorted image representing simulated reality (SR) content to be displayed in a display space, the distorted image having a plurality of pixels at corresponding positions evenly spaced in a grid pattern in the distorted space, wherein the plurality of pixels are respectively associated with a plurality of corresponding pixel values ​​and a plurality of corresponding scaling factors, the plurality of corresponding scaling factors indicating a plurality of corresponding resolutions at a plurality of corresponding positions in the display space.

[0265] In various implementations, the plurality of respective scaling factors define a mapping between the warp space and the display space. For example, in various implementations, non-uniformly spaced regions of SR content corresponding to different amounts of regions in the display space are represented by the same amount of regions in the warped image.

[0266] In various implementations, each pixel of the plurality of pixels is respectively associated with a separately received pixel value. In various implementations, each pixel of the plurality of pixels is respectively associated with a separately received scaling factor (or a set of scaling factors including a horizontal scaling factor and a vertical scaling factor). In various implementations, each tile of the plurality of tiles of the plurality of pixels is respectively associated with a separately received scaling factor (or a set of scaling factors including a horizontal scaling factor and a vertical scaling factor). Thus, a plurality of pixels (e.g., those pixels of a single tile) are associated with a single received scaling factor (or a set of scaling factors including a single horizontal scaling factor and a single vertical scaling factor).

[0267] Thus, in various implementations, the warped image includes a plurality of tiles at corresponding positions evenly spaced in a grid pattern in the warped space, wherein each tile in the plurality of tiles is associated with a corresponding one or more scaling factors. Additionally, in various implementations, one or more scaling factors in the plurality of corresponding scaling factors include a horizontal scaling factor and a vertical scaling factor.

[0268] The method 2500 continues at block 2520, where the module processes the warped image in the warp space based on the plurality of corresponding scaling factors to generate a processed warped image. In various implementations, like the warped image, the processed warped image includes a plurality of pixels at corresponding locations evenly spaced in a grid pattern in the warp space. In various implementations, the plurality of pixels are respectively associated with a plurality of corresponding pixel values ​​(which may be different from the plurality of corresponding pixel values ​​of the warped image) and the plurality of corresponding scaling factors, the plurality of corresponding scaling factors indicating a plurality of corresponding resolutions at a plurality of corresponding locations in the display space.

[0269] In various implementations, processing the warped image in the warp space does not include processing the warped image in the warp space. Thus, processing the warped image in the warp space does not include converting the warped image into a display space image (which includes a plurality of pixels at corresponding locations evenly spaced in a grid pattern in the display space), does not include processing such a display space image, and does not include converting the processed display space image into a processed warped image in the warp space. Avoiding these steps (e.g., converting, processing, and reverse conversion) saves a significant amount of processing time and power. Thus, in a mobile device (such as an HMD), battery power is saved by processing the warped image in the warp space rather than in the display space.

[0270] In various implementations, the warped image is processed entirely in the warp space by a rendering module that generates the warped image to a display panel that displays an image based on the processed warped image. Thus, in various implementations, receiving the warped image (in block 2510) includes receiving the warped image from the rendering module. In some implementations, transmitting the processed warped image (as described below in block 2530) includes transmitting the processed warped image to the display panel. In various implementations, processing the warped image in the warp space includes processing the warped image in the warp space according to a plurality of processes in the warp space (e.g., performing two or more of compression, error correction coding, distortion [such as lens compensation], filtering, etc. in the warp space).

[0271] In various implementations, processing the warped image in the warp space includes compressing the warped image based on the plurality of corresponding scaling factors. Fig.15 Examples are described. In various specific implementations, processing the warped image in the warped space includes error correction encoding the warped image based on the plurality of corresponding scaling factors. Fig.15 Examples are described.

[0272] In various implementations, processing the warped image in the warped space includes distorting the warped image based on the plurality of corresponding scaling factors. Fig. 20 Examples are described.

[0273] In various implementations, processing the warped image in the warped space includes filtering the warped image based on the plurality of corresponding scaling factors. For example, applying a simple low pass filter defined by a normalized 3×3 matrix consisting of ones will inappropriately weight pixels having scaling factors indicating low resolution (where the pixels represent a larger area in the display space) compared to pixels having scaling factors indicating high resolution (where the pixels represent a smaller area in the display space). Thus, in various implementations, filtering the warped image in the warped space based on the plurality of scaling factors includes, for a particular pixel value, determining the filtered pixel value as a weighted sum of neighboring pixel values ​​of neighboring pixels, where the weights are based on a filter kernel (e.g., a normalized matrix consisting of ones, a Gaussian blur filter kernel) and the corresponding scaling factors of the neighboring pixels.

[0274] In various implementations, processing the warped image in the warp space includes color correcting the warped image based on the plurality of corresponding scaling factors.

[0275] Thus, in various implementations, processing the warped image in the warp space (e.g., distorting, filtering, or color correcting the warped image) includes determining, for a particular pixel of the warped image, a pixel value of a corresponding pixel of the processed warped image (e.g., at the same position in the warp space) based on corresponding pixel values ​​of pixels of the warped image in a neighborhood of the particular pixel and corresponding scaling factors of pixels of the warped image in the neighborhood of the particular pixel.

[0276] The method 2500 continues at block 2530, whereupon the module transmits the processed warped image. In various implementations, the module transmits the processed warped image along with a corresponding plurality of scaling factors.

[0277] Fig.26 26 is a flowchart representation of a method 2600 for processing a distorted image based on multiple scaling factors according to some implementations. In some implementations (and as described in detail below as an example), the method 2600 is composed of a module (or a portion thereof) such as Figure 2 In various implementations, the method 2600 is performed by the HMD (such as Figure 1 HMD 100) or a portion thereof (such as Figure 2 In various implementations, the method 2600 is performed by a device having one or more processors, non-transitory memory, and one or more SR displays. In some implementations, the method 2600 is performed by a processing logic component (including hardware, firmware, software, or a combination thereof). In some implementations, the method 2600 is performed by a processor executing instructions (e.g., code) stored in a non-transitory computer-readable medium (e.g., memory).

[0278] The method 2600 begins at block 2610 when the module receives a warped image representing simulated reality (SR) content, the warped image having a plurality of pixels at respective locations uniformly spaced in a grid pattern, wherein the plurality of pixels are respectively associated with a plurality of respective pixel values ​​and a plurality of respective scaling factors, the plurality of respective scaling factors indicating a plurality of respective resolutions at the plurality of respective locations of the SR content. In various implementations, different portions of the SR content corresponding to non-uniformly spaced regions of the SR content are represented by the same amount of regions in the warped image.

[0279] In various implementations, each pixel of the plurality of pixels is respectively associated with a separately received pixel value. In various implementations, each pixel of the plurality of pixels is respectively associated with a separately received scaling factor (or a set of scaling factors including a horizontal scaling factor and a vertical scaling factor). In various implementations, each tile of the plurality of tiles of the plurality of pixels is respectively associated with a separately received scaling factor (or a set of scaling factors including a horizontal scaling factor and a vertical scaling factor). Thus, a plurality of pixels (e.g., those pixels of a single tile) are associated with a single received scaling factor (or a set of scaling factors including a single horizontal scaling factor and a single vertical scaling factor).

[0280] Thus, in various implementations, the warped image includes a plurality of tiles at corresponding positions evenly spaced in a grid pattern, wherein each tile in the plurality of tiles is associated with a corresponding one or more scaling factors. Additionally, in various implementations, one or more scaling factors in the plurality of corresponding scaling factors include a horizontal scaling factor and a vertical scaling factor.

[0281] The method 2600 continues at block 2620, where the module processes the warped image based on the plurality of corresponding scaling factors to generate a processed warped image. In various implementations, like the warped image, the processed warped image includes a plurality of pixels at corresponding locations that are evenly spaced in a grid pattern. In various implementations, the plurality of pixels are respectively associated with a plurality of corresponding pixel values ​​(which may be different from the plurality of corresponding pixel values ​​of the warped image) and the plurality of corresponding scaling factors, the plurality of corresponding scaling factors indicating a plurality of corresponding resolutions at a plurality of corresponding locations of the SR content.

[0282] In various implementations, processing the warped image based on the respective plurality of scaling factors does not include generating an unwarped image in which different portions of the SR content corresponding to the evenly spaced regions of the SR content are represented by the evenly spaced regions of the unwarped image.

[0283] In various implementations, the distorted image is processed entirely by a rendering module based on the plurality of corresponding scaling factors, the rendering module generating the distorted image to a display panel, the display panel displaying the image based on the processed distorted image. Thus, in various implementations, receiving the distorted image (in block 2510) includes receiving the distorted image from the rendering module. In some implementations, transmitting the processed distorted image (as described below in block 2630) includes transmitting the processed distorted image to the display panel. In various implementations, processing the distorted image based on the plurality of corresponding scaling factors includes processing the distorted image based on the plurality of corresponding scaling factors according to a plurality of processes (e.g., performing two or more of compression, error correction coding, distortion [such as lens compensation], filtering, etc. based on the plurality of corresponding scaling factors).

[0284] In various specific implementations, processing the distorted image based on the plurality of corresponding scaling factors includes compressing the distorted image based on the plurality of corresponding scaling factors. Fig.15 Examples are described. In various specific implementations, processing the distorted image based on the plurality of corresponding scaling factors includes error correction encoding the distorted image based on the plurality of corresponding scaling factors. Fig.15 Examples are described.

[0285] In various implementations, processing the distorted image based on the plurality of corresponding scaling factors includes distorting the distorted image based on the plurality of corresponding scaling factors. Fig. 20 Examples are described.

[0286] In various implementations, processing the warped image based on the plurality of corresponding scaling factors includes filtering the warped image based on the plurality of corresponding scaling factors. For example, applying a simple low pass filter defined by a normalized 3×3 matrix consisting of ones will inappropriately weight pixels having scaling factors indicating low resolution (where the pixels represent a larger area in the display space) compared to pixels having scaling factors indicating high resolution (where the pixels represent a smaller area in the display space). Thus, in various implementations, filtering the warped image based on the plurality of scaling factors includes, for a particular pixel value, determining the filtered pixel value as a weighted sum of neighboring pixel values ​​of neighboring pixels, where the weights are based on a filter kernel (e.g., a normalized matrix consisting of ones, a Gaussian blur filter kernel) and the corresponding scaling factors of the neighboring pixels.

[0287] In various implementations, processing the distorted image based on the plurality of corresponding scaling factors includes color correcting the distorted image based on the plurality of corresponding scaling factors.

[0288] Thus, in various implementations, processing the warped image based on the plurality of corresponding scaling factors (e.g., distorting, filtering, or color correcting the warped image) includes, for a particular pixel of the warped image, determining a pixel value of the corresponding pixel of the processed warped image based on corresponding pixel values ​​of pixels of the warped image in a neighborhood of the particular pixel and corresponding scaling factors of pixels of the warped image in the neighborhood of the particular pixel.

[0289] The method 2600 continues at block 2630, whereupon the module transmits the processed warped image. In various implementations, the module transmits the processed warped image along with a corresponding plurality of scaling factors.

[0290] Although various aspects of specific implementations within the scope of the appended claims are described above, it should be apparent that the various features of the above-mentioned specific implementations can be embodied in a variety of forms, and any specific structures and / or functions described above are merely illustrative. Based on this disclosure, it should be understood by those skilled in the art that the aspects described herein can be implemented independently of any other aspects, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement a device and / or a method can be practiced. In addition, in addition to or different from one or more aspects set forth herein, other structures and / or functions can be used to implement such a device and / or such a method can be practiced.

[0291] It will also be understood that, although the terms "first", "second", etc. may be used to describe various elements in this article, these elements should not be limited by these terms. These terms are only used to distinguish one element from another element. For example, a first node may be referred to as a second node, and similarly, a second node may be referred to as a first node, which changes the meaning of the description, as long as all occurrences of the "first node" are consistently renamed and all occurrences of the "second node" are consistently renamed. Both the first node and the second node are nodes, but they are not the same node.

[0292] The terms used herein are only for describing specific implementations and are not intended to limit the claims. As used in the description of this specific implementation and the appended claims, the singular forms of "a" and "the" are intended to also cover the plural forms, unless the context clearly indicates otherwise. It will also be understood that the term "and / or" used herein refers to and covers any and all possible combinations of one or more items in the associated listed items. It will also be understood that the term "comprising" when used in this specification specifies the presence of stated features, integers, steps, operations, elements and / or parts, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, parts, and / or their grouping.

[0293] As used herein, the term “if” may be interpreted to mean “when the antecedent is true” or “when the antecedent is true” or “in response to determining” or “upon determining” or “in response to detecting” that the antecedent is true, depending on the context. Similarly, the phrase “if it is determined that [the antecedent is true]” or “if [the antecedent is true]” or “when [the antecedent is true]” is interpreted to mean “upon determining that the antecedent is true” or “in response to determining” or “upon determining” that the antecedent is true or “when detecting that the antecedent is true” or “in response to detecting” that the antecedent is true, depending on the context.

Claims

1. A method, comprising: Get the content to be rendered into the display space; obtaining a rendering resolution function that defines a mapping between a display space and a warp space, wherein the warp space includes uniformly spaced regions that map to non-uniformly spaced regions in the display space; as well as A rendered image is generated based on the content and the rendering resolution function, wherein the rendered image includes a warped image having a plurality of pixels at respective locations evenly spaced in a grid pattern in a warped space, wherein the plurality of pixels are respectively associated with a plurality of respective pixel values ​​and a plurality of respective scaling factors, the plurality of respective pixel values ​​being based on the content, and the plurality of respective scaling factors defining an area in a display space based on the rendering resolution function.

2. The method according to claim 1, wherein obtaining the rendering resolution function comprises: obtaining eye tracking data indicative of a user's gaze; as well as The rendering resolution function is generated based on the eye tracking data. 3 . The method of claim 2 , wherein the eye tracking data comprises at least one of data indicating a gaze angle of a user or data indicating a gaze point of a user.

4. The method of claim 1 , wherein the rendering resolution function has a maximum value that depends on eye tracking data and a summed value that is independent of the eye tracking data. The method of claim 1 , wherein the rendering resolution function is further based on eye tracking metadata. The method of claim 5 , wherein the eye tracking metadata comprises data indicating a confidence level of the eye tracking data. The method of claim 1 , wherein a rendering resolution function is further based on the content.

8. The method of claim 1, wherein the warped image comprises a plurality of tiles at respective locations evenly spaced in a grid pattern in the warped space, wherein each tile of the plurality of tiles is associated with a respective one or more scaling factors.

9. The method of claim 1, wherein one or more of the plurality of corresponding scaling factors comprises a horizontal scaling factor and a vertical scaling factor.

10. The method of claim 1, further comprising transmitting the warped image including the plurality of corresponding pixel values ​​in association with the plurality of corresponding scaling factors. The method of claim 1 , wherein the rendering resolution function characterizes a resolution in display space that varies with angle.

12. A method comprising: receiving a warped image having a plurality of pixels at respective locations evenly spaced in a grid pattern in a warp space, wherein the plurality of pixels are respectively associated with a plurality of respective pixel values ​​and a plurality of respective scaling factors indicating a plurality of respective resolutions at the plurality of respective locations in the display space; as well as The warped image is transmitted through a channel such that a bandwidth of the channel used for transmission of the corresponding portion of the warped image through the channel is based on a scaling factor of a plurality of scaling factors corresponding to the corresponding portion of the warped image.

13. The method of claim 12, further comprising compressing the corresponding portion of the warped image based on the scaling factor of the plurality of scaling factors corresponding to the corresponding portion of the warped image.

14. The method of claim 12, further comprising error correction encoding the corresponding portion of the distorted image based on the scaling factor of the plurality of scaling factors corresponding to the corresponding portion of the distorted image.

15. The method of claim 12, wherein the probability of transmission of the corresponding portion of the warped image is based on the scaling factor of the plurality of scaling factors that corresponds to the corresponding portion of the warped image. The method according to claim 15 , wherein the probability of transmission is the probability of retransmission in case of packet loss.

17. A method comprising: receiving at least a subset of a plurality of data packets corresponding to a portion of a warped image having a plurality of pixels at corresponding locations evenly spaced in a grid pattern in a warped space; receiving a plurality of scaling factors indicating respective resolutions of the portion of the warped image at a plurality of respective locations in a display space; detecting an error condition of a particular data packet among the plurality of data packets; as well as The error condition is resolved based on a particular scaling factor of the plurality of scaling factors corresponding to the particular one of the plurality of data packets.

18. The method of claim 17, wherein detecting the error condition comprises receiving the particular one of the plurality of scaling factors without receiving the particular one of the plurality of data packets.

19. The method of claim 17, wherein resolving the error condition comprises determining whether to send a retransmission request for the particular one of the plurality of data packets based on the particular one of the plurality of scaling factors.

20. The method of claim 17, wherein resolving the error condition comprises determining whether to drop or store the particular one of the plurality of packets based on the particular one of the plurality of scaling factors.

21. A method comprising: At a device having one or more processors, non-volatile memory, and a display panel: receiving a warped image representing content to be displayed by a display panel in a display space, the display space comprising non-uniformly spaced regions mapped to uniformly spaced regions of the display panel, the warped image having a plurality of pixels at respective locations uniformly spaced in a grid pattern in a warp space different from the display space, and wherein the warp space comprises uniformly spaced regions that map to the non-uniformly spaced regions in the display space; obtaining a plurality of corresponding scaling factors for the warped image, wherein the plurality of pixels of the warped image are respectively associated with a plurality of corresponding pixel values ​​and the plurality of corresponding scaling factors, the plurality of corresponding scaling factors indicating a plurality of corresponding resolutions at a plurality of corresponding locations in a display space associated with a display panel, and wherein the plurality of corresponding scaling factors define a mapping between the display space and the warped space; determining a plurality of respective boundaries in a display space and a plurality of respective resolution reduction factors for a plurality of resolution regions based on the plurality of respective scaling factors; as well as Data indicative of the plurality of respective boundaries in a display space, the plurality of respective resolution reduction factors, and corresponding pixel values ​​based on the plurality of respective pixel values ​​associated with the warped image are transmitted to a display panel.

22. The method of claim 21, wherein at least one scaling factor of the plurality of corresponding scaling factors is not an integer.

23. The method according to claim 21, further comprising: The warped image in the warp space is processed based on the plurality of respective scaling factors to generate a processed warped image, wherein processing the warped image comprises at least one of distorting the warped image based on the plurality of respective scaling factors or filtering the warped image based on the plurality of respective scaling factors, wherein processing the warped image in the warp space does not include processing the image in the display space.

24. The method of claim 23, wherein processing the warped image in the warped space comprises: For a specific pixel among the plurality of pixels, a pixel value of a corresponding pixel of the processed warped image is determined based on corresponding pixel values ​​of pixels of the warped image in a neighborhood of the specific pixel and corresponding scaling factors of pixels of the warped image in the neighborhood of the specific pixel.

25. An apparatus comprising: one or more processors; non-transitory memory; one or more displays; as well as Means for causing the device to perform the method according to any one of claims 1 to 24.

26. A non-transitory computer-readable medium having encoded thereon instructions that, when executed by one or more processors of a device, cause the device to perform the method of any one of claims 1 to 24.

27. A computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method according to any one of claims 1 to 24.