Method for calculating head-mounted display image and foveal region brightness

Abstract

An image that can be displayed at a head-mounted display (HMD) is downsampled into image pixels representing the image. The illuminance of the image pixels is calculated from the color values of the image pixels to determine the image brightness. The foveal region of the image is calculated from the eye tracking information of the user of the HMD. The foveal region of the image is downsampled into foveal pixels representing the foveal region. The illuminance of the foveal pixels is calculated from the color values of the foveal pixels to determine the foveal region brightness.

Description

Background Art

[0001] Extended Reality (XR) technologies include Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR) technologies, and quite literally extend the reality of the user experience. XR technologies can employ a Head-Mounted Display (HMD). An HMD is a display device that can be worn on the head. In VR technology, the HMD wearer is immersed in a completely virtual world, while in AR technology, the direct or indirect view of the physical, real-world environment of the HMD wearer is enhanced. In MR or Mixed Reality technology, the HMD wearer experiences a fusion of the real and virtual worlds. Brief Description of the Drawings

[0002] Figure 1 is a diagram of an exemplary topology for calculating Head-Mounted Display (HMD) images and foveal region luminance.

[0003] Figure 2 is a diagram of an exemplary image and its foveal region.

[0004] Figure 3 is a flowchart of an exemplary method for determining image pixels representing an image and foveal pixels representing the foveal region of the image.

[0005] Figure 4 is a diagram of an exemplary downsampling of an image or foveal region to a representative image or foveal region pixel.

[0006] Figure 5 is a flowchart of an exemplary method for calculating the illuminance of an image or foveal region as the luminance of the image or foveal region from image or foveal region pixels.

[0007] Figure 6 is a diagram of an exemplary non-transitory computer-readable data storage medium.

[0008] Figure 7 is a flowchart of an exemplary method. Detailed Description

[0009] As described in the background art, a Head-Mounted Display (HMD) can be used as an Extended Reality (XR) technology to extend the reality of the HMD wearer's experience. The HMD can include small display panels in front of each eye of the wearer, as well as various sensors for detecting or sensing the wearer and / or the wearer's environment, such that the images on the display panels convince the wearer to be immersed in the XR, which is Virtual Reality (VR), Augmented Reality (AR), Mixed Reality (MR), or another type of XR. The HMD can include one or more lenses in the optical path between each display panel and the corresponding eye of the user.

[0010] The HMD can be used as an accessory for biometric inference processing, which is the analysis of biometric information of the wearer of the HMD to make inferences about the state of the wearer. One type of such biometric inference processing is the estimation or determination of cognitive load. The cognitive load of a user can be defined non - restrictively as a multi - dimensional construct that represents the load imposed on the user's cognitive system by performing a specific task. Tasks may objectively require more or less, people can have different cognitive abilities, and certain tasks may be easier for those who are proficient in the task.

[0011] Cognitive load can be measured from pupil changes, which makes it well - suited for estimation for HMD wearers since the eye - tracking information typically measured by the HMD can include such pupil metrics. However, in order to use pupil changes as an accurate estimate of cognitive load, the effects of image and foveal region brightness must first be removed or otherwise compensated for. This is because in addition to the cognitive load of the HMD wearer, both image and foveal region brightness also affect pupil changes.

[0012] Image brightness is the brightness of the entire image that is presented to and thus viewed by the eye. In the case where the HMD is in the form of goggles or a helmet with an eye - cup that is positioned adjacent to the wearer's face during use, the wearer's eyes are not exposed to light other than the light from the panel of the HMD that displays the image. In contrast, in the case where the HMD is in the form of glasses, the wearer's eyes may also be exposed to external or ambient surrounding light.

[0013] The foveal region of the image can be estimated as the part of the image that the HMD wearer is fixating on. More precisely, the foveal region is the part of the displayed image whose light enters the pupil of the eye and impinges on the foveal region of the eye. The foveal region of the eye is located at the inner rear of the eye, where the eye is most sensitive to light. That is, the foveal region of the eye is the part of the eye that has the highest cone density.

[0014] The techniques described herein provide a way to calculate image and foveal region brightness in real - time without affecting the image display at the HMD. For example, when the eye - tracking information is refreshed and when the image displayed to each eye is refreshed, the image and foveal region brightness can be calculated for each eye. More specifically, the image and foveal region brightness can be calculated as the illuminance of the image and foveal region of each eye, as displayed at the display panel of the HMD.

[0015] Thus, four illuminances can be calculated: the illuminance of the image for the left eye and the illuminance of the foveal region of that image, as well as the illuminance of the image for the right eye and the illuminance of the foveal region of that image. If the illuminance is actually measurable in real time, each illuminance can accurately correspond to the actual measured illuminance at the HMD. This illuminance is contrasted with the perceived illuminance, which is the illuminance perceived by the human perceptual system in the brain and can vary from person to person.

[0016] Figure 1 An exemplary topology 100 of the overall process is shown by which left and right images and foveal region luminance can be calculated in conjunction with the HMD 102. The HMD 102 includes a left eyepiece assembly 104L and a right eyepiece assembly 104R, which are collectively referred to as the eyepiece assembly 104. The left eyepiece assembly 104L and the right eyepiece assembly 104R have respective display panels 106L and 106R at ends 108L and 108R opposite the ends 110L and 110R where the user of the HMD 102 positions his or her eyes, which are collectively referred to as the display panel 106. The display panel 106 can be a flat panel display, such as a liquid crystal display (LCD) or a light emitting diode (LED) display, such as an organic LED (OLED) and a micro LED display.

[0017] The HMD 102 also includes a left eye tracking sensor 109L and a right eye tracking sensor 109R, which are collectively referred to as the eye tracking sensors 109. The eye tracking sensors 109 track the gaze of the HMD user's eyes relative to the display panel 108 and can include, for example, cameras and other types of hardware components. The HMD 102 itself can also include other components, such as one or more various lenses in the optical path between the user's eyes at ends 110L and 110R of the eyepiece assembly 104 and the display panel 106 at ends 108L and 108R of the assembly 104.

[0018] The left image 112L and the right image 112R, collectively referred to as the image 112, are respectively displayed on the display panels 108L and 108R. The images 112 are different from each other but correspond to each other such that when the eyes view the images 112 simultaneously, the HMD user's brain stereoscopically perceives a single image. The left eye tracking information 114L and the right eye tracking information 114R, collectively referred to as the eye tracking information 114, are obtained from the HMD 102, based on which the left foveal region 116L and the right foveal region 116R of the images 112L and 112R can be estimated and determined. The foveal regions 116L and 116R are collectively referred to as the foveal region 116.

[0019] The left image 112L is downsampled or compressed to determine left image pixels 118L that represent the image 112L, and the right image 112R is similarly downsampled or compressed to determine right image pixels 118R that represent the image 112R. The image pixels 118L and 118R are collectively referred to as image pixels 118. Each image pixel 118 represents its corresponding image 112 because the image pixel 118 has a color value that is the average of the color values of the constituent pixels of the corresponding image 112. The color value can be expressed as a tuple of three color values (such as red, green, and blue values).

[0020] Similarly, the left fovea region 116L is downsampled or compressed to determine left fovea pixels 120L that represent the fovea region 116L of the left image 112L, and the right fovea region 116R is similarly downsampled or compressed to determine right fovea pixels 120R that represent the fovea region 116R of the right image 112R. The fovea pixels 120L and 120R are collectively referred to as fovea pixels 120. Each fovea pixel 120 represents its corresponding fovea region 116 because the fovea pixel 120 has a color value that is the average of the color values of the pixels of the corresponding fovea region 116 of the image 112 under discussion.

[0021] The left image luminance 122L and the right image luminance 122R are calculated from the left image pixels 118L and the right image pixels 118R, respectively. The left image luminance 122L and the right image luminance 122R are collectively referred to as image luminance 122. Each image luminance 122 represents the total luminance of its corresponding image 112 and can be determined as the illuminance of its corresponding image pixel 118 from the color value of the pixel 118.

[0022] Similarly, the left fovea region luminance 124L and the right fovea region luminance 124R are calculated from the left fovea pixels 120L and the right fovea pixels 120R, respectively. The left fovea region luminance 124L and the right fovea region luminance 124R are collectively referred to as fovea region luminance 124. Each fovea region luminance 124 represents the total luminance of its corresponding fovea region 116 of the image 112 under discussion and can be determined as the illuminance of its corresponding fovea pixel 120 from the color value of the pixel 120.

[0023] The biometric inference processing 126 can be performed based on the image luminance 122 and the fovea region luminance 124 and other information. For example, the eye tracking information 114 can include pupil changes (e.g., pupil diameter changes) of each eye of the HMD user, based on which the effects of the luminance 122 and 124 are removed or otherwise compensated. As described, the biometric inference processing 126 can include determining or estimating the cognitive load of the HMD user.

[0024] In one embodiment, the image 112 presented to the user by the HMD 102 can be adjusted based on the results of biometric inference processing 126. As an example, an HMD user can perform a task on a machine by following instructions presented at the HMD 102. Based on the estimated cognitive load of the HMD user while performing the task and other information such as how well or correctly the user performs the task, the instructions can be simplified or presented in more detail. The likelihood that the HMD user will make an error can even be predicted before the error actually occurs.

[0025] The biometric inference processing 126 and the resulting adjustment of the displayed image 126 can be performed by an application software external to and separate from the program code that determines the brightness 122 and 124. That is, the biometric inference processing 126 performed and how the displayed image 112 is adjusted based on the inference processing 126 use the determined brightness 122 and 124, but the techniques described herein for determining the brightness 122 and 124 are independent of such inference processing 126 and image 126 adjustment. This inference processing 126 and adjustment of the displayed image 126 can occur in real time or offline with respect to the determination of the brightness 122 and 124.

[0026] Figure 2 An exemplary image 212 is shown that has a foveal region 216 identified therewith. The image 212 can be a left image shown to the left eye of the HMD user or a right image shown to the right eye of the HMD user. The foveal region 216 is the portion of the image 212 that the user's gaze is directed to, which can be determined from eye tracking information captured by the HMD. When the user directs his or her focused attention to different parts of the image 212, the foveal region 216 changes accordingly.

[0027] Figure 3 An exemplary method 300 is shown for determining image pixels representative of an image and foveal pixels representative of the foveal region of the image. The method 300 is described with respect to one such image, but is performed for each of a left image shown to the left eye of the HMD user and a right image shown to the right eye of the HMD user. The method 300 can be performed by a host computing device communicatively coupled to the HMD, such as a desktop or laptop computer or a mobile computing device, such as a smart phone or a tablet computing device. The method 300 can be implemented as program code stored on a non-transitory computer-readable data storage medium and executable by a processor of the host computing device.

[0028] In one embodiment, the method 300 is performed by a graphics processing unit (GPU) of the host computing device as opposed to a different processor such as a central processing unit (CPU) of the device. The method 300 is in Figure 3Examples are specifically shown for such embodiments and utilize the processing capabilities unique to a GPU (compared to a CPU) to permit real-time image and foveal pixel determination (and thus image and foveal region luminance) without slowing down the display of the image at the HMD. In contrast, performing such processing at the CPU can overburden the CPU and may not permit real-time determination of the image and foveal pixels.

[0029] Method 300 is described with respect to an example where the refresh rate at which eye tracking information is updated is faster than the refresh rate at which the image displayed at the HMD is updated. In such an example, for both the left and right eyes, method 300 is repeated upon each refresh of the eye tracking information. At the refresh of the eye tracking information (302), the image to be displayed at the HMD may or may not have been refreshed since the last refresh of the eye tracking information. If an image refresh has occurred (304), then method 300 includes then obtaining the image (306) from, for example, a compositor that generates the image for display.

[0030] Method 300 includes copying and posting the image for display at the display panel of the HMD (308). For example, the image can be sent (i.e., bit blit or bitmap transfer) from one location block of the GPU's memory to another. The bitmap transfer is performed very quickly. The remainder of method 300 operates on the copy of the image that has been bit blit. Thus, the original image can be posted for display at the display panel without having to wait for the completion of method 300. In this way, the display of the image at the HMD is not slowed down because in practice the bitmap transfer can be performed at a much faster rate than the image refresh rate.

[0031] Method 300 can include applying an occlusion mask to the image (310). Although the image may be perfectly rectangular or square, the physical geometry of the HMD or its current physical configuration may result in occlusion of portions of the image. The eyes of the HMD user are not exposed to the occluded portions of the image and thus these portions are not considered when determining the total image luminance. The occlusion mask defines which portions of the image are visible and which portions are occluded, and the application of the mask removes the occluded portions from the image. However, in another embodiment, there may be no occlusion at the HMD, or the image obtained in part 306 may already have accounted for occlusion, in which case the occlusion mask must be applied.

[0032] Method 300 then includes downsampling the image to image pixels representative of the image (312). As described above, the downsampled image pixels have color values that are the average of the color values of the constituent images of the image. The GPU is particularly well-suited for such downsampling because it can perform texture mapping (mipmap). Texture mapping is the process of progressively downsampling an image by orders of magnitude of two, which can culminate in a one-pixel-by-one-pixel texture map of the image that can correspond to the determined image pixels. The downsampling process can also be referred to as a compression process because the GPU can recursively perform wavelet or other compression on the image to achieve a one-pixel-by-one-pixel texture map. Example image downsampling is illustratively described later in the detailed description.

[0033] Then, the luminance of the image can be determined based on the image pixels (314). The specific manner of calculating the image luminance from the determined image pixels is described later in the detailed description. The image luminance calculation can be performed at the CPU of the host computing device or at the GPU of the host computing device. For example, compared to the processing capabilities of the GPU, the calculations performed to calculate the image luminance from the image pixels may be more suited to the processing capabilities of the CPU, thus causing itself to be performed by the CPU to both calculate the image luminance in real time with the display of the image at the HMD and avoid an unnecessary burden on the GPU.

[0034] Regarding foveal region luminance determination, method 300 includes obtaining eye tracking information from the HMD (316). Method 300 then includes using the obtained eye tracking information to determine the foveal region of the image being displayed at the HMD (318). If the image has been refreshed since the last refresh of the eye tracking information when the eye tracking information is refreshed, the foveal region is determined for the image obtained in part 306 in the current iteration of method 300. As a comparison, if the image has not been refreshed since the last refresh of the eye tracking information when the eye tracking information is refreshed, the foveal region is determined for the image obtained in part 306 in the previous iteration of method 300.

[0035] The eye tracking information can specify a vector extending outward from the center of the user's eye pupil of the HMD in three-dimensional space toward the display panel of the HMD that displays the image. Thus, determining the foveal region of the image can require projecting the three-dimensional vector onto the two-dimensional surface representing the display panel. The foveal region is a contiguous portion of the image as a whole. As described above, the foveal region is the portion of the image toward which the user's gaze is directed, as an estimate of the portion of the image incident on the foveal region of the user's eye.

[0036] Method 300 then includes downsampling the foveal region of the image to foveal pixels (320) representative of the foveal region. The downsampling of the foveal region is performed in the same manner as the downsampling of the image, but starting from the foveal region of the image rather than the image as a whole. Once the foveal region has been downsampled or compressed to foveal pixels, the luminance of the foveal region can be determined in the same manner as determining the image luminance from image pixels, based on the foveal pixels (322).

[0037] As described, Figure 3 An exemplary implementation of method 300 is shown, where the eye tracking information is refreshed faster than the image displayed at the HMD. However, in another implementation, the image refresh rate can be faster than the rate at which the eye tracking information is refreshed. In this case, the execution of method 300 is based on image refresh prediction, so method 300 is repeated each time the image is refreshed. Even if the eye tracking information may not have been refreshed, the foveal region is determined with each iteration of method 300 because the image of which the foveal region is a part will have changed.

[0038] Figure 4 An example of image to image pixel downsampling as may be performed by the GPU of the host computing device in part 312 of Figure 3 is shown. The downsampling process starts with an image that may have a resolution of n pixels by n pixels. (Such as in part 318, downsampling the foveal region of the image to foveal pixels is performed in the same manner, but starting from the foveal region of the image rather than the entire image.) The image can be considered as an initial image representation 400A at full n pixel by n pixel resolution.

[0039] The image representation 400A is downsampled by an order of two to an image representation 400B having a resolution of n / 2 pixels by n / 2 pixels. Then, the image representation 400B is downsampled by an order of two to an image representation 400C having a resolution of n / 4 pixels by n / 4 pixels. This downsampling or compression is continuously repeated, resulting in a four pixel by four pixel image representation 400L, and then a two pixel by two pixel image representation 400M before culminating in a compressed one pixel by one pixel image representation 400N (which is a single image pixel representing the entire image).

[0040] The image representations 400A, 400B, 400C,..., 400L, 400M, and 400N are collectively referred to as image representations 400. Image representations other than the first image representation 400A can be progressive texture mappings with a resolution reduced by an order of two, which are calculated by the GPU using its texture mapping function. Texture mapping is a lower representation of the image (i.e., the image corresponding to the initial image representation 400A).

[0041] As described above, an image pixel (i.e., a one-pixel by one-pixel image representation 400N) represents the entire image and has a color value that is the average color value of the constituent n by n pixels of the image. The brightness of the overall image can be determined from this image pixel. However, in another embodiment, there may be more than one image pixel for which the image brightness is determined. For example, the image brightness can be determined for each image pixel of a two-pixel by two-pixel image representation 400M.

[0042] In this case, the upper-left image pixel of the image representation 400M has a color value that is the average color value of the corresponding upper-left quadrant of the constituent pixels of the image. Similarly, the upper-right, lower-right, and lower-left image pixels of the image representation 400M have color values that are the average color values of the corresponding upper-right, lower-right, and lower-left quadrants of the pixels of the image, respectively. If the pupil change is not uniformly affected by brightness, it can be beneficial to determine the image brightness of more than one image pixel (and similarly, the foveal region brightness of more than one foveal pixel).

[0043] The image can be downsampled to pixels using a method different from texture mapping. For example, a compute shader that does not use texture mapping can be used. A compute shader is a computer program (e.g., a programmable stage, including vertex and geometry stages) executed by the GPU, and instances of it can be executed simultaneously by the corresponding hardware processing units of the GPU. For example, an example of such a hardware processing unit is a single instruction multiple data (SIMD) unit.

[0044] Figure 5 An exemplary method 500 for calculating the illuminance of an image or foveal pixel to obtain the illuminance of an image or foveal region is shown. The calculated image illuminance corresponds to and is used as the determined brightness of the image, and similarly, the calculated foveal region illuminance corresponds to and is used as the determined brightness of the foveal region. Thus, method 500 can be executed to implement Figure 3 parts 314 and 322. Method 500 is described with respect to calculating image illuminance, but is equally applicable to calculating foveal region illuminance.

[0045] Method 500 can be executed by a host computing device communicatively connected to the HMD. Similar to method 300, method 500 can be implemented as program code stored on a non-transitory computer-readable data storage medium and executable by a processor of the host computing device. As described, the image brightness can be more appropriately determined by the CPU of the host computing device rather than by the GPU of the device, such that method 500 can be executed by the CPU accordingly.

[0046] Method 500 includes obtaining image pixels (512). For example, the CPU of a host computing device may read the image pixels determined by the GPU. The color space of the image and thus the color values of the image pixels may be in a linear color space or a non-linear color space. If the color space is non-linear (504), then method 500 includes converting the color values to a linear color space (506) such that the illuminance of the image pixels is properly calculated. The red-green-blue (RGB) color space is an example of a linear color space, while the standard RGB (sRGB) color space is an example of a non-linear color space.

[0047] Method 500 may include adjusting the image pixels to compensate for the HMD display panel (508) that displays the image. More specifically, the color values of the image pixels are adjusted to compensate for the brightness variation characteristics of the display panel when displaying different colors. Thus, an experimentally derived look-up table (LUT) may be used or the color values of the image pixels may be adjusted in another way to compensate for this variation. Different characteristics of the display panel may cause these brightness variations.

[0048] For example, the physical pixels of the display panel may consist of red, green, and blue sub-pixels, some of which may be shared with adjacent physical pixels of the panel. Although an ideal display panel displays each sub-pixel color with uniform brightness, in reality some colors may be brighter or darker than others. The number of physical sub-pixel elements (such as LCD cells or LEDs) corresponding to each sub-pixel color may vary. The physical arrangement of the sub-pixel elements in forming the sub-pixels may be different for different colors of the same display panel and for different panels.

[0049] Method 500 ends by calculating the illuminance of the image pixels from the color values of the pixels (510). Since the color values are for a linear color space, equations may be used to calculate the illuminance. For example, for the RGB color space, the illuminance may be calculated as the maximum of the red, green, and blue color values. As another example, the illuminance may be the maximum value minus the minimum of the red, green, and blue color values, and the resulting difference divided by two. As described above, the calculated illuminance is used as the brightness of the image. Method 500 is performed for both the left image brightness and the right image brightness and for both the left foveal region brightness and the right foveal region brightness. If more than one pixel is determined for each image and each foveal region, then method 500 is similarly performed for each image pixel and each foveal region.

[0050] Figure 6An exemplary non - transitory computer - readable data storage medium 600 storing program code 602 is shown. The program code 602 can be executed by a computing device, such as a host computing device connected to an HMD. The processing includes calculating an illuminance of an image pixel representing an image displayable at the HMD from a color value of the image pixel to determine an image brightness (604). The processing includes calculating an illuminance of a foveal pixel representing a foveal region of the image from a color value of the foveal pixel to determine a foveal region brightness (606). The foveal region is determined from eye - tracking information of a user of the HMD.

[0051] Figure 7 An exemplary method 700 is shown, which can be executed by a computing device, such as a host computing device connected to an HMD. Method 700 includes downsampling an image displayable at the HMD to image pixels representing the image (702), and calculating an illuminance of the image pixels from the color values of the image pixels to determine an image brightness (704). Method 700 includes determining a foveal region of the image from eye - tracking information of a user of the HMD (706). Method 700 includes downsampling the foveal region of the image to foveal pixels representing the foveal region of the image (708), and calculating an illuminance of the foveal pixels from the color values of the foveal pixels to determine a foveal region brightness (710).

[0052] Techniques have been described for determining image and foveal region brightness in real - time when displaying an image at an HMD. The image and foveal region brightness are determined as image and foveal region illuminances calculated respectively from image and foveal pixels representing the image and the foveal region. The determined image and foveal region brightness can be used when performing biometric inference processing on a user of the HMD.

Claims

1. A method for calculating the image and foveal region brightness of a head-mounted display (HMD), comprising: downsampling, by a computing device, an image that can be displayed at the head-mounted display (HMD) into image pixels representing the image; calculating, by the computing device, the illuminance of the image pixels from the color values of the image pixels to determine the image brightness; determining, by the computing device, the foveal region of the image from the eye tracking information of the user of the HMD; downsampling, by the computing device, the foveal region of the image to foveal pixels representing the foveal region of the image; and calculating, by the computing device, the illuminance of the foveal pixels from the color values of the foveal pixels to determine the foveal region brightness; wherein the display panel of the HMD that displays the image has varying brightness characteristics for different sub-pixel colors, and the method further comprises: before determining the illuminance of the image pixels from the color values of the image pixels, adjusting, by the computing device, the color values of the image pixels to compensate for the varying brightness characteristics of the display panel for different sub-pixel colors; and before determining the illuminance of the foveal pixels from the color values of the foveal pixels, adjusting, by the computing device, the color values of the foveal pixels to compensate for the varying brightness characteristics of the display panel for different sub-pixel colors.

2. The method according to claim 1, wherein, the image is a left-eye image, the foveal region is the left-eye foveal region, the image brightness is the left-eye image brightness, and the foveal region brightness is the left-eye foveal region brightness, and wherein the method is repeated to determine the right-eye image brightness of the right-eye image and the right-eye foveal region brightness of the right-eye foveal region of the right-eye image.

3. The method according to claim 1, wherein, during a biometric inference process regarding the user of the HMD, the calculated image and foveal region brightness are removed when evaluating pupil diameter changes.

4. The method according to claim 3, wherein, the biometric information processing includes measuring the cognitive load of the user of the HMD.

5. The method according to claim 3, wherein, the image that can be displayed at the HMD is adjusted based on a biometric inference process regarding the user of the HMD.

6. The method according to claim 1, wherein, the eye tracking information has a higher refresh rate than the image, and the method is repeated at each refresh of the eye tracking information, and wherein, if the image has been refreshed since the previous refresh of the eye tracking information, the image is downsampled and the image brightness is determined at each refresh of the eye tracking information.

7. The method according to claim 1, wherein, the image has a higher refresh rate than the eye tracking information, and the method is repeated at each refresh of the image.

8. The method according to claim 1, wherein, the image is downsampled by mapping the image texture to a one-pixel by one-pixel texture map as the image pixels, And wherein, the foveal region of the image is downsampled by mapping the image texture to a one-pixel by one-pixel texture map as foveal pixels.

9. The method according to claim 1, wherein, the image pixels are selected image pixels, the image is downsampled by mapping the image texture to an n-pixel by n-pixel texture map including the selected image pixels, and the illuminance of each image pixel is calculated, wherein the foveal pixels are selected foveal pixels, the image is downsampled by mapping the foveal region texture of the image to an n-pixel by n-pixel texture map including the selected foveal image pixels, and the illuminance of each foveal pixel is calculated.

10. The method according to claim 1, wherein, the eye tracking information includes a three-dimensional vector radiating outward from the user's eye in the direction of gaze towards the display panel of the head-mounted display (HMD) for the displayed image, and wherein, the foveal region is determined by projecting the three-dimensional vector onto a two-dimensional surface representing the display panel.

11. The method according to claim 1, wherein, the image is displayed at the HMD in an occluded manner, and the method further includes: before downsampling the image, applying an occlusion mask by the computing device to the image corresponding occluded portion when displayed at the HMD.

12. A non-transitory computer-readable data storage medium storing program code executable by a computing device to perform a process including the following: Calculating the illuminance of the image pixels representing an image displayable at a head-mounted display (HMD) from the color values of the image pixels to determine the image brightness ； and Calculating the illuminance of the foveal pixels representing the foveal region of the image from the color values of the foveal pixels to determine the foveal region brightness, the foveal region being determined from the eye tracking information of the user of the HMD; wherein the display panel of the HMD for displaying the image has varying brightness characteristics for different sub-pixel colors, and the method further includes: Before determining the illuminance of the image pixels from the color values of the image pixels, adjusting the color values of the image pixels by the computing device to compensate for the varying brightness characteristics of the display panel for different sub-pixel colors; and Before determining the illuminance of the foveal pixels from the color values of the foveal pixels, adjusting the color values of the foveal pixels by the computing device to compensate for the varying brightness characteristics of the display panel for different sub-pixel colors.

13. The non-transitory computer-readable data storage medium according to claim 12, wherein, the process further includes: Determining the image pixels by continuously downsampling the image by an order of two at the graphics processing unit (GPU) of the computing device to produce a compressed image representation including the image pixels; and Determining the foveal pixels by continuously downsampling the image by an order of two at the GPU of the computing device to produce a compressed foveal region representation including the foveal pixels.

14. The non-transitory computer-readable data storage medium according to claim 13, wherein, the illuminance of the image and the foveal pixels is calculated at a central processing unit (CPU) of the computing device.

Images