Foveated rendering of an extended reality environment

Abstract

There is provided techniques for foveated rendering of an XR environment. A method is performed by a controller. The controller is configured to select between using a first eye-tracker and a second eye-tracker for eye-tracking of a user in the XR environment. The method comprises obtaining, using the first eye-tracker, a gaze direction of the user towards a first region in the XR environment. The method comprises selecting, based on a predicted gaze behavior of the user and an energy consumption cost for switching from using the first eye-tracker to the second eye-tracker for continued eye-tracking of the user, which of the first eye-tracker and the second eye-tracker to use. The gaze behavior indicates statistics of how long the user is expected to maintain the gaze direction. The method comprises performing foveated rendering of the XR environment in accordance with eye-tracking of the user as performed using the selected eye-tracker.

Description

[0001] FOVEA TED RENDERING OF AN EXTENDED REALITY ENVIRONMENT

[0002] TECHNICAL FIELD

[0003] Embodiments presented herein relate to a method, a controller, a computer program, and a computer program product for foveated rendering of an extended reality environment.

[0004] BACKGROUND

[0005] Camera-based eye-tracking, is a well-used technology in e.g., laptops and extended reality (XR) headsets, to track the eyes of the user for several different types of applications or user analyses. Stereo-based eyetracking uses two cameras as well as several illuminators using near-infrared light to enhance the eyetracking.

[0006] Whereas camera-based eye-tracking has the ability to provide detailed gaze tracking, camera-based eyetracking is quite computationally complex, have a high cost, and requires high power consumption. Camera-based eye-tracking is often based on statically mounted cameras. There are simple camera-based eye-tracking techniques available, see for example E. Whitmire et al. “EyeContact: Scleral Coil Eye Tracking for Virtual Reality” in Proceedings of the International Symposium of Wearable Computers (ISWC), September 12-16, 2016, but these eye-tracking techniques are typically less accurate than camera-based eye-tracking with statically mounted cameras, are more difficult to place optimally in lightweight equipment, such as XR glasses, and still suffer from a significant power consumption due to the operation of multiple cameras, processing of image data at certain framerate and the use of near-infrared illumination (if included).

[0007] Originating for medical purposes, another technology to track the user’s eyes is electrooculography, or EOG. Electrooculography is a technique for measuring the comeo-retinal standing potential that exists between the front and the back of the human eye. The resulting signal is called the electrooculogram. Primary applications are in ophthalmological diagnosis and in recording eye movements. To measure eye movement, pairs of electrodes are typically placed either above and below the eye or to the left and right of the eye.

[0008] One advantage of electrooculography-based eye-tracking over camera-based eye-tracking is the significantly lower power consumption due to the measurement of low-dimensional signals at a fairly low bitrate and significantly less processing the analyze those signals compared to camera-based eye-tracking.

[0009] However, there are some shortcomings of electrooculography-based eye-tracking compared to camerabased eye-tracking. For example, depending on implementation and / or placement of the electrodes, etc., electrooculography-based eye-tracking can have worse accuracy in vertical dimension than in horizontal dimension. For example, electrooculography-based eye-tracking might need calibration (which is persondependent). The calibration also depends on e.g., sweat and adaption of the eyes to certain lightning conditions. In turn, this means that for the best performance there is a need to re-calibrate at occasions. For example, electrooculography-based eye-tracking is generally less accurate in terms of gaze angle compared to camera-based eye-tracking. Furthermore, the accuracy might depend on how the electrodes are attached to, or touches, the skin, which might vary depending on the industrial design of the electrodes, etc.

[0010] Foveated rendering is a video rendering technique based on eye movements. Only the area, sometimes referred to as the foveal area, corresponding to the user’s gaze is rendered in full resolution, and the remaining part of the view is rendered with gradually decreasing quality. This does not affect the user experience, but the computational load is significantly reduced. While rendering techniques for general camera-based eye-tracking and electrooculography-based eye-tracking are known, current technologies for foveated rendering are mainly based on camera-based eye-tracking.

[0011] SUMMARY

[0012] An object of embodiments herein is to address the above short-comings with camera-based eye-tracking and electrooculography-based eye-tracking, especially in the context of foveated rendering.

[0013] A particular object is to provide energy-efficient foveated rendering in the context of eye-tracking.

[0014] According to a first aspect there is presented a method for foveated rendering of an XR environment. The method is performed by a controller. The controller is configured to select between using a first eyetracker and a second eye-tracker for eye-tracking of a user in the XR environment. The method comprises obtaining, using the first eye-tracker, a gaze direction of the user towards a first region in the XR environment. The method comprises selecting, based on a predicted gaze behavior of the user and an energy consumption cost for switching from using the first eye-tracker to the second eye-tracker for continued eye-tracking of the user, which of the first eye-tracker and the second eye-tracker to use. The gaze behavior indicates statistics of how long the user is expected to maintain the gaze direction. The method comprises performing foveated rendering of the XR environment in accordance with eye-tracking of the user as performed using the selected eye-tracker.

[0015] According to a second aspect there is presented a controller for foveated rendering of an XR environment. The controller is configured to select between using a first eye-tracker and a second eye-tracker for eyetracking of a user in the XR environment. The controller comprises processing circuitry. The processing circuitry is configured to cause the controller to obtain, using the first eye-tracker, a gaze direction of the user towards a first region in the XR environment. The processing circuitry is configured to cause the controller to select, based on a predicted gaze behavior of the user and an energy consumption cost for switching from using the first eye-tracker to the second eye-tracker for continued eye-tracking of the user, which of the first eye-tracker and the second eye-tracker to use. The gaze behavior indicates statistics of how long the user is expected to maintain the gaze direction. The processing circuitry is configured to cause the controller to perform foveated rendering of the XR environment in accordance with eyetracking of the user as performed using the selected eye-tracker. According to a third aspect there is presented a computer program for foveated rendering of an XR environment. The computer program comprises computer code which, when run on processing circuitry of a controller, causes the controller to perform actions. One action comprises the controller to obtain, using the first eye-tracker, a gaze direction of the user towards a first region in the XR environment. One action comprises the controller to select, based on a predicted gaze behavior of the user and an energy consumption cost for switching from using the first eye-tracker to the second eye-tracker for continued eye-tracking of the user, which of the first eye-tracker and the second eye-tracker to use. The gaze behavior indicates statistics of how long the user is expected to maintain the gaze direction. One action comprises the controller to perform foveated rendering of the XR environment in accordance with eyetracking of the user as performed using the selected eye-tracker.

[0016] According to a fourth aspect there is presented a computer program product comprising a computer program according to the third aspect and a computer readable storage medium on which the computer program is stored. The computer readable storage medium could be a non-transitory computer readable storage medium.

[0017] Advantageously, these aspects provide eye-tracking that does not suffer from the above-identified issues.

[0018] Advantageously, these aspects enable energy-efficient foveated rendering in the context of eye-tracking.

[0019] Advantageously, these aspects enable efficient foveated rendering using electrooculography-based eyetracking as well as camera-based eye-tracking.

[0020] Advantageously, these aspects enable the foveated rendering to be adapted for an eye-tracker that requires only low power consumption.

[0021] Advantageously, these aspects enable the time that electrooculography-based eye-tracking is used for foveated rendering to be maximized.

[0022] Advantageously, these aspects enable a trade-off to be made with respect to the lower energy consumption of using electrooculography-based eye-tracking and the higher accuracy of using camerabased eye-tracking during the foveated rendering.

[0023] Other objectives, features and advantages of the enclosed embodiments will be apparent from the following detailed disclosure, from the attached dependent claims as well as from the drawings.

[0024] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, module, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, module, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

[0025] BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The inventive concept is now described, by way of example, with reference to the accompanying drawings, in which:

[0027] Fig. 1 is a schematic diagram illustrating user devices according to embodiments;

[0028] Fig. 2 is a block diagram of a system with a controller according to an embodiment;

[0029] Figs. 3, 6, 7, and 8 are flowcharts of methods according to embodiments;

[0030] Fig. 4 schematically illustrates gaze areas in a user interface according to an embodiment;

[0031] Fig. 5 schematically illustrates eye-tracking resolution maps according to embodiments;

[0032] Fig. 9 is a schematic diagram showing structural units of a controller according to an embodiment;

[0033] Fig. 10 is a schematic diagram showing functional modules of a controller according to an embodiment; and

[0034] Fig. 11 shows one example of a computer program product comprising computer readable storage medium according to an embodiment.

[0035] DETAILED DESCRIPTION

[0036] The inventive concept will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the inventive concept are shown. This inventive concept may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. Like numbers refer to like elements throughout the description. Any step or feature illustrated by dashed lines should be regarded as optional.

[0037] As noted above, an object of embodiments herein is to address the above short-comings with camerabased eye-tracking and electrooculography-based eye-tracking, especially in the context of foveated rendering.

[0038] The embodiments disclosed herein in particular relate to techniques for foveated rendering of an XR environment. In order to obtain such techniques there is provided a controller, a method performed by the controller, a computer program product comprising code, for example in the form of a computer program, that when run on a controller, causes the controller to perform the method. Fig. 1 is a schematic diagram illustrating user devices 110, represented by a pair of smart glasses, or XR glasses, for eye-tracking according to embodiments. The eye-tracking might be performed in the context of gaming, navigation, or tracking user behaviour in an XR environment. Further, the eye-tracking might be performed as part of a meeting, a learning event, etc. In these examples, a first eye-tracking mode is represented by electrooculography-based eye-tracking and a second eye-tracking mode is represented by camera-based eye-tracking. The electrooculography-based eye-tracking is based on signals obtained from electrodes 120. In more detail, Fig. 1 schematically illustrates different types of implementations of electrooculography-based eye-tracking, and particularly the placements of the electrodes 120. In Fig. 1(a) six electrodes 120 (denoted HR (horizontal right), VL (vertical lower), HL (horizontal left), VU (vertical upper), REF (reference), GND (ground)) are fixed to the skin of a user 150 wearing the user device 110. In Fig. 1(b) the electrodes (as represented by single electrode 120) are part of the user device 110 itself. In Fig. 1(c) the electrodes (as represented by single electrode 120) are placed on an in-ear headset 140. In Fig. 1(c) the electrodes might be integrated with audio and microphone circuitry in the in-ear headset 140. The camera-based eye-tracking is based on signals obtained from a camera 130.

[0039] Reference is next made to Fig. 2 which shows a block diagram of a controller 200 for eye-tracking using electrooculography-based eye-tracking and camera-based eye-tracking. The controller 200 therefore comprises an electrooculography-based eye-tracker (EET) 210 and a camera-based eye-tracker (CET) 220. The electrooculography-based eye-tracker 210 is configured to receive input from one or more electrodes 260. The camera-based eye-tracker 220 is configured to receive input from one or more cameras 250. Further, the camera-based eye-tracker 220 might be operatively connected to one or more light sources, such as infrared light-emitting diodes (LEDs) 270 for illuminating the face of the user so that the one or more cameras 250 can capture images of the user that can be accurately analyzed. An image-signal-processor (ISP) 230 is configured to process image data as captured by the one or more cameras 250. In this respect the image-signal -processor 230 might be configured to perform pixel-based adjustments from raw-sensor data to calibrated pixel-data used the camera-based eye-tracker 220. A central processing unit (CPU) 240 is configured to select which of the eye-trackers to be used.

[0040] In some aspects, the camera-based eye-tracker 220 (as well as the electrooculography-based eye-tracker 210) can be configured to operate in multiple power modes. Therefore, in some embodiments, each of the first eye-tracker and the second eye-tracker is associated with a respective set of at least two power modes, where each of the power modes represents a respective energy consumption for performing the eye-tracking of the user.

[0041] In some examples, one of the power modes is a deep-sleep mode. In this power mode there is no or little power available to the eye-tracker. In some examples, one of the power modes is an inactive mode. In this power mode, the eye-tracker has access to power but is not operational. In some examples, one of the power modes is a fully active mode. In this power mode, the eye-tracker is fully operational to perform eye-tracking. Hence, in some examples, one of the at least two power modes is an active mode and another of the at least two power modes is a sleep mode, and the eye-tracker is in the active mode only when selected for performing the continued eye-tracking of the user. As the skilled person understands, these are just some examples of power modes, and the eye-trackers might have also further, or alternate, power modes. For example, there might be more than one sleep mode, where one of the sleep modes is a deep-sleep mode. The term (deep-)sleep mode will be used throughout to cover any sleep mode.

[0042] Further, since the energy consumption generally is higher for the camera-based eye-tracker 220 than the electrooculography-based eye-tracker 210, specific focus will next be on different power modes of the camera-based eye-tracker 220. However, the general aspects of the different power modes apply also for the electrooculography-based eye-tracker 210. In the (deep-)sleep mode, the one or more cameras 250, LEDs 270, the image-signal-processor 230, and the camera-based eye-tracker 220, are in deep sleep. In the active mode, the one or more cameras 250, LEDs 270, the image-signal-processor 230, and the camera-based eye-tracker 220, are active and as well as the interfaces to these entities, are active and deliver eye-tracking output to the central processing unit 240. In general terms, at least for the camerabased eye-tracker 220, the time needed for switching between the different modes is not negligible. Therefore, in the inactive mode, those modules of the one or more cameras 250, LEDs 270, the imagesignal -processor 230, and the camera-based eye-tracker 220, are ready to perform eye-tracking that requires a comparatively long start-up time to from (deep-)sleep mode to active mode are started up and are potentially active. The remaining modules can still be in deep sleep. The inactive mode can therefore be regarded as providing a compromise between the (deep-)sleep mode and the active mode, and thus a compromise between energy consumption and how fast camera-based eye-tracking can start up and deliver eye-tracking output.

[0043] Further aspects of eye-tracking application-specific (or context-specific in the more general case) control of three different power modes of an eye-tracker will be disclosed below with reference to Fig. 6.

[0044] In this respect, it is noted that there is an energy consumption for the eye tracking, and there is an energy consumption for the foveated rendering and data transfer related to the foveated rendering. In general terms, it is the sum of these energy consumptions that should be optimized.

[0045] Reference is next made to the flowchart in Fig. 3 illustrating embodiments of methods for foveated rendering of an XR environment. The methods are performed by the controller 200, 900, 1000. The controller 200, 900, 1000 is configured to select between using a first eye-tracker and a second eyetracker for eye-tracking of a user in the XR environment. The methods are advantageously provided as computer programs 1120. The methods are based on selecting between at least two available eye-trackers based on statistics of how long the user is expected to maintain the gaze direction and in relation to energy consumption of switching between the different eye-trackers.

[0046] S102: The controller 200, 900, 1000 obtain, using the first eye-tracker, a gaze direction of the user towards a first region in the XR environment. S 106: The controller 200, 900, 1000 selects, based on a predicted gaze behavior of the user and an energy consumption cost for switching from using the first eye-tracker to the second eye-tracker for continued eye-tracking of the user, which of the first eye-tracker and the second eye-tracker to use. The gaze behavior indicates statistics of how long the user is expected to maintain the gaze direction.

[0047] S108: The controller 200, 900, 1000 performs foveated rendering of the XR environment in accordance with eye-tracking of the user as performed using the selected eye-tracker.

[0048] In some aspects, this method aims at optimizing the energy consumption, taking into consideration different power modes of the different eye-trackers as well as the typical user behavior. Overall, this can reduce the overall energy consumption of the user device 110. Therefore, in some embodiments, the energy consumption cost favors selection of the eye-tracker yielding lowest energy consumption for eyetracking of the user for the predicted gaze behavior.

[0049] Embodiments relating to further details of foveated rendering of an XR environment as performed by the controller 200, 900, 1000 will now be disclosed with continued reference to Fig. 3.

[0050] As disclosed above, the controller 200 in Fig. 2 comprises an electrooculography-based eye-tracker 210 and a camera-based eye-tracker 220. Therefore, in some embodiments, one of the first eye-tracker and the second eye-tracker is an electrooculography-based eye-tracker 210 and the other of the first eye-tracker and the second eye-tracker is a camera-based eye-tracker 220.

[0051] In some aspects, statistics on how the user is gazing are collected and made available to the controller 200, 900, 1000. The controller might then use such statistics when predicting the gaze behavior of the user. That is, in some embodiments, the gaze behavior is predicted based on collected statistics of gaze directions of the user. The statistics can be collected and trained offline for many users and eye-tracking applications. However, such statistics can also be collected per user and per eye-tracking application to personalize and therefore individually optimize the selection of eye-tracker. Hence, depending on personalized profiles, different thresholds as utilized when determining whether to switch between the different power modes might vary from one user to the next and / or from one eye-tracking application to the next. Reference is here made to Fig. 4 and in Table 1 illustrating an example of such statistics for a specific user and eye-tracking application. Hence, in some embodiments, the eye-tracking of the user is performed for an eye-tracking application selected from a set of available eye-tracking applications, and there are separate statistics of gaze directions for each of the eye-tracking applications. Fig. 4 schematically illustrates different gaze areas 410, 420a: 420d, 430a:430d in a user interface 400. According to this example, the statistics were quantified into nine different gaze areas 410, 420a: 420d, 430a:430d. As will be further disclosed below, the statistics of gaze directions might further be dependent on the layout of visual objects placed on the user interface of each of the eye-tracking applications. Table 1 shows the fraction of time and the average time before switch for each of the gaze areas 410, 420a: 420d, 430a:430d. Thus, in some embodiments, the statistics of gaze directions is defined by a set of different gaze areas and an average time spent by the user in each of the different gaze areas. As can be seen in Table 1, the fraction of time spent in each gaze area (where the fraction of time spent in gaze areas 420a: 420d is summed into one value and the fraction of time spent in gaze areas 430a:430d also is summed into one value) is highest in the center-most gaze area 410 and then decreases towards the edges of the user interface 400. Likewise, the average time spent in the center-most gaze area 410 is highest and then decreases towards the edges of the user interface 400. It can for example be seen that the user is expected to on average spend 80% of his / her time gazing in a direction covered by gaze area 410, and that the user on average is expected to spend 5 seconds gazing in this gaze area 410 before turning his / her gaze to a direction covered by any other gaze areas 420a:420d, 430a:430d, and so on.

[0052] Table 1 : Example of statistics for user and eye-tracking application

[0053] From Fig. 4 and Table 1 it thus follows that the user is expected to not gaze very long in extreme directions (e.g., in the comers represented by gaze areas 430a:430d). Rather, it is expected that user prefers to move his / her head to neutralize the gaze position (e.g., such that the gaze direction is in the boresight direction). Also, the change in gaze direction is generally a relatively fast movement and it can give indication of the coming change in head rotation well before the rotation starts. In other words, it is likely that the user will start gazing in a certain direction towards an object before the user turns his / her head towards the object so as to neutralize the gaze position. In this respect, the controller 200, 900, 1000 might therefore be configured to predict an upcoming gaze direction. In particular, in some embodiments, the controller 200, 900, 1000 is configured to perform (optional) step SI 04.

[0054] S 104: The controller 200, 900, 1000 predicts, based on the gaze direction, and the gaze behavior of the user, at least one second region in the XR environment towards which the user is expected to gaze upon having gazed towards the first region. Which of the first eye-tracker and the second eye-tracker to use is then selected also based on the gaze direction towards the predicted at least one second region.

[0055] In general terms, the gaze accuracy for using the eye-tracker to perform the eye-tracking can be represented by an eye-tracking resolution map 500a, 500b, 500c. That is, in some embodiments, the gaze accuracy varies over the XR environment 500 according to an eye-tracking resolution map 500a, 500b, 500c. The foveated rendering can then be performed with a higher resolution inside a region in the XR environment towards which the user is gazing than outside this region, where the size of the region decreases with increased gaze accuracy. That is, in the eye-tracking resolution map 500a, 500b, 500c, certain areas represent a higher gaze accuracy (i.e., better resolution) for performing the eye-tracking. Examples of three such eye-tracking resolution maps 500a, 500b, 500c are provided in Fig. 5. The eyetracking resolution maps 500a, 500b, 500c can thus be used to provide information of where the field-of- view of the gaze detection is of good enough accuracy and, consequently, which objects can be distinguished with good-enough probability for the used eye-tracker. The eye-tracking resolution map 500a is an example of a highly irregular eye-tracking resolution map, where the gaze accuracy varies over the field-of-view, whereas the eye-tracking resolution maps 500b, 500c are examples of regular eyetracking resolution maps, where the gaze accuracy is more or less constant over the whole field-of-view. This is in Fig. 5, schematically illustrated by the eye-tracking resolution maps 500a, 500b, 500c comprising regions representing low, medium and / or high gaze accuracy / re solution of the used eyetracker. For illustrative purposes, diameters dl, d2, d3, d4, d5, d6 of different ellipses 510a, 510b, 510c represent the smallest distance required for the used eye-tracker to successfully distinguish between two objects in the XR environment. That is, the smaller the ellipses 510a, 510b, 510c is, the higher the gaze accuracy / re solution of the used eye-tracker is. That the eye-tracking resolution map 500a is a highly irregular eye-tracking resolution map is represented by the ellipses 510a being of different sizes, i.e., the diameters dl, d2, d3, d4 all are different from each other. In contrast, in each of the eye-tracking resolution maps 500b, 500c, the diameters d5, d6 are the same, irrespectively of where the ellipses 510b, 510c would be placed within the eye-tracking resolution maps 500b, 500c. As an example, when electrooculography-based eye-based eye tracking is used for foveated rendering, the foveal area (meaning the area with the highest rendered resolution) needs typically be larger than for camera-based eyetracking, and the lower the accuracy of the specific gaze direction the larger the area must be. In general terms, for a larger foveal area, the power consumption of the rendering and related processing and data movements is higher than for a small foveal area, but the power consumption required to for the eyetracking is significantly lower with electrooculography-based eye-tracking compared to camera-based eye-tracking. Therefore, in some embodiments, the energy consumption cost favors selection of the electrooculography-based eye -tracker 210 over selection of the camera-based eye-tracker 220.

[0056] The gaze accuracy of the electrooculography-based eye-tracker might be recalibrated as a background process while the camera-based eye-tracker is active. This will naturally strive for optimized energy consumption; when the gaze accuracy of the electrooculography-based eye-tracker has degraded, the camera-based eye-tracker is activated. This could, in turn, activate a recalibration of the electrooculography-based eye-tracker and thus yield improved gaze accuracy of the electrooculographybased eye-tracker. This could, in turn, lead to the electrooculography-based eye-tracker being activated whilst the camera-based eye-tracker is put back in its (deep-)sleep mode.

[0057] Further aspects, embodiments, and examples of foveated rendering of an XR environment as performed by the controller 200, 900, 1000 will now be disclosed with reference to the flowcharts of Figs. 6, 7, and Fig. 6 is a flowchart of an eye-tracking application-specific (or context-specific in the more general case) control of three different power modes of an eye-tracker. Fig. 6 represent an embodiment where the camera-based eye-tracker 220 is selected for continued eye-tracking of the user when, according to the predicted gaze behavior, the eye-tracking is to be performed longer than a time duration threshold value in an area where a gaze accuracy of the electrooculography-based eye-tracker 210 is below an accuracy threshold value, and the electrooculography-based eye-tracker 210 is otherwise selected. Applicationspecific statistics of gaze directions of the user is collected and / or updated (step S202). In this way, the control of the power mode can be re-assessed when there are major changes of behavior of the user, and / or there is a switch between eye-tracking applications, and / or when content associated with fundamentally different user behavior is displayed to the user on a graphical user interface. Based on the statistics and input from an eye-tracking resolution map for the electrooculography-based eye-tracker (as acquired in step S204) it is in step S206 checked if the fraction of time spent in a region where the gaze accuracy of the electrooculography-based eye-tracker 210 is below a first accuracy threshold value (“Threshold 1“). If yes, step S208 is entered. Else, step S210 is entered. In step S208 the camera-based eye-tracker remains in the (deep-)sleep mode. In step S210 it is checked if the fraction of time spent in a region where the gaze accuracy of the electrooculography-based eye-tracker 210 is below a second accuracy threshold value (“Threshold 2“). If yes, step S212 is entered. Else, step S214 is entered. In step S212 the camera-based eye-tracker is set to be in the inactive mode. In step S214 the camera-based eyetracker is set to be in the active mode. In general terms, the energy consumption of the different power modes of the camera-based eye-tracker are The switching time from the (deep-)sleep mode to the active mode is Tdeep-sieep, and from inactive mode to the active mode is Tractive- Furthermore, (AEOG) denotes the energy consumption overhead of rendering a larger foveal area for a certain electrooculography-based eye-tracking gaze accuracy AEOG than if camera-based eyetracking would be used. The lower the gaze accuracy is, the larger the foveal area will have to be, and hence the (AEOG) is increased with a lower AEOG.

[0058] It is noted that if the camera-based eye-tracker could switch between the (deep-)sleep mode and the active mode in zero time (i.e., where 0), a decision criterion to use electrooculography-based eye-tracking instead of camera-based eye-tracking based on energy consumption would be if (AEOG) for the current AEOG as determined by the current gaze in relation to the gaze accuracy map of the electrooculography-based eye-tracker. However, it is in the present disclosure assumed that this is not the case.

[0059] For a system where is long and if the user spends a very low fraction of the time gazing where the gaze accuracy of the electrooculography-based eye-tracker is low, then the energy overhead of switching to instead using the camera-based eye-tracker before it can be used (approximated as Pactive • although the energy consumption is not Pactive during the complete switching time) plus the overhead of later switching it off does not pay off, especially if the average time the user spends gazing in such areas is very low. In one example, as in the example of Fig. 6, only the statistical fraction of time spent in different gaze accuracy regions together with the average time before switching is taken into consideration, and the threshold levels are determined for when it makes sense to have the camera-based eye-tracker in (deep-)sleep mode, even if that means that the system sometimes uses electrooculographybased eye-tracking with large foveal areas, since the overhead of switching back and forth between using different eye-trackers is larger than the small duration of time the energy consumption is slightly larger with the electrooculography-based eye-tracker. This is achieved by means of the comparison to Threshold 1 in Fig. 6. In a similar way, it can be determined when it is more favorable to have the camera-based eye-tracker in inactive mode and then switch to active mode when it is more favorable, or when the statistics is such that it is better to have the camera-based eye-tracker in active mode. This is achieved by means of the comparison to Threshold 2 in Fig. 6.

[0060] With this principle, and for a certain statistics and gaze accuracy maps, it might be better for the camerabased eye-tracker to remain in deep-sleep than toggling between the (deep-)sleep mode and the other power modes. However, in other situations, it will be more advantageous to have the electrooculographybased eye-tracker toggling between active modes and inactive mode depending on the current gaze direction and the gaze accuracy map, but not to put the camera-based eye-tracker in the (deep-)sleep mode due to the overhead and latency to switch the camera-based eye-tracker back to active mode from the (deep-)sleep mode.

[0061] Since the statistics can be made application-specific (since different eye-tracking applications might have very different gaze behavior), and that the gaze accuracy map might change over time (for example, it might degrade due to sweat, and might improve due to re-calibration). This means that which eye-tracker that is used might change at a change of eye-tracking application or at a significantly changed accuracy gaze map. One example might be that a recalibration of the electrooculography-based eye-tracker with an improved gaze accuracy map might imply that it is a better tradeoff to put the camera-based eye-tracker in (deep-)sleep mode. Further, if an eye-tracking application is started where the user frequently gazes at areas with typically low accuracy of the electrooculography-based eye-tracker, this might imply that the camera-based eye-tracker is set to adaptively move between being in the inactive mode and the active since that yields a better tradeoff.

[0062] Fig. 7 is a flowchart of a method for determining whether to use electrooculography-based eye-tracking or camera-based eye-tracking. Fig. 7 represents an example where the gaze accuracy for using the electrooculography-based eye -tracker 210 to perform the eye-tracking might vary over the XR environment according to the eye-tracking resolution map 500a, 500b, 500c in Fig. 5. The foveated rendering can then be performed with a higher resolution inside a region in the XR environment towards which the user is gazing than outside the region, and the size of the region decreases with increased gaze accuracy. Application-specific statistics of gaze directions of the user is collected and / or updated (step S302). In this way, the control of the power mode can be re-assessed when there are major changes of behavior of the user, and / or there is a switch between eye-tracking applications, and / or when content associated with fundamentally different user behavior is displayed to the user on a graphical user interface. Input is received from an eye-tracking resolution map for the electrooculography-based eyetracker (as acquired in step S302). The gaze direction of the user is determined (step S306). It is in step S308 checked if the gaze direction is within a region where the gaze accuracy of the electrooculographybased eye-tracker is higher than some gaze accuracy threshold. If yes, step S312 is entered. Else, step S310 is entered. In step S312 the electrooculography-based eye-tracker is used with a comparatively small foveal area. In step S310 it is checked if the user is expected to keep his / her gaze direction in the determined gaze direction shorter than some duration threshold. If yes, step S314 is entered. Else, step S316 is entered. In step S314 the electrooculography-based eye-tracker is used with a comparatively large foveal area (at least larger than in step S312). In step S316 the camera-based eye-tracker is selected to be used for the eye-tracking (instead of the electrooculography-based eye-tracker).

[0063] In step S308, the gaze accuracy threshold can be defined by those gaze accuracy levels of the electrooculography-based eye-tracker corresponding to the energy overhead of the camera-based eyetracker, determined as PvoG-overhead = (Pactive - Psieep ), being larger than PEOG-overhead (AEOG). Furthermore, for lower gaze accuracy levels, where PEOG-overhead (AEOG) > PvoG-overhead, there is an energy consumption overhead of switching the camera-based eye-tracker from inactive mode to active mode and back to inactive mode. This energy consumption overhead is ESWitCh = Tsieep • (Pactive - Psieep). If the average duration, d, that the user spends in a region where the gaze accuracy of the electrooculography-based eyetracker is lower than the gaze accuracy threshold before returning to a region where the gaze accuracy of the electrooculography-based eye-tracker is higher than the gaze accuracy threshold is so short that ESWitch > d ■ (PEOG-overhead(AEOG) - PvoG-overhead), it is statistically more favourable energy-wise to continue using the electrooculography-based eye-tracker with a larger foveal area than to use camera-based eye-tracker. This is achieved by means of the comparison to the duration threshold in Fig. 7.

[0064] The flowchart of Fig. 7 represents an adaptive scenario where the power modes of the camera-based eyetracker are handled as in the flowchart of Fig. 6. This implies that for system with a long Tdeep-sieep with a corresponding high energy overhead for switching between (deep-)sleep mode and active mode for the camera-based eye-tracker, the camera-based eye-tracker might remain in the (deep-)sleep mode as long as the gaze accuracy of the electrooculography-based eye-tracker has not degraded. However, if the user behavior of the eye-tracking application is suddenly changed, the controller might determine to put the camera-based eye-tracker in inactive mode for a more rapid switching to the camera-based eye-tracker, if needed.

[0065] Fig. 8 is a flowchart of active switching between (deep-)sleep mode and inactive / active mode for the camera-based eye-tracker. Fig. 8 represents an example where, based on the predicted gaze behavior and the energy consumption cost, the camera-based eye-tracker 220 is selected for continued eye-tracking of the user when, according to the predicted gaze behavior, the eye-tracking is to be performed longer than a time duration threshold value in an area where a gaze accuracy of the electrooculography-based eye- tracker 210 is below an accuracy threshold value. For example, for system where the Tdeep-sieep is very short, and the corresponding energy overhead for switching between different power modes is fairly low, the controller might perform more actively switching between the (deep-)sleep mode and the inactive / active modes. This dynamic behavior is illustrated in Fig. 8.

[0066] Application-specific statistics of gaze directions of the user is collected and / or updated (step S402). In this way, the control of the power mode can be re-assessed when there are major changes of behavior of the user, and / or there is a switch between eye-tracking applications, and / or when content associated with fundamentally different user behavior is displayed to the user on a graphical user interface. Input is received from an eye-tracking resolution map for the electrooculography-based eye-tracker (as acquired in step S404). The gaze direction of the user is determined (step S406). It is in step S408 checked if the gaze direction is within a region where the gaze accuracy of the electrooculography-based eye-tracker is higher than some gaze accuracy threshold. If yes, step S412 is entered. Else, step S410 is entered. In step S412 the electrooculography-based eye-tracker is used with a comparatively small foveal area. The camera-based eye-tracker is in inactive mode. In step S410 it is checked if it would be advantageous to continue to use electrooculography-based eye-tracking (with details as will be specified below). If yes, step S414 is entered. Else, step S416 is entered. In step S414 the electrooculography-based eye-tracker is used with a comparatively large foveal area (at least larger than in step 440). In step S416 the camerabased eye-tracker is selected to be used for the eye-tracking (instead of the electrooculography-based eyetracker). In step S418 it is checked if it would be advantageous to set the camera-based eye-tracker in (deep-)sleep mode (with details as will be specified below). If yes, step S420 is entered. Else, step S422 is entered. In step S420 the camera-based eye-tracker is kept in the inactive mode. In step S422 the camerabased eye-tracker is put in (deep-)sleep mode.

[0067] In this example, if the gaze direction is in region where the gaze accuracy of the electrooculographybased eye-tracker is lower than the gaze accuracy threshold, the decision whether to use electrooculography-based eye-tracking or camera-based eye-tracking depends on the expected time duration until the gaze direction returns to a region where the gaze accuracy of the electrooculographybased eye-tracker again is higher than the gaze accuracy threshold. This expected time duration in turn depends (as in step S410) on whether the camera-based eye-tracker is in the (deep-)sleep mode or in the inactive mode. This expected time duration, referred to as Dyn-threshold, has two levels; a shorter duration when the camera-based eye-tracker is in the inactive mode (due to a lower energy overhead to activate it) and a longer duration when the camera-based eye-tracker is in the (deep-)sleep mode. For example, if the gaze direction is in one of the gaze areas 430a: 43 Od, it might be a longer duration until the gaze direction returns to the gaze area 410 than if the gaze direction is in one of the gaze areas 420a:420d, and it might be better to turn on the camera-based eye-tracker even if the camera-based eye-tracker is in the (deep-)sleep mode. However, if the gaze is in one of the gaze areas 420a:420d, the gaze direction might return to the gaze area 410 faster and the camera-based eye-tracker is set to the active mode only if the camera-based eye-tracker currently is in the inactive mode (and not in the (deep-)sleep mode). Step S410 can therefore be regarded as the criterion for determining whether the camera-based eye-tracker is to be set in the active mode or not. The decision to put the camera-based eye-tracker in the (deep-)sleep mode is taken when the gaze direction is in a region where the gaze accuracy of the electrooculographybased eye-tracker is higher than the gaze accuracy threshold (e.g., in the gaze area 410), and it is expected that it will take longer than a certain threshold (exemplified by threshold2) before the gaze direction will be in a region where the gaze accuracy of the electrooculography-based eye-tracker is lower than the gaze accuracy threshold, and hence it would be advantageous to continue using the electrooculography-based eye-tracker (as in step S418).

[0068] The examples of Fig. 6, Fig. 7, and Fig. 8 illustrate that the switching between different eye-trackers can be dynamically controlled based on different parameters, such as the predicted gaze behavior of the user and the energy consumption cost for switching from using one eye-tracker to using another eye-tracker, where the decision can be taken as the user changes gaze direction (as in the example in Fig. 7), or based on more static approaches, for example when there is a (deep-)sleep mode with long time between sleep mode and active mode (as in the example in Fig. 6).

[0069] In many systems, or eye-tracking applications, the user is faced with content that is not anchored in the XR environment, such as toolbars, widgets, user menus, clocks, and maps (such as in gaming applications). For a given eye-tracking application, such anchored objects are always rendered in the same position relative to the field of vision. However, the type of anchored objects, as well as the positions where the anchored object are rendered, might change from one eye-tracking application to the next. As such, the user behavior may vary, as the user is forced to gaze in specific directions to interact with these objects, depending on the eye-tracking application. Consequently, such objects can be treated differently than other objects displayed in the XR environment. In some examples, in case such objects are present, information that specifies positions where the anchored objects are to be rendered is fetched. Since these anchored objects are likely to be gazed towards more frequently, user statistics can be collected to determine how these anchored objects are to be rendered. Such statistics can thus be used for pre-rendering anchored objects and thereby reducing latency for when an anchored object is to be rendered using foveated rendering. If, according to the statistics, it is likely that the user is to frequently gaze towards the anchored objects, these anchored objects may always be rendered in full (or almost) resolution, irrespectively of which eye-tracker is used, thereby preventing unnecessary switches to using the camera-based eye -tracker.

[0070] Fig. 9 schematically illustrates, in terms of a number of structural units, the components of a controller 900 according to an embodiment. Processing circuitry 910 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 1110 (as in Fig. 11), e.g. in the form of a storage medium 930. The processing circuitry 910 may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA). Particularly, the processing circuitry 910 is configured to cause the controller 900 to perform a set of operations, or steps, as disclosed above. For example, the storage medium 930 may store the set of operations, and the processing circuitry 910 may be configured to retrieve the set of operations from the storage medium 930 to cause the controller 900 to perform the set of operations. The set of operations may be provided as a set of executable instructions.

[0071] Thus the processing circuitry 910 is thereby arranged to execute methods as herein disclosed. The storage medium 930 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory. The controller 900 may further comprise a communications (comm.) interface 920 at least configured for communications with other entities, functions, nodes, and devices (such as the user device 110, the electrodes 120, 260, the camera 130, 250, the headset 140, and the LEDs 270) as needed for the controller 900 to be able to perform the foveated rendering of an XR environment in accordance with the herein disclosed embodiments. As such the communications interface 920 may comprise one or more transmitters and receivers, comprising analogue and digital components. The processing circuitry 910 controls the general operation of the controller 900 e.g. by sending data and control signals to the communications interface 920 and the storage medium 930, by receiving data and reports from the communications interface 920, and by retrieving data and instructions from the storage medium 930. Other components, as well as the related functionality, of the controller 900 are omitted in order not to obscure the concepts presented herein.

[0072] Fig. 10 schematically illustrates, in terms of a number of functional modules, the components of a controller 1000 according to an embodiment. The controller 1000 of Fig. 10 comprises a number of functional modules; an obtain module 1010 configured to perform step S102, a select module 1030 configured to perform step S106, and a render module 1040 configured to perform step S108. The controller 1000 of Fig. 10 may further comprise a number of optional functional modules, as a predict module 1020 configured to perform step S104. In general terms, each functional module 1010: 1040 may in one embodiment be implemented only in hardware and in another embodiment with the help of software, i.e., the latter embodiment having computer program instructions stored on the storage medium 930 which when run on the processing circuitry makes the controller 200, 900, 1000 perform the corresponding steps mentioned above in conjunction with Fig 9. It should also be mentioned that even though the modules correspond to parts of a computer program, they do not need to be separate modules therein, but the way in which they are implemented in software is dependent on the programming language used. Preferably, one or more or all functional modules 1010: 1040may be implemented by the processing circuitry 910, possibly in cooperation with the communications interface 920 and / or the storage medium 930. The processing circuitry 910 may thus be configured to from the storage medium 930 fetch instructions as provided by a functional module 1010: 1040and to execute these instructions, thereby performing any steps as disclosed herein. The controller 200, 900, 1000 may be provided as a standalone device or as a part of at least one further device. For example, the controller 200, 900, 1000 may be provided in the user device 110.

[0073] Fig. 11 shows one example of a computer program product 1110 comprising computer readable storage medium 1130. On this computer readable storage medium 1130, a computer program 1120 can be stored, which computer program 1120 can cause the processing circuitry 910 and thereto operatively coupled entities and devices, such as the communications interface 920 and the storage medium 930, to execute methods according to embodiments described herein. The computer program 1120 and / or computer program product 1110 may thus provide means for performing any steps as herein disclosed.

[0074] In the example of Fig. 11, the computer program product 1110 is illustrated as an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. The computer program product 1110 could also be embodied as a memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM) and more particularly as a non-volatile storage medium of a device in an external memory such as a USB (Universal Serial Bus) memory or a Flash memory, such as a compact Flash memory. Thus, while the computer program 1120 is here schematically shown as a track on the depicted optical disk, the computer program 1120 can be stored in any way which is suitable for the computer program product 1110.

[0075] The inventive concept has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the inventive concept, as defined by the appended patent claims.

Claims

CLAIMS1. A controller (200, 900, 1000) for foveated rendering of an XR environment, wherein the controller (200, 900, 1000) is configured to select between using a first eye-tracker and a second eye-tracker for eye-tracking of a user in the XR environment, wherein the controller (200, 900, 1000) comprises processing circuitry (240, 910), and wherein the processing circuitry (240, 910) is configured to cause the controller (200, 900, 1000) to: obtain, using the first eye-tracker, a gaze direction of the user towards a first region in the XR environment select, based on a predicted gaze behavior of the user and an energy consumption cost for switching from using the first eye-tracker to the second eye-tracker for continued eye-tracking of the user, which of the first eye-tracker and the second eye-tracker to use, wherein the gaze behavior indicates statistics of how long the user is expected to maintain the gaze direction; and perform foveated rendering of the XR environment in accordance with eye-tracking of the user as performed using the selected eye-tracker.

2. The controller (200, 900, 1000) according to claim 1, wherein the processing circuitry (240, 910) further is configured to cause the controller (200, 900, 1000) to: predict, based on the gaze direction, and the gaze behavior of the user, at least one second region in the XR environment towards which the user is expected to gaze upon having gazed towards the first region, and wherein which of the first eye-tracker and the second eye-tracker to use is selected based on a gaze direction towards the predicted at least one second region.

3. The controller (200, 900, 1000) according to claim 2, wherein the gaze behavior is predicted based on collected statistics of gaze directions of the user.

4. The controller (200, 900, 1000) according to claim 3, wherein the statistics of gaze directions is defined by a set of different gaze areas and an average time spent by the user in each of the different gaze areas.

5. The controller (200, 900, 1000) according to claim 3 or 4, wherein the eye-tracking of the user is performed for an eye-tracking application selected from a set of available eye-tracking applications, and where there are separate statistics of gaze directions for each of the eye-tracking applications.

6. The controller (200, 900, 1000) according to claim 5, wherein the statistics of gaze directions further is dependent on a layout of visual objects placed on a graphical user interface of each of the eyetracking applications.

7. The controller (200, 900, 1000) according to claim 1, wherein each of the first eye-tracker and the second eye-tracker is associated with a respective set of at least two power modes, wherein each of the power modes represents a respective energy consumption for performing the eye-tracking of the user.

8. The controller (200, 900, 1000) according to claim 7, wherein one of the at least two power modes is an active mode and another of the at least two power modes is a sleep mode, and wherein the eyetracker is in the active mode only when selected for performing the continued eye-tracking of the user.

9. The controller (200, 900, 1000) according to claim 1, wherein the energy consumption cost favors selection of the eye-tracker yielding lowest energy consumption for eye-tracking of the user for the predicted gaze behavior.

10. The controller (200, 900, 1000) according to claim 1, wherein one of the first eye-tracker and the second eye-tracker is an electrooculography-based eye-tracker (210) and the other of the first eye-tracker and the second eye-tracker is a camera-based eye-tracker (220).

11. The controller (200, 900, 1000) according to claim 10, wherein the energy consumption cost favors selection of the electrooculography-based eye-tracker (210) over selection of the camera-based eyetracker (220).

12. The controller (200, 900, 1000) according to claim 10, wherein the camera-based eye-tracker (220) is selected for continued eye-tracking of the user when, according to the predicted gaze behavior, the eyetracking is to be performed longer than a time duration threshold value in an area where a gaze accuracy of the electrooculography-based eye-tracker (210) is below an accuracy threshold value, and wherein the electrooculography-based eye-tracker (210) otherwise is selected.

13. The controller (200, 900, 1000) according to claim 9, 10 or 11, wherein a gaze accuracy for using the electrooculography-based eye-tracker (210) to perform the eye-tracking varies over the XR environment according to an eye-tracking resolution map (500a, 500b, 500c), and wherein the foveated rendering is performed with a higher resolution inside a region in the XR environment towards which the user is gazing than outside the region, and wherein the size of the region decreases with increased gaze accuracy.

14. The controller (200, 900, 1000) according to claim 1, wherein the first eye-tracker is an electrooculography-based eye-tracker (210) and the second eye-tracker is a camera-based eye-tracker (220), and wherein, based on the predicted gaze behavior and the energy consumption cost, the camerabased eye-tracker (220) is selected for continued eye-tracking of the user when, according to the predicted gaze behavior, the eye-tracking is to be performed longer than a time duration threshold value in an area where a gaze accuracy of the electrooculography-based eye-tracker (210) is below an accuracy threshold value.

15. A method for foveated rendering of an XR environment, wherein the method is performed by a controller (200, 900, 1000), wherein the controller (200, 900, 1000) is configured to select between using a first eye-tracker and a second eye-tracker for eye-tracking of a user in the XR environment, and wherein the method comprises: obtaining (SI 02), using the first eye-tracker, a gaze direction of the user towards a first region in the XR environment selecting (S 106), based on a predicted gaze behavior of the user and an energy consumption cost for switching from using the first eye-tracker to the second eye-tracker for continued eye-tracking of the user, which of the first eye-tracker and the second eye-tracker to use, wherein the gaze behavior indicates statistics of how long the user is expected to maintain the gaze direction; and performing (S 108) foveated rendering of the XR environment in accordance with eye-tracking of the user as performed using the selected eye-tracker.

16. The method according to claim 15, wherein the method further comprises: predicting (SI 04), based on the gaze direction, and the gaze behavior of the user, at least one second region in the XR environment towards which the user is expected to gaze upon having gazed towards the first region, and wherein which of the first eye-tracker and the second eye-tracker to use is selected based on a gaze direction towards the predicted at least one second region.

17. The method according to claim 16, wherein the gaze behavior is predicted based on collected statistics of gaze directions of the user.

18. The method according to claim 17, wherein the statistics of gaze directions is defined by a set of different gaze areas and an average time spent by the user in each of the different gaze areas.

19. The method according to claim 17 or 18, wherein the eye-tracking of the user is performed for an eye-tracking application selected from a set of available eye-tracking applications, and where there are separate statistics of gaze directions for each of the eye-tracking applications.

20. The method according to claim 19, wherein the statistics of gaze directions further is dependent on a layout of visual objects placed on a graphical user interface of each of the eye-tracking applications.

21. The method according to claim 15, wherein each of the first eye-tracker and the second eye-tracker is associated with a respective set of at least two power modes, wherein each of the power modes represents a respective energy consumption for performing the eye-tracking of the user.

22. The method according to claim 21, wherein one of the at least two power modes is an active mode and another of the at least two power modes is a sleep mode, and wherein the eye-tracker is in the active mode only when selected for performing the continued eye-tracking of the user.

23. The method according to claim 15, wherein the energy consumption cost favors selection of the eye-tracker yielding lowest energy consumption for eye-tracking of the user for the predicted gaze behavior.

24. The method according to claim 15, wherein one of the first eye-tracker and the second eye-tracker is an electrooculography-based eye-tracker (210) and the other of the first eye-tracker and the second eyetracker is a camera-based eye-tracker (220).

25. The method according to claim 24, wherein the energy consumption cost favors selection of the electrooculography-based eye-tracker (210) over selection of the camera-based eye-tracker (220).

26. The method according to claim 24, wherein the camera-based eye-tracker (220) is selected for continued eye-tracking of the user when, according to the predicted gaze behavior, the eye-tracking is to be performed longer than a time duration threshold value in an area where a gaze accuracy of the electrooculography-based eye-tracker (210) is below an accuracy threshold value, and wherein the electrooculography-based eye-tracker (210) otherwise is selected.

27. The method according to claim 23, 24 or 25, wherein a gaze accuracy for using the electrooculography-based eye-tracker (210) to perform the eye-tracking varies over the XR environment according to an eye-tracking resolution map (500a, 500b, 500c), and wherein the foveated rendering is performed with a higher resolution inside a region in the XR environment towards which the user is gazing than outside the region, and wherein the size of the region decreases with increased gaze accuracy.

28. The method according to claim 15, wherein the first eye-tracker is an electrooculography-based eye-tracker (210) and the second eye-tracker is a camera-based eye-tracker (220), and wherein, based on the predicted gaze behavior and the energy consumption cost, the camera-based eye-tracker (220) is selected for continued eye-tracking of the user when, according to the predicted gaze behavior, the eyetracking is to be performed longer than a time duration threshold value in an area where a gaze accuracy of the electrooculography-based eye-tracker (210) is below an accuracy threshold value.

29. A computer program (1120) for foveated rendering of an XR environment, the computer program comprising computer code which, when run on processing circuitry (240, 910) of a controller (200, 900, 1000), wherein the controller (200, 900, 1000) is configured to select between using a first eye-tracker and a second eye-tracker for eye-tracking of a user in the XR environment, causes the controller (200, 900, 1000) to:obtain (S 102), using the first eye-tracker, a gaze direction of the user towards a first region in the XR environment select (S106), based on a predicted gaze behavior of the user and an energy consumption cost for switching from using the first eye-tracker to the second eye-tracker for continued eye-tracking of the user, which of the first eye-tracker and the second eye-tracker to use, wherein the gaze behavior indicates statistics of how long the user is expected to maintain the gaze direction; and perform (S 108) foveated rendering of the XR environment in accordance with eye-tracking of the user as performed using the selected eye-tracker.

30. A computer program product (1110) comprising a computer program (1120) according to claim 29, and a computer readable storage medium (1130) on which the computer program is stored.

Images