An extended reality device and a method for providing an improved extended reality interface

Abstract

An extended reality, XR, device arranged to predict a head movement based on eye movement and generate predicted content for the predicted head movement.

Description

[0001] AN EXTENDED REALITY DEVICE AND A METHOD FOR PROVIDING AN IMPROVED EXTENDED

[0002] REALITY INTERFACE

[0003] TECHNICAL FIELD

[0004] The present invention relates to an arrangement comprising computer software modules, an extended reality, XR, device and a method for providing an improved extended reality interface, and in particular to an arrangement comprising computer software modules, a device and a method for providing an improved extended reality interface utilizing gaze tracking as well as head movement sensors.

[0005] BACKGROUND

[0006] Augmented reality (AR) is a technology that overlays digital information or computergenerated graphics onto the real world. It involves adding virtual elements to the user's actual environment, typically viewed through a camera (Video see-through VST) or a transparent screen (Optical see-through OST). AR can enhance the perception of reality by adding layers of digital information that can be informative, entertaining, or both, or neither.

[0007] Virtual reality (VR), on the other hand, creates a completely immersive, computergenerated environment that can be experienced through a headset or a full-body suit. In VR, users are completely isolated from the real world and transported to a simulated environment that can be interactive and often responsive to user input.

[0008] The key difference between AR and VR is that AR enhances the real world by overlaying digital elements onto it, whereas VR replaces the real world with a computergenerated environment. AR is therefore a more natural and intuitive experience that blends the virtual with the real world, while VR is a more immersive but completely synthetic experience.

[0009] Both AR and VR have a wide range of applications, including gaming, education, advertising, training, and simulation, among others. However, they offer distinct experiences that are best suited to different types of applications. Mixed reality (MR) is a term used to describe the merging of a real-world environment and a computer-generated one. Physical and virtual objects may co-exist in mixed reality environments and interact in real time.

[0010] Extended reality (XR) is a catch-all term to refer to augmented reality (AR), virtual reality (VR), and mixed reality (MR).

[0011] Head-worn devices, such as optical-see-through, OST, devices or other head-mounted displays, are used more and more in AR and XR systems, and are thus becoming more and more commonplace. Therefore, the public's demand is ever-increasing of both quality of image displayed and of wearability.

[0012] When using head-mounted displays in extended reality (XR), there are strict requirements relating to price and weight which in turn puts requirements on fast processing even when using limited computing resources.

[0013] SUMMARY

[0014] In addition to what is discussed above, all head-mounted displays (HMDs) suffer from latency. If the latency is too long the user experience is negatively impacted. A common scenario is when a user wearing an HMD is turning his or her head, and the new virtual environment must follow the head movement. To reduce problems with motion sickness, it is desirable to minimize the delay from physical movement to the movement of the image, also called Motion-to-Photon (M2P) latency. The delay is caused by multiple factors including but not limited to: sensor sampling, data filtering and image rendering.

[0015] There is thus a strong need for a solution to the problems regarding latency discussed in the above, that is made without increasing the weight or cost significantly.

[0016] The inventors have realized after insightful and inventive reasoning that a solution may be provided by monitoring shifting of a user's eye(s). By monitoring the user's eye movements, it can be predetermined where the user will look to next, and virtual content can thus be rendered (and / or transmitted to the head-mounted display) in advance and be ready as the user actually shifts the gaze to a new view by turning the user's head. According to one aspect there is provided an extended reality, XR, device comprising a controller, wherein the controller is configured to obtain eye-tracking information for a user's eye (E) through an eye-tracking sensor, the eye-tracking information comprising an indication of a line of view (LV) for the eye, obtain head-movement information for the user's head (H) through a head-movement sensor, the head-movement information comprising an indication of a center line for the head (CL), determine that the user's eye (E) is moving relative the user's head (H), by detecting an increase in a viewing angle (A), wherein the viewing angle (A) is an angle between a line of view (LV) for the eye and a center line for the head (CL), and determining that the increase in viewing angle is exceeding a first eye movement angle threshold, and, in response thereto generate a predicted content, wherein the predicted content represents content outside a current view being presented through the image presenting arrangement in a direction of the eye movement, determine a change in movement for the user's head (H), and in response thereto cause the predicted content to be presented.

[0017] Other embodiments are discussed in below and are also as per the appended claims.

[0018] According to another aspect there is provided a method for use in an extended reality, XR, device as herein, wherein said method comprises obtaining eye-tracking information for a user's eye (E) through an eye-tracking sensor, the eye-tracking information comprising an indication of a line of view (LV) for the eye, obtaining head-movement information for the user's head (H) through a head-movement sensor, the head-movement information comprising an indication of a center line for the head (CL), determining that the user's eye (E) is moving relative the user's head (H), by detecting an increase in a viewing angle (A), wherein the viewing angle (A) is an angle between a line of view (LV) for the eye and a center line for the head (CL), and that determining that the increase in viewing angle is exceeding a first eye movement angle threshold, and, in response thereto generating predicted content, wherein the predicted content represents content outside a current view being presented through the image presenting arrangement in a direction of the eye movement, determining a change in movement for the user's head (H), and in response thereto causing the predicted content to be presented. According to another aspect there is provided a computer-readable medium carrying computer instructions that when loaded into and executed by a controller of an extended reality, XR, device enables the XR device to execute a method according to herein.

[0019] According to another aspect there is provided an extended reality, XR, device as herein comprising a software code module arrangement, wherein the software code module arrangement comprises a software component for obtaining eye-tracking information for a user's eye (E) through an eye-tracking sensor, the eye-tracking information comprising an indication of a line of view (LV) for the eye, a software component for obtaining headmovement information for the user's head (H) through a head-movement sensor, the headmovement information comprising an indication of a center line for the head (CL), a software component for determining that the user's eye (E) is moving relative the user's head, by detecting an increase in a viewing angle, wherein the viewing angle is an angle between a line of view for the eye and a center line for the head, and that determining that the increase in viewing angle is exceeding a first eye movement angle threshold, a software component for generating predicted content in response to determining that the user's eye is moving, wherein the predicted content represents content outside a current view being presented through the image presenting arrangement (106) in a direction of the eye movement, a software component for determining a change in movement for the user's head, and a software component for causing the predicted content to be presented in response to determining that the user's head is moving.

[0020] For the context of the teachings herein a software code module may be replaced or supplemented by a software module. For the context of the teachings herein a software code module may alternatively be replaced or supplemented by a circuit for performing a corresponding function.

[0021] Some advantages and benefits of the teachings herein are that it reduces latency which is important to avoid virtual objects or scenes moving unaligned with head movements which risk causing nausea / motion sickness. Furthermore, this alleviates one of the key traditional drawbacks of virtual reality or augmented reality experiences - that the virtual objects are not well-aligned with the surroundings as the user moves. The teachings herein thus provide a solution which predicts when a movement is about to happen from a stationary position (or, similarly, end when moving). This is made possible utilizing eye-tracking. Already when a person starts to gaze in a direction an indication of possible movement occurs, which can be used as input to a head-pose estimation algorithm, i.e. to predict a head movement. Similarly, when the head movement is almost complete, the eyes return to a normal state and this allows the user to avoid overshooting.

[0022] Further embodiments and advantages of the present invention will be given in the detailed description. It should be noted that the teachings herein find use in XR devices.

[0023] BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Embodiments of the invention will be described in the following, reference being made to the appended drawings which illustrate non-limiting examples of how the inventive concept can be reduced into practice.

[0025] Figure 1A shows a schematic view of an extended reality, XR, device according to some embodiments of the present invention,

[0026] Figure IB shows a schematic view of a head-mounted extended reality, XR, device according to some embodiments of the present invention,

[0027] Figure 2A, 2B, 2C, 2D and 2E each shows a schematic view illustrating eye shifting and head movements of a user,

[0028] Figure 3 shows a schematic view of a timeline,

[0029] Figures 4A, 4B, 4C, 4D and 4E each shows a schematic view of how virtual content is rendered according to eye and head movements according to some embodiments of the teachings herein,

[0030] Figure 5 shows a flowchart of a general method according to some embodiments of the present invention,

[0031] Figure 6 shows a component view for a software component arrangement according to some embodiments of the teachings herein, and Figure 7 shows a schematic view of a computer-readable medium carrying computer instructions that when loaded into and executed by a controller of an arrangement enables the arrangement to implement some embodiments of the present invention.

[0032] DETAILED DESCRIPTION

[0033] Figure 1A shows a schematic view of an extended reality, XR, device 100 according to some embodiments of the present invention. The XR device 100 comprises a controller 101 and a memory 102. In some embodiments the XR device 100 furthermore comprises a communication interface 103. The XR device 100 furthermore comprises an eye-tracking sensor 104 and a head movement sensor 105. The XR device 100 furthermore comprises or is connected to an image presenting arrangement 106, such as a display 106. In some embodiments the XR device 100 furthermore comprises or is connected to an audio presenting arrangement 107.

[0034] The controller 101 is configured to control the overall operation of the XR device 100. In some embodiments, the controller 101 may be a general-purpose controller, wherein general purpose refers to hardware (and / or software) that performs a variety of tasks. As a skilled person would understand there are many alternatives for how to implement a controller, such as using Field-Programmable Gate Arrays circuits, ASICs, GPUs, etc. in addition to or as an alternative. For the purpose of this application, all such possibilities and alternatives will be referred to simply as the controller 101.

[0035] The memory 102 is configured to store instruction data, graphics data, User Interface (Ul) settings, and / or communication data as well as computer-readable instructions that when loaded into the controller 101 indicate how the XR device 100 is to be controlled. The memory 102 may comprise several memory units or devices, but they will be perceived as being part of the same overall memory 102. There may be one memory unit for the image presenting device storing graphics data, one memory unit for an eye movement sensor (more on such below) for storing settings if present, one memory for a communications interface (if such is present) for storing settings, and so on. As a skilled person would understand, there are many possibilities of how to select where data should be stored. In one embodiment, the XR device 100 may comprise a general memory 102 in turn comprising any and all such memory units for the purpose of this application. As a skilled person would understand there are many alternatives of how to implement a memory, for example using non-volatile memory circuits, such as EEPROM memory circuits, or using volatile memory circuits, such as RAM memory circuits. For the purpose of this application all such alternatives will be referred to simply as the memory 102.

[0036] The image presenting arrangement 106 comprises a display arranged to display virtual (graphical) content. In some embodiments the display 106 is an optical see-through display.

[0037] The audio presenting arrangement 107 comprises one or more speakers, such as headphone(s).

[0038] The communication interface 103 is arranged to enable communication with other devices, such as other smart devices 100 or a server (not shown) for receiving content, instructions and / or settings or other data. In some embodiments the communication interface 103 is arranged to receive data regarding the imagery that is to be displayed. The communication interface 103 may be wired and / or wireless. The communication interface may comprise several interfaces. In some embodiments, the communication interface 103 may comprise a radio frequency (RF) communications interface. In some such embodiments, the communication interface 103 may comprise a Bluetooth™ interface, a WiFi™ interface, a ZigBee™ interface, a RFID™ (Radio Frequency Identification) interface, Wireless Display (WiDi) interface, Miracast interface, and / or other RF interface commonly used for short range RF communication. In alternative or supplemental such embodiments, the communication interface 103 may comprise a cellular communications interface such as a fifth generation (5G) cellular communication interface, an LTE (Long Term Evolution) interface, a GSM (Global System for Mobile Communication) interface and / or other interface commonly used for cellular communication. In some embodiments, the communication interface 103 may be configured to communicate using the UPnP (Universal Plug n Play) protocol. In some embodiments, the communication interface 103 may be configured to communicate using the DLNA (Digital Living Network Appliance) protocol. In some embodiments, the communication interface 103 may comprise a USB (Universal Serial Bus) interface. In some embodiments, the communication interface 103 may comprise an HDMI (High-Definition Multimedia Interface) interface. In some embodiments the communication interface 103 comprises a Display Port interface. In some embodiments the communication interface 103 may comprise an Ethernet interface. In some embodiments, the communication interface 103 may comprise a MIPI (Mobile Industry Processor Interface) interface. In some embodiments, the communication interface may comprise an analog interface, a CAN (Controller Area Network) bus interface, an I3C (Inter- Integrated Circuit) interface, or other communication interface.

[0039] In some embodiments the communication interface 103 is configured to establish a connection to the display 106 for transmitting graphic content to be displayed. In some embodiments the communication interface 103 is configured to establish a connection to the speaker(s) 107 for transmitting audio content to be output.

[0040] The OST 100 also comprises or is connected to the eye tracking sensor 104 that is arranged to track movements (or changes in movement) of the eye(s) of a user. More precisely, the eye tracking sensor 104 is configured to provide sensor information comprising an indication of a line of view of the user's eye(s). The indication of a line of view of the user's eye(s) may in some embodiments comprise an actual line of sight, i.e. a direction that the user is looking at. In some embodiments the indication of a line of view of the user's eye(s) may comprise a change in line of view, i.e. an indication that the user's eye is moving.

[0041] The eye tracking sensor 104 may in some embodiments be an infrared light sensor, that transmits infrared light towards the eye and measure eye's line of view based on the received reflection of the infrared light.

[0042] The eye tracking sensor 104 may in some embodiments be an image sensor, such as a camera or image sensor module, arranged to provide an image (or stream of images) of the user, wherein images of the user may be analyzed using image processing techniques known in the art in order to determine a line of view, LV, of the user, and more specifically the user's eye(s) E.

[0043] Optionally the eye tracking sensor 104 may in some embodiments be other types of electro-optical sensors. In some embodiments the eye tracking sensor 104 comprises one or more electro-oculography (EOG) sensors. As a skilled person would understand, the XR device 100 may comprise one controller 101 and the sensor 104 may comprise another controller, but for the purpose of the teachings herein, they will be considered to be the same controller 101 in order to cover all possible variations of exactly where the determination of movement or motion takes place.

[0044] The OST 100 also comprises or is connected to the head movement sensor 105 that is arranged to track movements (or changes in movement) of the head H of a user. The headmovement sensor 105 is configured to determine a center line, CL, for where the user's head is pointing (looking at) or more precisely, determine a change in a center line. The headmovement sensor 105 may in some embodiments be an inertial measurement unit, IMU.

[0045] The controller 101 of the XR device 100, alone or in cooperation with controllers of the eye tracking sensor 104 and / or head movement sensor 105, is configured to, based on the sensor data from the eye tracking sensor 104 and head movement sensor 105, determine a change in viewing angle wherein the viewing angle (A) is an angle between a line of view (LV) for the eye and a center line for the head (CL).

[0046] In some embodiments, the controller 101 is further configured to determine a change in movement of the user's head H by detecting a change in speed of rotation around at least one axis based on the sensor information received from the head movement sensor 105, and that the change in speed rotation of the head H is exceeding a first head movement threshold. In some embodiments the head movement threshold is 1, 2, 3 or 5 degrees or in any range there- in-between.

[0047] Figure IB shows a schematic view of an XR device 100 as shown in figure 1A, wherein the XR device 100 of figure IB is a head-mounted display, HMD. One example of such HMD is XR (AR or VR) goggles, for example an optical see-through display. As is illustrated in figure IB, the XR device 100 comprises the display 106 (and possibly the speaker(s) 107).

[0048] The inventors have realized that by utilizing the eye movements or shifts in eye movement of a user, it can be predicted when the user is about to start moving its head, and also in which direction, whereby latency can be reduced by generating predicted content in advance. It can also be predicted that the head is about to stop moving, whereby an overshoot of movement on the display 106 is prevented (or reduced). The eye shifts position frequently and scans our environment, e.g. saccading. Changes are not always indications of coming head movements but can for instance be a quick look to the side to see if something turned up. The inventors have therefore realized that the eye tracking data needs to be complemented with behavioral analysis before being interpreted as an indication of a coming head movement. However, such behavioral analysis is difficult to achieve in HMDs, especially since HMDs are usually worn by more than one user. The inventors are therefore proposing a simple solution where saccadic movements are simply filtered out and where eye movements are only taken into consideration when the viewing angle exceeds a first eye movement threshold, such as 1, 2, 3 or 5 degrees, or in any range there-in-between. In some embodiments the eye movement needs also be longer than a certain time duration, for example 5, 10 or 20ms or in any range there-in-between.

[0049] The teachings herein will be described with simultaneous reference to figures 2A-2E as well as 4A to 4E.

[0050] Figure 2A shows a schematic view of a user's head H. Indicated is the line of view, i.e. the direction of the user's eye(s) and also the center line, CL, for the head H, i.e. the direction that the user's head is pointing (looking at). In figure 2A the user is looking dead ahead and the line of view, LV is parallel to the center line, CL, of the head, i.e. the viewing angle as defined herein is 0.

[0051] Utilizing XR device 100 a view in the direction that the user is looking, such as defined by the center line, is presented on the image presenting arrangement, the display 106. Such a view 410 is shown in figure 4A showing a schematic view of an XR view presented to a user. In the example of figure 4A some content 420 is displayed in the view 410. In the description herein, focus will be on graphical content, but it should be noted that the teachings may also be applied to audio content to be presented by being output (played) through the speaker(s) 107.

[0052] As a user's eye starts moving to one side (up, down, left, and / or right), the angle between the line of view and the center line is no longer 0. As the change in angle is of importance, the absolute value of the angle is of importance (a negative angle is simply a view in an opposite direction) for determining how much the eye is moving. Figure 2B shows how the angle A has changed from zero as the eye E of the user starts moving to the side (change in line of view LV), while the head is still pointing in the same direction (no change in center line CL). While the eye is moving the (absolute) value of the viewing angle will keep increasing. It can thus be predicted in which way a user will be looking or rather turning to look, by determining that the viewing angle is increasing - and increases above the first eye movement threshold. It is thus possible to predict content that will be presented to the user as the head also starts to move, thereby shifting the view. Figure 4B shows a schematic view of an extended view 410X where predicted content 420P, i.e. content that was predicted to be presented, is presented.

[0053] As the head starts moving, as is shown in figure 2C, the viewing angle A stops changing, or at least changes at a lesser rate. As the head moves, the view 410 to be presented will also change, and the predicted content 420P will be presented to the user. This is shown in figure 4C schematically where the shifted view 410s now also comprises the predicted content 420P. While the head keeps moving, the XR device 100 keeps predicting content to be presented. This is shown in figure 4C schematically where further predicted content 420P' is shown. In some embodiments the controller 101 is further configured to continue generating predicted content 420P while the viewing angle exceeds the first eye movement angle threshold.

[0054] In some embodiments, the XR device 100 is configured to determine predict a speed of movement and generate or render the predicted content based on the speed. If the user is looking to one side very fast, more content may be predicted than if the user is looking slowly, as it can be predicted that a fast (initial) movement will result in a bigger movement. This applies both to the speed of eye movement and to the (later) speed of head movement.

[0055] In this manner, a reduced latency may be achieved by predicting that the user is going to look to a side and thereby rendering or generating predicted content. In cases where the user does not move the head, the predicted content may be discarded or saved for a later time, when the user may move the head to that side.

[0056] As the inventors have also realized, it is also possible in a similar manner to anticipate or predict when the user is to stop moving the head by determining a change in viewing angle, by determining or detecting a decrease in viewing angle. As the user's eye finds what it is looking for, the eye will rest on that, and the head movement will align with the eyes once again. This is shown in figures 2D where the line of view is drifting away from the center line, and in figure 4E where the line of view is again parallel to the center line CL. This realization may thus be utilized to prevent - or at least reduce - the amount of overshoot of presenting shifted views to a user, as it can be determined that the user is to stop moving the head based on the eye movement, the XR device, knows or can predict when to stop presenting shifted views. Specifically, the XR device can stop predicting content 420P to be presented. As is illustrated in figures 4D, as the viewing angle decreases (and the decrease falls below a second eye movement threshold), the XR device can determine or anticipate a stop in head movement and thereby determine which view 410 will be the final view and also stop predicting content 420P. In some embodiments the XR device stops generating or rendering predicted content as it is predicted that the movement is to stop. In some embodiments the XR device stops generating or rendering predicted content when the movement stops. In some embodiments the second eye movement threshold is 1, 2, 3 or 5 degrees or in any range there-in-between. In some embodiments the first and the second eye movement thresholds are the same.

[0057] Figure 3 shows a schematic timeline which helps explain the teachings herein. During an initial time A (corresponding to figures 2A and 4A) movement of a user's eye(s) is tracked and as it is determined that the viewing angle A exceeds the first eye movement threshold, the XR device 100 can predict that a head movement will follow, and also in which direction (same as change in line of view). This gives the XR device a time B (corresponding to figures 2B and 4B) to generate or render predicted content 420P. As the head starts moving, the XR device continues rendering further predicted content 420P' and presents the predicted content 420P during the period when the head is moving C (corresponding to figures 2C and 4C).

[0058] As a change is again detected for the viewing angle, and now a decrease in (absolute) viewing angle A, and the decrease exceeds the second eye movement threshold, the XR device 100 can predict a resting or ending view that the user will stop to look at. This can be done during a time D (corresponding to figures 2D and 4D) before the head stops, whereby the final or ending view is displayed E (corresponding to figures 2E and 4E).

[0059] The inventors are thus proposing an extended reality, XR, device 100 comprising a controller 101, wherein the controller 101 is configured to obtain eye-tracking information for a user's eye E through an eye-tracking sensor 104, the eye-tracking information comprising an indication of a line of view LV for the eye, obtain head-movement information for the user's head H through a head-movement sensor 105, wherein the head-movement information comprises an indication of a center line for the head CL. As discussed above, the controller 101 is also configured to determine that the user's eye E is moving relative the user's head H, by detecting an increase in a viewing angle A. As discussed above, the viewing angle A is an angle between a line of view LV for the eye and a center line for the head CL. As discussed above, the controller 101 is also configured to determine that the increase in viewing angle is exceeding a first eye movement angle threshold, and, in response thereto generate a predicted graphical content 420P. As discussed above, the predicted content 420P represents content outside a current view 410 being presented through the image presenting arrangement 106 in a direction of the eye movement. As discussed above, the controller 101 is also configured to determine a change in movement for the user's head H, and in response thereto cause the predicted content 420X to be presented.

[0060] In some embodiments the predicted content is only to be displayed if the head movement sensed actually corresponds to the predicted head movement. In such some embodiments the controller 101 is thus further configured to predict a change in movement of the user's head and in response to determining that the user's eye E is moving relative the user's head H, and to determine that the change in movement for the user's head H corresponds to the predicted change in movement, and then, in response thereto, cause the predicted content to be presented.

[0061] In some embodiments the XR device 100 is further configured to determine that the user's head E is to stop moving by monitoring eye-movements and head movements over time and train the controller to be able to determine when the user's eye E is to stop, and determine that the users head is to stop moving based on the monitored eye-movements and head movements. In some such embodiments, the training is to establish the first (and optionally the second) eye movement threshold. In some such embodiments, the training is achieved through machine learning. As mentioned above, the predicted content is in some embodiments predicted graphical content to be presented by being displayed on the display 106. And, in embodiments where XR device 100 comprises or is connected to the display 106, the XR device 100 is also configured to display the predicted graphical content 420P on the display 106. In embodiments where the XR device is not connected to or comprises the display, the XR device simply generates or renders the content and provides it to a controller connected to the display 106.

[0062] One example of graphical content is content representing advertisement localized outside the current view. This enables advertisements to be generated before a user turns the head in that direction, and especially for generating targeted advertisements. Another example is content representing safety warnings localized outside the current view. This enables warnings to be generated before a user turns the head in that direction, and especially for determining some dangers and generating warnings for such dangers.

[0063] As is also mentioned above, the predicted content in some embodiments comprises audio content to be presented by being played through the speaker(s) 107.

[0064] As all predicted content is not to be displayed, at least some of the predicted content is generated at a lower resolution than the other content 410. This reduces the processing and memory needed for generating and storing the predicted content. For graphical content the resolution is image resolution, and for audio content the resolution relates to audio quality.

[0065] As mentioned above, the XR device is in some embodiments configured to connect to a different device, such as a server. By causing the server to generate the predicted content, the local controller 101 can be offloaded. The XR device is therefore, in some embodiments, configured to generate the predicted content 420P by causing a server to generate the predicted content 420P and receiving the predicted content 420P from the server.

[0066] Figure 5 shows a general flowchart for a method according to the teachings herein. The method corresponds to the operation of the XR device 100 as discussed in the above, wherein said method comprises obtaining 510 eye-tracking information for a user's eye E through an eye-tracking sensor 104, obtaining 520 head-movement information for the user's head H through a head-movement sensor 105, and determining 530 that the user's eye E is moving relative the user's head H. The method further comprises generating 540 predicted content 420P in response thereto, and then determining 550 a change in movement for the user's head H, and in response thereto causing 560 the predicted content 420X to be presented.

[0067] It should be noted that the other functionalities discussed herein may be included in some or all of the functionalities discussed in relation to figure 5 and the method of figure 5 is a general method and also allows for implementing other features as disclosed in above as sub functionality of any part of the method as disclosed.

[0068] Figure 6 shows a component view for a software component or module arrangement 600 according to some embodiments of the teachings herein. The software component arrangement 600 is adapted to be used in an XR device 100 as taught herein and corresponds to the operation of the XR device 100 in the above. The software component arrangement 600 comprises a software component for obtaining 610 eye-tracking information for a user's eye E through an eye-tracking sensor 104, a software component for obtaining 620 head-movement information for the user's head H through a head-movement sensor 105, and a software component for determining 630 that the user's eye E is moving relative the user's head H. The software component arrangement 600 further comprises a software component for generating 640 predicted content 420P in response thereto, and a software component for determining 650 a change in movement for the user's head H, and a software component for causing 660 the predicted content 420X to be presented in response thereto.

[0069] In some embodiments the software component arrangement 600 also comprises software component(s) 670 for further functionalities as discussed in the teachings herein.

[0070] For the context of the teachings herein a software code module may be replaced or supplemented by a software component. Alternatively for the context of the teachings herein a software code module may be replaced or supplemented by a circuit configured for performing a corresponding function.

[0071] Figure 7 shows a schematic view of a computer-readable medium 700 carrying computer instructions 710 that when loaded into and executed by a controller 101 of an XR device 100 enables the XR device 100 to operate according to the teachings herein. The computer-readable medium 700 may be tangible such as a hard drive or a flash memory, for example a USB memory stick or a cloud server. Alternatively, the computer- readable medium 700 may be intangible such as a signal carrying the computer instructions enabling the computer instructions to be downloaded through a network connection, such as an internet connection. In the example of figure 7, a computer-readable medium 700 is shown as being a computer disc 700 carrying computer-readable computer instructions 710, being inserted in a computer disc reader 720. The computer disc reader 720 may be part of a cloud server 730 - or other server - or the computer disc reader may be connected to a cloud server 730 - or other server. The cloud server 730 may be part of the internet or at least connected to the internet. The cloud server 730 may alternatively be connected through a proprietary or dedicated connection. In one example embodiment, the computer instructions are stored at a remote server 730 and be downloaded to the memory 102 of the XR device 100 for being executed by the controller 101. The computer disc reader 720 may also or alternatively be connected to (or possibly inserted into) a XR device 100 for transferring the computer-readable computer instructions 710 to a controller 101 of the XR device 100 via a memory 102 of the XR device. Figure 7 shows both the situation when a XR device 100 receives the computer-readable computer instructions 710 via a server connection and the situation when another XR device 100 receives the computer-readable computer instructions 710 through a wired interface. This enables for computer-readable computer instructions 710 being downloaded into an XR device 100 thereby enabling the XR device 100 to operate according to and implement the invention as disclosed herein.

Claims

CLAIMS1. An extended reality, XR, device (100) comprising a controller (101), wherein the controller (101) is configured to obtain eye-tracking information for a user's eye (E) through an eye-tracking sensor (104), the eye-tracking information comprising an indication of a line of view (LV) for the eye, obtain head-movement information for the user's head (H) through a head-movement sensor (105), the head-movement information comprising an indication of a center line for the head (CL), determine that the user's eye (E) is moving relative the user's head (H), by detecting an increase in a viewing angle (A), wherein the viewing angle (A) is an angle between a line of view (LV) for the eye and a center line for the head (CL), and determining that the increase in viewing angle is exceeding a first eye movement angle threshold, and, in response thereto generate a predicted content (420P), wherein the predicted content (420P) represents content outside a current view (410) being presented through the image presenting arrangement (106) in a direction of the eye movement, determine a change in movement for the user's head (H), and in response thereto cause the predicted content (420X) to be presented.2.The XR device according to claim 1, wherein the controller (101) is further configured to determine a change in movement of the user's head (H) by detecting a change in speed of rotation around at least one axis and that the change in speed rotation of the head (H) is exceeding a first head movement threshold.3.The XR device according to any preceding claim, wherein the controller (101) is further configured to predict a change in movement of the user's head and in response to determining that the user's eye (E) is moving relative the user's head (H), and to determine that the change in movement for the user's head (H) corresponds to the predicted change in movement, and then, in response thereto, cause the predicted content to be presented.4.The XR device according to any preceding claim, wherein the controller (101) is further configured to continue generating predicted content (420P) while the viewing angle exceeds the first eye movement angle threshold.S.The XR device according to any preceding claim, wherein the controller (101) is further configured to determine a speed of the head's movement and to generate the predicted graphic content (420P) based on the speed of the head's movement.6.The XR device according to any preceding claim, wherein the controller (101) is further configured to predict that the user's head (E) is about to stop moving by determining that the viewing angle (A) is decreasing and that the viewing angle falls under a second eye movement angle threshold.7.The XR device according to claim 6, wherein the controller (101) is further configured to discontinue generating predicted content (420P) in response to that it is determined that the user's eye (E) is about to stop moving.8.The XR device according to any preceding claim, wherein the controller (101) is further configured to determine that the user's head (E) is to stop moving by monitoring eyemovements and head movements over time and train the controller to be able to determine when the user's eye (E) is to stop, and determine that the users head is to stop moving based on the monitored eye-movements and head movements.9.The XR device according to any preceding claim, wherein the predicted content comprises predicted graphical content to be presented by being displayed on the image presenting arrangement (106).10.The XR device according to claim 9, wherein the XR device further comprises or is connected to the image presenting arrangement (106), and wherein the controller (101) is further configured to display the predicted graphical content (420P) on the image presenting arrangement (106).11.The XR device according to claim 9 or 10, wherein the predicted graphical content comprises content representing advertisement localized outside the current view or wherein the predicted graphical content comprises content representing safety warnings localized outside the current view.12.The XR device according to any preceding claim, wherein the XR device further comprises or is connected to an audio device (107), and wherein the predicted content comprises audio content to be presented by being played through the audio device (107).

13. The XR device (100) according to any preceding claim, wherein the controller (101) is further configured to generate the predicted content (420P) by generating at least some of the predicted content (420P) in a lower resolution than the content (420) currently presented.14.The XR device according to any preceding claim, wherein the controller (101) is further configured to generate the predicted content (420P) by causing a server to generate the predicted content (420P) and receiving the predicted content (420P) from the server.

15. The XR device (100) according to any preceding claim, wherein the XR device is a head mounted display.

16. A method for use in an XR device (100), wherein the method comprises obtaining eye-tracking information for a user's eye (E) through an eye-tracking sensor (104), the eye-tracking information comprising an indication of a line of view (LV) for the eye,obtaining head-movement information for the user's head (H) through a headmovement sensor (105), the head-movement information comprising an indication of a center line for the head (CL), determining that the user's eye (E) is moving relative the user's head (H), by detecting an increase in a viewing angle (A), wherein the viewing angle (A) is an angle between a line of view (LV) for the eye and a center line for the head (CL), and that determining that the increase in viewing angle is exceeding a first eye movement angle threshold, and, in response thereto generating predicted content (420P), wherein the predicted content (420P) represents content outside a current view (410) being presented through the image presenting arrangement (106) in a direction of the eye movement, determining a change in movement for the user's head (H), and in response thereto causing the predicted content (420X) to be presented.

17. A computer-readable medium (720) carrying computer instructions (721) that when loaded into and executed by a controller (101) of an XR device (100) enables the XR device (100) to implement the method according to claim 16.

18. A software component arrangement for controlling projection of graphic content in an XR device (100), wherein the software component arrangement (600) comprises a software component for obtaining eye-tracking information for a user's eye (E) through an eye-tracking sensor (104), the eye-tracking information comprising an indication of a line of view (LV) for the eye, a software component for obtaining head-movement information for the user's head (H) through a head-movement sensor (105), the head-movement information comprising an indication of a center line for the head (CL), a software component for determining that the user's eye (E) is moving relative the user's head (H), by detecting an increase in a viewing angle (A), wherein the viewing angle (A) is an angle between a line of view (LV) for the eye and a center line for the head (CL), and thatdetermining that the increase in viewing angle is exceeding a first eye movement angle threshold, a software component for generating predicted content (420P) in response to determining that the user's eye is moving, wherein the predicted content (420P) represents content outside a current view (410) being presented through the image presenting arrangement (106) in a direction of the eye movement, a software component for determining a change in movement for the user's head (H), and a software component for causing the predicted content (420X) to be presented in response to determining that the user's head is moving.

Images