Methods and devices for providing feedback to a user of an extended reality device

Abstract

The present disclosure relates to a method (200) for providing feedback to a user of an extended reality device, wherein the extended reality device is configured to display an extended reality environment to the user. The method (200) comprises: tracking (S204) a position of a physical object in a physical environment surrounding the extended reality device; and in response to the physical object entering a space around the extended reality device, providing (S210) feedback to the user indicative of the position of the physical object in relation to the extended reality device, wherein the feedback comprises an audio and / or haptic feedback. The present disclosure further relates to an electronic device (300), and an extended reality device (400).

Description

[0001] P106566W001

[0002] 1

[0003] METHODS AND DEVICES FOR PROVIDING FEEDBACK TO A USER OF AN EXTENDED REALITY DEVICE

[0004] TECHNICAL FIELD

[0005] The present disclosure relates to the field of extended reality applications, and in particular to methods

[0006] 5 and devices for providing feedback to a user of an extended reality device.

[0007] BACKGROUND

[0008] Extended reality (XR) devices have revolutionized the way we interact with digital content, providing immersive experiences that bridge the gap between the physical and virtual worlds. The field of XR has witnessed significant advancements in recent years, driven by technological breakthroughs and increased consumer demand for more immersive and interactive experiences. XR provides a wide range of applications across industries, including gaming, entertainment, education, training, healthcare, architecture, and more. It offers opportunities for realistic simulations, virtual prototyping, interactive storytelling, telepresence, and collaborative experiences, among others.

[0009] Extended Reality is an umbrella term that encompasses a spectrum of immersive technologies, including

[0010] 15 virtual reality (VR), Augmented Reality (AR), and Mixed Reality (MR). VR completely immerses a user in computer-generated environments, blocking out the physical world and replacing it with a simulated reality. AR overlays digital information or virtual objects onto the real-world environment, enhancing the user's perception of reality. Lastly, MR combines elements of both VR and AR, allowing virtual objects to interact with the real world and vice versa.

[0011] 20 XR applications are often experienced through various devices equipped with displays and motion tracking systems, such as headsets (sometimes accompanied by handheld controllers for facilitating interaction with the virtual world), smartphones, tablets, smart glasses or the like. These devices typically further comprise cameras or sensors enabling the device to capture data of the physical world and overlay digital content onto it.

[0012] One of the key challenges in XR is achieving a high level of realism and immersion while maintaining a seamless integration with the real world. However, with increased immersion into the virtual world (in AR and MR, and especially in VR applications), it is increasingly more difficult for the user to stay aware of its surroundings. For instance, virtual objects in the XR environment may risk blocking impressions from the physical world.

[0013] 30 One current attempt for mitigating this problem is the so-called Space Sense system of the Oculus Quest VR headset. This approach involves displaying outlines of e.g. a person or pet in front of a user immersed P106566W001

[0014] 2 in VR. However, this process greatly reduces the VR user experience. There is therefore need for improvements in how to provide feedback of the physical world to a user of an XR device.

[0015] SUMMARY

[0016] The technology disclosed herein seeks to mitigate, alleviate, or eliminate one or more deficiencies and

[0017] 5 disadvantages of the prior art singly or in any combination.

[0018] In extended reality applications, the user is typically fully immersed in the extended reality world (e.g. virtual reality application) or partly immersed (e.g. mixed or augmented reality application). In any case, the user is at least partly disconnected from the physical world, in the sense that the user loses awareness of his or her surroundings. This may pose potential safety risks to the user, or to any physical objects in the physical environment surrounding the user. The inventor has realized a new and improved way of providing feedback from the physical world to the user of an extended reality device, having less effect on the extended reality experience.

[0019] Various aspects and embodiments of the present disclosure or the technology disclosed herein are defined below and in the accompanying independent and dependent claims.

[0020] 15 According to a first aspect of the present disclosure, there is provided a method for providing feedback to a user of an extended reality device. The extended reality device is configured to display an extended reality environment to the user. The method comprises tracking a position of a physical object in a physical environment surrounding the extended reality device. The method further comprises, in response to the physical object entering a space around the extended reality device, providing feedback

[0021] 20 to the user indicative of the position of the physical object in relation to the extended reality device. The feedback comprises an audio and / or haptic feedback.

[0022] The presently disclosed technology provides an audio and / or haptics-based space sense approach based on spatial audio and / or haptics technology. This provides feedback from objects in the physical world without affecting or interrupting a visual experience of the extended reality environment. This in turn allows for an ever more immersive extended reality experience compared to current solutions, while the user experience can be increased due to the user having improved awareness of the surrounding physical environment.

[0023] According to a second aspect of the present disclosure, there is provided a computer program, comprising instructions which, when executed on at least one processor, cause the at least one processor

[0024] 30 to carry out the method according to any one of the embodiments of the first aspect. With this second aspect of the disclosed technology, similar advantages and preferred features are present as in the first aspect. In order to avoid undue repetition, reference is made to the above. P106566W001

[0025] 3

[0026] According to a third aspect of the present disclosure, there is provided a (non-transitory) computer- readable storage medium storing one or more computer program code configured to be executed by one or more processors of a processing device, the one or more computer program codes comprising instructions for performing the method according to any one of the embodiments of the first aspect.

[0027] 5 With this third aspect of the disclosed technology, similar advantages and preferred features are present as in the other aspects. In order to avoid undue repetition, reference is made to the above.

[0028] The term "non-transitory," as used herein, is intended to describe a computer-readable storage medium (or "memory") excluding propagating electromagnetic signals, but is not intended to otherwise limit the type of physical computer-readable storage device that is encompassed by the phrase computer-

[0029] 10 readable medium or memory. For instance, the terms "non-transitory computer readable medium" or "tangible memory" are intended to encompass storage devices that do not necessarily store information permanently, including for example, random access memory (RAM). Program instructions and data stored on a tangible computer-accessible storage medium in non-transitory form may further be transmitted by transmission media or signals such as electrical, electromagnetic, or digital signals, which may be conveyed via a communication medium such as a network and / or a wireless link. Thus, the term "non-transitory", as used herein, is a limitation of the medium itself (i.e., tangible, not a signal) as opposed to a limitation on data storage persistency (e.g., RAM vs. ROM).

[0030] According to a fourth aspect of the present disclosure, there is provided an electronic device for providing feedback to a user of an extended reality device. The extended reality device is configured to

[0031] 20 display an extended reality environment to the user. The electronic device comprises control circuitry. The control circuitry is configured to track a position of a physical object in a physical environment surrounding the extended reality device. The control circuitry is further configured to, in response to the physical object entering a space around the extended reality device, provide a feedback to the user, indicative of the position of the physical object in relation to the extended reality device. The feedback comprises an audio and / or haptic feedback. With this fourth aspect of the disclosed technology, similar advantages and preferred features are present as in the other aspects. In order to avoid undue repetition, reference is made to the above.

[0032] According to a fifth aspect of the present disclosure, there is provided an extended reality device. The extended reality device comprises a display for displaying an extended reality environment to a user of

[0033] 30 the extended reality device. The extended reality device further comprises a sensor system for tracking a position of a physical object in a physical environment surrounding the extended reality device. The extended reality device further comprises an electronic device according to any one of the embodiments of the fourth aspect. With this fifth aspect of the disclosed technology, similar advantages and preferred P106566W001

[0034] 4 features are present as in the other aspects. In order to avoid undue repetition, reference is made to the above.

[0035] Further embodiments of the disclosure are defined in the dependent claims. It should be emphasized that the term "comprises / comprising" when used in this specification is taken to specify the presence of

[0036] 5 stated features, integers, steps, or components. It does not preclude the presence or addition of one or more other features, integers, steps, components, or groups thereof. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.

[0037] A possible advantage of some embodiments is that feedback from the physical world can be provided

[0038] 10 without affecting or interrupting a visual experience of the extended reality environment. Thus, user experience can be improved.

[0039] A further possible advantage of some embodiments is that safer operation of the extended reality device can be achieved, due to providing improved awareness of the physical environment around the extended reality device.

[0040] These and other features and advantages of the present disclosure will in the following be further clarified with reference to the embodiments described hereinafter.

[0041] BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The foregoing will be apparent from the following more particular description of the example embodiments, as illustrated in the accompanying drawings in which like reference characters refer to

[0043] 20 the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the example embodiments. In the drawings:

[0044] Figure 1 illustrates, by way of example, a perspective view of a physical environment.

[0045] Figure 2 is a flowchart illustrating some embodiments of a method for providing feedback of a physical world to a user of an extended reality device.

[0046] 25 Figure 3 is a schematic illustration of an electronic device for providing feedback of a physical world to a user of an extended reality device, according to some embodiments.

[0047] Figure 4 is a schematic illustration of an extended reality device according to some embodiments.

[0048] DETAILED DESCRIPTION P106566W001

[0049] 5

[0050] The present disclosure is described below with reference to the accompanying drawings, in which certain aspects of the present disclosure are shown. The present disclosure may, however, be embodied in many different forms and should not be construed as limited to the embodiments and aspects set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and

[0051] 5 complete, and will fully convey the scope of the present disclosure to those skilled in the art. Like numbers refer to like elements throughout the description.

[0052] It is to be understood that the present disclosure is not limited to the embodiments described herein and illustrated in the drawings; rather, those skilled in the art will recognize that many changes and modifications may be made within the scope of the appended claims.

[0053] For example, those skilled in the art will appreciate that the steps, services and functions explained herein may be implemented using individual hardware circuitry, using software functioning in conjunction with a programmed microprocessor or general-purpose computer, using one or more Application Specific Integrated Circuits (ASICs) and / or using one or more Digital Signal Processors (DSPs). It will also be appreciated that when the present disclosure is described in terms of a method, it may

[0054] 15 also be embodied in one or more processors and one or more memories coupled to the one or more processors, wherein the one or more memories store one or more programs that perform the steps, services and functions disclosed herein when executed by the one or more processors.

[0055] Hereinafter, various techniques for providing feedback of a physical world to a user of an extended reality device will be described with reference to Figs. 1 to 4, in which a method, an electronic device

[0056] 20 configured for performing the method, and an extended reality device thereof, are presented.

[0057] Figure 1 illustrates, by way of example, a physical environment surrounding a user 110 of an extended reality (XR) device 400, in perspective view. The XR device 400 herein comprises a headset 102, and a first and second hand controller 104a, 104b. It should be mentioned, however, that there are examples of XR devices where hand controllers are not necessary, and which are operated by hand or finger gestures. The XR device 400 may be communicatively connected to a host device 106, such as a computer, game console or the like. The host device 106 may be configured to host at least parts of the extended reality application. Thus, the XR device 400 can leverage the processing power of the host device 106 to enable a more efficient and advanced XR experience. The host device 106 may be tasked with handling the rendering and processing of the XR content, while the XR device 400 itself focuses on

[0058] 30 displaying the rendered content and tracking the user's movement. The XR device 400 is herein connected to the host device 106 wirelessly. This type of XR device 400 is commonly referred to as wireless, or standalone XR devices. However, the XR device 400 and host device 106 may be communicatively connected by a wired connection. This type of XR device 400 is commonly referred to P106566W001

[0059] 6 as a tethered, or PC-based XR device. As will be further explained below in connection with Fig. 4, other types of XR devices are applicable as well. The XR device 400 illustrated herein should thus only be seen as a non-limiting example for illustrative purposes.

[0060] The physical environment 100 surrounding the extended reality device 400 (or user 110) is herein

[0061] 5 illustrated as an indoor environment as way of example. The principles described herein are applicable also to other indoor, as well as outdoor, environments. The physical environment surrounding the XR device is herein to be understood as a general area around the XR device in which physical objects (such as other persons, animals or inanimate objects, etc.) can be detected, tracked, and / or identified by sensors of the XR device. The physical environment may i.e. be within a sensor range of the XR device.

[0062] 10 The physical environment is thus a part of the physical world in which the user 110 is present.

[0063] Further illustrated in Fig. 1 is a space 108 around the extended reality device 400 (or around the user 110), which from now on will be called configured space 108. The configured space 108 may be a three- dimensional space around the XR device 400. The configured space 108 is herein represented by a cuboid in broken lines. The configured space may be a pre-configured, or pre-defined space. The configured

[0064] 15 space 108 may be a user-defined area. More specifically, the configured space 108 may be seen as a user-defined play area or (play) space. In other words, the configured space 108 may be set by the user 110 as an area in the physical world within which the user can move and interact while using the XR device 400. In some embodiments, the configured space 108 is a space fixed within the physical environment (or the physical world). In other words, the configured space may be anchored to a location

[0065] 20 in the physical environment (or the physical world). Thus, the configured space 108 may be fixed, while the XR device 400 moves in space. In some embodiments, the configured space 108 may be configured as a set space around the XR device 400. In other words, the configured space 108 can be anchored to the XR device. Thus, the configured space 108 may move together with the XR device 400. The configured space is typically set up in space free from obstacles, to ensure user safety and provide for improved XR experience. The configured space may help to prevent users from accidentally colliding with walls or objects in the physical world while they are immersed in the extended reality experience. The user 110 can explore and navigate XR environments by physically moving within the configured space 108. It is to be appreciated that the size and shape of the configured space 108 is not limited to the illustrated example. Instead, the size and shape of the configured space 108 may vary depending e.g. on the XR

[0066] 30 device 400, XR experience, and the available physical space.

[0067] The configured space 108 may be set up before using an XR headset. The user 110 may define the boundaries of the configured space by marking it with virtual or physical boundaries. It is to be P106566W001

[0068] 7 appreciated that the configured space 108, even though illustrated in the physical environment 100 of Fig. 1, is to be understood as a virtual space, implemented in software.

[0069] Now, the way the configured space 108 is utilized in the presently disclosed technology will be further explained below in connection with Fig. 2. In short, the user 110 may be provided with a feedback in

[0070] 5 response to a physical object entering the configured space 108 (or put differently, crossing a boundary of the space 108). The physical object may e.g. be a person 112 or an animal 114 (such as a pet) as illustrated herein. The physical object may further be an inanimate object. Thereby, the user 110 can be provided with information about any physical objects entering the configured space 108.

[0071] In some embodiments, the configured space 108 may be anchored in the physical environment

[0072] 10 surrounding the user wearing the XR device, such that it does not move with the XR device. In this case, the sensors in the XR device (such as image sensors, lidar and other distance measurement sensors) can be able to detect movement of the user towards the edges (or boundaries) of the configured space 108 and recognize one or more physical objects located in the direction of the edge of the configured space 108 towards which the user is moving. The XR device may issue an audio and / or haptic feedback

[0073] 15 depending on how close the user is to the object or how fast he or she is approaching the object, as measured e.g. by its sensors. The object could be a person, pet or a physical object outside but close the edge of the configured space 108. Here, the term "near" may be defined as a distance d between the edge of the configured space 108 and the object.

[0074] In some embodiments, the configured space 108 may be anchored in the XR device and outward from

[0075] 20 the XR device in the physical environment, but moving with the XR device. When the sensors of the XR device detect physical objects which the user is approaching or which the configured space 108 is approaching due to the movement of the user (and hence also the XR device) towards the physical object, the XR device may issue an audio and / or haptic feedback to the user which may be dependent on the distance of the edge of the configured space 108 anchored in the XR device or the speed of movement of the configured space 108 towards the physical object.

[0076] Figure 2 is a flowchart illustrating some embodiments of a method 200 for providing feedback of a physical world to a user of an extended reality (XR) device. The XR device being configured to display an XR world to the user. As explained in the foregoing, Fig. 1 illustrates an example of a scenario in which the techniques of the method 200 can be performed.

[0077] 30 In some embodiments, the XR device may be a head-worn device. In some embodiments, the XR device may be a hand-held device. By the wording "extended reality environment" (or "extended reality world") it is herein meant the world experienced by the user while using the XR device. The XR environment may P106566W001

[0078] 8 be a completely virtual world. This may be the case e.g. in virtual reality (VR) applications. Alternatively, the XR world may be a partly virtual world. This may be the case e.g. in mixed reality (MR) or augmented reality (AR) applications.

[0079] Below, the different steps of the method 200 are described in more detail. Even though illustrated in a

[0080] 5 specific order, the steps of the method 200 may be performed in any suitable order as well as multiple times. Thus, although Fig. 2 shows a specific order of method steps, the order of the steps may differ from what is depicted. In addition, two or more steps may be performed concurrently or with partial concurrence. For example, the step denoted S202 can be performed independently of the step denoted S204, and in any order, based on a specific implementation. Such variations will depend on the software and hardware systems chosen and on design choices. All such variations are within the scope of the invention. Likewise, software implementations could be accomplished with standard programming techniques with rule-based logic and other logic to accomplish the various steps.

[0081] The embodiments mentioned and described herein are only given as examples and should not be construed as limiting to the present invention. Other solutions, uses, objectives, and functions within

[0082] 15 the scope of the patent claims should be apparent for the person skilled in the art. Moreover, it should be appreciated that the method 200 of Fig. 2 may comprise some steps which are illustrated as boxes in solid lines and some steps which are illustrated in dashed lines. The steps which are shown in solid lines are steps which are comprised in the broadest example embodiment of the method 200. The steps which are comprised in dashed lines are examples of a number of optional steps which may form part of a

[0083] 20 number of alternative embodiments. It should be appreciated that not all of the steps need to be performed. The example steps may be performed in any order and in any combination.

[0084] The method comprises tracking S204 the position of a physical object in a physical environment surrounding the extended reality device. Tracking S204 the position of the physical object may be performed in response to detecting that the physical object has entered into the physical environment or that the user-is approaching the edge of the configured area 108 where one or more physical objects are located. In another variant, especially in the case of a configured area 108 anchored to the XR device and the physical world, the tracking step S204 may comprise the image and / or distance sensors of the XR device detecting the configured area 108 approaching one or more physical objects in the physical word by means of movement of the user of XR device. The wording position herein refers to a specific

[0085] 30 location or placement of the physical object in a given space or coordinate system and / or the position of the user of the XR device in relation to the placement of the physical object. The given space herein being the physical environment of the XR device. Moreover, the position may define a spatial relationship of a physical object relative to a reference point or frame of reference. Position is typically P106566W001

[0086] 9 described using a set of coordinates that specify the physical object's location along different axes, such as x, y, and z coordinates in a three-dimensional space. The position of the physical object may be a position of the physical object relative the XR device. Thus, the reference point may be the XR device. Alternatively, the position of the physical object may be a position in a local coordinate system, such as

[0087] 5 within the physical environment surrounding the XR device, or within the configured space around the XR device. In yet another example, the position may be a position in a global coordinate system, such as Global Navigation Satellite System (GNSS). The XR device may track its own position and orientation within the same local (or global) coordinate system. The XR device may for instance track its own position and orientation by use of an inertial measurement unit, comprising one or more sensors, such as an

[0088] 10 accelerometer and / or a gyroscope. Alternatively, or in combination, the XR device may track its own position by use of built-in or external sensors for perceiving the physical environment, such as the one or more cameras, LIDAR, and RADAR mentioned above, or by a Global Positioning System (GPS). By the XR device knowing its own position and orientation within the local (or global) coordinate system, a position of the physical object in relation to the XR device can be transformed into a position in the local

[0089] 15 (or global) coordinate system.

[0090] The step denoted S204 may be formulated as tracking S204 a physical object in a physical environment surrounding the XR device. Thus, the step of tracking S204 the position of the physical object may encompass tracking the physical object in any sense, which includes information being indicative of the position of the physical object. Tracking S204 the position of the physical object may also encompass

[0091] 20 determining a plurality of positions of the physical object over a number of time points. Moreover, tracking S204 the position of the physical object may comprise tracking a movement direction and / or a distance of the physical object in relation to the XR device. Thus, the position of the physical object may comprise information about the movement direction and / or the distance of the physical object in relation to the extended reality device. The wording "movement direction" herein refers to the path or trajectory (current or expected trajectory) followed by a physical object as it moves from one location to another over time, but also the movement of the user of the XR device in relation to the physical object. When persons or pets are involved it will be more often than not the mutual movement of the user of the XR device and the physical object in relation to each other. Moreover, the movement direction may describe the orientation or course of movement relative to a reference point or frame of

[0092] 30 reference. The movement direction can be characterized by various factors, such as speed, velocity, acceleration, and the angle at which the movement occurs in relation to a reference point or frame of reference. By also using information about the movement direction and / or the distance of the physical object in relation to the XR device, a more relevant tracking of physical objects which move towards the configured space can be achieved. P106566W001

[0093] 10

[0094] Tracking S204 the position of the physical object may be done using built-in sensors of the XR device itself. Alternatively, or in combination, tracking S204 the position of the physical object may be done using sensors external to the XR device. The external sensors may e.g. be sensors arranged in the physical environment surrounding the XR device, and being communicatively connected to the XR device. The

[0095] 5 sensors used for tracking S204 (both built-in and external) may e.g. be one or more cameras, LIDAR, RADAR, etc. Further examples include SONAR, directional microphones, and directional Bluetooth. The tracking S204 of the physical object may further employ algorithms, such as image processing techniques, computer vision, and or machine learning models such as neural networks for object detection and / or tracking.

[0096] 10 The physical object may be any type of movable or static object. In other words, the physical object may be any type of object that can enter into the configured space whether by virtues of its own movement of by virtue of the user of the extended reality device moving towards it. For example, the physical object may be a person, an animal or an inanimate object.

[0097] The method 200 further comprises, in response to the physical object entering a space (or configured

[0098] 15 space) around the extended reality device, providing S210 feedback to the user indicative of the position of the physical object in relation to the extended reality device, wherein the feedback is an audio and / or haptic feedback. Put differently, the method comprises, in response to the physical object entering a configured space around the extended reality device, providing S210 a feedback indicative of the position of the physical object in relation to the extended reality device, wherein the feedback comprises

[0099] 20 an audio and / or haptic feedback. The feedback may be provided S210 by outputting a signal indicative of the feedback to an audio system and / or a haptic system of the XR device. The audio system and haptic system may be configured to generate an audio and haptic feedback respectively. Thus, the audio system may herein encompass any mechanism for generating an audio output, such as speakers or headphones. Similarly, the haptic system may herein encompass any mechanism for generating a haptic output, such as haptic motors or vibrators. By the configured space being "around" the extended reality device, it is herein meant that the extended reality device is within a configured space, such as the configured space 108 shown in Fig. 1 earlier. The configured space may be a user-defined play area. For further details about the configured space, reference is made to the above.

[0100] By using an audio and / or haptic feedback, the feedback of the physical world can be provided to the user

[0101] 30 without affecting or interrupting the visual aspects of the XR experience. Thereby, the user experience can be enhanced.

[0102] By the feedback being indicative of the position of the physical object, it is herein meant that the feedback is such that the user can get an indication of where the physical object is spatially located. In P106566W001

[0103] 11 other words, the feedback can be seen as a spatial feedback. The audio feedback may e.g. be provided by spatial audio. The haptic feedback may be provided by 3D haptics. Thereby, the feedback may reflect the position of the physical object in three dimensions. The feedback may for instance reflect a distance between the XR device and the physical object by increasing a sound level or vibration strength of the

[0104] 5 haptic signal the smaller the distance becomes. In general, at least one audio characteristic of the audio feedback may be adjusted to reflect the position of the physical object. Put differently, an audio characteristic of the audio feedback may be set based on the position of the physical object in relation to the extended reality device. An audio characteristic may be understood as a property of the audio feedback. The audio characteristic may e.g. be a pitch, a loudness (or volume), a sound quality, a tempo

[0105] 10 (e.g. beats per minute), a tune, or a melody. The audio characteristic of the audio feedback may further be a duration of the audio feedback, i.e. for how long the audio feedback is played. Thus, the at least one audio characteristic may be selected from the group comprising of pitch, loudness, quality, tune and melody. The group may further comprise a tempo and / or a duration of the audio feedback. As an example, the pitch, loudness, quality, duration, and / or tempo of the audio signal may e.g. be increased

[0106] 15 or decreased depending e.g. on the position of the physical object in relation to the XR device, or depending on a distance between the XR device and the physical object. In particular, the physical object may in some cases move towards the user (and the XR device). In other cases, the user (and the XR device) may move towards the physical object. In some cases, both the physical object and the user (and the XR device) may move in relation to each other. The feedback (e.g. audio or haptic characteristics)

[0107] 20 can be adjusted to reflect any such movement. In a more specific example, the feedback may indicate the position of the physical object by having a lower volume for positions further away from the XR device and by increasing volume as the physical object gets closer to the XR device. Further, the audio feedback may indicate an origin or direction of the object (e.g. by spatial audio) in relation to the XR device. The tune and / or melody of the audio signal may be changed e.g. depending on the position of the physical object in relation to the XR device or the distance between the XR device and the physical object. Similarly, at least one haptic characteristic of the haptic feedback may be adjusted to do the same. For example, an intensity or vibration pattern of the haptic signal can be adjusted to reflect the position of the physical object.

[0108] Moreover, the audio feedback may be provided to the user through headphones or speakers of the XR

[0109] 30 device. The haptic feedback may be provided to the user through one or more haptic motor(s) arranged e.g. in hand controllers, a headset of the XR device, or in the XR device itself in case it is hand-held.

[0110] In some embodiments, the method 200 further comprises determining S206 a type of the physical object. The type of the physical object may e.g. be a person, an animal or an inanimate object. Determining S206 the type of the physical object may be performed e.g. by visual techniques, such as P106566W001

[0111] 12 use of image recognition / classification networks, or through sound-based recognition, or any combination thereof.

[0112] The feedback may be further indicative of the type of object. In other words, one or more characteristics (or properties) of the feedback may vary depending on what type of object it is. More specifically, at

[0113] 5 least one audio characteristic of the audio feedback may be set based on the type of the physical object. Thus, depending on the feedback, the user may know what type of object it is. Put differently, the step of providing S210 the feedback to the user may comprise, in response to the physical object being of a first type, providing S212 a first type of feedback to the user, and in response to the physical object being of a second type, providing S214 a second type of feedback to the user. In other words, the step of

[0114] 10 providing S210 the feedback may comprise, in response to the physical object being of a first type, providing S212 a first type of feedback, and in response to the physical object being of a second type, providing S214 a second type of feedback. Providing S212, S214 the first and second type of feedback may comprise outputting a first and second signal indicative of the respective first and second feedback to the audio system and / or the haptic system of the XR device. The first and second type of feedback may differ from each other such that the user can distinguish the first and second feedback from each other. As an example, the feedback may comprise a certain sound (or certain haptic pattern) for a certain physical object, and another sound (or haptic pattern) for another physical object. Thereby, the user can distinguish between different types of physical objects entering the configured space by providing S210 a feedback which differs between types of objects. More specifically, a type of feedback may be defined

[0115] 20 by at least one audio characteristic of the audio feedback, and / or at least one haptic characteristic of the haptic feedback of the feedback. As already mentioned earlier, this type of audio and / or haptic feedback may also be produced in the case the user of the extended reality device approaches the edge of the configured space, such as the configured space 108 in Fig. 1, and is within a certain distance d from a physical object (human, animal or inanimate object). Likewise, this type of audio and / haptic feedback may also be produced in case the configured space is anchored in the extended reality device itself and moves with the extended reality device, such that the configured space is approaching a physical object and is within a distance d from the physical object.

[0116] Moreover, in line with what is described above with regard to the type of the physical object, the method 200 may further provide for distinguishing between different instances of the same type of physical

[0117] 30 object. For example, if two (or more) persons (or any other type of physical object) are present in the physical environment surrounding the XR device, each person (i.e. each instance of the physical object of a certain type) may be tracked S204. Further, in response to the two persons entering the configured space, the provided feedback may be indicative of the different persons. For example, a different feedback may be provided for the different persons. P106566W001

[0118] 13

[0119] Further, the feedback to the user may be indicative of that the physical object is a person or an animal and / or that the person or animal is looking at the user. Put differently, the feedback may be indicative of that the physical object is a person or an animal and / or that the person or animal is looking at the user. One or more characteristics of the audio or haptic feedback can be adapted to indicate this. For

[0120] 5 example, a melody or tune of the audio feedback, or an intensity of the haptic feedback may change when the person or animal starts to look at the user. This may provide for that the user can be informed of that the physical object is a person or animal that may want the attention of the user. In particular, if the person or animal is looking at the user. The method 200 may comprise detecting S208 if the person or pet is looking at the user. This may e.g. be performed by image processing techniques, e.g. based on the use of neural networks.

[0121] The method 200 may further comprise receiving S202 user input of a specific feedback relating to a specific physical object. The user input may also be referred to as the input. In response to detecting said specific physical object, the feedback may be provided S210 as the specific feedback. Put differently, a specific feedback can be assigned to a specific physical object (e.g. a specific person). When detecting

[0122] 15 that a tracked physical object corresponds to the specific physical object, the specific feedback assigned to that specific physical object can be provided as the feedback to the user. Thereby, improved user experience can be achieved by providing a personalized feedback on an object level. Detecting the specific physical object may e.g. be performed by object detection / recognition algorithms or the like.

[0123] The feedback may be continuously provided to the user while the physical object is within the configured

[0124] 20 space. In other words, the feedback may be continuously provided while the physical object is within the configured space. Thus, the signal indicative of the feedback may be outputted to the audio and / or haptic system continuously while the physical object is within the configured space. More specifically, the physical object may be continuously tracked while it is within the configured space (or within the physical environment surrounding the XR device). The provided feedback may then be continuously adjusted (or updated) as the physical object moves around. Put differently, the feedback may be continuously adjusted to be indicative of the position of the physical object in relation to the XR device. The provided feedback may then be interrupted if / when the physical object leaves the configured space. Thus, the method 200 may be performed iteratively (or repeatedly), as illustrated in Fig. 2, until the physical object leaves the configured space 108 or physical environment surrounding the XR device.

[0125] 30 Executable instructions for performing these functions are, optionally, included in a computer-accessible medium such as a non-transitory computer-readable storage medium or other computer program product configured for execution by one or more processors. P106566W001

[0126] 14

[0127] Generally speaking, a computer-accessible medium may include any tangible or non-transitory storage media or memory media such as electronic, magnetic, or optical media— e.g., disk or CD / DVD-ROM coupled to computer system via bus. The terms "tangible" and "non-transitory," as used herein, are intended to describe a computer-readable storage medium (or "memory") excluding propagating

[0128] 5 electromagnetic signals, but are not intended to otherwise limit the type of physical computer-readable storage device that is encompassed by the phrase computer-readable medium or memory. For instance, the terms "non-transitory computer-readable medium" or "tangible memory" are intended to encompass types of storage devices that do not necessarily store information permanently, including for example, random access memory (RAM). Program instructions and data stored on a tangible computer- accessible storage medium in non-transitory form may further be transmitted by transmission media or signals such as electrical, electromagnetic, or digital signals, which may be conveyed via a communication medium such as a network and / or a wireless link.

[0129] Figure 3 is a schematic illustration of an electronic device 300 for providing feedback to a user of an XR device, according to some embodiments. In other words, the electronic device 300 is configured to

[0130] 15 provide feedback from the physical world. In particular, the electronic device 300 is configured to perform the techniques of the method 200 described in the foregoing in connection with Fig. 2.

[0131] The electronic device 300 comprises control circuitry 302. The control circuitry 302 may physically comprise one single circuitry device. Alternatively, the control circuitry 302 may be distributed over several circuitry devices. Functions and operations of the control circuitry 302 may thus be distributed

[0132] 20 over the different circuitry devices.

[0133] As shown in the example of Fig. 3, the electronic device 300 may further comprise a transceiver 306 and a memory 308. The control circuitry 302 is communicatively connected to the transceiver 306 and the memory 308. The control circuitry 302 may comprise a data bus. The control circuitry 302 may communicate with the transceiver 306 and / or the memory 308 via the data bus.

[0134] The control circuitry 302 may be configured to carry out overall control of functions and operations of the electronic device 300. The control circuitry 302 may be any suitable type of computation unit. The control circuitry 302 may comprise a processor 304, such as a central processing unit (CPU), microcontroller, microprocessor, digital signal processor (DSP), field programmable gate array (FPGA), application specific integrated circuit (ASIC) or any other form of circuit. The processor 304 may be

[0135] 30 configured to execute program code stored in the memory 308, in order to carry out functions and operations of the electronic device 300. The control circuitry 302 is configured to perform the steps of the method 200 as described above in connection with Fig. 2. The steps may be implemented in one or more functions stored in the memory 308. P106566W001

[0136] 15

[0137] The transceiver 306 may be configured to enable the electronic device 300 to communicate with other devices. The transceiver 306 may thus both transmit and receive data. Even though illustrated as a single unit, the transceiver 306 may be distributed over several transceiver units of the electronic device 300. The transceiver 306 may be configured to communicate over one or more communication

[0138] 5 protocol known in the art. Examples include, but are not limited to, long range radio communication technologies (e.g. cellular radio technologies such as GSM, GPRS, EDGE, LTE, LTE-Advanced, 5G, 5G NR, 6G, and so on), as well as short to mid-range technologies such as Wi-Fi, Bluetooth, Wireless Local Area (LAN), e.g. IEEE 802.11 etc. In particular, the electronic device 300 may be communicatively connected to the XR device.

[0139] 10 Alternatively, the electronic device 300 may be part of the XR device 400, as explained further below, in connection with Fig. 4. It is to be noted that some functions of the electronic device 300 described herein may, depending on a specific realization, be implemented in the XR device 400. For example, tracking a position of a physical object (as further described below) may be performed by the XR device 400. The device 300 may then determine a feedback to the user, and send instructions to the XR device

[0140] 15 400 to provide the user with said feedback. Put differently, the device 300 may determine the feedback, and output a signal indicative of the feedback to the XR device 400. More specifically, the signal may be outputted to an audio and / or haptic system of the XR device).

[0141] Even though not explicitly illustrated in Fig. 3, the electronic device 300 may further comprise means for receiving user input, such as one or more of a keyboard, a mouse, and a touchscreen etc. The

[0142] 20 electronic device 300 may further comprise means for displaying information (e.g. to the user), such as a display. The electronic device 300 may further comprise means for obtaining sensory data. The sensory data may be received from another device or sensory system. Alternatively, the electronic device 300 may comprise a sensory system configured to record the sensory data from one or more sensors of the sensory system. The sensory system (either being part of the electronic device 300 or communicatively connected thereto) may comprise sensors for perceiving a physical environment, such as one or more cameras, LIDAR, RADAR, etc. The sensory system may further comprise an inertial measurement unit.

[0143] The memory 308 may be configured to store received or transmitted data and / or executable program instructions. The memory 308 may be any suitable type of computer readable memory and may be of

[0144] 30 volatile and / or non-volatile type. The memory 308 may for instance be one or more of a buffer, a flash memory, a hard drive, a removable media, a volatile memory, a non-volatile memory, a random access memory (RAM), or another suitable device. The memory 308 may be a non-transitory computer- readable storage medium. In a typical arrangement, the memory 308 may include a non-volatile memory for long term data storage and a volatile memory that functions as system memory for the P106566W001

[0145] 16 electronic device 300. The memory 308 may exchange data with the circuitry 302 over the data bus. Accompanying control lines and an address bus between the memory 308 and the circuitry 302 also may be present.

[0146] Functions and operations of the electronic device 300 may be implemented in the form of executable

[0147] 5 logic routines (e.g., lines of code, software programs, etc.) that are stored on a non-transitory computer readable recording medium (e.g., the memory 308) of the electronic device 300 and are executed by the circuitry 302 (e.g. using the processor 304). Put differently, when it is stated that the circuitry 302 is configured to execute or perform a specific function or operation, the processor 304 of the circuitry 302 may be configured to execute program code portions stored on the memory 308,

[0148] 10 wherein the stored program code portions correspond to the specific function or operation. Furthermore, the functions and operations of the circuitry 302 may be a stand-alone software application or form a part of a software application that carries out additional tasks related to the circuitry 302. The described functions and operations may be considered a method which the corresponding device is configured to carry out, such as the method 200 discussed above with reference to Fig. 2. Also, while the described functions and operations may be implemented in software, such functionality may as well be carried out via dedicated hardware or firmware, or some combination of one or more of hardware, firmware, and software. In the following, the function and operations of the electronic device 300 is described.

[0149] The control circuitry 302 is configured to track a position of a physical object in a physical environment

[0150] 20 surrounding the extended reality device. Tracking the position of the physical object may comprise receiving sensory data of the physical environment from the XR device. As an example, the device 300 may receive image data depicting at least a part of the physical environment, and captured by one or more cameras of the XR device. The device 300 may further receive LIDAR or RADAR data of the physical environment. Alternatively, the device 300 may transmit instructions to one or more sensors of the XR device to collect desired sensory data. The control circuitry 302 may be further configured to process the sensory data in order to detect and track the physical object. The sensory data may e.g. be processed through known machine learning based object detection and / or tracking techniques. The sensory data may be continuously, or iteratively collected, in order to track the position of the physical object over time. Moreover, the control circuitry may store a position of the physical object in relation

[0151] 30 to the XR device.

[0152] The control circuitry 302 is further configured to, in response to the physical object entering a configured space around the extended reality device, provide a feedback to the user, indicative of the position of the physical object in relation to the extended reality device. Put differently, the control P106566W001

[0153] 17 circuitry 302 is further configured to, in response to the physical object entering a configured space around the extended reality device, provide a feedback, indicative of the position of the physical object in relation to the extended reality device. As mentioned earlier, the feedback may also be provided in response to the user of the extended reality device approaching the edge of a configured space, such

[0154] 5 as the configured space 108 near which a physical object is located, but which is outside of the configured space. Here, an as also mentioned earlier, the term "near" is defined as within a distance d from the edge of the configured space. Alternatively, the feedback may also be produced in case the configured space is anchored in the extended reality device worn by the user, such that the configured space moves with the user. When the configured space approaches a physical object outside of it and

[0155] 10 comes within a distance d from it, the feedback may also be produced.

[0156] The feedback is an audio and / or haptic feedback. The feedback may be provided by outputting a signal indicative of the feedback to an audio and / or haptic system of the XR device. The feedback may thus be provided to the user without modifying or interrupting a visual experience of the XR world. The feedback may be provided e.g. by transmitting instructions to an audio (e.g. headphones, or speakers)

[0157] 15 and / or haptic system (e.g. haptic motors) of the XR device. The configured space may be a defined play area. The configured space may be a user-defined play area. The control circuitry 302 may be configured continuously provide the feedback to the user while the physical object is within the configured space. In other words, the control circuitry 302 may be configured continuously provide the feedback while the physical object is within the configured space. Thus, the signal indicative of the

[0158] 20 feedback may be outputted continuously.

[0159] The control circuitry 302 may be further configured to, in response to the physical object being of a first type, provide a first feedback to the user, and in response to the physical object being of a second type, provide a second feedback to the user. Put differently, the control circuitry 302 may be further configured to, in response to the physical object being of a first type, provide a first feedback, and in response to the physical object being of a second type, provide a second feedback. Providing the first and second feedback may comprise outputting a first and second signal indicative of the first and second feedback respectively, to the audio and / or haptic system of the XR device.

[0160] The control circuitry 302 may be further configured to determine a type of the physical object. The feedback may then be further indicative of the type of object.

[0161] 30 The control circuitry 302 may be further configured to receive user input of a specific feedback relating to a specific object, and in response to detecting said specific object, provide the feedback as the specific feedback. The user input may be interchanged with input. P106566W001

[0162] 18

[0163] At least one audio characteristic of the audio feedback may be set based on the type of the physical object. Moreover, an audio characteristic of the audio feedback may be set based on the position of the physical object in relation to the extended reality device. The at least one audio characteristic may be selected from the group comprising of pitch, loudness, quality, tune and melody.

[0164] 5 The position of the physical object may comprise information about a movement direction and / or a distance of the physical object in relation to the extended reality device. The physical object may be a person, an animal or an inanimate object. Moreover, the feedback to the user may be indicative of that the physical object is a person or an animal and / or that the person or the animal is looking at the user. In other words, the feedback may be indicative of that the physical object is a person or an animal and / or that the person or the animal is looking at the user.

[0165] It should be appreciated that any features, principles, aspects and advantages of the method 200 as described above in connection with Fig. 2, are applicable also to the electronic device 300 described herein. To avoid undue repetition, reference is made to the above.

[0166] Figure 4 is a schematic illustration of an extended reality device 400 according to some embodiments.

[0167] 15 The XR device 400 is enabled for performing the principles of the method 200 for providing feedback of a physical world to a user of the XR device 400.

[0168] The XR device 400 may be any type of XR device commonly known in the art. In some embodiments, the XR device 400 is a head-worn device. As an example, the XR device 400 may be smart glasses. As another example, the XR device 400 may comprise a headset (as illustrated in Fig. 1). The XR device

[0169] 20 400 may further comprise one or more hand controllers. The one or more hand controllers may allow users to interact with virtual objects and navigate through virtual spaces. The hand controllers may employ various tracking technologies, such as infrared or optical sensors, to capture movements of the user's hands. The hand controllers may further comprise haptic feedback mechanisms, such as haptic motors. The haptic feedback mechanisms can further enhance the realism of the XR experience by providing tactile sensations and simulating the sense of touch. In some embodiments, the XR device 400 is a hand-held device. The XR device 400 may for instance be a smartphone, a tablet, or the like.

[0170] The XR device 400 may be a stand-alone device. This means that it incorporates the necessary hardware and software components for performing the intended task, within the device itself. Thus, reducing the need for additional wires or external devices. Alternatively, the XR device 400 may be a

[0171] 30 tethered, or wirelessly connected device. This means that the XR device 400 is connected (either by wired or wireless connection) to an external device. The external device may be seen as a secondary or P106566W001

[0172] 19 host device. The XR device can utilize processing capabilities of the external device to host or perform at least some of the functionalities of the XR world.

[0173] The XR device 400 comprises a display 402. The display 402 is configured to display an XR world to a user of the XR device 400. It is to be noted that the XR device 400 may comprise more than one

[0174] 5 display. For example, the XR device 400 may comprise two displays, one for each eye of the user.

[0175] The XR device 400 further comprises a sensory system 404. The sensory system 404 comprises one or more sensors 406. The one or more sensors 406 may comprise sensors for tracking the user's movement, such as an inertial measurement unit comprising e.g. an accelerometer, and a gyroscope. The one or more sensors 406 may further comprise sensors for perceiving the physical environment

[0176] 10 surrounding the XR device 400. Such sensors may comprise one or more cameras, LIDAR sensors, RADAR sensors, proximity sensors etc. Thus, the sensory system 404 may be configured to obtain sensory data for tracking a position of the physical object in the physical environment surrounding the XR device 400, as well as tracking a movement of the user. The sensory system 404 may comprise additional sensors than those described herein.

[0177] 15 Moreover, the XR device 400 comprises the electronic device 300 as described above in connection with Fig. 4. Thus, the device 300 enables the XR device to perform the functions of the method 200 described above in connection with Fig. 2. The electronic device 300 may be provided as a separate unit within the XR device 400. Alternatively, the electronic device 300 and the XR device 400 may have a common control circuitry for performing the overall control of the XR device 400, including the electronic device 300. In some embodiments, the electronic device 300 is provided as an external unit to the XR device 400, but communicatively connected thereto.

[0178] In the drawings and specification, there have been disclosed exemplary aspects of the disclosure. However, many variations and modifications can be made to these aspects without substantially departing from the principles of the present disclosure. Thus, the disclosure should be regarded as

[0179] 25 illustrative rather than restrictive, and not as being limited to the particular aspects discussed above. Accordingly, although specific terms are employed, they are used in a generic and descriptive sense only and not for purposes of limitation.

Claims

1. P106566W00120CLAIMS1. A method (200) for providing feedback to a user of an extended reality device, wherein the extended reality device is configured to display an extended reality environment to the user, the method (200) comprising:5 tracking (S204) a position of a physical object in a physical environment surrounding the extended reality device; in response to the physical object entering a space around the extended reality device, providing (S210) feedback to the user indicative of the position of the physical object in relation to the extended reality device, wherein the feedback comprises an audio and / or haptic feedback.

102. The method (200) according to claim 1, wherein providing (S210) the feedback to the user comprises: in response to the physical object being of a first type, providing (S212) a first type of feedback to the user.

153. The method (200) according to claim 1 or 2, wherein providing (S210) the feedback to the user comprises: in response to the physical object being of a second type, providing (S214) a second type of20 feedback to the user.

4. The method (200) according to claim 1, wherein the position comprises information about a movement direction and / or a distance of the physical object in relation to the extended reality device.

5. The method (200) according to any one of the claims 1 to 4, wherein the physical object is a person, an animal or an inanimate object.

6. The method (200) according to any one of the claims 1 to 5, wherein the feedback to the user is indicative of that the physical object is a person or an animal and / or that the person or animal is30 looking at the user.

7. The method (200) according to any one of the claims 1 to 5, wherein the feedback to the user is indicative of that the physical object is an inanimate object.P106566W001218. The method (200) according to any one of the claims 1 to 6, further comprising receiving (S202) user-defined feedback relating to a specific physical object, and in response to detecting said specific physical object, providing (S210) the feedback as the user-defined feedback.5 9. The method (200) according to to claim 1, wherein the space is a user-defined space.

10. The method (200) according to claims 9, wherein the feedback is continuously provided to the user while the physical object is within the space10 11. The method (200) according to any of the claims 1-10, wherein the feedback is defined by at least one audio characteristic of the audio feedback, and / or at least one haptic characteristic of the haptic feedback.

12. The method (200) according to any one of the claims claim 1 to 11, further comprising15 determining (S206) the type of the physical object, and wherein the feedback is indicative of the type of object.

13. The method (200) according to claim 11, wherein at least one audio characteristic of the audio feedback is based on the type of the physical object.2014. The method (200) according toclaim 11 or 13, wherein the at least one audio characteristic of the audio feedback is based on the position of the physical object in relation to the extended reality device.

15. The method (200) according to any of the claims 11 or 13-14, wherein the at least one audio characteristic is selected from the group comprising of pitch, loudness, quality, tune and melody.

16. The method (200) according to claim 11, wherein the at least one haptic characteristic of the haptic feedback is based on the type of the physical object.3017. The method (200) according to claim 11 or 16, wherein the at least one haptic characteristic of the haptic feedback is based on the position of the physical object in relation to the extended reality device.P106566W0012218. The method (200) according to any of the claims 11, 16 or 17, wherein the at least one haptic characteristic is selected from a group comprising of intensity, and pattern of the haptic feedback.5 19. A computer program product, comprising instructions which, when executed on at least one processor, cause the at least one processor to carry out the method (200) according to any one of the claims 1-18.

20. An electronic device (300) for providing feedback to a user of an extended reality device (400), wherein the extended reality device (400) is configured to display an extended reality environment to the user, wherein the electronic device (300) comprises control circuitry (302) configured to: track a position of a physical object in a physical environment surrounding the extended reality device;15 in response to the physical object entering a space around the extended reality device, provide a feedback to the user, indicative of the position of the physical object in relation to the extended reality device, wherein the feedback comprises an audio and / or haptic feedback.

21. An extended reality device (400) comprising: a display (402) for displaying an extended reality environment to a user of the extended reality device, a sensory system (404) for tracking a position of a physical object in a physical environment surrounding the extended reality device, and an electronic device (300) according to claim 20.

Images