Providing feedback to a user of an XR client

Abstract

A method of an eXtended Reality (XR) server (103) of enabling an XR client (102) to provide feedback to a user (101) of the XR client (102) interacting with a virtual object presented to the user (101) by the XR client (102) in a 3D space is provided. The method comprises receiving (S201), from the XR server (103), a visual representation of the virtual object, data indicating a pose of the virtual object (105) in the 3D space, a representation of a shape of the virtual object (105), and an indication of feedback to be provided by the XR client (102) upon the user (101) interacting with the virtual object (105), acquiring (S203) data indicating a pose and a shape of a body part of the user (101) in the 3D space, determining (S204) from the data indicating the pose and the shape of the body part, the pose of the virtual object (105), and the representation of the shape of the virtual object (105), that the user (101) is interacting with the virtual object (105), and providing (S205), to the user (101), the feedback indicated by the XR server (103).

Description

PROVIDING FEEDBACK TO A USER OF AN XR CLIENTTECHNICAL FIELD[oooi] The present disclosure relates to a method of an extended Reality (XR) client of providing feedback to a user of the XR client interacting with a virtual object presented by the XR client in a 3D space, and an XR client performing the method.

[0002] The disclosure further relates to a method of an extended Reality (XR) server of enabling an XR client to provide feedback to a user of the XR client interacting with a virtual object presented to the user by the XR client in a 3D space, and an XR server performing the method.

[0003] The disclosure further relates to corresponding computer programs and computer program products.BACKGROUND

[0004] extended Reality (XR) is an umbrella term that encompasses various immersive technologies, including Virtual Reality (VR), Augmented Reality (AR), and Mixed Reality (MR). These technologies aim to provide new and interactive experiences for users which merge the physical and digital worlds.

[0005] XR applications are processing-heavy. Typically, data which is to be displayed to a user of the XR application, such as frames remotely rendered by an XR server and displayed to user via an XR client in the form of for instance an XR headset worn by the user, is sent downstream from the XR server to an XR headset worn by the user.

[0006] For instance, consider a user who is playing a video game on an XR headset. Data from sensors mounted on the XR headset is sent upstream to the XR server, which computes a frame of a next XR scene to be sent downstream to the XR headset. The XR server then renders the XR scene, encodes the scene, and sends it to the XR headset. The XR headset decodes the XR scene and optionally performs some pose adjustments before displaying the adjusted XR scene to the user.

[0007] Hence, the user must wait for the XR server to reply before receiving an updated XR scene and / or interaction feedback, e.g., a haptic response if touching a virtual object. It should be noted that data sent downstream from the XR server to the XR headset may require a significant amount of bandwidth. This may negativelyimpact the user experience, in particular in the case of scarce network resources, as the (haptic) interaction feedback may come with noticeable delay.SUMMARY

[0008] One objective is to solve, or at least mitigate, this problem in the art and thus to provide an improved method of providing feedback to a user of an XR client upon the user interacting with virtual objects presented by the XR client.

[0009] This objective is attained in a first aspect by a method of an XR client of providing feedback to a user of the XR client interacting with a virtual object presented by the XR client in a 3D space. The method comprises receiving, from an XR server, a visual representation of the virtual object, data indicating a pose of the virtual object in the 3D space, a representation of a shape of the virtual object, and an indication of feedback to be provided by the XR client upon the user interacting with the virtual object. The method further comprises acquiring data indicating a pose and a shape of a body part of the user in the 3D space. The method further comprises determining from the data indicating the pose and the shape of the body part, the pose of the virtual object, and the representation of the shape of the virtual object, that the user is interacting with the virtual object. The method further comprises providing, to the user, the feedback indicated by the XR server.

[0010] This objective is attained in a second aspect by an XR client configured to provide feedback to a user of the XR client interacting with a virtual object presented by the XR client in a 3D space. The XR client comprises a processing unit and a memory. Said memory contains instructions executable by said processing unit, whereby the XR client is operative to receive, from an XR server, a visual representation of the virtual object, data indicating a pose of the virtual object in the 3D space, a representation of a shape of the virtual object, and an indication of feedback to be provided by the XR client upon the user interacting with the virtual object. The XR client is further operative to acquire data indicating a pose and a shape of a body part of the user in the 3D space. The XR client is further operative to determine from the data indicating the pose and the shape of the body part, the pose of the virtual object, and the representation of the shape of the virtual object, that the user is interacting with the virtual object. The XR client is further operative to provide, to the user, the feedback indicated by the XR server.[oon] This objective is attained in a third aspect by a method of an XR server of enabling an XR client to provide feedback to a user of the XR client interacting with a virtual object presented to the user by the XR client in a 3D space. The method comprises sending, to the XR client, a visual representation of the virtual object, data indicating a pose of the virtual object in the 3D space, a representation of a shape of the virtual object, and an indication of feedback to be provided to the user by the XR client upon the user interacting with the virtual object.

[0012] This objective is attained in a fourth aspect by an XR server configured to enable an XR client to provide feedback to a user of the XR client interacting with a virtual object presented to the user by the XR client in a 3D space. The XR server comprises a processing unit and a memory. Said memory contains instructions executable by said processing unit, whereby the XR server is operative to send, to the XR client, a visual representation of the virtual object, data indicating a pose of the virtual object in the 3D space, a representation of a shape of the virtual object, and an indication of feedback to be provided to the user by the XR client upon the user interacting with the virtual object.

[0013] Advantageously, since the XR client receives, from the XR server, a visual representation of a virtual object, data indicating a pose of the virtual object in a 3D space, a representation of a shape of the virtual object, and an indication of feedback to be provided by the XR client upon the user interacting with the virtual object, the XR client can provide feedback to the user immediately upon detecting that the user interacts with the virtual object. Hence, there is no need for the XR client to turn to the XR server upon detecting the interaction in order to be informed of the feedback to be provided. This reduces any delay in providing the feedback to the user and therefor enhances user experience.

[0014] In an embodiment, the acquiring of data indicating a pose and shape of a body part of the user in the 3D space comprises receiving the data from a sensor capturing movement of the user and / or one or more body parts of the user.

[0015] In an embodiment, the virtual object is presented to the user on a display connected to the XR client.

[0016] In an embodiment, the method further comprises sending, to the XR server, updated information indicating a changed pose and / or shape of the virtual object in response to the user’s interaction with the virtual object.

[0017] In an embodiment, the method further comprises receiving, from the XR server, an updated representation of the virtual object reflecting a change in pose and / or shape of the virtual object.

[0018] In an embodiment, the data indicating a pose of said virtual object in the 3D space comprises a position and / or an orientation of said virtual object in the 3D space.

[0019] In an embodiment, the user is determined to interact with the virtual object upon a body part of the user contacting the virtual object or moving within a threshold distance of the virtual object.

[0020] In an embodiment, the feedback is one or more of visual feedback, audial feedback, and haptic feedback.

[0021] In an embodiment, the data indicating a pose of the virtual object in the 3D space, the representation of a shape of the virtual object, and the indication of feedback to be provided to the user by the XR client, is sent to the XR client for one or more of a virtual object which is closer than a threshold distance from a body part of the user, a virtual object which is approaching a body part of the user or is approached by a body part of the user, and a virtual object which the user is likely to interact with.

[0022] In a fifth aspect, a computer program is provided comprising computerexecutable instructions for causing an XR client of the second aspect to perform steps recited in the method of the first aspect when the computer-executable instructions are executed on a processing unit included in the XR client.

[0023] In a sixth aspect, a computer program product is provided comprising a computer readable medium, the computer readable medium having the computer program according to the fifth aspect embodied thereon.

[0024] In a seventh aspect, a computer program is provided comprising computer-executable instructions for causing an XR server of the fourth aspect to perform steps recited in the method of the second aspect when the computerexecutable instructions are executed on a processing unit included in the XR server.

[0025] In an eighth aspect, a computer program product is provided comprising a computer readable medium, the computer readable medium having the computer program according to the seventh aspect embodied thereon.

[0026] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, step, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, step, etc., unless explicitly stated otherwise. The steps of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Aspects and embodiments are now described, by way of example, with reference to the accompanying drawings, in which:

[0028] Figure 1 illustrates an XR system in which embodiments may be implemented;

[0029] Figure 2 illustrates a user wearing an XR headset presenting a virtual object;

[0030] Figure 3 shows a signalling diagram illustrating a known method;

[0031] Figure 4 shows a signalling diagram illustrating a method of an embodiment;

[0032] Figure 5 shows a signalling diagram illustrating a method of a further embodiment;

[0033] Figure 6 illustrates an XR client according to an embodiment; and

[0034] Figure 7 illustrates an XR server according to an embodiment.DETAILED DESCRIPTION

[0035] The aspects of the present disclosure will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the invention are shown.

[0036] These aspects may, however, be embodied in many different forms and should not be construed as limiting. Rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and to fully convey the scope of all aspects of invention to those skilled in the art. Like numbers refer to like elements throughout the description.

[0037] Figure 1 schematically illustrates an XR system 100 comprising an XR server 103 supplying XR content, herein also referred to as XR scenes, to an XR client 102 in the form of an XR headset to be worn by a user (not shown), in which the described embodiments may be implemented. The XR headset 102 illustrated in Figure 1 is equipped with a display 104 for presenting the XR scenes provided by the XR server 103 to the user wearing the headset 102.

[0038] Thus, the natural field of view of the user is replaced with a computergenerated field of view in case of VR, and is augmented / enhanced in case of AR, and as a result, a virtual 3D space is presented to the user on the display 104 upon the user wearing the headset 102.

[0039] In the art, various XR technologies are known:• Virtual Reality (VR): VR is a fully immersive technology that places users in a computer-generated environment. Users wear headsets with built-in displays and sensors to experience the virtual world, often using handheld controllers for interaction.• Augmented Reality (AR): AR overlays digital content onto the user’s view of the real world, augmenting their perception of reality. AR can be experienced through various devices, such as head-mounted displays, smartphones, tablets, and smart glasses.• Mixed Reality (MR): MR combines elements of both VR and AR, allowing users to interact with digital objects that are seamlessly integrated into the real world. MR devices, such as the Microsoft HoloLens, use advanced sensors and cameras to map the environment and enable interactions by the user with virtual objects.

[0040] XR technologies have numerous applications across various industries, including entertainment, education, healthcare, manufacturing, and retail. As thesetechnologies continue to evolve, they are expected to play an increasingly significant role in shaping the future of human-computer interaction.

[0041] The 3rd Generation Partnership Project (3GPP) document TR 26928, V18.0.0, describes architectures to implement XR in the context of a 5thGeneration (5G) or subsequent-generation cellular network.

[0042] With reference to Figure 1, and further with reference to Figure 2, illustrating a user 101 wearing an XR headset 102 and being presented a virtual object 105, the XR processing is typically arranged such that the XR server 103 provides the XR client 102 with XR content, i.e., XR scenes, to be presented to a user 101 via the display 104 of the XR client 102 for creating the virtual 3D space.

[0043] In the XR scenes presented on the display 104 of the XR client 102, one or more virtual objects are displayed, such as the virtual spherical object 105 illustrated in Figure 2.

[0044] Upon the user 101 reaching out to touch the virtual object 105 in the 3D space, the XR system 100 commonly provides haptic feedback, such as a vibration caused via, e.g., a handset or haptic glove worn by the user 101. A common application in XR is that of users playing video games presented on the display 104 of the XR client 102. Haptic feedback in the form of for instance vibrations may thus be provided to the user via a video game controller held by the user and acting as a user input device for playing the video game. Other types of feedback may also be provided, such as audial or visual feedback provided via the XR client 102. A combination of audial, visual, and haptic, feedback is envisaged.

[0045] Movements of the user are in the art tracked by the XR system 100, for instance using a camera, lidar, etc., of the XR client 102, possibly in combination with an accelerometer of the XR client 102, accelerometers of a handheld controller, and / or sensors of a glove worn by the user, and the tracking data is sent to the XR server 103 to allow the XR server 103 to determine that the user 101 touches the virtual object 105. In response thereto, the XR server 103 sends haptic feedback data to the XR client 102 and the XR client 102 in turn provides the haptic feedback to the user 101, such as causing a handset to vibrate or a haptic glove rendering haptic feedback to the user’s hand, as if the hand had touched a real, physical world object of the same shape as the virtual object 105.

[0046] However, a problem with this approach is that the haptic feedback data is generated at the XR server 103 and sent downstream to the XR client 102 in response to the XR client 102 sending the user tracking data in upstream to the XR server 103. Thereby, a substantial delay between the instant of the user 101 touching the virtual object 105 and the instant of the XR client 102 providing the haptic feed, e.g., by causing the handset to vibrate, may occur. Such delay may, e.g., be caused by network congestion and / or server overloading.

[0047] Figure 3 illustrates the scenario of Figures 1 and 2 in more detail, where the user 101 of the XR client 102 grabs the virtual 3D object 105 being presented to the user 101 by the XR client 102 via the display 104. As mentioned, the XR client 102 will provide haptic feedback to the user 101 upon the user 101 interacting with the virtual 3D object 105, for instance via a handset being in communication with the XR client 102 and held by the user 101.

[0048] In a first step S101, the user 101 performs a movement with her hand and touches the virtual object presented on the display 104 by the XR client 102, i.e., the user’s hand intersects with a surface of the virtual object. This movement may e.g., be tracked by a camera or other sensor of the XR client 102, a handheld controller, or a glove worn by the user 101. Data indicating a position (and / or pose), or a change in position (and / or pose), of the user’s hand in 3D space is hence registered by the XR client 102. In response thereto, the XR client 102 sends the hand position data (which is herein understood to optionally include pose data) upstream to the XR server 103 in step S102.

[0049] The XR server 103 determines in step S103 from the hand position data whether or not the user’s hand intersects with a surface of the virtual object, the 3D pose and shape of which is known to the XR server 103, whereupon the XR server 103 concludes that the user 101 indeed touches, or interacts with, the virtual object 105 in the XR scene previously provided by the XR server 103 and displayed by the XR client 102, and if so provides suitable haptic feedback data (e.g., an instruction "cause handset to vibrate”). The XR server 103 may optionally compute a (next) frame of an updated XR scene to be presented by the XR client 102 based on the hand movement information and send the updated XR scene downstream to the XR client 102 in step S104, in addition to data representing the haptic feedback. As is understood, depending on the particular XR system being used, upon providing updated XRscenes to the XR client 103, the XR server 103 may send complete frames of new XR scenes or just provide a difference (referred to as delta) between frames of the old XR scenes and the new XR scenes, similar to what is known from video coding technology.

[0050] Upon receiving the data representing the haptic feedback to be provided to the user 101 from the XR server 103 in step S104, the XR client 102 displays the updated XR scene to the user 101 via the XR client 102 and further provides haptic feedback to the user 101, e.g., in the form of a handset vibration in step S105, thereby indicating that the user 101 indeed touches the virtual object 105 presented by the XR client 102 via the display 104.

[0051] As previously mentioned, data representing the XR scenes sent downstream from the XR server 103 to the XR client 102 in step S104 may require a significant amount of bandwidth, and in the case of scarce network resources, this may negatively impact the user experience, in particular the interaction feedback - in this example in the form of remote controller vibration - which may come with noticeable delay. As is understood, hand position data sent in S102 also requires provision of network bandwidth, however far less than sending an XR scene.

[0052] Figure 4 shows a signalling diagram illustrating a method of an XR client according to an embodiment of providing feedback to a user regarding an object with which the user interacts, for mitigating the above stated issue.

[0053] The XR client 102 presents one or more visual representations in the form of XR scenes to the user 101 via the display 102, which XR scenes have been provided to the XR client 102 by the XR server 103. In the presented XR scenes, there may be one or more virtual objects 105 with which the user 101 potentially may interact by touch with a hand or other body part.

[0054] In contrast to the approach of Figure 3, the XR server 103 sends in step S201 data indicating a pose of the virtual object 105 with which the user 101 potentially may interact, a representation of the shape of the virtual object 105 (e.g., a sphere with diameter 10 cm), and an indication of feedback to be provided by the XR client 102 upon the user 101 interacting with the object 105 occurring in the XR scene presented on the display 104 of the XR client 102. In this particular example, the feedback is embodied in the form of vibration of a handset held by the user 101.Alternatively, the feedback may, e.g., be haptic feedback provided by a haptic glove which the user 101 is wearing, the haptic feedback being similar to the sensation of touch if the user 101 would touch a physical, real-world object corresponding to the virtual object 105.

[0055] The representation of the shape of the virtual object 105 may, for instance, for a spherical object include data indicating that the virtual object indeed is a sphere and a radius of the sphere. The spherical object can then be represented in 3D space since its position also is known.

[0056] In an example of a cubical object being presented, data indicating side length of the cubical object may be provided along with its pose, i.e., its position and orientation in 3D space.

[0057] As is understood, there are numerous known ways which can be utilized for representing the pose and shape of virtual objects in 3D space, such as for instance 3D point clouds, 3D meshes, voxel-based models, parametric models, depth maps, etc. In other words, in these approaches, information is provided from which the shape of an object may be attained.

[0058] Now, the XR server 103 may conclude from one or more previous XR scenes, or based on rules of a game which the user 101 is playing, that the user 101 is about to interact with the virtual object 105 and thus anticipates that the XR client 102 will be required to react to the interaction by providing haptic feedback, at which instant the XR client 102 has already received the required data of step S201. In another example, the data of S201 is sent by the XR server 103 for any virtual object in a presented XR scene that the user 101, or a body part of the user, is approaching, or any virtual object which moves towards a body part of the user. For example, as soon as the user 101 moves within a 20 cm range of a virtual object, the XR server 103 may send the data of step S201.

[0059] The XR client 102 may not necessarily require that the user 101 touches the virtual object 105 for the haptic feedback to be provided. Rather, the XR client 102 may provide a vibration when the user 101 approaches the object 105, e.g., if the user 101 moves within a 5-cm range of the virtual object 105. However, in this particular example, the user 101 is considered to interact with the object 105 upon actually touching the virtual object 105.

[0060] In response to the user 101 performing a movement with her hand in step S202 and touching the virtual object 105 presented via the display 104 of the XR client 102, the XR client 102 acquires (via e.g., its camera) data indicating a pose and shape of the user’s hand in 3D space in step S203.

[0061] Advantageously, based on the information received from the XR server 103 in step S201, the XR client 102 is already aware of the pose and the shape of the object 105.

[0062] As a result, in step S204, the XR client 102 determines from the hand position data acquired in step S203 representing the pose and shape of the hand and the data indicating the shape and pose of the object 105 received in step S201 that the user’s hand intersects with a surface of the virtual object 105, the 3D pose and shape of which is known to the XR client 102 from the data received in S201, whereupon the XR client 102 concludes that the user 101 indeed is interacting with the virtual object 105.

[0063] In response to detecting the interaction in step S204, the XR client 102 provides haptic feedback to the user 101 in step S205, in accordance with the indication of feedback to be provided by the XR client 102 upon the user 101 interacting with the virtual object 105, e.g., in the form of a vibration of the handset held by the user 101, thereby indicating that the user 101 indeed touches the virtual 3D object 105 presented via the display 104 of the XR client 102.

[0064] In embodiments, the XR client 102 may not necessarily require that the user 101 touches the virtual object 105 for the haptic feedback to be provided. For instance, the XR client 102 may provide a vibration when the user 101 approaches the object 105, e.g., if the user 101 moves within a 5-cm range of the virtual object 105, which becomes stronger the closer the user 101 moves to the object 105.

[0065] The embodiment of Figure 4 is advantageous for numerous reasons:- The user 101 of the XR client 102 is given feedback, e.g., haptic or sound, with a reduced delay after he / she engages with the object 105 occurring in the presented XR scene. Avoiding a round-trip to the XR server 102 will greatly improve user experience.- The XR client 102 will have knowledge of the shape and position of objects occurring in the presented XR scene.- The XR client 102 may execute interaction feedback based on the instructions provided by the XR server 103, which further allows the XR server 103 to, at least to some extent, select when to provide the information to the XR client 102 in case bandwidth temporarily is scarce.- The processing performed by the XR client 102 requires a fraction of the processing power required by the XR server 103, since user-object interaction detection is a lightweight processing task, requiring the XR client 102 to determined that an outer position of the user’s hand coincides with a position of the object’s surface in 3D space. The advantages in terms of user experience are significant as compared to turning to the XR server 103 each time user-object interaction occurs.- The XR server 103 can offload the user-object interaction detection task to the XR client 102 and save computing resources in an effective way.

[0066] With the spherically shaped virtual object 105 illustrated in Figure 2, it is sufficient for the XR server 103 to provide the position of the object 105 in 3D space along with its shape in S201, in addition to the feedback data. That is, position and shape (including a size) of the spherical object 105 is sufficient to determine that the user 101 touches the object 105 if the user’s hand position data is known.

[0067] In case of a non-spherical virtual object, e.g., in the form of a cube, the pose information received in S201 is required to comprise both a position and orientation of the cube in 3D space, as well as its shape. For instance, the user 101 may approach the cube at a corner from one angle, while approaching the cube from a side from another. Thus, depending on the shape of the virtual object, the pose information supplied by the XR server 103 in S201 may have to include data indicating both the position and the orientation of the object in 3D space.

[0068] Further, while the user 101 in the above-described embodiments is exemplified to interact with the virtual object 105 using a hand, other body parts may be envisaged for interacting with the object 105, such as a finger, a foot, or the head, of the user 101. In such an example, the XR client 102 will in step S203, using its camera or other sensors, acquire data indicating a pose and shape of the finger, foot, or head, respectively, in 3D space.

[0069] Figure 5 illustrates that, in embodiments, the XR client 102 may send updated information about user interactions with the virtual object 105 (or other virtual objects) to the XR server 103 in step S206. This may be required if it is expected that the object 105 will change shape and / or pose in response to the user interaction with the object 105. As an example, trajectory information (i.e. position vs. time for a movement) may be sent to the XR server 103 in S206, e.g., when a body part touches a virtual object, for instance in an AR ball game.

[0070] As a response, in step S207, the XR server 103 may send an updated XR scene back to the XR client 102. This updated scene includes the result of user interactions with the virtual object with regard to changed object pose and / or shape and potentially further virtual objects located in the 3D space provided to the user via the display 104 of the XR client 102.

[0071] A further indicated in Figure 5 with the dotted arrow from step S207 back to step S202, steps S202-S207 may be performed repetitively with the XR client 102 providing haptic feedback the user 105 interacting with one or more objects 105 in the updated XR scenes presented via the display 104 of the XR client 102, while it may be envisaged that that the sending in S201 of the object pose and shape along with the haptic feedback information is performed only once for all the rounds of presenting updated XR scenes to the user 101 and providing haptic feedback accordingly related to object interaction in the updated XR scenes.

[0072] In other words, while step S201 may occur only once, steps S202-S207 may be repeated until the XR application has finished. As is understood, step S201 may also occur multiple times. It may be advantageous to send 3D object information only when required, since such information may represent complex objects and be large in terms of data amount to transfer.

[0073] The data sent from the XR server 103 in S201 indicating the pose of the virtual object 105 in the 3D space, a representation of a shape of the virtual object 105, and feedback to be provided by the XR client 102 upon the user 101 interacting with the virtual object 105 may be sent in a Wavefront Object (OBJ) file format. As is understood, several other data formats are also available, e.g., the Filmbox (FBX) file format or Graphics Language Transmission Format (gITF).However, the OBJ format can be exported by most used 3D modelling software, e.g.,Blender, and is widely supported by all the major game engines used to develop XR applications, e.g., Unity, Unreal, Godot.

[0074] The OBJ format is a text-based format that contains information about the geometry of the 3D object to model. For example, geometric data is defined using vertex coordinates, which represent the points in 3D space that make up the model’s shape. Normals define the orientation of the surface at each vertex, aiding in rendering and lighting calculations. Texture coordinates determine how the model’s surface is mapped with textures.

[0075] An OBJ file representing a cube may look like the code below:# Vertices v 0.0 0.00.0 # l a v 0.0 1.00.0 # 2 b v 1.0 1.00.0 # 3 c v 1.00.00.0 # 4 d v o.o 0.0 1.0 # 5 e v o.o 1.0 1.0 # 6 f v 1.0 1.0 1.0 # 7 g v 1.00.0 1.0 # 8 h# Faces# Face 1: cghd = cgh + chd f 378 f 384# Face 2: aefb = aef + afb f i 56 f 16 2# Face 3: gcbf = gcb + gbf f 73 2f 726# Face 4: dhea = dhe + dea£485 f45 i# Face 5: hgfe = hgf + hfe£876£8 65# Face 6: cdab = cda + cab f 34 i£3 12

[0076] In the example above, a cube is represented in a 3D space, where the v lines represent vertices, specifying the 3D coordinates of each point, while the f lines define the faces of the cube.

[0077] Overall, the simplicity and compatibility of OBJ files make them easy to read, edit, and share, among different 3D modelling software, and serve as a standard format for exchanging 3D models, facilitating collaboration and interoperability across various platforms and applications.

[0078] Figure 6 illustrates an XR client 102, such as an XR headset or XR glasses, according to an embodiment, where the steps of the method performed by the XR client 102 in practice are performed by a processing unit 111 embodied in the form of one or more microprocessors arranged to execute a computer program 112 downloaded to a storage medium 113 associated with the microprocessor, such as a Random Access Memory (RAM), a Flash memory or a hard disk drive. The processing unit 111 is arranged to cause the XR client 102 to carry out the method according to embodiments when the computer program 112 comprising computer-executable instructions is downloaded to the storage medium 113 and executed by the processingunit in. The storage medium 113 may also be a computer program product comprising the computer program 112. Alternatively, the computer program 112 may be transferred to the storage medium 113 by means of a suitable computer program product, such as a Digital Versatile Disc (DVD) or a memory stick. As a further alternative, the computer program 112 may be downloaded to the storage medium 113 over a network. The processing unit 111 may alternatively be embodied in the form of a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a complex programmable logic device (CPLD), etc.

[0079] The XR client 102 may further comprise a display 104 via which XR content is presented to the user 101, and a communication interface 114 (wired and / or wireless) over which the XR client 102 is configured to transmit and receive data, in particular to / from the XR server 104. Moreover, the XR client 102 may be equipped with one or more sensors 115 in the form of cameras, lidars, etc., for registering movements of the user 101 and the user’s body parts. The XR client 102 may further be equipped with, or may be in connection with, a device 116 such has a handset via which the XR client 102 may provide the user 101 with feedback, e.g., haptic feedback in the form of vibrations. The handset 116 may also serve as user input data device to the XR client 102.

[0080] Figure 7 illustrates an XR server 104 according to an embodiment, where the steps of the method performed by the XR server 104 in practice are performed by a processing unit 211 embodied in the form of one or more microprocessors arranged to execute a computer program 212 downloaded to a storage medium 213 associated with the microprocessor, such as a RAM, a Flash memory or a hard disk drive. The processing unit 211 is arranged to cause the XR server 104 to carry out the method according to embodiments when the computer program 212 comprising computerexecutable instructions is downloaded to the storage medium 213 and executed by the processing unit 211. The storage medium 213 may also be a computer program product comprising the computer program 212. Alternatively, the computer program 212 may be transferred to the storage medium 213 by means of a suitable computer program product, such as a DVD or a memory stick. As a further alternative, the computer program 212 may be downloaded to the storage medium 213 over a network. The processing unit 211 may alternatively be embodiedin the form of a DSP, an ASIC, an FPGA, a CPLD, etc. The XR server 104 further comprises a communication interface 214 (wired and / or wireless) over which the XR server 104 is configured to transmit and receive data.

[0081] The aspects of the present disclosure have mainly been described above with reference to a few embodiments and examples thereof. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the invention, as defined by the appended patent claims.

[0082] Thus, while various aspects and embodiments have been disclosed herein, other aspects and embodiments will be apparent to those skilled in the art. The various aspects and embodiments disclosed herein are for purposes of illustration and are not intended to be limiting, with the true scope and spirit being indicated by the following claims.

Claims

CLAIMS1. A method of an extended Reality, XR, client (102) of providing feedback to a user (101) of the XR client (102) interacting with a virtual object presented by the XR client (102) in a 3D space, comprising: receiving (S201), from an XR server (103), a visual representation of the virtual object, data indicating a pose of the virtual object (105) in the 3D space, a representation of a shape of the virtual object (105), and an indication of feedback to be provided by the XR client (102) upon the user (101) interacting with the virtual object (105); acquiring (S203) data indicating a pose and a shape of a body part of the user (101) in the 3D space; determining (S204) from the data indicating the pose and the shape of the body part, the pose of the virtual object (105), and the representation of the shape of the virtual object (105), that the user (101) is interacting with the virtual object (105); and providing (S205), to the user (101), the feedback indicated by the XR server (103).

2. The method of claim 1, the acquiring (S203) of data indicating a pose and shape of a body part of the user (101) in the 3D space comprising: receiving the data from a sensor (115) capturing movement of the user (101) and / or one or more body parts of the user (101).

3. The method of claims 1 or 2, the virtual object (105) being presented to the user (101) on a display (104) connected to the XR client (102).

4. The method of any one of claims 1 to 3, further comprising: sending (S206), to the XR server (103), updated information indicating a changed pose and / or shape of the virtual object (105) in response to the user’s interaction with the virtual object (105).

5. The method of claim 4, further comprising: receiving (S207), from the XR server (103), an updated representation of the virtual object (105) reflecting a change in pose and / or shape of the virtual object (105).

6. The method of any one of claims 1 to 5, the data indicating a pose of said virtual object (105) in the 3D space comprising a position and / or an orientation of said virtual object (105) in the 3D space.

7. The method of any one of claims 1 to 6, wherein the user (101) is determined (S203) to interact with the virtual object (105) upon a body part of the user (101) contacting the virtual object (105) or moving within a threshold distance of the virtual object (105).

8. The method of any one of claims 1 to 7, the feedback being one or more of visual feedback, audial feedback, and haptic feedback.

9. A method of an extended Reality, XR, server (103) of enabling an XR client (102) to provide feedback to a user (101) of the XR client (102) interacting with a virtual object presented to the user (101) by the XR client (102) in a 3D space, comprising: sending (S201), to the XR client (102), a visual representation of the virtual object (105), data indicating a pose of the virtual object (105) in the 3D space, a representation of a shape of the virtual object (105), and an indication of feedback to be provided to the user (101) by the XR client (102) upon the user (101) interacting with the virtual object (105).

10. The method of claim 9, wherein the data indicating a pose of the virtual object (105) in the 3D space, the representation of a shape of the virtual object (105), and the indication of feedback to be provided to the user by the XR client (102), is sent (S201) to the XR client (102) for one or more of: a virtual object (105) which is closer than a threshold distance from a body part of the user (101), a virtual object (105) which is approaching a body part of the user (101) or is approached by a body part of the user (101), and a virtual object (105) which the user is likely to interact with.

11. The method of claims 9 or 10, further comprising: receiving (S206) from the XR client (102), updated information indicating a changed pose and / or shape of the virtual object (105) in response to the user’s interaction with the virtual object (105).

12. The method of claim 11, further comprising: sending (S207), to the XR client (102), an updated representation of the virtual object reflecting a change in pose and / or shape of the virtual object.

13. A computer program (112) comprising computer-executable instructions for causing an XR client (102) to perform steps recited in any one of claims 1-8 when the computer-executable instructions are executed on a processing unit (111) included in the XR client (102).

14. A computer program product comprising a computer readable medium (113), the computer readable medium having the computer program (112) according to claim 13 embodied thereon.

15. A computer program (212) comprising computer-executable instructions for causing an XR server (103) to perform steps recited in any one of claims 10-12 when the computer-executable instructions are executed on a processing unit (211) included in the XR server (103).

16. A computer program product comprising a computer readable medium (213), the computer readable medium having the computer program (212) according to claim 15 embodied thereon.

17. An extended Reality, XR, client (102) configured to provide feedback to a user (101) of the XR client (102) interacting with a virtual object presented by the XR client (102) in a 3D space, the XR client (102) comprising a processing unit (111) and a memory (113), said memory containing instructions (112) executable by said processing unit (111), whereby the XR client (102) is operative to: receive, from an XR server (103), a visual representation of the virtual object, data indicating a pose of the virtual object (105) in the 3D space, a representation of a shape of the virtual object (105), and an indication of feedback to be provided by the XR client (102) upon the user (101) interacting with the virtual object (105); acquire data indicating a pose and a shape of a body part of the user (101) in the 3D space; determine from the data indicating the pose and the shape of the body part, the pose of the virtual object (105), and the representation of the shape of the virtualobject (105), that the user (101) is interacting with the virtual object (105); and provide, to the user (101), the feedback indicated by the XR server (103).

18. The XR client (102) of claim 17, further being operative to acquire data indicating a pose and shape of a body part of the user (101) in the 3D space by: receiving the data from a sensor (115) capturing movement of the user (101) and / or one or more body parts of the user (101).

19. The XR client (102) of claims 17 or 18, further being operative to present the virtual object (105) to the user (101) on a display (104) connected to the XR client (102).

20. The XR client (102) of any one of claims 17 to 19, further being operative to: send, to the XR server (103), updated information indicating a changed pose and / or shape of the virtual object (105) in response to the user’s interaction with the virtual object (105).

21. The XR client (102) of claim 20, further being operative to: receive, from the XR server (103), an updated representation of the virtual object (105) reflecting a change in pose and / or shape of the virtual object (105).

22. The XR client (102) of any one of claims 17 to 21, the data being configured to indicate a pose of said virtual object (105) in the 3D space comprising a position and / or an orientation of said virtual object (105) in the 3D space.

23. The XR client (102) of any one of claims 17 to 22, further being operative to determine that the user (101) interacts with the virtual object (105) upon a body part of the user (101) contacting the virtual object (105) or moving within a threshold distance of the virtual object (105)24. The XR client (102) of any one of claims 17 to 23, the feedback being configured to comprise one or more of visual feedback, audial feedback, and haptic feedback.

25. An extended Reality, XR, server (103) configured to enable an XR client (102) to provide feedback to a user (101) of the XR client (102) interacting with a virtual object presented to the user (101) by the XR client (102) in a 3D space, the XR server (103) comprising a processing unit (211) and a memory (213), said memory containing instructions (212) executable by said processing unit (211), whereby the XR server (103) is operative to:send, to the XR client (102), a visual representation of the virtual object (105), data indicating a pose of the virtual object (105) in the 3D space, a representation of a shape of the virtual object (105), and an indication of feedback to be provided to the user (101) by the XR client (102) upon the user (101) interacting with the virtual object (105).

26. The XR server (103) of claim 25, further being operative to send the data indicating a pose of the virtual object (105) in the 3D space, the representation of a shape of the virtual object (105), and the indication of feedback to be provided to the user by the XR client (102) to the XR client (102) for one or more of: a virtual object (105) which is closer than a threshold distance from a body part of the user (101), a virtual object (105) which is approaching a body part of the user (101) or is approached by a body part of the user (101), and a virtual object (105) which the user is likely to interact with.

27. The XR server (103) of claims 25 or 26, further being operative to: receive from the XR client (102), updated information indicating a changed pose and / or shape of the virtual object (105) in response to the user’s interaction with the virtual object (105).

28. The XR server (103) of claim 27, further being operative to: sending, to the XR client (102), an updated representation of the virtual object reflecting a change in pose and / or shape of the virtual object.

Images