A method and device for rendering extended reality objects

Abstract

A method for rendering Extended Reality, XR, objects for an XR application for an XR device is disclosed. The method is performed in a control device and comprises determining at least one property of one or more XR objects in an XR environment; determining at least one property of an XR rendering system providing the one or more XR objects; and controlling rendering of the one or more XR objects to be performed locally and / or remotely based on one or both of: the determined properties of the one or more XR objects, and the one or more properties of the XR rendering system. A corresponding device, computer program and computer program product are also disclosed.

Description

[0001] A METHOD AND DEVICE FOR RENDERING EXTENDED REALITY OBJECTS

[0002] TECHNICAL FIELD

[0003] The technology disclosed herein relates generally to the field of Extended Reality, and in particular to devices and methods for rendering Extended Reality, XR, objects for an XR application for an XR device

[0004] BACKGROUND

[0005] An Extended Reality (XR) is rendered by converting three-dimensional (3D) objects into two-dimensional (2D) stereoscopic images, which are displayed in head mounted displays (HMDs). The rendering may be done locally in the same device in which the 3D objects are to be displayed (e.g., e.g., XR-glasses / display), or in a separate unit linked to the device. As a further example, the rendering may be done remotely in a server that is located e.g., in the cloud or edge cloud. Telefonaktiebolaget LM Ericsson has presented ideas to offload the rendering to edge cloud servers over 5G in order to provide more rendering power and therefore increase the quality or frequency of rendered images.

[0006] It is known in the industry and academia to use hybrid rendering, i.e., rendering some 3D-objects locally and some objects (other or same) remotely and then combine the rendered objects locally. The hybrid combination could be a simple image overlay or a more elaborate object occlusion and different model fidelity aware combinations. This may be done in order to render user interface locally while the environment or avatars are rendered remotely. Remote rendering may provide faster rendered objects with higher quality but will also be more prone to encounter problems due to network congestion, disconnection, or fluctuating latency.

[0007] From the above it is realized that there is a need for improvements in rendering 3D objects, and in particular in relation to such rendering using hybrid solutions.

[0008] SUMMARY

[0009] An objective of embodiments herein is to address and improve various aspects relating to hybrid rendering of Extended Reality (XR) objects. A particular objective is to provide a method for hybrid rendering overcoming the prior art problems of planning. Another particular objective is to enable a dynamic rendering in that the objects to be rendered locally and remotely, respectively, can be dynamically changed based on various criteria. A still further objective is to enable an uninterrupted XR session, by dynamically changing between remote rendering and local rendering depending on current network connectivity status. These objectives and others are achieved by the methods, devices, computer programs and computer program products according to the appended independent claims, and by the embodiments according to the dependent claims.

[0010] According to a first aspect there is presented a method for rendering XR objects for an XR application for an XR device. The method is performed in a control device and comprises determining at least one property of one or more XR objects in an XR environment; determining at least one property of an XR rendering system that provides the one or more XR objects; and controlling the rendering of the one or more XR objects to be performed locally and / or remotely based on one or both of: the determined properties of the one or more XR objects, and the one or more properties of the XR rendering system.

[0011] According to a second aspect there is presented a computer program for control device. The computer program comprises computer code which, when run on processing circuitry of the control device, causes the control device to perform a method according to the first aspect.

[0012] According to a third aspect there is presented a computer program product comprising a computer program as above, and a computer readable storage medium on which the computer program is stored.

[0013] According to a fourth aspect there is presented a control device for rendering Extended Reality, XR, objects for an XR application that is run in an XR device. The control device is configured to: determine one or more properties of one or more XR objects in an XR environment; determine one or more properties of an XR rendering system providing the one or more XR objects; and control rendering of the one or more XR objects to be performed locally based on one or both of: the determined properties of the one or more XR objects, and the one or more properties of the XR rendering system. Advantageously, these aspects provide an improved XR experience for a user in providing an increased resilience against connection issues, e.g., when rendering the XR objects remotely. An enhanced performance in view of latency and fidelity of rendered 3D-objects is also provided. Further, the provided methods and devices enables an adaptive dynamic onloading for XR rendering optimized based on various parameters, such as device capabilities (energy and resources), network conditions and reliability, object priority, rendering quality loss, and rendering frequency loss.

[0014] Other objectives, features and advantages of the enclosed embodiments will be apparent from the following detailed disclosure, from the attached dependent claims as well as from the drawings.

[0015] Generally, all terms used in the claims are to be interpreted according to their ordinary meaning in the technical field, unless explicitly defined otherwise herein. All references to "a / an / the element, apparatus, component, means, module, action, etc." are to be interpreted openly as referring to at least one instance of the element, apparatus, component, means, module, action, etc., unless explicitly stated otherwise. The actions of any method disclosed herein do not have to be performed in the exact order disclosed, unless explicitly stated.

[0016] BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The inventive concept is now described, by way of example, with reference to the accompanying drawings, in which:

[0018] Fig. 1 illustrates a system according to embodiments.

[0019] Fig. 2 is a block diagram illustrating embodiments of property retrieval.

[0020] Fig. 3 is a flowchart of various embodiments of a method.

[0021] Fig. 4 is a schematic diagram showing functional units of a device according to an embodiment.

[0022] Fig. 5 is a schematic diagram showing functional modules of a device according to an embodiment. Fig. 6 shows one example of a computer program product comprising computer readable means according to an embodiment.

[0023] DETAILED DESCRIPTION

[0024] The inventive concept will now be described more fully hereinafter with reference to the accompanying drawings, in which certain embodiments of the inventive concept are shown. This inventive concept may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein; rather, these embodiments are provided by way of example so that this disclosure will be thorough and complete, and will fully convey the scope of the inventive concept to those skilled in the art. Like numbers refer to like elements throughout the description. Any action or feature illustrated by dashed lines should be regarded as optional.

[0025] Briefly, method and means such as devices, software and / or hardware, are disclosed for providing an improved XR hybrid rendering, and in particular, a local and / or remote XR rendering. In contrast to the known hybrid solutions, which are static, the present teachings provide a dynamic hybrid solution. A dynamic evaluation of split rendering is provided and based on different priorities. Moreover, depending on device capabilities and energy constraints, local rendering might or might not be preferred. Moreover, depending on current network- or application context conditions priority of which objects to render locally or remotely may be changed over time.

[0026] Priority and degree of importance of an XR object changes during an XR session. For instance, when an object, such as a tool or avatar, is currently unused or far away, it is typically less important, it is still part of the environment but has a lower priority. However, when the object is picked up or interacted with by a user (user avatar) it has a higher priority. According to one aspect of the present teachings, for an object with high enough priority it may be decided that is should be rendered locally in order to avoid potential downside(s) of remote rendering, such as, for instance, connectivity issues or potentially longer latency. A priority threshold for local rendering may be derived based on parameters such as local processing capacity, expected battery duration time remaining, probability of connectivity issues, object expected rendering speed (movement speed), proximity to the user (objects closer to the user may need to be updated faster than objects far away from the user), required level of detail etc. A valuable advantage that is provided is that remote rendering may be used continuously for most objects, while key objects may be rendered locally during (unexpected) connectivity issues. This allows almost unnoticeable bridging of temporary increased latency during e.g., handovers, without implicating a large, complicated offload / unoffload event as in prior art.

[0027] As a particular example, when network reliability is questionable and / or energy and / or resources of a local device is plentiful, then more objects may be rendered locally. On the other hand, when connectivity reliability can, at least to some degree, be guaranteed and / or a remote rendering largely benefits quality and / or rendering speed owing to e.g., that device resources are constrained, then the decision threshold may be changed such that only highly critical objects are rendered locally.

[0028] Fig. 1 illustrates a hybrid XR rendering system according to embodiments. Extended reality (XR), as used herein, comprises augmented reality (AR), virtual reality (VR), and mixed reality (MR). XR is a technology that combines (or mirrors) the physical world with a "digital twin world", in which a user interacts with objects in the XR. The user gets a digital experience wherein physical and digital object coexist in real-time.

[0029] The hybrid XR rendering system io is arranged to dynamically select objects to be rendered in an XR device 6, and in particular select whether a particular object should be rendered locally, e.g., in the XR device 6 or in a local device 4 located e.g., in the same room as the XR device 6, or be rendered remotely in a remote rendering server 2. To accomplish this, object- and system-level properties are obtained and a set of candidate objects that fulfil property criteria for this time period is derived. Then a set of objects that are to be rendered locally are selected, based e.g., on priority. The hybrid rendering is controlled such as to render the set of objects locally and / or remotely.

[0030] The hybrid XR rendering system 10 comprises a network connectivity between the local device 4 in the environment in which a user uses his XR device 6 and the remote rendering server 2, or between the hybrid XR rendering system 10 and the XR device 6. The network connectivity enables remote rendering and may be wired and / or wireless. The XR device 6 comprises various means for user interaction input. Examples of such means comprise XR controllers and gaze tracking, hand gestures, standard input devices such as mouse and / or keyboard. The XR device 6 comprises an XR application 5 utilizing the network connectivity to the remote rendering server 2, and / or a module 9 thereof, for creating the XR experience.

[0031] The remote rendering server 2 comprises means, such as hardware, software and any combinations thereof, for simulating of a virtual environment, and discrete virtual object knowledge, i.e., properties of a virtual object, such as for instance, position, rotation and / or scale of the virtual object relative to the XR device 6 or in the virtual world. Further, the remote rendering server 2 may also render still further properties such as priority, mesh, textures, semantic information etc.

[0032] In the hybrid XR rendering system 10, the XR device 6, to be worn by a user, is running the XR application 5 and is connected to the remote rendering server 2. The remote rendering server 2 provides video streams for visualization in the XR device 6.

[0033] A base state of the hybrid XR rendering system 10 may, in an embodiment, be full remote rendering. In such full remote rendering, all the visual content is rendered on the remote rendering server 2, e.g., in a server module 9 thereof. When certain criteria are met, virtual objects are instead instanced locally on the XR device 6, e.g., in a XR device module 8 thereof. This gives all the benefits that follow with such locally rendered XR content, e.g., not dependent on availability, quality or speed of a connection to a remote rendering server. The present teachings provide a way to select where rendering is to be made, based on evaluation of various criteria as will be described.

[0034] A method is described herein, which may be implemented locally in the local module 7 of the local control device 4, e.g., located in the same room as the XR device 6, or in the XR device module 8 of the XR device 6, or it may be implemented remotely in the cloud module 9 in the remote rendering server 2. In the method, the usage of the XR application 5 is continuously, periodically, at specified events or at specified points in time, evaluated for several criteria that may prompt local rendering to be enabled, e.g., for specific XR objects. These criteria may concern system-level properties and / or XR object -level properties; system-level properties concern the state of the system itself such as network conditions, e.g., reliability, latency, bandwidth, etc., or local rendering capacities, e.g., energy and resource constrains, local rendering quality and speed, etc., while XR object-level properties concern the state of virtual objects, such as innate object prioritization, virtual distance, occlusions, complexity, level of detail needed and user interaction / attention with the XR object. While certain cases may allow one criterion to have greater influence than another, generally the hybrid XR rendering system io may be set up such that when one or more system-level properties reach their criteria for local rendering to be enabled (e.g., poor network connection reaches a certain threshold), the XR object-level properties are used to determine which XR objects should be prioritized for local rendering (e.g. an XR object held in the user’s hand). An exemplary case of XR object-level properties having higher priority is a preference to use the lower delay that local interaction can provide for certain XR objects, combined with system-level properties when a network tail latency is above a certain level.

[0035] The hybrid XR rendering system io is provided with a list of criteria that are continuously evaluated, polling the system for properties, and comparing them to thresholds set, for instance, by the developer of the XR application 5 or by the developer of the XR device 6. As an example, Boolean criteria may be defined as:

[0036] Criteria 001:

[0037] Examples of system-level criteria:

[0038] 1. The network connection bitrate falls under x Mb / s.

[0039] 2. The XR device 6 local memory usage for the application is below y MB.

[0040] 3. The XR device 6 available battery is above b%.

[0041] 4. The XR device 6 rendering speed can deliver more than v frames per second (fps).

[0042] 5. The XR device 6 rendering quality can be delivered above e mean-squared error (MSE) and s peak signal-to-noise ratio (PSNR).

[0043] Examples of object -level criteria: 6. XR object currently / recently within or near user’s field of view (within camera render view).

[0044] 7. XR object has an object priority value at least z or is part of a group / dass / type ofobjed.

[0045] 8. XR object has currently / recently been interacted with by user (e.g., has property IsHeld = true).

[0046] Examples of criteria combination of system-level criteria and object-level criteria:

[0047] 9. Boolean operations between the XR object- and system-level criteria e.g., AND-operation.

[0048] Such criteria and other criteria may be evaluated for applicable XR objects, thereby producing a set of at least one candidate object. Additionally, a prioritization among these candidate objects may be needed due to one or more of: the total resource usage on the device, a companion device and a network connection. The candidate objects may establish a set of to-be-locally rendered objects.

[0049] In other embodiments, a criteria type to be used may comprise a priority / rank criteria providing a level (e.g., o.0-1.0) that can be compared to a threshold. For example:

[0050] Criteria 002:

[0051] Examples of system-level criteria:

[0052] 1. Percentage of battery level

[0053] 2. Available device local memory percentage (e.g., 0.5 if 4 GB available out of 8GB device capacity)

[0054] 3. Relative device to server rendering speed ratio (e.g., 0.5 if 30 fps on XR device and 60 fps on rendering server 10)

[0055] 4. Relative XR device to server rendering resolution ratio (e.g., 0.5 if io8op on device and 4K on render server). 5- Relative network predicted / estimated / measured tail or average latency compared with requested latency.

[0056] 6. Relative network predicted / estimated / measured tail or average bitrate compared with requested bitrate.

[0057] Examples of object -level criteria:

[0058] 7. Normalized distance to XR object (e.g., relative to a fixed “maximum” distance)

[0059] 8. Likelihood of user interaction with an XR object, e.g., based on how recent / often it was used.

[0060] 9. Normalized inverted object model size / complexity (e.g., relative to an average simple model that can be rendered locally)

[0061] Examples of criteria combinations:

[0062] 10. Weighting of the XR object- and system-level criteria into the final criteria level. Other combination methods are also possible, e.g., taking a max, min or even a full AI / ML derived level based on the object- and system-level criteria features.

[0063] Such level-based criteria then are compared to a threshold, which threshold may be static, or it can depend on e.g., system-level information. An exemplary threshold may be based on the current or expected network communication latency, wherein a higher latency gives a lower threshold. The network latency may, in other embodiments, instead be incorporated as a system-level criteria. The objects that are evaluated whether or not being above a threshold are the candidate objects. Then, these candidate objects are selected based on priority level until a total device resources usage is full, and thus establishing a set of to-be-locally rendered objects.

[0064] The criteria evaluation and XR object rendering control may be performed in any of the described nodes, devices, network, modules, services, servers in the hybrid XR rendering system 10. For instance, the rendering control maybe implemented in and performed in a remote rendering server 2, e.g., a cloud server, or in a separate local control device 4 for rendering XR objects in the environment (e.g., same room) wherein the XR device 6 is used, or it may be a rendering control module 8 in the XR device 6, worn by a user. For simplicity, the rendering control module 8 is here described as being executed in the rendering control module 8 in the XR device 6. Hence, a pull of object data is then to be replaced with a push of object data and control of local rendering by a remote service. However, both alternatives and a hybrid thereof are suitable implementations.

[0065] If the criteria are met, the XR device 6 either polls the remote rendering server 2 for object data to be sent to the XR device 6, which instantiates the object, or the object is instantiated using existing local object data. It is noted that these criteria examples are just a few criteria, and that the system may require several more for a satisfying implementation to differentiate between different levels of property values.

[0066] Next, a list of exemplary evaluation properties is given, that may be used when evaluating and prioritizing local rendering of objects. It is noted that this is not an exhaustive list, and there may be other properties not mentioned that may be utilized.

[0067] In a further embodiment, the hybrid XR rendering system 10 is also concerned with other outputs than visual outputs, for instance audio, haptics, scent data etc., which may be streamed from the remote rendering server 2, or the data rendering may, according to the present teachings, instead be instanced locally.

[0068] Fig. 2 is a block diagram illustrating embodiments of property retrieval. The XR device 6 comprises, as described earlier, an XR application 5. The XR application 5 receives various properties (property values) from the XR device 6, as well as various network properties (property values). Both of these property values have been exemplified earlier. The XR application 5 may further receive server properties from the remote rendering server 2, together with object properties of the objects created in the virtual environment.

[0069] In the XR application 5, one or more of the property values Prop 001, Prop. 002,..., Prop, oon, the object values Object 001, Object 002,..., Object n, the server properties and object properties of XR object are used for determining if and which of one or more criteria are fulfilled. The XR application 5 then decides on where the XR objects are to be rendered.

[0070] Fig. 3 is a flowchart of various embodiments of a method 40 according to the herein presented teachings. The method 40 may be used for rendering Extended Reality, XR, objects for an XR application 5 that is run in an XR device 6. The method 40 is performed in a control device 2, 4, 6, such as, for instance, in a remotely located cloud server 2, in a local control module 8 for an XR device 6, or in an XR device 6.

[0071] The method 40 comprises determining 42 at least one property of one or more XR objects in an XR environment. Examples on such XR objects are location of an XR object, priority of an XR object, type of XR object, class of XR object, group of XR object, and time since XR object has been interacted with by user. The method comprises determining 44 at least one property of an XR rendering system 10 providing the one or more XR objects. Examples of such properties are bitrate of a connection between a remotely located server that rendering the XR objects remotely and the XR device, memory usage of the XR device, battery status of the XR device, rendering speed of the XR device, and rendering quality of the XR device.

[0072] The method comprises controlling 46 the rendering of the one or more XR objects to be performed locally and / or remotely based on one or both of: the determined properties of the one or more XR objects, and on the one or more properties of the XR rendering system 10. The controlling of the rendering can be made dynamically, i.e., the rendering can change dynamically between remotely rendered and / or locally rendered XR objects.

[0073] In an embodiment, the controlling 46 comprises selecting one or more XR objects to be rendered locally when the properties of the XR rendering system 10 fulfills at least one of: bitrate of a network connection between a remotely located server 2 and XR device 6 falls below a set threshold value, latency of a network connection between a remotely located server 2 and XR device 6 exceeds a set threshold value, reliability of a network connection between a remotely located server 2 and XR device 6 falls below a set threshold value, available battery capacity of XR device 6 is above a set threshold value, rendering speed of XR device 6 is above a set threshold value, rendering quality of XR device 6 to be provided is above a peak signal -to-noise ratio, object property value of an XR object meeting a set threshold.

[0074] In various embodiments, the controlling 46 further comprises prioritizing local rendering before remote rendering based on one or more of: user interactivity with XR object, priority value of XR object, rendering distance from user camera, and velocity of XR object. When local rendering is used, the user experience may be improved in that essentially all latency issues are avoided, which is important as the XR experience is sensitive to delays of any kind. Delays in, for instance, moving of the virtual objects by the user is highly annoying for the user experience.

[0075] In various embodiments, the one or more properties of the XR rendering system 10 are selected among: location of an XR object, priority of an XR object, type of XR object, class of XR object, group of XR object, and time since XR object has been interacted with by user.

[0076] In various embodiments, the one or more properties of the XR rendering system io are selected among: bitrate of a connection between a remotely located server 2 rendering the XR objects remotely and the XR device 6, memory usage of the XR device 6, battery status of the XR device 6, rendering speed of the XR device 6, and rendering quality of the XR device 6.

[0077] In various embodiments, the method 40 comprises performing the rendering 46 in one or more of: a cloud server 2, a control module 8 of an XR device 6, and a local control module 4 in an environment wherein the XR device 6 is used.

[0078] In various embodiments, the method 40 comprises repeating the steps 42 and 44 at pre-determined points in time.

[0079] In various embodiments, the method 40 comprises repeating one or both of steps 42 and 44, and upon at least one property of one or more XR objects in an XR environment has changed more than a set value, or at least one property of an XR rendering system 10 providing the one or more XR objects has changed more than a set value, then performing the step of controlling 46.

[0080] A control device 2, 4, 6 for rendering Extended Reality, XR, objects for an XR application 5 run in an XR device 6 is provided. The control device 2, 4, 6 is configured to determine one or more properties of one or more XR objects in an XR environment. Examples of such properties has been given earlier.

[0081] The control device 2, 4, 6 is configured to determine one or more properties of an XR rendering system 10 providing the one or more XR objects. Also this type of properties has been exemplified earlier.

[0082] The control device 2, 4, 6 is configured to control the rendering of the one or more XR objects to be performed locally based on one or both of: the determined properties of the one or more XR objects, and the one or more properties of the XR rendering system 10. In an embodiment, the control device 2, 4, 6 is configured to control by selecting one or more XR objects to be rendered locally, when the one or more of XR objects fulfils at least one of: XR object currently being within field of view of user of XR device 6, XR object having a priority value above a set threshold, XR object being part of a specified type of object, XR object being part of a specified type of class, XR object being part of a specified group, XR object is currently or recently being interacted with by user of XR device 6.

[0083] In various embodiments, the control device 2, 4, 6 is configured to control by selecting one or more XR objects to be rendered locally when the properties of the XR rendering system 10 fulfills at least one of: bitrate of a network connection between a remotely located server 2 and XR device 6 falls below a set threshold value, latency of a network connection between a remotely located server 2 and XR device 6 exceeds a set threshold value, reliability of a network connection between a remotely located server 2 and XR device 6 falls below a set threshold value, available battery capacity of XR device 6 is above a set threshold value, rendering speed of XR device 6 is above a set threshold value, rendering quality of XR device 6 to be provided is above a peak signal-to-noise ratio, object property value of an XR object meeting a set threshold.

[0084] In various embodiments, the control device 2, 4, 6 is configured to control by prioritizing local rendering before remote rendering based on one or more of: user interactivity with XR object, priority value of XR object, rendering distance from user camera, and velocity of XR object.

[0085] In various embodiments, the control device 2, 4, 6 is configured to perform the rendering in one or more of: a cloud server 2, a control module 8 of an XR device 6, and a control module 4 in an environment wherein the XR device 6 is used.

[0086] In various embodiments, the control device 2, 4, 6 is a remotely located cloud server 2, a local control module 8 of an XR device 6, or an XR device 6. In still further embodiments, a distributed control device is implemented. In such embodiments, the control features are distributed among two or more of the control devices 2, 4, 6.

[0087] In various embodiments, the control device 2, 4, 6 is configured to repeat the steps 42 and 44 at pre-determined points of time. In various embodiments, the control device 2, 4, 6 is configured to repeat one or both of steps 42 and 44 and configured to perform the controlling upon at least one property of one or more XR objects in an XR environment has changed more than a set value, or at least one property of an XR rendering system 10 providing the one or more XR objects has changed more than a set value.

[0088] Fig. 4 schematically illustrates, in terms of a number of functional units, the components of a module 2, 4, 6 as has been described, according to an embodiment. Processing circuitry 110 is provided using any combination of one or more of a suitable central processing unit (CPU), multiprocessor, microcontroller, digital signal processor (DSP), etc., capable of executing software instructions stored in a computer program product 330 (as shown in Fig. 6), e.g., in the form of a storage medium 130. The processing circuitry 110 may further be provided as at least one application specific integrated circuit (ASIC), or field programmable gate array (FPGA).

[0089] Particularly, the processing circuitry 110 is configured to cause the module 2, 4, 6 to perform a set of operations, or actions, as disclosed herein. For example, the storage medium 130 may store the set of operations, and the processing circuitry 110 may be configured to retrieve the set of operations from the storage medium 130 to cause the module 2, 4, 6 to perform the set of operations. The set of operations may be provided as a set of executable instructions. The processing circuitry 110 is thereby arranged to execute methods as herein disclosed.

[0090] The storage medium 130 may also comprise persistent storage, which, for example, can be any single one or combination of magnetic memory, optical memory, solid state memory or even remotely mounted memory.

[0091] The module 2, 4, 6 may further comprise a communications interface 120 for communications with other entities, functions, nodes, and devices, over suitable interfaces. As such the communications interface 120 may comprise one or more transmitters and receivers, comprising analogue and digital components.

[0092] The processing circuitry 110 controls the general operation of the module 2, 4, 6 e.g., by sending data and control signals to the communications interface 120 and the storage medium 130, by receiving data and reports from the communications interface 120, and by retrieving data and instructions from the storage medium 130. Other components, as well as the related functionality, of the module 2, 4, 6 are omitted in order not to obscure the concepts presented herein.

[0093] Fig. 5 schematically illustrates, in terms of a number of functional modules, the components of a module 2, 4, 6 according to an embodiment. The module 2, 4, 6 of Fig. 5 comprises a number of functional modules; an determine module 210 may, for instance, be configured to determine at least one property of one or more XR objects in an XR environment; an determine module 220 configured to determine at least one property of an XR rendering system providing the one or more XR objects; and an control module 230, configured to control rendering of the one or more XR objects to be performed locally and / or remotely based on one or both of: the determined properties of the one or more XR objects, and the one or more properties of the XR rendering system. The module 2, 4, 6 of Fig. 5 may further comprise a number of optional functional modules, for performing the method as disclosed herein. In general terms, each functional module 210 - 230 may be implemented in hardware or in software. Preferably, one or more or the functional modules 210 - 230 may be implemented by the processing circuitry 110, possibly in cooperation with the communications interface 120 and the storage medium 130. The processing circuitry 110 may thus be arranged to from the storage medium 130 fetch instructions as provided by a functional module 210 - 230 and to execute these instructions, thereby performing any actions of the module 2, 4, 6 as disclosed herein.

[0094] Fig. 6 shows one example of a computer program product 330 comprising computer readable means 340. On this computer readable means 340, a computer program 320 can be stored, which computer program 320 can cause the processing circuitry 110 and thereto operatively coupled entities and devices, such as the communications interface 120 and the storage medium 130, to execute methods according to embodiments described herein. The computer program 320 and / or computer program product 330 may thus provide means for performing any actions of the module 2, 4, 6 as disclosed herein.

[0095] In the example of Fig. 6, the computer program product 330 is illustrated as an optical disc, such as a CD (compact disc) or a DVD (digital versatile disc) or a Blu-Ray disc. The computer program product 330 could also be embodied as a memory, such as a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), or an electrically erasable programmable read-only memory (EEPROM) and more particularly as a non-volatile storage medium of a device in an external memory such as a USB (Universal Serial Bus) memory or a Flash memory, such as a compact Flash memory. Thus, while the computer program 320 is here schematically shown as a track on the depicted optical disk, the computer program 320 can be stored in any way which is suitable for the computer program product 330.

[0096] The inventive concept has mainly been described above with reference to a few embodiments. However, as is readily appreciated by a person skilled in the art, other embodiments than the ones disclosed above are equally possible within the scope of the inventive concept, as defined by the appended patent claims.

Claims

CLAIMS1. A method (40) for rendering Extended Reality, XR, objects for an XR application (5) for an XR device (6), the method (40) being performed in a control device (2, 4, 6) and comprising:- determining (42) at least one property of one or more XR objects in an XR environment,- determining (44) at least one property of an XR rendering system (10) providing the one or more XR objects, and- controlling (46) rendering of the one or more XR objects to be performed locally and / or remotely based on one or both of: the determined properties of the one or more XR objects, and the one or more properties of the XR rendering system (10).

2. The method (40) as claimed in claim 1, wherein the controlling (46) comprises selecting one or more XR objects to be rendered locally, when the one or more of XR objects fulfils at least one of: XR object currently being within field of view of user of XR device (6), XR object having a priority value above a set threshold, XR object being part of a specified type of object, XR object being part of a specified type of class, XR object being part of a specified group, XR object is currently or recently being interacted with by user of XR device (6).

3. The method (40) as claimed in claim 1 or 2, wherein the controlling (46) comprises selecting one or more XR objects to be rendered locally when the properties of the XR rendering system (10) fulfills at least one of: bitrate of a network connection between a remotely located server (2) and XR device (6) falls below a set threshold value, latency of a network connection between a remotely located server (2) and XR device (6) exceeds a set threshold value, reliability of a network connection between a remotely located server (2) and XR device (6) falls below a set threshold value, available battery capacity of XR device (6) is above a set threshold value, rendering speed of XR device (6) is above a set threshold value, rendering quality of XR device (6) to be provided is above a peak signal -to- noise ratio, object property value of an XR object meeting a set threshold.

4. The method (40) as claimed in any of the preceding claims, wherein the controlling (46) further comprises prioritizing local rendering before remote rendering based on one or more of: user interactivity with XR object, priorityvalue of XR object, rendering distance from user camera, and velocity of XR object.

5. The method (40) as claimed in any of the preceding claims, comprising performing the rendering (46) in one or more of: a cloud server (2), a control module (8) of an XR device (6), and a control module (4) in an environment wherein the XR device (6) is used.

6. The method (40) as claimed in any of the preceding claims, wherein the control device (2, 4, 6) is one of: a remotely located cloud server (2), a local control module (8) of an XR device (6), and an XR device (6).

7. The method (40) as claimed in any of the preceding claims, comprising repeating the steps (42) and (44) at pre-determined points of time.

8. The method (40) as claimed in any of the preceding claims, comprising repeating one or both of steps 42 and 44, and upon at least one property of one or more XR objects in an XR environment has changed more than a set value, or at least one property of an XR rendering system (10) providing the one or more XR objects has changed more than a set value, then performing the step of controlling (46).

9. A computer program (320) for rendering Extended Reality, XR, objects for an XR application (5) run in an XR device (6), the computer program comprising computer code which, when run on processing circuitry (110) of a control device (2, 4, 6) causes the control device (2, 4, 6) to:- determine one or more properties of one or more XR objects in an XR environment,- determine one or more properties of an XR rendering system (10) providing the one or more XR objects, and control rendering of the one or more XR objects to be performed locally based on one or both of: the determined properties of the one or more XR objects, and the one or more properties of the XR rendering system (10).

10. A computer program product (330) comprising a computer program (320) according to claim 9, and a computer readable storage medium (340) on which the computer program (320) is stored.

11. A control device (2, 4, 6) for rendering Extended Reality, XR, objects for an XR application (5) run in an XR device (6), the control device (2, 4, 6) being configured to:- determine one or more properties of one or more XR objects in an XR environment,- determine one or more properties of an XR rendering system (io) providing the one or more XR objects, and control rendering of the one or more XR objects to be performed locally based on one or both of: the determined properties of the one or more XR objects, and the one or more properties of the XR rendering system (io).

12. The control device (2, 4, 6) as claimed in claim 11, configured to control by selecting one or more XR objects to be rendered locally, when the one or more of XR objects fulfils at least one of: XR object currently being within field of view of user of XR device (6), XR object having a priority value above a set threshold, XR object being part of a specified type of object, XR object being part of a specified type of class, XR object being part of a specified group, XR object is currently or recently being interacted with by user of XR device (6).

13. The control device (2, 4, 6) as claimed in claim 11 or 12, configured to control by selecting one or more XR objects to be rendered locally when the properties of the XR rendering system (10) fulfills at least one of: bitrate of a network connection between a remotely located server (2) and XR device (6) falls below a set threshold value, latency of a network connection between a remotely located server (2) and XR device (6) exceeds a set threshold value, reliability of a network connection between a remotely located server (2) and XR device (6) falls below a set threshold value, available battery capacity of XR device (6) is above a set threshold value, rendering speed of XR device (6) is above a set threshold value, rendering quality of XR device (6) to be provided is above a peak signal-to-noise ratio, object property value of an XR object meeting a set threshold.

14. The control device (2, 4, 6) as claimed in any of claims 11 - 13, configured to control by prioritizing local rendering before remote rendering based on one or more of: user interactivity with XR object, priority value of XR object, rendering distance from user camera, and velocity of XR object.

15. The control device (2, 4, 6) as claimed in any of claims 11 - 14, configured to perform the rendering in one or more of: a cloud server (2), a control module (8) of an XR device (6), and a control module (4) in an environment wherein the XR device (6) is used.

16. The control device (2, 4, 6) as claimed in any of claims 11 - 15, wherein the control device (2, 4, 6) is a remotely located cloud server (2), a local control module (8) of an XR device (6), or an XR device (6).

17. The control device (2, 4, 6) as claimed in any of claims 11 - 16, configured to repeat the steps (42) and (44) at pre-determined points of time.

18. The control device (2, 4, 6) as claimed in any of claims 11 - 17, comprising repeating one or both of steps 42 and 44, and upon at least one property of one or more XR objects in an XR environment has changed more than a set value, or at least one property of an XR rendering system (10) providing the one or more XR objects has changed more than a set value, then performing the step of controlling

Images