Adaptive split rendering

Abstract

An apparatus includes: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: monitoring or receiving information about factors that affect rendering capabilities of a device or about factors that affect a quality of experience of an ongoing split rendering session; creating a signal based on changes in the rendering capabilities or factors that affect the quality of experience of the ongoing split rendering session; and reacting, acknowledging, negotiating, or forwarding the signal.

Description

ADAPTIVE SPLIT RENDERINGTECHNICAL FIELD

[0001] The examples and non-limiting embodiments relate generally to graphics rendering and, more particularly to, adaptive split rendering.BACKGROUND

[0002] It is known to provide graphics rendering.SUMMARY

[0003] Example 1. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: monitoring or receiving information about factors that affect rendering capabilities of a device or about factors that affect a quality of experience of an ongoing split rendering session; creating a signal based on changes in the rendering capabilities or factors that affect the quality of experience of the ongoing split rendering session; and reacting, acknowledging, negotiating, accepting, rejecting, or forwarding the signal. In one or more examples, the signal comprises a new distribution of rendering tasks. In some examples, the signal further comprises at least one first message identifier, wherein the at least one first message identifier uniquely identifies the signal or a message comprised in the signal within a split rendering session or the ongoing split rendering session. In some other examples, the signal further comprises at least one second message identifier, wherein the at least one second message identifier identifies the signal or the message comprised in the signal as one or more of: a request for, offers of, acknowledgement of, acceptance of, or rejection of the new distribution of the rendering tasks.

[0004] Example 2. The apparatus of example 1, wherein the factors that affect the quality of experience of the ongoing split rendering session comprise one or more of the following: changes in operating environment; or changes in a scene being rendered; and wherein reacting, acknowledging, negotiating, or forwarding comprises one or more of requesting for changes in a distribution of rendering tasks; requesting for a desired distribution of the rendering tasks; or accepting the distribution of the rendering tasks.

[0005] Example 3. The apparatus of any of the examples 1 or 2, wherein to react, the apparatus is further caused to perform one or more of the following: accepting a received distribution of rendering tasks; acknowledging the acceptance; actuating the received distribution of the rendering tasks; signaling the actuation; replying with acceptable the distribution of the rendering tasks;1SUBSTITUTE SHEET (RULE 26)rejecting a request for change in the distribution of rendering tasks; or verifying ability to change the distribution of the rendering tasks and forwarding the received signals to another network entity.

[0006] Example 4. The apparatus of any of the examples 2 to 3, wherein a distribution of the rendering task comprises one or more of the following: a scene description or a scene graph comprising a portion of a scene to be rendered; a scene description resource or a scene graph in which objects comprise a parameter which identifies rendering entity for an object; a data structure comprising key-value pairs, wherein the key identifies the rendering entity and the value comprises a list of identifiers for objects in the scene to be rendered at the identified rendering entity; or a data structure comprising the key-value pairs, wherein the key identifies the rendering entity and the corresponding value comprises the scene description or the scene graph to be rendered at the identified rendering entity.

[0007] Example 5. The apparatus of any of the examples 2 to 4, wherein the distribution of the rendering tasks is indicated by one or more universal resource identifiers.

[0008] Example 6. The apparatus of any of the previous examples, wherein the apparatus is further caused to perform: negotiating and initiating the split rendering session.

[0009] Example 7. The apparatus of example 6, wherein the apparatus is further caused to perform: starting the split rendering process; and establishing transport channels for media and metadata exchange.

[0010] Example 8. The apparatus example 7, wherein the apparatus is further caused to perform: signaling or receiving the meta-data required for the split rendering operations.

[0011] Example 9. The apparatus example 8, wherein the apparatus is further caused to perform: rendering the media; and transmitting or receiving the media.

[0012] Example 10. The apparatus of example 9, wherein the apparatus is further caused to perform: receiving, signaling, or detecting a split rendering adaptation trigger, wherein the split scene rendering trigger comprises a change in the capabilities of the device, a change in the ongoing split rendering session, a change in a split rendering server, a change in a scene being rendered in the ongoing split rendering session, the factors affecting quality of experience, or energy efficiency of the ongoing split rendering session.

[0013] Example 11. The apparatus of example 10, wherein the apparatus is further caused to perform: calculating a desirable split rendering task distribution based on the split rendering adaptation trigger; signaling or receiving a request for a new desired split rendering task distribution;2SUBSTITUTE SHEET (RULE 26)and receiving or signaling acknowledgement or acceptance of the received request.

[0014] Example 12. The apparatus of example 10, wherein the apparatus is further caused to perform: calculating a desirable split rendering task distribution; signaling or receiving a request for a new desired split rendering task distribution; receiving or signaling acknowledgement of the received request; receiving or signaling a rejection of the request for the new desired split rendering; signaling or receiving an offer of one or more offered split rendering task distributions; selecting one of the one or more offered split rendering task distributions as agreeable; receiving or signaling acknowledgement of the rejection; ; and receiving or signaling acceptance of a selected split rendering task distribution.

[0015] Example 13. The apparatus of example 10, wherein the apparatus is further caused to perform: calculating a desirable split rendering task distribution based on the split rendering adaptation trigger and one or more factors comprising device capabilities, network conditions, a scene being rendered, quality of service, charging policy considerations, quality of experience considerations, or energy efficiency factors.

[0016] Example 14. The apparatus of any of the examples 9 to 12, wherein the split rendering adaptation trigger comprises one or more of the following: change in the device battery charge below a threshold; change in the device temperature above a threshold; change in available memory or computational capacity for rendering at the device; changes in the scene being rendered; change in nature or number of participants in an augmented reality call; a render-to-photon latency of quality of experience critical parts of the scene; a render-to-photon latency of parts of the scene; resources available; changes in network link conditions; changes in a scene description; changes in a scene graph; upgrade or downgrade in quality or service; changes in quality of rendered graphics; or changes in energy efficiency factors at the device.

[0017] Example 15. The apparatus of example 4, wherein the scene description or a scene graph comprises one or more objects in accordance with one or more of the following: Open Universal Scene Description (OpenUSD); Extensible 3D (X3D); Vulkan Scene Graph; Open Scene Graph; Collada; or glTF.

[0018] Example 16. The apparatus of any of the previous examples, wherein the apparatus or the device comprise one or more of a split rendering client or a split rendering server.

[0019] Example 17. A method comprising: monitoring or receiving information about factors that affect rendering capabilities of a device or about factors that affect a quality of experience of an ongoing split rendering session; creating a signal based on changes in the rendering capabilities or3SUBSTITUTE SHEET (RULE 26)factors that affect the quality of experience of the ongoing split rendering session; and reacting, acknowledging, negotiating, accepting, rejecting, or forwarding the signal. In one or more examples, the signal comprises a new distribution of rendering tasks. In some examples, the signal further comprises at least one first message identifier, wherein the at least one first message identifier uniquely identifies the signal or a message comprised in the signal within a split rendering session or the ongoing split rendering session. In some other examples, the signal further comprises at least one second message identifier, wherein the message identifier identifies the signal or a message comprised in the signalas one or more of: a request for, offers of, acknowledgement of, acceptance of, or rejection of the new distribution of the rendering tasks.

[0020] Example 18. The method of example 17, wherein the factors that affect the quality of experience of the ongoing split rendering session comprise one or more of the following: changes in operating environment; or changes in a scene being rendered; and wherein reacting, acknowledging, negotiating, or forwarding comprises one or more of requesting for changes in a distribution of rendering tasks; requesting for a desired distribution of the rendering tasks; or accepting the distribution of the rendering tasks.

[0021] Example 19. The method of any of the examples 17 or 18 further comprising: accepting a received distribution of rendering tasks; acknowledging the acceptance; actuating the received distribution of the rendering tasks; signaling the actuation; replying with acceptable the distribution of the rendering tasks; rejecting a request for change in the distribution of rendering tasks; or verifying ability to change the distribution of the rendering tasks and forwarding the received signals to another network entity.

[0022] Example 20. The method of any of the examples 18 to 19, wherein a distribution of the rendering task comprises one or more of the following: a scene description or a scene graph comprising a portion of a scene to be rendered; a scene description resource or a scene graph in which objects comprise a parameter which identifies rendering entity for an object; a data structure comprising key-value pairs, wherein the key identifies the rendering entity and the value comprises a list of identifiers for objects in the scene to be rendered at the identified rendering entity; or a data structure comprising the key-value pairs, wherein the key identifies the rendering entity and the corresponding value comprises the scene description or the scene graph to be rendered at the identified rendering entity.

[0023] Example 21. The method of any of the examples 18 to 20, wherein the distribution of the rendering tasks is indicated by one or more universal resource identifiers.4SUBSTITUTE SHEET (RULE 26)

[0024] Example 22. The method of any of the examples 17 to 21, wherein the method is further caused to perform: negotiating and initiating the split rendering session.

[0025] Example 23. The method of example 22 further comprising: starting the split rendering process; and establishing transport channels for media and meta-data exchange.

[0026] Example 24. The method example 23 further comprising: signaling or receiving the metadata required for the split rendering operations.

[0027] Example 25. The method example 24 further comprising: rendering the media; and transmitting or receiving the media.

[0028] Example 26. The method of example 25 further comprising: receiving, signaling, or detecting a split rendering adaptation trigger, wherein the split scene rendering trigger comprises a change in the capabilities of the device, a change in the ongoing split rendering session, a change in a split rendering server, a change in a scene being rendered in the ongoing split rendering session, the factors affecting quality of experience, or energy efficiency of the ongoing split rendering session.

[0029] Example 27. The method of example 26 further comprising: calculating a desirable split rendering task distribution based on the split rendering adaptation trigger; signaling or receiving a request for a new desired split rendering task distribution; and receiving or signaling acknowledgement or acceptance of the received request.

[0030] Example 28. The method of example 27 further comprising: calculating a desirable split rendering task distribution; signaling or receiving a request for a new desired split rendering task distribution; receiving or signaling acknowledgement of the received request; receiving or signaling a rejection of the request for the new desired split rendering; signaling or receiving an offer of one or more offered split rendering task distributions; selecting one of the one or more offered split rendering task distributions as agreeable; receiving or signaling acknowledgement of the rejection; and receiving or signaling acceptance of a selected split rendering task distribution.

[0031] Example 29. The method of example 27 further comprising: calculating a desirable split rendering task distribution based on the split rendering adaptation trigger and one or more factors comprising device capabilities, network conditions, a scene being rendered, quality of service, charging policy considerations, quality of experience considerations, or energy efficiency factors.

[0032] Example 30. The method of any of the examples 25 to 28, wherein the split rendering adaptation trigger comprises one or more of the following: change in the device battery charge below5SUBSTITUTE SHEET (RULE 26)a threshold; change in the device temperature above a threshold; change in available memory or computational capacity for rendering at the device; changes in the scene being rendered; changes in nature or number of participants in an augmented reality call; ; a render-to-photon latency of quality of experience critical parts of the scene; a render-to-photon latency of parts of the scene; resources available; changes in network link conditions; changes in a scene description; changes in a scene graph; upgrade or downgrade in quality or service; changes in quality of rendered graphics; or changes in energy efficiency factors at the device.

[0033] Example 31. The method of example 20, wherein the scene description or a scene graph comprises one or more objects in accordance with one or more of the following: Open Universal Scene Description (OpenUSD); Extensible 3D (X3D); Vulkan Scene Graph; Open Scene Graph; Collada; or glTF.

[0034] Example 32. The method of any of the examples 17 to 31, wherein the device comprise one or more of a split rendering client or a split rendering server.

[0035] Example 33. An apparatus comprising: means for monitoring or receiving information about factors that affect rendering capabilities of a device or about factors that affect a quality of experience of an ongoing split rendering session; means for creating a signal based on changes in the rendering capabilities or factors that affect the quality of experience of the ongoing split rendering session; and means for reacting, acknowledging, negotiating, accepting, rejecting, or forwarding the signal. In one or more examples, the signal comprises a new distribution of rendering tasks.

[0036] Example 34. The apparatus of example 33, wherein the apparatus is further caused to perform the methods as described in any of the examples 17 to 32.

[0037] Example 35. A computer readable medium comprising program instructions which, when executed by an apparatus, cause the apparatus to perform at least the following: monitoring or receiving information about factors that affect rendering capabilities of a device or about factors that affect a quality of experience of an ongoing split rendering session; creating a signal based on changes in the rendering capabilities or factors that affect the quality of experience of the ongoing split rendering session; and reacting, acknowledging, negotiating, accepting, rejecting, or forwarding the signal. In one or more examples, the signal comprises a new distribution of rendering tasks.

[0038] Example 36. The computer readable medium of example 35, wherein the computer readable medium comprises a non-transitory computer readable medium.

[0039] Example 37. The computer readable medium of any of the examples 35 or 36, wherein6SUBSTITUTE SHEET (RULE 26)the computer readable medium causes the apparatus to further perform the methods as described in any of the examples 17 to 32.BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The foregoing embodiments and other features are explained in the following description, taken in connection with the accompanying drawings, wherein:

[0041] FIG. 1 shows schematically an electronic device employing embodiments of the examples described herein.

[0042] FIG. 2 shows schematically a user equipment suitable for employing embodiments of the examples described herein.

[0043] FIG. 3a illustrates an example graphics library transmission format (glTF)2.0 top level scene graph.

[0044] FIG. 3b illustrates an moving picture experts group (MPEG)-I extensions to the glTF2.0 scene graph.

[0045] FIG. 4a illustrates split rendering architecture in a split rendering (SR) media service enabler (MSE).

[0046] FIG. 4b. illustrates split rendering architecture in an internet protocol multimedia subsystem based augmented reality conversational services (IB ACS).

[0047] FIG. 5 illustrates high-level call flow for client driven split rendering session setup.

[0048] FIG. 6 illustrates the operation of a split rendering session.

[0049] FIGs. 7a and 7b illustrate split rendering call flow in IB ACS.

[0050] FIGs. 8a and 8b illustrate a call flow for split rendering, in accordance with an embodiment.

[0051] FIG. 9 is an example apparatus configured to implement the examples described herein.

[0052] FIG. 10 shows a representation of an example of non-volatile memory media used to store instructions that implement the examples described herein.

[0053] FIG. 11 is an example method, based on the examples described herein.7SUBSTITUTE SHEET (RULE 26)DETAILED DESCRIPTION OF EXAMPLE EMBODIMENTS

[0054] Described herein is a method and apparatus for implementing adaptive split rendering.

[0055] The following describes in detail a suitable apparatus, method and possible mechanisms for adaptive split rendering according to embodiments. In this regard reference is first made to FIG. 1 and FIG. 2, where FIG. 1 shows an example block diagram of an apparatus 50. The apparatus may be an Internet of Things (loT) apparatus configured to perform various functions, such as for example, gathering information by one or more sensors, receiving or transmitting information, analyzing information gathered or received by the apparatus, or the like. The apparatus may comprise a video coding system, which may incorporate a codec. FIG. 2 shows a layout of an apparatus according to an example embodiment. The elements of FIG. 1 and FIG. 2 are explained next.

[0056] The apparatus 50 may for example be a mobile terminal or user equipment of a wireless communication system, a sensor device, a tag, or other lower power device. However, it may be appreciated that embodiments of the examples described herein may be implemented within any electronic device or apparatus which may process data by neural networks.

[0057] The apparatus 50 may comprise a housing 30 for incorporating and protecting the device. The apparatus 50 further may comprise a display 32 in the form of a liquid crystal display, a virtual reality headset, and / or a transparent / translucent display. In other embodiments of the examples described herein the display may be any suitable display technology suitable to display an image or video. The apparatus 50 may further comprise a keypad 34. In other embodiments of the examples described herein any suitable data or user interface mechanism may be employed. For example, the user interface may be implemented as a virtual keyboard, data entry system as part of a touch- sensitive display, a wired / wireless controller, or gestures based user interfaces.

[0058] The apparatus may comprise a microphone 36 or any suitable audio input which may be a digital or analog signal input. The apparatus 50 may further comprise an audio output device which in embodiments of the examples described herein may be any one of: an earpiece 38, speaker, or an analog audio or digital audio output connection. The apparatus 50 may also comprise a battery (or in other embodiments of the examples described herein the device may be powered by any suitable mobile energy device such as solar cell, fuel cell or clockwork generator). The apparatus may further comprise a camera 42 capable of recording or capturing images and / or video. The apparatus 50 may further comprise an infrared port for short range line of sight communication to other devices. In other embodiments the apparatus 50 may further comprise any suitable short range communication solution such as, for example, a Bluetooth wireless connection or a USB / firewire wired connection.8SUBSTITUTE SHEET (RULE 26)

[0059] The apparatus 50 may comprise a controller 56, processor or processor circuitry for controlling the apparatus 50. The controller 56 may be connected to memory 58 which in embodiments of the examples described herein may store both data in the form of image and audio data and / or may also store instructions for implementation on the controller 56. The controller 56 may further be connected to codec circuitry 54 suitable for carrying out coding and / or decoding of audio and / or video data or assisting in coding and / or decoding carried out by the controller.

[0060] The apparatus 50 may further comprise a card reader 48 and a smart card 46, for example a UICC and UICC reader for providing user information and being suitable for providing authentication information for authentication and authorization of the user at a network.

[0061] The apparatus 50 may comprise radio interface circuitry 52 connected to the controller and suitable for generating wireless communication signals for example for communication with a cellular communications network, a wireless communications system or a wireless local area network. The apparatus 50 may further comprise an antenna 44 connected to the radio interface circuitry 52 for transmitting radio frequency signals generated at the radio interface circuitry 52 to other apparatus(es) and / or for receiving radio frequency signals from other apparatus(es).

[0062] The apparatus 50 may comprise a camera capable of recording or detecting individual frames which are then passed to the codec 54 or the controller for processing. The apparatus may receive the video image data for processing from another device prior to transmission and / or storage. The apparatus 50 may also receive either wirelessly or by a wired connection the image for coding / decoding. The structural elements of apparatus 50 described above represent examples of means for performing a corresponding function.

[0063] Graphics rendering is compute, render and power intensive. These requirements are especially high for immersive applications like virtual reality, augmented reality, mixed reality, and the like, collectively called extended reality (XR) which require high quality graphics for a reasonable quality of experience (QoE). Resource constrained devices like smartphones and headmounted displays may not have the capacity to provide high quality XR experiences due to limitations of compute, render and energy resources. Split rendering (SR) may be a potential solution to this issue. In the split rendering paradigm, the rendering workload is divided between two or more devices, for example, a split rendering client (SRC) and a split rendering server (SRS). The SRS may have substantially more resources for performing the rendering and other operations. The SRS and SRC exchange meta-data and media with each other. In the most basic case SRS receives metadata from the SRC and performs the rendering operations based on the received metadata. The SRS then transmits the rendered media to the SRC which composes the received media with locally9SUBSTITUTE SHEET (RULE 26)rendered media into a frame and displays the frame to a user. In a simplistic case of split rendering, the SRC does the compositing and possibly reprojection on the media received from the SRS without rendering any media locally.

[0064] Computationally rendering content, especially real time 3D content, requires that the computing apparatus is provided information, in a compatible form, about what objects to render and how to render them. Further, the apparatus may need information about how the objects interact with each other or how they evolve over time or how they react to user input. The objects to be possibly rendered may be conceptualized as existing in a virtual scene with the graphics to be rendered being captured by one or more virtual cameras.

[0065] The information required may be provided to a computing apparatus in the form of application logic, description of objects in the (2D / 3D) scene, and the like. The application logic may, non-exhaustively, provide information about one or more of the following:- How the objects in a scene interact with each other;- How they react to user input;- What rules they may be subjected to;- How they evolve over time;- How the scene is viewed, e.g., number of cameras, camera properties, projection, and the like;- How the objects interact with light, how colors in the scene mix and behave, and the like; or- What physics or game engine to use;

[0066] The scene description may, non-exhaustively, provide information about one or more of the following:- Relative location, orientation, and size of the objects in the scene being described;- How the objects are built, e.g., from 3D models, geometric primitives, and the like;- How assets like sprites are mapped to the objects; or10SUBSTITUTE SHEET (RULE 26)Light sources in the scene;

[0067] The information listed above is non-exhaustive, the division into application data and scene description is for exposition and illustrative only, the two groups may be overlapping, not discernible from each other, and some or all the information needed for rendering may be provided in a different manner. Further information, like simulated physics, rendering pipeline to use, and the like, may be provided. The 3D scene may be represented as a (hierarchical) scene graph or a scene description in a format that the rendering apparatus understands.

[0068] An example describe a method and apparatus for adjusting distribution of computations between a server and a client based on computational load of the application, computational capacity of the client and communication link quality, latency, and capacity. Further, the rendering operations are described as an embodiment of the computations and split rendering is also defined as an embodiment of the distribution of computations.

[0069] Another example describes a method and apparatus for offloading AR processing to a server which receives tracking information from sensors on the viewing client device as well as sensors outside the viewing device

[0070] Scene Description

[0071] An example method of providing a scene description to a rendering entity is by using MPEG-I Scene Description, available from [https: / / www.mpeg.org / wp- content / uploads / mpeg_meetings / 143_Geneva / w23004.zip (last accessed on September 26, 2023)]. This standard provides a standardized means for distribution of information about a real time immersive scene and the content therein. This standard defines extensions to glTF2.0, available from [https: / / registry.khronos.Org / glTF / specs / 2.0 / glTF-2.0.html (last accessed on September 26, 2023)] which itself is a standard format for distribution of 3D scene and models. The core capability of glTF2.0 allows description of a scene graph in an interoperable and extensible fashion as a JSON file which comprises a hierarchy of nodes representing the scene graph. A node may include objects like transformations, accessors to meshes, textures, other nodes, references to other resources, and the like. The extensions defined by MPEG allow glTF2.0 to reference immersive multimedia content which may be timed, dynamic textures, dynamic scene updates, and the like.

[0072] FIG. 3a illustrates an example glTF2.0 top level scene graph and FIG. 3b illustrates an MPEG-I extensions to the glTF2.0 scene graph.

[0073] In the JSON file describing a glTF2.0 / MPEG-I scene description based scene, the objects in a scene may be identified by their index values in the scene graph or by their “name” property.11SUBSTITUTE SHEET (RULE 26)Additional information may be defined for each node as an application specific “extra” property. An example glTF representation of a scene in JSON is shown below, and is adapted from glTF available from [https: / / github.com / KhronosGroup / glTF-Tutorials / blob / master / gltfTutorial / gltfTutorial_003_MinimalGltfFile.md (last accessed on September 26, 2023)]."scene": 0,"scenes" : [{"nodes" : [ 0,1, 3, 4, 5, 6, 7, 8,9,10]}],"nodes" : [{ "name": "ObjlO", ..., }],"meshes" : [],"buffers" : [{"uri" : "3gpp.sp.example / assetl","byteLength" : 4412SUBSTITUTE SHEET (RULE 26)],"bufferViews" : [ {...}, {...}],"accessors" : [{ ...}, {...}],"asset" : {"version" : "2.0"}}

[0074] Once the scene information is available to a rendering entity, the rendering process for a frame may conceptually be divided into following three example stages:

[0075] Application: In this stage, the scene may undergo transformations based on the execution logic. This may include effects of user interaction, such as key press / controller / mouse event; effects of time dependent animation logic; physics events, such as collisions and collision detection. In case the XR scene is rendered on a head mounted display (HMD), the user interaction may also include user’s head and eye movements. These transformations may change the relative position and orientation of some or all the objects in a scene. The output of these transformations may be a new scene. The new scene or a portion of it, together with associated mesh information (decomposed into primitives like polygons, lines or triangles) may be passed to the next stage.

[0076] Geometry stage: In this stage, the mesh information undergoes various geometric transforms to obtain vertices which are within the view frustrum of the virtual camera. These transforms may include model to world transform, camera transform, projection, clipping and culling, window viewport transforms, and the like. There may be additional operations like tessellation which modify the number of primitives associated with a particular mesh. Some effect of light sources on the objects (in terms of their vertices) may also be calculated. Some or all of these operations may be executed in parallel for each primitive, for example, using so called shader programs. The output of this stage may be primitives that fall within the view frustrum of camera specified in device coordinates.

[0077] Rasterization: In this stage, pixels in a frame buffer are filled based on transformed vertices from the geometry stage, their associated information like color, illumination, depth from the camera near plane, and the like13SUBSTITUTE SHEET (RULE 26)

[0078] This is an example rendering process, various other rendering processes exist, such as raytracing based rendering, point cloud based rendering, image based rendering. The stages of rendering enumerated here, and the operations discussed are non-exhaustive and may be carried out in various configurations and sequences.

[0079] 3GPP working group SA 4

[0080] Split rendering is envisioned to be available in 5G networks. The use case and possible high level architecture is defined in clause 6.2.5 of TR 26.928 available from [https: / / www.3gpp.org / ftp / Specs / archive / 26_series / 26.928 / 26928-i00.zip (last accessed on September 26, 2023)] which maps XR to 5G. 3GPP SA4 is considering split rendering as a service in many work and study items like MeCAR {permanent document available from [https: / / www.3gpp.org / ftp / TSG_SA / WG4_CODEC / 3GPP_SA4_AHOC_MTGs / SA4_VIDEO / Doc s / S4aV230050.zip (last accessed on September 26, 2023)] and TS 26.119 available from [https: / / www.3gpp.org / ftp / TSG_SA / WG4_CODEC / TSGS4_125_Gotheneburg / Docs / S4- 231560.zip (last accessed on September 26, 2023)] }, FS_SmartAR {(TR 26.806 available from(last accessed on September 26, 2023)] }, FS_XRTraffic {TR 26.926available from [https: / / www.3gpp.org / ftp / TSG_SA / WG4_CODEC / TSGS4_125_Gotheneburg / Docs / S4-231413.zip (last accessed on September 26, 2023)] }. Two ongoing work items in 3GPP SA4 working group specifically define split rendering architectures in 5G. These work items (WIs) are Split Rendering Media Service Enabler (SR_MSE) and IMS based conversational services (IBACS). In SR_MSE {TS 26.565 available from[https: / / www.3gpp.org / ftp / TSG_SA / WG4_CODEC / TSGS4_125_Gotheneburg / Docs / S4-231520.zip (last accessed on September 26, 2023)] } a general-purpose split rendering media service enabler is being developed and in IBACS (PD and TS) IMS based split rendering services and architectures are being developed specifically for conversational AR calls. Both WIs define an SRS (residing in the 5G network) and a SRC residing in the UE.

[0081] In SR_MSE, the SRS is defined as a 5G media function residing in a 5G edge server which is responsible for negotiation of an SR session with an SRC, monitoring edge server resource usage and managing / running the split rendering process.

[0082] In IBACS, the SRS is actuated as an AR media function (ARMF) and is envisioned as an IMS based media function (MF) or a media resource function (MRF) that has AR media processing capabilities.

[0083] In both architectures, split rendering may be conceptualized as being performed by an14SUBSTITUTE SHEET (RULE 26)SRC inside a UE and an SRS residing inside the 5G network. Both architectures envision an Application provider or Application service provider entity which may provide split rendering based applications to users.

[0084] FIG. 4a. illustrates split rendering architecture in an SR MSE. The split rendering architecture is described in TS 26.565 v0.6, introduces the following functions:- Split-Rendering Client (SRC) 402: This function is responsible to acquire the UE media capabilities and negotiates with an 5G-Real Time communication ( RTC) Application Server (AS) 404 to agree on the split-rendering process at the RTC AS. It should be noted that the RTC-AS may comprise a SRS and other functions, for example, for signaling.- Split-Rendering Server (SRS) 406: This function is responsible for negotiation of SR session with SRC 402, monitoring the server’s edge resource usage, and managing / running the split rendering process.- Application Function (AF) 408: responsible for provisioning, QoS allocation, and edge resource discovery.- Application Service Provider / Application Provider (AP) 410: The application provider that offers the service.- Application 412: The application running on UE 414- Media Session Handler (MSH) 416: is the entity on UE 414 that is responsible for the control plane communication with the AF 408.

[0085] The split rendering architecture in an SR MSE is described as following:- The application provider 410, e.g., an 5G Application Provider (AP) provisions the split-rendering through RTC-1.- In the use cases in which the application provider 410 is involved in the media delivery, the RTC-2 interface is used for this purpose.- The communication between AF 408 and SRS 406 is through RTC- 3. This interface may for instance include the EDGE-3 interface.- The signaling as well as the media delivery between SRC 402 and15SUBSTITUTE SHEET (RULE 26)SRS 406 is though RTC-4.- The AF 408 may provide the split-rendering information to the Media Session Handler (MSH) 416 defined by RTC-5, defined in TS 26.506.- SRC 402 in the UE 414 discovers the application through RTC-6 and handles an XR runtime 418.- The SRC 402 discovers the client media capabilities through the RTC-7 interface.- The application 412, e.g., an 5G application and application provider 410 interact through RTC-8.

[0086] FIG. 4b illustrates split rendering architecture in IBACS. Split rendering is described in IB ACS PD vO.5.0 as following:

[0087] In this architecture:- The split-rendering client (SRC) 402 is responsible on behalf of the UE 414 for negotiations with IMS 420 to find the split-rendering configuration.- The split-rendering function (SRF) 422 is responsible on behalf of the IMS 420 for negotiations with the UE 414 to find the split-rendering configuration.

[0088] The SRF 422 needs to allocate ARMF 424 media resources through DCSF 426 / MMTel AS 428 when receiving the split rendering request, to confirm whether there is enough media rendering resource.

[0089] The above entities use the following interfaces to interact with other entities:- SR-2 for an Augmented Reality (AR) (Application Server (AS) 430 and the ARMF 424 media delivery.SR-4 for the SRC 402 and the SRF 422 through DCMF / MRF 432 for negotiation on split: This interface is used for negotiation at the beginning of the media delivery session and / or during the media delivery session to update / change the split.- SR-6 for the SRC 402 to interact with an AR Application 434: This16SUBSTITUTE SHEET (RULE 26)interface is used by the AR Application 434 to request the SRC 402 to manage a split and to retrieve the status of the split management.- SR-7 for the SRC 402 to discover the client’s capabilities: The device capabilities are retrieved by the SRC 402 with this interface. The interface may provide static and dynamic capabilities, e.g., capabilities that do not change or may change during the media delivery session.

[0090] Split rendering session setup and operation in the SR_MSE WI is described in TS 26.565 v0.6 in the following two call flows illustrated in FIGs. 5 and 6.

[0091] FIG. 5 illustrates high-level call flow for client driven split rendering session setup. The high-level call flow for client driven split rendering session setup it is described in TS 26.565 v0.6 as:- The application service provider 502 requests the SRF, e.g., the RTC AF 504 the provisioning 503 a split management session.- The split management session is announced to the Application 506 as part of the Service Access Information 505.- The Application 506 requests 507 for split rendering of the client media functions from the SRC 508.- The SRC 508 inquires 509 the Media Session Handler 510 about media capabilities of the split rendering client 508.- The SRC 508 and SRS 512 negotiate 511 on the acceptable capabilities for the device and agree on the split option.- The SRS 512 starts the split rendering process 513.- The SRC 508 establishes 515 the WebRTC session.- The SRC 508 informs 517 the application 506 that the split-rendering on edge is running.- The SRC 508 sends uplink metadata 519, such as pose and action information.- The SRS 512 sends 521 the rendered media to the SRC.

[0092] FIG. 6 illustrates a high level operation of a split rendering session. FIG. 6, for example, is shown to include split rendering call flow and operation; and is described in TS 26.565 v0.6 as:- A Presentation Engine 602 discovers 603 a split rendering server 604 and sets up a connection to it. It provides information about its rendering capabilities and the XR runtime configuration, e.g., the OpenXR configuration may be used for this purpose.17SUBSTITUTE SHEET (RULE 26)- In response, the split rendering server 604 creates a description of the split rendering output 605 and the input it expects to receive from the UE / split rendering client 606.- The Presentation Engine 602 requests the buffer streams from a media access function (MAF) 610, which in turn establishes a connection 607 to the split rendering server 604 to stream pose and retrieve split rendering buffers.- A Source Manager 608 retrieves pose information and user actions 609 from an XR runtime 612.- The Source Manager 608 shares the pose information and user input / actions 611 with the split rendering server 604.- The split rendering server 604 uses that information to render 613 the frame.- The rendered frame is encoded and streamed down 615 to the MAF 610.- 617, 619, 621 correspond to decoding the received frame 617, passing it 619 to the XR runtime 612 and composing and displaying it 621, respectively.

[0093] The procedure for split rendering session set up and operation in IBACS WI is described in IBACS pd v0.5.0 as the call flow shown in FIG. 7.

[0094] FIG. 7 illustrates split rendering call flow in IBACS. IBACS is described in IBACS pd v0.5.0 as:- A UE-A 702 initiates an AR communication session and establishes audio and video session connections 703 with a UE-B 704. Then the bootstrap and application data channels are established for the UE-A 702 and UE-B 704.- When an AR Application 706 on the UE-A 702 discovers that media capabilities of a client cannot meet AR communication related media rendering requirements, the AR application 706 decides to start split rendering call flow 705.- The AR application 706 initiates the application data channels 707 between a SRC 708 and a SRF(DCMF) 710, for the split rendering confirmation request and metadata transmission.- The AR application 706 calculates which AR objects can be rendered by itself based on its status, and then decides some part of the AR objects to be rendered in the UE-A 702 and the others to be rendered in an IMS network 712. The AR application 706 sends a split rendering request 709 to the SRC 708, the SRC 708 sends a Split Rendering Confirmation Request 711 to the SRF / DCMF 710 through an established application data channel, the request includes the information of the AR objects to be rendered in the IMS network 712, the SRF / DCMF 710 transfers the request 713 to the AR AS 714.18SUBSTITUTE SHEET (RULE 26)- The AR AS 714 decides 715 whether to provide AR media rendering function based on the request message received from the UE-A 702, the media rendering resource available on the ARMF 716 and the split rendering provisioning status for the UE-A 702.- The AR AS 714 sends a request 717 to a DCSF 718 to allocate media rendering resource on the ARMF 716 for the AR objects that should be rendered in IMS network 714, the DCSF 718 transfers the request to an MMTel AS 720.- The MMTel AS 720 forwards the media resource allocation request 719 to the ARMF 716, to reserve AR media rendering resource for the UE-A 702.- When an ARMF resource is applied successfully, the MMTel AS 720 return a successful response 721 to the DCSF 718, the DCSF 718 transfers the response to the AR AS 714.- The AR AS 714 returns a successful Split Rendering Confirmation Response 723 carrying the confirmation result to the SRF / DCMF 710. The SRF(DCSF) 718 then transfers the response message 725 to the UE-A 702 through the application data channel.- The SRC 708 acknowledges 727 the AR Application 706 that the split- rendering on IMS network is running- The MMTel AS 720 then initiates a media re-negotiation procedure 729, to anchor an audio / video connection of the UE-A 702 and UE-B 704 731 to the ARMF 716 for media rendering.- The AR application 706 requests the SRC 708 to start AR media split rendering procedure 733 based on the response received in 727.- The SRC starts AR media rendering 735 based on the confirmation result.- The SRC 708 retrieves metadata 737 such as pose and user input from a Runtime 722.- The SRC 708 sends metadata 739 to the SRF / DCMF 710 through the dedicated application data channel(s), the metadata includes the information of the AR objects that should be rendered in the IMS network 712, then the SRF / DCMF 710 transfer the metadata 741 to the ARMF 716.- The ARMF 716 renders a part of the AR objects 743 based on the metadata received.- The SRC 708 renders the part of AR objects 745 (optional) that should be done in the UE-A 702 according to the confirmation result in 725. 745 is not needed when all the AR objects are rendered in the IMS network 712.- The SRC 708 sends audio / video media stream including rendered AR objects 747 to the ARMF, when all the AR objects are rendered in the IMS network 712, the UE-A 702 should send original audio / video media stream to the ARMF 716.19SUBSTITUTE SHEET (RULE 26)- The ARMF 716 decodes the audio / video media stream received from the UE-A 702, and combines 749 with the AR media rendered by the ARMF, then sends the combined audio / video media stream 751 to the UE-B.- The ARMF 716 sends the rendered audio / video overlay RTP media stream 753 to the SRC 708 when it needs to send video back to the UE-A 702 through the video back channel.- The SRC 708 composes and renders the frame 755, then transfers to the AR application 706 for display.

[0095] Split-rendering as developed in SR_MSE is static in nature when considering the split of the rendering workload between the SRS and SRC. In fact, the only scenario discussed so far is when the whole scene is rendered at the SRS, no mechanisms of true split rendering operations have been developed yet. This minimalistic case of split rendering fails to fully leverage the rendering resources that may be available at the UE. The split is negotiated at the start of the session and stays like that till the end of the session. This approach fails in considering the variable nature of resources available to the SRC, the network conditions, nature of the content being rendered and the conditions at the SRS. For example, during a split rendering session, the UE (SRC) may start running low on battery power or the capacity of the communication channel between the SRS and SRC may change.

[0096] In IB ACS pd v0.5.0, rendering is envisioned to be true split rendering in that a portion of the rendering workload may be handled by the UE. Further, in clause 2.3.1, it is mentioned that UE may decide and change the split ratio depending on its status, the means, mechanisms and supporting signaling is not developed.

[0097] Currently, there are no considerations and mechanisms for adapting the split ratio of the rendering operations between the SRS and the SRC.

[0098] The problems discussed above may be solved by adaptive split rendering between an SRS and an SRC. This may, for example, be achieved by:

[0099] Enabling mechanisms at the SRS and SRC to monitor factors affecting their operating environment rendering abilities and to either monitor or receive information about factors that affect the QoE of the SR session.

[0100] Enabling the SRS and SRC to signal to each other (or other entities involved in the SR session) changes in their operating environment, requests for change in the distribution of the rendering tasks, desired distribution of rendering tasks, acceptable distributions of rendering tasks, and the like.20SUBSTITUTE SHEET (RULE 26)

[0101] Enabling the SRS, SRC or other network entities involved in the SR session to react, acknowledge, negotiate, or forward these signals. The reactions may include accepting the received distribution of rendering tasks and acknowledging the acceptance; actuating the received distribution of rendering tasks and signaling the actuation, replying with acceptable distributions of rendering tasks, rejecting a request for change in distribution of rendering tasks; verifying ability to change the distributions and forwarding the received signals to another network entity, e.g., an SRS / AR MF.

[0102] FIGs. 8 and 8b illustrate a call flow for split rendering, in accordance with an embodiment. In this implementation, in the context of a split rendering architecture, an SRC is located in a UE and an SRS may be any 5G network entity with rendering capabilities and capable of supporting SR operations for one or more UEs.

[0103] At 802, a split rendering session is negotiated and initialized. Entities not shown in the call flow may be involved in this step or prior to this step 1, e.g., for provisioning the necessary resources on the 5G network side, or for assessing the UE capabilities. Application and scene information, e.g., using a scene description document in the appropriate format may also be delivered to the SRC or SRS prior or during 802.

[0104] An SRS 806 and an SRC 804 may agree on a split for SR task distribution. Information about the media and meta data to be exchanged between the SRS 806 and SRC 804 during the SR session may be agreed in at 802 as well. The SRS 806 and SRC 804 may acquire the actual assets referenced in the scene description document based on the agreed split.

[0105] In some embodiments, the initial SR task distribution may be signaled within the initial scene description that is exchanged, for example, as a property field for each node in the scene description or scene graph, indicating whether the rendering operations for the object are executed at the SRS or SRC.

[0106] An example SR task distribution for a scene comprising nodes or objects indexed from 1 to 10 may look like:

[0107] Alternatively, it could be expressed as {“SRS”: [ 1, 3, 5, 7, 8,9,10],”SRC”: [2,4,6] } .

[0108] In some embodiments, an SR task distribution may comprise one or more URIs which21SUBSTITUTE SHEET (RULE 26)identify an SR task distribution resource. An SR task distribution resource identified by a URI may comprise one or more of:• A scene description or a scene graph which comprises a portion of the whole scene to be rendered. In this exemplary embodiment, a scene description may be indicated for a specific rendering entity.• A scene description resource or a scene graph in which objects comprise a parameter which identifies rendering entity for the object. In an example embodiment, the parameter may be “rendering entity” with possible values from [ “SRS”, “SRC”], wherein “rendering-entity”=”SRS” indicates that the object is to be rendered by the SRS and “ rendering-entity”=”SRC” indicates the object is to be rendered by the SRC.• A data structure, comprising key-value pairs, wherein the key identifies the rendering entity and the value comprises a list of identifiers for objects in a scene to be rendered at the rendering entity.As an exemplary embodiment, this data structure could be represented as JSON formatted text, for example, for a scene comprising objects or nodes indexed from 1 to 10, the URI may identify a task distribution: {“SRS”:[ 1, 3, 5, 7, 8,9,10],”SRC”: [2,4,6] }, indicating that when this SR task distribution is actuated by the SRS and SRC, objects / nodes 1,3,5,7,8,9,10 are rendered at SRS and objects 2,4 and 6 are rendered at SRC.• A data structure comprising key-value pairs, wherein a key identifies the rendering entity and a corresponding value comprises a scene description or a scene graph to be rendered at the identified rendering entity.

[0109] In another embodiment, an SR task distribution may be expressed as the portion of total rendering operations (e.g., number of polygons or triangles, shader passes, rendered volume, and the like) executed by the server vs the portion rendered by the client. For example, the scene may be composed of objects with a total of 10k triangles; 8k would be rendered by the server and 2k by the client. In that case, the task distribution would 80% vs 20%.

[0110] In an embodiment, the task distribution may be represented as URIs as described in previous embodiments.

[0111] In some embodiments, URIs may further comprise a field for proportion of rendering operations, for example, triangles, polygons, and the like. An exemplary SR task distribution may be of the form : distl: { “SRS”: [80, URI1],”SRC”: [20,URI2] }22SUBSTITUTE SHEET (RULE 26)

[0112] In another embodiment, the SR task distribution is expressed as bounding volumes to be rendered by the SRS and SRC.

[0113] In an embodiment, the bounding volumes are represented as bounding volume hierarchies.

[0114] In some embodiments, the SRS and SRC may agree on a list of SR task distributions that are invokable during the session. SRS and SRC may the acquire actual assets referenced in the scene description document based on the agreed list of invokable SR task distributions.

[0115] An example SR task distribution list could take the form of a list with each member comprising an SR task distribution., e.g., an agreed list of invokable SR task distributions could look like [“distl”: {“SRS”: [ 1, 3, 5, 7, 8, 9, 10],”SRC”: [2, 4, 6] }, “dist2”: {...}, ..., ”distn”: {...}] . Here distl, dist2, ... distn are the identifiers of the task distributions. In an embodiment, the task distribution list is represented with a URI.

[0116] At 808: The SR process is started and transport channels for the media and meta-data exchange are established. The transport channels may be, e.g., WebRTC media streams, IMS media streams, WebRTC or IMS data channels.

[0117] At 810 and 810 the rendering loop of the SR session is initiated.

[0118] At 810: The SRC 804 sends metadata required for rendering operations to the SRS 806.

[0119] At 812: The SRS renders media and transmits it to the SRC 804.

[0120] Concurrent to the rendering loop of 810 and 812, the SR split adaptation proposed in some example embodiment, may be carried out as follows.

[0121] At 814: The SRC 804, SRS 806 and other possible entities, either in the UE or in the network monitor the factors which may impact either the rendering capabilities of the SRS 806 or the SRC 804 or the user’s QoE.

[0122] At 816: An example SR adaptation trigger is detected or received by the SRC 804. Alternatively, the SR adaptation trigger may be detected or received by the SRS 806 or by any other entity involved in the SR session and relayed to either the SRS 806 or the SRC 804. The trigger may be a change in the operating environment of the UE, the SR session, the SRS 806, the scene being rendered in the SR session, or a factor affecting QoE or the energy efficiency of the SR session.

[0123] In one embodiment, 816 is followed by 818, 820 and 822.23SUBSTITUTE SHEET (RULE 26)

[0124] At 818: The SRC 804 on receiving or detecting the SR adaptation trigger, calculates a desirable SR task distribution considering the nature of the SR adaptation trigger, e.g., factors like UE’s capabilities, network conditions, SRS capabilities, scene being rendered, QoE considerations, energy efficiency factors, and the like.

[0125] At 820: The SRC 804 signals a request for the new desired SR task distribution to the SRS 806.

[0126] At 822: The SRS 806 acknowledges the received request and agrees to the new SR task distribution.

[0127] In another embodiment, 816 is followed by 824, 826, and 828.

[0128] At 824: The SRC 804 calculates a desirable SR task distribution similar to that in 822.

[0129] At 826: The SRC 804 signals a request for the new desired SR task distribution to the SRS 806.

[0130] At 828: The SRS 806 acknowledges the received request, signals a rejection of the request. The SRS 806 may signal an offer of one or more SR task distributions to the SRC 804.

[0131] At 830: The SRC 804 acknowledges the rejection and the offer and accepts one of the offered task distributions. In some embodiments, this may be followed the SRS 806 acknowledging the acceptance by SRC 804.

[0132] In another embodiment, 816 may be followed by 832, 834, 836, and 838.

[0133] At 832: The SRC 804 reports the SR adaptation trigger to the SRS 806. In some embodiments, the current UE capabilities are also shared along with the SR adaptation trigger.

[0134] At 834: The SRS 806 calculates a desirable SR task distribution considering the nature of the SR adaptation trigger and factors like UE’s capabilities, network conditions, SRS capabilities, scene being rendered, QoS and charging policy considerations, QoE considerations, energy efficiency factors, and the like.

[0135] At 836: The SRS 806 sends acknowledgement of split adaptation trigger to the UE. Furthermore, it offers one or more task distributions based on the trigger.

[0136] At 838: The SRC 804 (e.g., UE) acknowledges the offer with its preferred task distribution from the ones offered.24SUBSTITUTE SHEET (RULE 26)

[0137] It should be noted that while the discussion above exemplarily shows SRC as the location of certain operations and origin of certain signals, these operations may be executed at, and these signals may originate from the SRS. The same possibility holds for operations or signals shown to execute at or originate from the SRS and these operations may be executed at the SRC and the signals may originate from the SRS.

[0138] In some embodiments, a trigger for SR adaptation may be one or more of the following:• Change in UE battery charge below a threshold.• Change in UE temperature beyond a threshold.• Change in available memory or computational capacity for rendering at the UE (e.g., due to increase in spatial computation needs of an AR application, execution of other applications)• Changes in the scene being rendered which may affect the QoE of the user at the current SR task distribution e.g., some objects in a scene or parts of a scene which are rendered at the SRS under the current SR task distribution are now closer in the viewer’s viewport and the motion-to-photon latency for these objects manifests as judder in the frames composited at the UE. o and may be more / less adequate for offloading to SRS. o Number of participants in an AR call. o Motion-to-render-to-photon latency. o Render-to-photon latency of QoE critical parts of the scene. o Render-to-photon latency of parts of the scene rendered at SRS o Compute resources available at the SRS or SRC.• Changes in network link conditions including capacity, latency and packet loss, and the like• Changes in the SRS operating conditions e.g., number of concurrent SRCs being served, total capacity, change in resources provisioned, and the like• Changes in the scene description e.g., a scene description update o Changes in scene graph, e.g. as enabled and indicated by MPEG_scene_dynamic extension in a MPEG-I Scene description.• Additional triggers may be user-initiated e.g., via UI interaction or HMD / controller / peripheral device gesture or motion pattern: o Quality or service upgrade / downgrade requested by user. o Rendered graphics quality change requested by the user.25SUBSTITUTE SHEET (RULE 26)• Changes in the energy efficiency factors at the SRS and / or the SRC. The energy efficiency factors (not just based on battery consumption, but on more complex energy estimations) may be used as triggers for re-balancing the proportion of distributed tasks between SRS and SRC.

[0139] In some embodiments, the SRS and SRC exchange a list of possible SR task distributions during SR session initialization and setup. Upon detection or reception of an SR adaptation trigger, the desirable SR task distribution is chosen from the list and an identifier of the SR task distribution is signaled.

[0140] In some embodiments, the SRS and the SRC have access to a common scene description, which is, for example, shared at session initialization and possibly dynamically updated during the session. In such cases upon detection or reception of an SR adaptation trigger, the desirable SR task distribution comprises choosing elements identified in the scene description to be rendered locally or to be offloaded. In such cases the request signal for a new SR task distribution comprises these object identifiers from the scene description.

[0141] In some embodiments, only those objects whose rendering location is affected by the new SR task distribution are signaled when requesting a new task distribution.

[0142] In some embodiments, the scene description is in accordance with glTF2.0.

[0143] In some embodiments, the scene description is in accordance with glTF2.0 extensions defined by MPEG-I in ISO / IEC 23090-14.

[0144] In some embodiments, wherein the scene description is based on glTF2.0, the object identifiers used in signaling the SR task distribution update requests, acceptance, offers, acknowledgements are indices of the objects in the scene description.

[0145] In some embodiments, the scene description comprises one or more objects in accordance with one or more of:• Open Universal Scene Description (OpenUSD);• Extensible 3D (X3D);• Vulkan Scene Graph;• Open Scene Graph;• Collada;• glTF.26SUBSTITUTE SHEET (RULE 26)

[0146] In some embodiments, the scene rendered is available to the SRS and SRC in the form of a scene graph wherein nodes of the scene graph may represent objects in the scene. In such cases the desirable SR task distribution may be signaled as identifiers of the nodes representing objects desired to be rendered locally or to be offloaded.

[0147] In some embodiments, the desirable SR task distribution signal comprises only identifiers of the nodes corresponding to objects whose rendering entity is affected by the desired task distribution for example, if an object was rendered by the SRS in an earlier task distribution and will be rendered by the SRC in the new task distribution, its identifier may be included in the SR task distribution signal.

[0148] In some embodiments, the signals carrying requests for, offers of, acknowledgement of, acceptance of, or rejection of new SR task distributions are transmitted in RTCP packets.

[0149] In some embodiments, an RTCP feedback packet comprises an application layer feedback message as defined in RFC4585, which may be signalled during an SDP based session set up using the rtcp-fb attribute, for example, a=rtcp-fb:*sr-split. The SR task distribution information may be carried in Feedback Control Information field of the packet.

[0150] In some embodiments, an RTCP feedback packet is sent in immediate, early or regular feedback mode. In some embodiments, the signals carrying requests for, offers of, acknowledgement of, acceptance of, or rejection of new SR task distributions are transmitted in header extensions added to RTP packets.

[0151] In some embodiments, an RTCP feedback message requesting a new SR task distribution comprises information identifying at least portions of the scene being rendered in the SR session and information indicating desired rendering entity.

[0152] In an example of the above, the RTCP feedback message may include an indication that the server should perform the rendering for certain objects. For example, the device may detect that it is overheating or the allocated rendering resources for the split rendering session are decreasing. In that case, it may send the IDs of certain objects to the SRS indicating that those should be rendered by the SRS. In an alternative, the device may signal an updated task distribution in another form, e.g., an increased number of triangles to be rendered by the SRS.

[0153] In some embodiments, an RTP header extension acknowledging the receipt or acceptance of a new SR task distribution comprises information identifying at least portions of the scene being rendered in the SR session and information indicating current rendering entity.27SUBSTITUTE SHEET (RULE 26)

[0154] In an example of the above, the SRS may insert an RTP header extension to the RTP stream that carries the pre-rendered media. The RTP header extension may indicate the requests for, acknowledgement of, rejection of, or acceptance of updated task distribution in any of the forms mentioned above (e.g., rendered objects / polygons / triangles distribution). This extension does not necessarily need to be inserted to each RTP packet; it may be sent less frequently, e.g. once per frame or only when a change in the task distribution occurs, albeit preferably at least a few times to avoid losing information in case the packet carrying the extension is lost. In some embodiments, supplemental enhancement information (SEI) messages in the media stream may be used to signal requests for, acknowledgement of, rejection of, or acceptance of a new task distribution.

[0155] In some embodiments, the SRS is an IMS based media function with AR rendering capabilities, the SRC is in an MTSI client, and the signals carrying requests for, offers of, acknowledgement of, acceptance of, or rejection of new SR task distributions are transmitted over an IMS data channel.

[0156] In some embodiments, the SRS is an IMS based media function with AR rendering capabilities, the SRC is in an MTSI client, and the signals carrying requests for, offers of, acknowledgement of, acceptance of, or rejection of new SR task distributions are transmitted over an IMS media channel.

[0157] In some embodiments, the SRS and SRC are conformant to specifications in 3GPP TS 26.565 and the signals carrying requests for, offers of, acknowledgement of, acceptance of, or rejection of new SR task distributions are transmitted over the user plane data and signaling interface RTC-4.

[0158] In some embodiments, the SRS and SRC are conformant to specifications in 3GPP TS 26.565 and the signals carrying requests for, offers of, acknowledgement of, acceptance of, or rejection of new SR task distributions are transmitted over the user plane data and signaling interface RTC-4 using WebRTC.

[0159] In an embodiment, the signals carrying requests for, offers of, acknowledgement of, acceptance of, or rejection of new SR task distributions are transmitted over RTC-4s.

[0160] In an embodiment, the signals carrying requests for, offers of, acknowledgement of, acceptance of, or rejection of new SR task distributions are transmitted over RTC-4s using the SWAP protocol defined in TS 26.113.

[0161] In an embodiment, the signals carrying requests for, offers of, acknowledgement of, acceptance of, or rejection of new SR task distributions are transmitted as application specific SWAP28SUBSTITUTE SHEET (RULE 26)messages.

[0162] In an embodiment, the application specific SWAP messages are identified by the message type “urn:3gpp:sr-mse:” appended by a string, e.g., “sr-split”, “sr-ratio”, “sr-disf ’, and the like.

[0163] In an embodiment, the signals carrying requests for, offers of, acknowledgement of, acceptance of, or rejection of new SR task distributions are transmitted over a WebRTC data channel subprotocol identified as “3gpp-sr-metadata”, as defined in TS 26.565.

[0164] In an embodiment, the signals carrying requests for, offers of, acknowledgement of, acceptance of, or rejection of new SR task distributions are transmitted over a WebRTC data channel subprotocol identified as “3gpp-sr-splif ’.

[0165] In an embodiment, the sub-protocol messages comprising the signals carrying requests for, offers of, acknowledgement of, acceptance of, or rejection of new SR task distributions comprise a urn of the form “urn: 3gpp: split-rendering: vl:” appended by a string, e.g., “sr-split”, “sr-ratio”, “sr- dist”, etc..

[0166] In an embodiment, the signals carrying requests for, offers of, acknowledgement of, acceptance of, or rejection of new SR task distributions are transmitted over RTC-4s using the data channel subprotocol.

[0167] In an alternative embodiment, the signals carrying requests for, offers of, acknowledgement of, acceptance of, or rejection of new SR task distributions are transmitted over RTC-4m.

[0168] In some embodiments, where the SRS and SRC are conformant to TS 26.565, upon detection or reception of a trigger for SR task distribution if the SRS and SRC fail to agree on a new SR task distribution, the SRS or SRC trigger an update of the current Split-Rendering Configuration as provisioned by the application provider on the split rendering AF. SRS may request an update by communicating with the AF via RTC-3.

[0169] In some embodiments, the update of the current Split-Rendering Configuration changes the QoS resources allocated to the SR session, e.g., render resources at the SRS, data bearer resources, and the like.

[0170] In embodiments, the SRC signals the Split-Rendering Configuration update request to the application over RTC-7 and the application forwards the request to the application provider over RTC-8 which updates the Split-Rendering Configuration on the SR AF by invoking one of HTTP29SUBSTITUTE SHEET (RULE 26)PUT or PATCH requests with the relevant SplitRenderingConfiguration to the AF over RTC-1.

[0171] In some embodiments, the signals carrying requests for, offers of, acknowledgement of, acceptance of, or rejection of new SR task distributions comprise at least one message identifier. In some embodiments, the message identifier uniquely identifies the signal or a message comprised in the signal within a split rendering session. In some embodiments, the message identifier identifies the signal or the message comprised in the signal as one or more of: a request for, offers of, acknowledgement of, acceptance of, or rejection of the new SR task distributions.

[0172] In some embodiments, the phrase split rendering task distribution (SR task distribution) is used in the above text as a non-limiting way to describe the division of computational tasks and / or operations required to produce a frame of a computationally generated or augmented experience, for example a video game, a VR / AR / MR / XR experience, and the like. Such a division may produce two or more sets of rendering tasks, which when executed using appropriate rendering apparatus each produce an output. Outputs from two or more sets of rendering tasks may be combined by an appropriate apparatus to compose a rendered frame. Such a division of rendering tasks may be actualized in various dimensions and granularities for a given scene and a rendering process. The above examples illustrate the case wherein the scene (as indicated by a scene description) is divided to affect the division of rendering tasks. In alternative embodiments, the division is affected in the rendering pipeline, such that some of the rendering operations for the whole scene are executed at a first rendering entity, while the some or all of the remaining rendering operations for the whole scene are executed by a second rendering entity, the first rendering entity being separate from the second rendering entity. In some embodiments, the two approaches may be combined, so that some rendering operations for a first subset of objects in a scene are executed by a first rendering entity and while other rendering operations for the first subset of objects are executed by a second rendering entity. Some or all rendering operations of a second subset of objects in a scene may be executed wholly by the first rendering entity or by the second rendering entity or by a third rendering entity.

[0173] The transport protocols mentioned in this invention must be considered as examples and are not limiting the applicability of the invention to other protocols such as SDP, HTTP, RTSP, SIP, QUIC and alike.

[0174] FIG. 9 is an example apparatus 900, which may be implemented in hardware, configured to implement the examples described herein. The apparatus 900 comprises at least one processor 902 (e.g., an FPGA and / or CPU), one or more memories 904 including computer program code 905, the computer program code 905 having instructions to carry out the methods described herein, wherein the at least one memory 904 and the computer program code 905 are configured to, with30SUBSTITUTE SHEET (RULE 26)the at least one processor 902, cause the apparatus 900 to implement circuitry, a process, component, module, or function (implemented with control module 906) to implement the examples described herein, including implementing adaptive split rendering. Optionally included encoder 930 of the control module 906 performs encoding, and optionally included decoder 940 implements decoding. The memory 904 may be a non-transitory memory, a transitory memory, a volatile memory (e.g. RAM), or a non-volatile memory (e.g., ROM).

[0175] The apparatus 900 includes a display and / or I / O interface 908, which includes user interface (UI) circuitry and elements, that may be used to display features or a status of the methods described herein (e.g., as one of the methods is being performed or at a subsequent time), or to receive input from a user such as with using a keypad, camera, touchscreen, touch area, microphone, biometric recognition, one or more sensors, and the like. The apparatus 900 includes one or more communication, e.g., network (N / W) interfaces (I / F(s)) 910. The communication I / F(s) 910 may be wired and / or wireless and communicate over the Intemet / other network(s) via any communication technique including via one or more links 924. The communication I / F(s) 910 may comprise one or more transmitters or one or more receivers.

[0176] The transceiver 916 comprises one or more transmitters 918 and one or more receivers 920. The transceiver 916 and / or communication I / F(s) 910 may comprise standard well-known components such as an amplifier, filter, frequency-converter, (de)modulator, and encoder / decoder circuitries and one or more antennas, such as antennas 914 used for communication over wireless link 922.

[0177] The control module 906 of the apparatus 900 comprises one of or both parts 906-1 and / or 906-2, which may be implemented in a number of ways. The control module 906 may be implemented in hardware as control module 906-1, such as being implemented as part of the one or more processors 902. The control module 906-1 may be implemented also as an integrated circuit or through other hardware such as a programmable gate array. In another example, the control module 906 may be implemented as control module 906-2, which is implemented as computer program code (having corresponding instructions) 905 and is executed by the one or more processors 902. For instance, the one or more memories 904 store instructions that, when executed by the one or more processors 902, cause the apparatus 900 to perform one or more of the operations as described herein. Furthermore, the one or more processors 902, one or more memories 904, and example algorithms (e.g., as flowcharts and / or signaling diagrams), encoded as instructions, programs, or code, are means for causing performance of the operations described herein.

[0178] The apparatus 900 to implement the functionality of control 906 may correspond to any31SUBSTITUTE SHEET (RULE 26)of the apparatuses depicted herein. Alternatively, apparatus 900 and its elements may not correspond to any of the other apparatuses depicted herein, as apparatus 900 may be part of a self- organizing / optimizing network (SON) node or other node, such as a node in a cloud.

[0179] The apparatus 900 may also be distributed throughout the network (e.g., internet 28) including within and between apparatus 900 and any network element (such as a base station 24 and / or apparatus 90).

[0180] Interface 912 enables data communication and signaling between the various items of apparatus 900, as shown in FIG. 9. For example, the interface 912 may be one or more buses such as address, data, or control buses, and may include any interconnection mechanism, such as a series of lines on a motherboard or integrated circuit, fiber optics or other optical communication equipment, and the like. Computer program code (e.g., instructions) 905, including control 906 may comprise object-oriented software configured to pass data or messages between objects within computer program code 905. The apparatus 900 need not comprise each of the features mentioned, or may comprise other features as well. The various components of apparatus 900 may at least partially reside in a common housing 928, or a subset of the various components of apparatus 900 may at least partially be located in different housings, which different housings may include housing 928.

[0181] FIG. 10 shows a schematic representation of non-volatile memory media 1000a (e.g. computer / compact disc (CD) or digital versatile disc (DVD)) and 1000b (e.g. universal serial bus (USB) memory stick) and 1000c (e.g. cloud storage for downloading instructions and / or parameters 1002 or receiving emailed instructions and / or parameters 1002) storing instructions and / or parameters 1002 which when executed by a processor allows the processor to perform one or more of the operations of the methods described herein.

[0182] FIG. 11 is an example method 1100, based on the example embodiments described herein. At 1102, the method 1100 includes monitoring or receiving information about factors that affect rendering capabilities of a device or about factors that affect a quality of experience of an ongoing split rendering session. At 1104, the method 1100 includes creating a signal based on changes in the rendering capabilities or factors that affect the quality of experience of the ongoing split rendering session. At 1106, the method 1100 includes reacting, acknowledging, negotiating, accepting, rejecting, or forwarding the signal.

[0183] References to a ‘computer’, ‘processor’, etc. should be understood to encompass not only computers having different architectures such as single / multi-processor architectures and sequential / parallel architectures but also specialized circuits such as field-programmable gate arrays (FPGAs),32SUBSTITUTE SHEET (RULE 26)application specific circuits (ASICs), signal processing devices and other processing circuitry. References to computer program, instructions, code etc. should be understood to encompass software for a programmable processor or firmware such as, for example, the programmable content of a hardware device such as instructions for a processor, or configuration settings for a fixed- function device, gate array or programmable logic device, etc.

[0184] As used herein, the term ‘circuitry’, ‘circuit’ and variants may refer to any of the following: (a) hardware circuit implementations, such as implementations in analog and / or digital circuitry, and (b) combinations of circuits and software (and / or firmware), such as (as applicable): (i) a combination of processor(s) or (ii) portions of processor(s) / software including digital signal processor(s), software, and one or more memories that work together to cause an apparatus to perform various functions, and (c) circuits, such as a microprocessor(s) or a portion of a microprocessor(s), that require software or firmware for operation, even when the software or firmware is not physically present. As a further example, as used herein, the term ‘circuitry’ would also cover an implementation of merely a processor (or multiple processors) or a portion of a processor and its (or their) accompanying software and / or firmware. The term ‘circuitry’ would also cover, for example and when applicable to the particular element, a baseband integrated circuit or applications processor integrated circuit for a mobile phone or a similar integrated circuit in a server, a cellular network device, or another network device. Circuitry or circuit may also be used to mean a function or a process used to execute a method.

[0185] It should be understood that the foregoing description is only illustrative. Various alternatives and modifications may be devised by those skilled in the art. For example, features recited in the various dependent claims could be combined with each other in any suitable combination(s). In addition, features from different embodiments described above could be selectively combined into a new embodiment. Accordingly, the description is intended to embrace all such alternatives, modifications and variances which fall within the scope of the appended claims.

[0186] The following acronyms and abbreviations that may be found in the specification and / or the drawing figures are defined as follows (the abbreviations may be appended with each other or with other characters using, e.g., a hyphen or dash (-), and may be case insensitive): augmented reality ARIP multimedia subsystem IMS multimedia telephony service for IMS MTSI33SUBSTITUTE SHEET (RULE 26)mixed reality MR quality of experience QoE split rendering SR split rendering server SRS split rendering client SRC user equipment UE virtual reality VR extended reality XR34SUBSTITUTE SHEET (RULE 26)

Claims

CLAIMSWhat is claimed is:

1. An apparatus comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the apparatus at least to perform: monitoring or receiving information about factors that affect rendering capabilities of a device or about factors that affect a quality of experience of an ongoing split rendering session; creating a signal based on changes in the rendering capabilities or factors that affect the quality of experience of the ongoing split rendering session; and reacting, acknowledging, negotiating, accepting, rejecting, or forwarding the signal.

2. The apparatus of claim 1, wherein the signal comprises a new distribution of rendering tasks.

3. The apparatus of any of the claims 1 or 2, wherein the signal further comprises at least one first message identifier, wherein the at least one first message identifier uniquely identifies the signal or a message comprised in the signal within a split rendering session or the ongoing split rendering session.

4. The apparatus of any of the claims 1 to 3, wherein the signal further comprises at least one second message identifier, wherein the at least one second message identifier identifies the signal or a message comprised in the signal as one or more of: a request for, offers of, acknowledgement of, acceptance of, or rejection of the new distribution of the rendering tasks.

5. The apparatus any of the claims 1 to 4, wherein the factors that affect the quality of experience of the ongoing split rendering session comprise one or more of the following: changes in operating environment; or changes in a scene being rendered; and wherein reacting, acknowledging, negotiating, or forwarding comprises one or more of requesting for changes in a distribution of rendering tasks; requesting for a desired distribution of the rendering tasks; or accepting the distribution of the rendering tasks.35SUBSTITUTE SHEET (RULE 26)6. The apparatus of any of the claims 1 to 5, wherein to react, the apparatus is further caused to perform one or more of the following: accepting a received distribution of rendering tasks and acknowledging the acceptance; actuating the received distribution of the rendering tasks and signaling the actuation; replying with acceptable the distribution of the rendering tasks; rejecting a request for change in the distribution of rendering tasks; or verifying ability to change the distribution of the rendering tasks and forwarding the received signals to another network entity.

7. The apparatus of any of the claims 2 to 6, wherein a distribution of the rendering task comprises one or more of the following: a scene description or a scene graph comprising a portion of a scene to be rendered; a scene description resource or a scene graph in which objects comprise a parameter which identifies rendering entity for an object; a data structure comprising key-value pairs, wherein the key identifies the rendering entity and the value comprises a list of identifiers for objects in the scene to be rendered at the identified rendering entity; or a data structure comprising the key-value pairs, wherein the key identifies the rendering entity and the corresponding value comprises the scene description or the scene graph to be rendered at the identified rendering entity.

8. The apparatus of any of the claims 2 to 7, wherein the distribution of the rendering tasks is indicated by one or more universal resource identifiers.

9. The apparatus of any of the previous claims, wherein the apparatus is further caused to perform: negotiating and initiating the split rendering session.

10. The apparatus of claim 9, wherein the apparatus is further caused to perform: starting the split rendering process; and establishing transport channels for media and meta-data exchange.

11. The apparatus claim 10, wherein the apparatus is further caused to perform:36SUBSTITUTE SHEET (RULE 26)signaling or receiving the meta-data required for the split rendering operations.

12. The apparatus claim 11, wherein the apparatus is further caused to perform: rendering the media; and transmitting or receiving the media.

13. The apparatus of claim 12, wherein the apparatus is further caused to perform: receiving, signaling, or detecting a split rendering adaptation trigger, wherein the split scene rendering trigger comprises a change in the capabilities of the device, a change in the ongoing split rendering session, a change in a split rendering server, a change in a scene being rendered in the ongoing split rendering session, the factors affecting quality of experience, or energy efficiency of the ongoing split rendering session.

14. The apparatus of claim 13, wherein the apparatus is further caused to perform: calculating a desirable split rendering task distribution based on the split rendering adaptation trigger; signaling or receiving a request for a new desired split rendering task distribution; receiving or signaling acknowledgement of the received request; and agreeing to the new split rendering (SR) task distribution.

15. The apparatus of claim 13, wherein the apparatus is further caused to perform: calculating a desirable split rendering task distribution; signaling or receiving a request for a new desired split rendering task distribution; receiving or signaling acknowledgement of the received request; receiving or signaling a rejection of the request for the new desired split rendering; signaling or receiving an offer of one or more offered split rendering task distributions; receiving or signaling acknowledgement of the rejection; and selecting one of the one or more offered split rendering task distributions as agreeable.

16. The apparatus of claim 13, wherein the apparatus is further caused to perform: calculating a desirable split rendering task distribution based on the split37SUBSTITUTE SHEET (RULE 26)rendering adaptation trigger and one or more factors comprising device capabilities, network conditions, a scene being rendered, quality of service, charging policy considerations, quality of experience considerations, or energy efficiency factors.

17. The apparatus of any of the claims 12 to 15, wherein the split rendering adaptation trigger comprises one or more of the following: change in the device battery charge below a threshold; change in the device temperature above a threshold; change in available memory or computational capacity for rendering at the device; changes in the scene being rendered that affects the quality of experience; number of participants in an augmented reality call; a motion-to-render-to-photon latency; a render-to-photon latency of quality of experience critical parts of the scene; a render-to-photon latency of parts of the scene; resources available; changes in network link conditions; changes in a scene description; changes in a scene graph; upgrade or downgrade in quality or service; changes quality of the rendered graphics; or changes in the energy efficiency factors at the device.

18. The apparatus of claim 7, wherein the scene description or a scene graph comprises one or more objects in accordance with one or more of the following:Open Universal Scene Description (OpenUSD);Extensible 3D (X3D);Vulkan Scene Graph;Open Scene Graph;Collada; or glTF.

19. The apparatus of any of the previous claims, wherein the apparatus or the device comprise one or more of a split rendering client or a split rendering server.

20. A method comprising:38SUBSTITUTE SHEET (RULE 26)monitoring or receiving information about factors that affect rendering capabilities of a device or about factors that affect a quality of experience of an ongoing split rendering session; creating a signal based on changes in the rendering capabilities or factors that affect the quality of experience of the ongoing split rendering session; and reacting, acknowledging, negotiating, accepting, rejecting, or forwarding the signal.

21. The method of claim 20, wherein the signal comprises a new distribution of rendering tasks.

22. The method of any of the claims 20 or 21, wherein the signal further comprises at least one first message identifier, wherein the at least one first message identifier uniquely identifies the signal or a message comprised in the signal within a split rendering session or the ongoing split rendering session.

23. The method of any of the claims 20 to 22, wherein the signal further comprises at least one second message identifier, wherein the at least one second message identifier identifies the signal or a message comprised in the signal as one or more of: a request for, offers of, acknowledgement of, acceptance of, or rejection of the new distribution of the rendering tasks.

24. The method of any of the claims 20 or 23, wherein the factors that affect the quality of experience of the ongoing split rendering session comprise one or more of the following: changes in operating environment; or changes in a scene being rendered; and wherein reacting, acknowledging, negotiating, or forwarding comprises one or more of requesting for changes in a distribution of rendering tasks; requesting for a desired distribution of the rendering tasks; or accepting the distribution of the rendering tasks.

25. The method of any of the claims 20 to 24 further comprising: accepting a received distribution of rendering tasks and acknowledging the acceptance; actuating the received distribution of the rendering tasks and signaling the actuation; replying with acceptable the distribution of the rendering tasks; rejecting a request for change in the distribution of rendering tasks; or verifying ability to change the distribution of the rendering tasks and39SUBSTITUTE SHEET (RULE 26)forwarding the received signals to another network entity.

26. The method of any of the claims 21 to 25, wherein a distribution of the rendering task comprises one or more of the following: a scene description or a scene graph comprising a portion of a scene to be rendered; a scene description resource or a scene graph in which objects comprise a parameter which identifies rendering entity for an object; a data structure comprising key-value pairs, wherein the key identifies the rendering entity and the value comprises a list of identifiers for objects in the scene to be rendered at the identified rendering entity; or a data structure comprising the key-value pairs, wherein the key identifies the rendering entity and the corresponding value comprises the scene description or the scene graph to be rendered at the identified rendering entity.

27. The method of any of the claims 21 to 26, wherein the distribution of the rendering tasks is indicated by one or more universal resource identifiers.

28. The method of any of the claims 20 to 27, wherein the method is further caused to perform: negotiating and initiating the split rendering session.

29. The method of claim 28 further comprising: starting the split rendering process; and establishing transport channels for media and meta-data exchange.

30. The method claim 29 further comprising: signaling or receiving the meta-data required for the split rendering operations.

31. The method claim 30 further comprising: rendering the media; and transmitting or receiving the media.

32. The method of claim 31 further comprising: receiving, signaling, or detecting a split rendering adaptation trigger, wherein the split scene rendering trigger comprises a change in the capabilities of the device, a change in the ongoing split rendering session, a change in a split rendering server, a change in a scene being rendered in the ongoing split rendering session,40SUBSTITUTE SHEET (RULE 26)the factors affecting quality of experience, or energy efficiency of the ongoing split rendering session.

33. The method of claim 32 further comprising: calculating a desirable split rendering task distribution based on the split rendering adaptation trigger; signaling or receiving a request for a new desired split rendering task distribution; receiving or signaling acknowledgement of the received request; and agreeing to the new SR task distribution.

34. The method of claim 33 further comprising: calculating a desirable split rendering task distribution; signaling or receiving a request for a new desired split rendering task distribution; receiving or signaling acknowledgement of the received request; receiving or signaling a rejection of the request for the new desired split rendering; signaling or receiving an offer of one or more offered split rendering task distributions; receiving or signaling acknowledgement of the rejection; and selecting one of the one or more offered split rendering task distributions as agreeable.

35. The method of claim 33 further comprising: calculating a desirable split rendering task distribution based on the split rendering adaptation trigger and one or more factors comprising device capabilities, network conditions, a scene being rendered, quality of service, charging policy considerations, quality of experience considerations, or energy efficiency factors.

36. The method of any of the claims 31 to 34, wherein the split rendering adaptation trigger comprises one or more of the following: change in the device battery charge below a threshold; change in the device temperature above a threshold; change in available memory or computational capacity for rendering at the device;41SUBSTITUTE SHEET (RULE 26)changes in the scene being rendered that affects the quality of experience; number of participants in an augmented reality call; a motion-to-render-to-photon latency; a render-to-photon latency of quality of experience critical parts of the scene; a render-to-photon latency of parts of the scene; resources available; changes in network link conditions; changes in a scene description; changes in a scene graph; upgrade or downgrade in quality or service; changes quality of the rendered graphics; or changes in the energy efficiency factors at the device.

37. The method of claim 26, wherein the scene description or a scene graph comprises one or more objects in accordance with one or more of the following:Open Universal Scene Description (OpenUSD);Extensible 3D (X3D);Vulkan Scene Graph;Open Scene Graph;Collada; or glTF.

38. The method of any of the claims 20 to 37, wherein the device comprise one or more of a split rendering client or a split rendering server.

39. An apparatus comprising: means for monitoring or receiving information about factors that affect rendering capabilities of a device or about factors that affect a quality of experience of an ongoing split rendering session; means for signaling changes in the rendering capabilities or factors that affect the quality of experience of the ongoing split rendering session; and means for reacting, acknowledging, negotiating, accepting, rejecting, or forwarding the signal.

40. The apparatus of claim 39, wherein the apparatus is further caused to perform the methods as claimed in any of the claims 20 to 38.42SUBSTITUTE SHEET (RULE 26)41. A computer readable medium comprising program instructions which, when executed by an apparatus, cause the apparatus to perform at least the following: monitoring or receiving information about factors that affect rendering capabilities of a device or about factors that affect a quality of experience of an ongoing split rendering session; creating signal based on changes in the rendering capabilities or factors that affect the quality of experience of the ongoing split rendering session; and reacting, acknowledging, negotiating, accepting, rejecting, or forwarding the signal.

42. The computer readable medium of claim 41, wherein the computer readable medium comprises a non-transitory computer readable medium.

43. The computer readable medium of any of the claims 41 or 42, wherein the computer readable medium causes the apparatus to further perform the methods as claimed in any of the claims 20 to 38.43SUBSTITUTE SHEET (RULE 26)

Images