Artificial intelligence (AI) assisted split rendering

Abstract

The invention relates to artificial intelligence (AI) assisted split rendering. A method performed by a split rendering client on a user device is provided. The method includes establishing a split rendering session for rendering a scene, wherein objects in the scene are to be rendered by a split rendering server on a network device. In this regard, the scene is to be displayed by the user device at least at a first level of quality. The method includes receiving, from the split rendering server, rendering media generated by the split rendering server from objects rendered at a lower second level of quality. The method includes inputting rendered media into a machine learning (ML) model to restore the rendered media to at least a first level of quality. And, the method includes synthesizing, for display, a view of the scene, where the view of the scene includes at least a first level of quality of rendering media.

Description

Technical Field

[0001] This disclosure relates generally to telecommunications, and more particularly to split-graphics rendering in telecommunications systems. Background Technology

[0002] A telecommunications system can be viewed as a facility that enables a communication session between two or more entities (such as user terminals, base stations, and / or other nodes) by providing carrier waves between the various entities involved in the communication path. For example, a telecommunications system can be provided by means of a communication network and one or more compatible communication devices. For instance, a communication session can include communications for carrying data, such as voice, video, email, text messaging, multimedia, and / or content data. Non-limiting examples of the services provided include two-way or multi-way calling, data communication or multimedia services, and access to data network systems such as the Internet.

[0003] In wireless telecommunications systems, at least a portion of a communication session between at least two stations occurs via a wireless link. Examples of wireless systems include Public Land Mobile Networks (PLMNs), satellite-based communication systems, and various wireless local area networks (e.g., Wireless Local Area Networks (WLANs)). Some wireless systems can be divided into cells and are therefore collectively referred to as cellular systems.

[0004] Users can access telecommunications systems using appropriate communication equipment or terminals. A user's communication equipment may be referred to as user equipment (UE) or user gear. The communication equipment is provided with appropriate signal receiving and transmitting means to enable communication, for example, to enable access to a communication network or to communicate directly with other users. The communication equipment can access a carrier provided by a station (e.g., a base station in a cell) and transmit and / or receive communication on that carrier.

[0005] Telecommunication systems and associated equipment generally operate according to a given standard or specification that defines what the various entities associated with the system are allowed to do and how they should be implemented. The communication protocols and / or parameters that should be used for connectivity are also generally defined. An example of a telecommunications system is the Universal Mobile Telecommunications System (UMTS). Other examples of telecommunications systems are Long Term Evolution (LTE), LTE-Advanced, and so-called 5G or New Radio (NR) networks. NR is being standardized by the 3rd Generation Partnership Project (3GPP). Summary of the Invention

[0006] The example implementations of this disclosure relate to split-graphics rendering in telecommunications, and particularly in telecommunications systems. For illustrative purposes, this disclosure includes, but is not limited to, the following example implementations.

[0007] Some example implementations provide an apparatus for implementing a split-rendering client on a user device, the apparatus comprising: at least one memory configured to store instructions; and at least one processing circuitry configured to access the at least one memory and execute the instructions such that the apparatus at least: establishes a split-rendering session for rendering a scene, wherein objects in the scene will be rendered by a split-rendering server on a network device, wherein the scene will be displayed by the user device at least at a first quality level; receives rendering media generated by the split-rendering server from objects rendered at a second quality level, wherein the second quality level is lower than the first quality level; inputs the rendering media into a machine learning (ML) model to restore the rendering media to at least the first quality level; and synthesizes a view of the scene for display, wherein the view of the scene includes: rendering media at least the first quality level.

[0008] Some example implementations provide a method performed by a split rendering client on a user device, the method comprising: establishing a split rendering session for rendering a scene, wherein objects in the scene will be rendered by a split rendering server on a network device, wherein the scene will be displayed by the user device at least at a first quality level; receiving rendering media generated by the split rendering server from objects rendered at a second quality level, wherein the second quality level is lower than the first quality level; feeding the rendering media into a machine learning (ML) model to restore the rendering media to at least the first quality level; and compositing a view of the scene for display, wherein the view of the scene includes: rendering media at least at the first quality level.

[0009] Some example implementations provide an apparatus for implementing a split rendering server on a network device, the apparatus comprising: at least one memory configured to store instructions; and at least one processing circuitry configured to access the at least one memory and execute the instructions such that the apparatus at least: establishes a split rendering session for rendering a scene, wherein objects in the scene will be rendered by the split rendering server, wherein the scene will be displayed by a user device at least at a first quality level; renders the objects at a second quality level to generate rendering media, wherein the second quality level is lower than the first quality level; sends the rendering media to a split rendering client on the user device, at the split rendering client, where the rendering media will be fed into a machine learning (ML) model to restore the rendering media to at least the first quality level; performs an update of the ML model based on the rendering media; generates an update to be applied to the ML model based on the rendering media; and sends the update to the split rendering client, at the split rendering client, where the update will be applied to the ML model to create an updated ML model.

[0010] Some example implementations provide a method performed by a split rendering server on a network device, the method comprising: establishing a split rendering session for rendering a scene, wherein objects in the scene will be rendered by the split rendering server, wherein the scene will be displayed by a user device at least at a first quality level; rendering the objects at a second quality level to generate rendering media, wherein the second quality level is lower than the first quality level; sending the rendering media to a split rendering client on the user device, at the split rendering client, where the rendering media will be fed into a machine learning (ML) model to restore the rendering media to at least the first quality level; performing an update of the ML model based on the rendering media; generating an update to be applied to the ML model based on the rendering media; and sending the update to the split rendering client, at the split rendering client, where the update will be applied to the ML model to create an updated ML model.

[0011] These and other features, aspects, and advantages of this disclosure will become apparent from the following detailed description and accompanying drawings, which will be briefly described below. This disclosure includes any combination of two, three, four, or more features or elements set forth in this disclosure, regardless of whether such features or elements are explicitly combined or otherwise referenced in the specific example implementations described herein. This disclosure is intended to be read holistically, such that any separable feature or element of this disclosure should be considered composable in any aspect and example implementation thereof, unless the context of this disclosure expressly provides otherwise.

[0012] Therefore, it should be understood that the content of this invention is provided merely for the purpose of summarizing some exemplary implementations in order to provide a basic understanding of some aspects of this disclosure. Consequently, it should be understood that the above exemplary implementations are merely examples and should not be construed as limiting the scope or spirit of this disclosure in any way. Other exemplary implementations, aspects, and advantages will become apparent from the following detailed description taken in conjunction with the accompanying drawings, which illustrate by way of example the principles of some of the described exemplary implementations. Attached Figure Description

[0013] After providing a general description of an example implementation of this disclosure, reference will now be made to the accompanying drawings, which are not necessarily drawn to scale, and in the drawings:

[0014] Figure 1 The illustration shows a telecommunications system implemented according to some examples of this disclosure, including one or more public land mobile networks (PLMNs) coupled to one or more external data networks;

[0015] Figure 2 The illustration shows a 5G deployment of a PLMN, sometimes referred to as a 5G system (5GS), based on some examples.

[0016] Figure 3A more detailed description is given based on some examples. Figure 2 The 5GS aspect;

[0017] Figure 4 The diagram illustrates the Split Rendering (SR) Media Service Enabler (MSE) architecture based on some examples.

[0018] Figure 5 The diagram illustrates a split-rendering architecture for IP Multimedia Subsystem (IMS)-based Session Services (IBACS) implemented using some examples;

[0019] Figure 6 The diagram illustrates the SR architecture implemented based on some examples;

[0020] Figure 7 It is a signaling diagram that is rendered with artificial intelligence (AI) assistance, based on some examples.

[0021] Figure 8 It is an AI-assisted split-rendering signaling diagram implemented based on other examples;

[0022] Figure 9A , Figure 9B and Figure 9C This is a flowchart illustrating the various steps in a method executed by a split-rendering client on a user's device, implemented according to various examples;

[0023] Figure 10A and Figure 10B This is a flowchart illustrating the various steps in a method executed by a split-rendering server on a network device, implemented according to various examples; and

[0024] Figure 11 The diagram illustrates a device implemented based on some examples. Detailed Implementation

[0025] Some implementations of this disclosure will now be described more fully below with reference to the accompanying drawings, which show some, but not all, implementations of this disclosure. In fact, various implementations of this disclosure may be embodied in many different forms and should not be construed as limited to the implementations set forth herein; rather, these exemplary implementations are provided to make this disclosure thorough and complete and to fully convey the scope of this disclosure to those skilled in the art. The same reference numerals throughout refer to the same elements.

[0026] Unless otherwise specified or the context clearly indicates, references to "first," "second," etc., should not be construed as implying a particular order. A feature described as being above another feature (unless otherwise stated or the context clearly indicates) may instead be located below that other feature, and vice versa; and similarly, a feature described as being to the left of another feature may instead be located to the right of that other feature, and vice versa. Furthermore, while quantitative measurements, values, geometric relationships, etc., may be referenced herein, any one or more of these (if not all) may be absolute or approximate to account for acceptable variations that may occur, such as those due to engineering tolerances, etc.

[0027] As used herein, unless otherwise specified or the context clearly indicates otherwise, "or" in the operand set is "inclusive or," and is therefore true if and only if one or more of the operands are true, as opposed to "exclusive or" (which is false when all operands are true). Thus, for example, "[A] or [B]" is true if [A] is true, or if [B] is true, or if both [A] and [B] are true. Furthermore, unless otherwise specified or the context clearly indicates the singular form, the articles "a" and "an" mean "one or more." Additionally, it should be understood that, unless otherwise stated, the terms "data," "content," "digital content," "information," and similar terms are sometimes used interchangeably. The term "network" can refer to a group of interconnected computers, including clients and servers; and within a network, these computers can be interconnected directly or indirectly in various ways, including via one or more switches, routers, gateways, access points, etc.

[0028] This document may use terms specific to a particular system, architecture, etc., but it should be understood that the exemplary implementations of this disclosure are equally applicable to any of a variety of systems, architectures, etc. For example, reference may be made to 3GPP technologies such as Global System for Mobile Communications (GSM), UMTS, LTE, Advanced LTE, 5G NR, Advanced 5G, and 6G; however, it should be understood that the exemplary implementations of this disclosure are equally applicable to non-3GPP technologies such as IEEE 802, Bluetooth, and Low Energy Bluetooth.

[0029] Furthermore, as used in this application, the term "circuit system" may refer to one or more or all of the following: (a) a hardware circuit implementation only (such as an implementation only in analog and / or digital circuit systems); (b) a combination of hardware circuits and software, such as (if applicable): (i) a combination of (multiple) analog and / or digital hardware circuits with software / firmware, and (ii) any portion of (multiple) hardware processors (including (multiple) digital signal processors), software, and (multiple) memories having software, which work together to cause a device such as a mobile phone or server to perform various functions; or (c) (multiple) hardware circuits and / or (multiple) processors, such as (multiple) microprocessors or portions thereof, which require software (e.g., firmware) to operate, but may be absent when the software is not required to operate.

[0030] The above definition of circuit system applies to all uses of the term in this application, including in any claim. As another example, as used in this application, the term circuit system also covers only the implementation of hardware circuitry or a processor (or processors) or a portion thereof and its accompanying software and / or firmware. For example, if applicable to a particular claim element, the term circuit system also covers baseband integrated circuits or processor integrated circuits for mobile devices, or similar integrated circuits in servers, cellular network devices, or other computing or network devices.

[0031] Figure 1 The illustration depicts a telecommunications system 100 implemented according to various examples of this disclosure. The telecommunications system generally includes one or more telecommunications networks. As shown, for example, the system includes one or more Public Land Mobile Networks (PLMNs) 102, which are coupled to one or more other external data networks 104, particularly including wide area networks (WANs), such as the Internet. Each PLMN includes a core network (CN) 106 backbone, such as an Evolved Packet Core (EPC) for LTE, a 5G core network (5GC), etc.; and each core network and the Internet are coupled to one or more Radio Access Networks (RANs) 108, air interfaces, etc., implementing one or more Radio Access Technologies (RATs). As used herein, "network device" refers to any suitable device on the network side of the telecommunications network. Examples of suitable network devices will be described in more detail below.

[0032] Furthermore, the system includes one or more radio units, which may be referred to differently as User Equipment (UE) 110, terminal equipment, terminal device, mobile station, etc. A UE is generally a device configured to communicate with another UE in a network or telecommunications network. A UE can be a portable computer (e.g., laptop, notebook, tablet), a mobile phone (e.g., mobile phone, smartphone), a wearable computer (e.g., smartwatch), etc. In other examples, a UE can be an Internet of Things (IoT) device, an Industrial IoT (IIoT) device, a vehicle equipped with Vehicle-to-Everything (V2X) communication technology, etc. In some examples, as referenced by 3GPP, a UE can be a Narrowband IoT (NB-IoT) device, an enhanced machine-type communication (eMTC) device, a low-energy (RedCap) device, an environmental IoT device, etc.

[0033] In operation, these UEs can be configured to connect to one or more RANs 108 based on their specific radio access technology, thereby accessing a specific CN 106 of PLMN 102, or accessing one or more external data networks in external data network 104 (e.g., the Internet). External data networks can be configured to provide Internet access, operator services, third-party services, etc. For example, the International Telecommunication Union (ITU) classifies 5G mobile network services into three categories: enhanced mobile broadband (eMBB), ultra-reliable low-latency communications (URLLC), and massive machine-type communications (mMTC) or massive Internet of Things (MIoT).

[0034] Examples of radio access technologies include 3GPP radio access technologies such as GSM, UMTS, LTE, Advanced LTE, 5G NR, Advanced 5G, and 6G. Other examples of radio access technologies include IEEE 802 technologies such as IEEE 802.11 (Wi-Fi), IEEE 802.15 (including 802.15.1 (WPAN / Bluetooth), 802.15.4 (Zigbee), and 802.15.6 (WBAN)), Bluetooth, Bluetooth Low Energy (BLE), Ultra Wideband (UWB), etc. In general, radio access technology can refer to any 2G, 3G, 4G, 5G, 6G, or higher generation mobile communication technology and its different versions, as well as any other radio access technology that can be configured to interoperate with such mobile communication technologies to provide access to a mobile network operator (MNO) under CN 106.

[0035] In various examples, RAN 108 can be configured as one or more macro cells, micro cells, pico cells, femtocells, etc. The RAN generally includes one or more radio access nodes configured to interact with UE 110. In various examples, radio access nodes can be referred to as base stations (BS), access points (APs), base transceiver stations (BTS), node Bs (NBs), evolved NBs (eNBs), macro BSs, NBs (MNBs) or eNBs (MeNBs), home BSs, NBs (HNBs) or eNBs (HeNBs), next-generation NBs (gNBs), enhanced gNBs (en-gNBs) or next-generation eNBs (ng-eNBs), etc. The RAN can include some type of network control / management entity responsible for controlling the radio access nodes. The network control / management entity and the radio access nodes can be separate or integrated into a single device. The network control / management entity can include a processing circuitry system configured to perform various management functions, etc. The processing circuitry system can be associated with a memory, computer-readable storage medium, or database used to maintain the information required for the management functions.

[0036] RAN 108 can be centralized or distributed. In various examples, RAN components can be interconnected via Ethernet, Gigabit Ethernet, Asynchronous Transfer Mode (ATM), fiber optic, dark fiber, passive wavelength division multiplexing (WDM), WDM passive optical network (WDM-PON), optical transport network (OTN), time-sensitive network (TSN), and / or any other data link layer network (potentially including radio links). RAN can be connected to CN106 via one or more gateways, network functions, etc.

[0037] As will be understood, PLMN 102 can be deployed in a variety of different ways. In a 4G LTE deployment, the EPC is CN106, and the evolved UMTS Terrestrial Radio Access Network (E-UTRAN) is RAN 108; and the E-UTRAN includes one or more eNBs (radio access nodes) configured to connect UE 110 to the E-UTRAN, thereby accessing the EPC. Figure 2 The diagram illustrates deployment 200, sometimes referred to as a 5G system (5GS). As shown, 5GC 202 is the CN, and Next Generation (NG) Radio Access Network (NG-RAN) 204 is the RAN; and the NG-RAN includes one or more gNBs 206 (radio access nodes) configured to connect UE 110 to the NG-RAN, thereby accessing the 5GC. The term 'gNB' in 5G can correspond to the eNB in ​​4G LTE.

[0038] Some 4G LTE and 5G deployments are considered standalone (SA) deployments. Other deployments combine 4G LTE and 5G technologies and are called non-standalone (NSA) deployments. In some deployments, the E-UTRAN includes one or more ng-eNBs configured to communicate with the 5GC and may also be configured to communicate with one or more gNBs. Similarly, in another deployment, the NG-RAN may include one or more en-gNBs configured to communicate with the EPC and may also be configured to communicate with one or more eNBs. In various cases, a single UE 110 (dual-mode UE or multi-mode UE) can support multiple (two or more) RANs, thus being configured to connect to multiple RANs, such as 4G LTE and 5G.

[0039] Figure 3 A more specific depiction of aspects of a 5GS deployment 200 of an MNO implemented according to some examples is provided. As shown, this deployment includes a 5GC 202 and an NG-RAN 204 with one or more gNBs 206 configured to connect UE 110 to the NG-RAN, thereby accessing the 5GC. The 5GC may include multiple network functions (NFs) partitioned between the control plane and the user plane. Specifically, the 5GC may include, for example, Access and Mobility Management Function (AMF) 302, Session Management Function (SMF) 304, User Plane Function (UPF) 306, Policy and Charging Function (PCF) 308, Network Open Function (NEF) 310, and / or Application Function (AF) 312. Other examples of suitable NFs include Network Repository Function (NRF), Network Slice Selection Function (NSSF), Unified Data Management (UDM), etc. A server hosting the applications, referred to as an Application Server (AS) 314, is also shown.

[0040] In the control plane, AMF 302 is configured to provide UE-based authentication, authorization, mobility management, etc. PCF 308 can be responsible for policy enforcement and management, controlling Quality of Service (QoS), charging, and subscriber resource allocation. PCF can promote efficient resource utilization, optimize network performance, and provide a high-quality user experience.

[0041] SMF 304 is configured to provide various functions, including Session Management (SM), UE Internet Protocol (IP) address allocation and management, selection and control of (multiple) UPF 306, policy application and QoS control, lawful interception, termination of the SM portion of NAS messages, Downlink Data Notification (DDN), roaming functionality, handling locally enforced QoS with application service level agreements (SLAs), charging data collection, and charging interfaces. If the UE 110 has multiple sessions, a different SMF can be assigned to each session to manage them individually, and different functions can be provided for each session.

[0042] UPF 306 supports various user plane operations and functions, such as packet routing and forwarding, traffic processing (e.g., applying QoS policies), anchoring for intra / inter-RAT mobility (if applicable), packet inspection and policy rule application, lawful interception (UP collection), traffic calculation and reporting, etc. UPF is the interconnection point between the 5GC and at least one external data network (DN) 316 (i.e., the ingress or egress point of the DN), and routes packets to and from the DN. The DN can be configured to provide Internet access, carrier services, third-party services, etc.

[0043] The AF 312 can interact with the 5GC 202 to enable the deployment of specific services and applications. The AF communicates with other NFs to request and manage network resources, ensuring the network adapts to the requirements of different applications and services. The NEF 310 allows authorized third-party applications and services to access specific NFs and services in a controlled manner. The NEF enables network capabilities to be exposed to external entities, facilitating the innovation and development of new services.

[0044] In some deployments (such as deployment 200), the operation of the gNB 206 or other radio access nodes may be performed at least partially in a central / centralized unit (CU) (such as a server, host, or node) that is operatively coupled to a distributed unit (DU), such as a radio headend / node. Node operations may also be distributed across multiple servers, hosts, or nodes. It should also be understood that the work allocation between 5GC 202 (or other CN) operations and gNB (or other radio access node) operations may vary depending on the implementation.

[0045] 5G network architecture can be based on so-called CU-DU splitting. A gNB-CU (central node) can control one or more gNB-DUs. A gNB-CU can control multiple spatially separated gNB-DUs that at least act as transmit / receive (Tx / Rx) nodes. However, in some example implementations, a gNB-DU (also called a DU) can include, for example, the Radio Link Control (RLC), Media Access Control (MAC) layer, and Physical (PHY) layer, while a gNB-CU (also called a CU) can include layers above the RLC layer, such as the Packet Data Convergence Protocol (PDCP) layer, Radio Resource Control (RRC) layer, and IP layer. Other functional splitting is also possible. Those skilled in the art are believed to be familiar with the Open Systems Interconnection (OSI) model and the functions within each layer.

[0046] In some example implementations, the server or CU can generate a virtual network through which the server can communicate with radio nodes. In general, a virtual network can involve the process of combining hardware and software network resources, as well as network functions, into a single software-based management entity (virtual network). Such a virtual network can provide flexible operational distribution between the server and the radio head end / node. In practice, any digital signal processing task can be performed in the CU or DU, and the boundary of responsibility transfer between the CU and DU can be chosen depending on the implementation.

[0047] Networks, including 5G, are now beginning to support interactive media services such as extended reality (XR), which includes virtual reality (VR), augmented reality (AR), and mixed reality (MR). These, and many other media services, involve graphics rendering, which can have computationally, rendering, and power-intensive requirements to deliver high-quality graphics for a reasonable quality of experience (QoE). Due to limitations in computing, rendering, and energy resources, resource-constrained devices such as smartphones and head-mounted displays (HMDs) may not be able to deliver a high-quality XR experience.

[0048] Split rendering (SR) could be a potential solution to this problem. In the split rendering paradigm, the rendering workload is divided among two or more rendering entities, such as a split rendering client (SRC) and a split rendering server (SRS). In a split rendering scenario, the SRC periodically sends metadata, including control information, to the SRS, which uses this metadata to render frames of the scene. The SRS then sends the rendered media to the SRC, which decodes and displays it. However, the example above is the simplest implementation of SR, as the client performs only minimal rendering.

[0049] In a rendering (SR) scenario, multiple rendering entities (such as SRS) may require information about the scene, including graphics assets, which can be used for execution and rendering at runtime. In typical rendering applications, such as video games and XR applications, this information can be provided to the rendering entities in the form of a scene graph or scene description.

[0050] Scene graphs and scene descriptions represent a virtual 3D scene as a layered graph including nodes. Nodes can represent virtual objects in the scene. Nodes can be associated with rendering data, which includes: geometric information such as meshes, material information such as reflectivity, transparency, color, color maps (UV maps), and rendering logic. Meshes can include so-called geometric primitives such as points, lines, triangles, and polygons, while color information can include textures, bitmaps, and UV maps, and rendering logic can include shaders. Rendering entities may need these components of objects to render them. This information can be provided to rendering entities as a scene graph or scene description, which is a resource that describes the scene as a layered graph in a format that the rendering entity can understand.

[0051] Once scene information is available for rendering entities, the scene can be rendered according to the rendering process. In one example, the rendering process for a given frame can conceptually be divided into three main phases: the application phase, the geometry phase, and the rasterization phase. In the application phase, the scene and its objects are transformed based on application logic. In the geometry phase, various geometric transformations are performed on the mesh data, including the objects in the scene, to obtain a subset of vertices falling within the view frustum of the rendering (virtual) camera. In the rasterization phase, the pixels in the display frame buffer are filled based on the vertices from the geometry phase and associated information such as color (e.g., shape texture, sprite graph), lighting, depth near the camera face, etc.

[0052] The above is an example rendering process, and it should be understood that various other rendering processes exist. Examples of other rendering processes include: ray tracing-based rendering, point cloud-based rendering, image-based rendering, etc.

[0053] Split rendering is expected to be available in 5G networks. One example of a proposed split rendering architecture is the SR Media Service Enabler (MSE) architecture, a general split rendering MSE based on 5G systems. Another example is the IP Multimedia Subsystem (IMS)-based Session Service (IBACS) split rendering architecture, specifically developed for IMS-based session AR services. Another example currently under discussion is general split rendering on IMS, which is not limited to session AR.

[0054] Figure 4The illustration shows an SR MSE architecture 400 implemented according to some examples. The SR MSE includes multiple 5G media functions, such as an SR client (SRC) 402, an SR server (SRS) 404, a real-time communication (RTC) AF 406 (e.g., AF 312), an application provider 408, an application 410, and a media session processor (MSH) 412. It is also shown that, in some examples, the SR MSE includes an XR runtime 414.

[0055] SRC 402 can reside in UE 110, and SRS 404 can reside in a 5G edge server, such as an RTC AS (e.g., AS 314) in DN316. SRC can be responsible for acquiring UE media capabilities and negotiating with SRS (e.g., RTC AS) to agree on the split rendering process at SRS. SRS can be responsible for negotiating SR sessions with SRC, monitoring resource usage of the 5G edge server (such as RTC AS) where SRS resides, and managing / running the split rendering process. RTC AF 406 can be responsible for provisioning, Quality of Service (QoS) allocation, and edge resource discovery. Application Provider 408 can provide split rendering media services for application 410 running on UE 110. Furthermore, MSH 412 can be responsible for control plane communication with RTC AF 406.

[0056] In the SR MSE architecture 400, application provider 408 can supply resources for split rendering via RTC-1. Application providers can be authorized to use resources and features provided by 5GS 200. In this regard, application providers can request 5GS to allocate appropriate resources and QoS profiles for split rendering sessions. For example, these resources may include components for performing computation and rendering operations for split rendering sessions.

[0057] Application provider 408 can deliver (e.g., provide) media to SRS 404 via RTC-2. Communication between RTC AF 406 and SRS can be performed via RTC-3, which may include the EDGE-3 interface. User plane signaling (e.g., WebRTC session setup or reconfiguration) and media delivery between SRC 404 and SRS can be performed via RTC-4. RTC AF 406 can provide split rendering information to MSH 412 as defined by RTC-5. SRC can discover application 408 via RTC-6. SRC can handle XR runtime 414, and SRC can discover client media capabilities from MSH via the RTC-7 interface. Applications and application providers can interact via RTC-8.

[0058] Figure 5The diagram illustrates an IBACS split-rendering architecture 500 implemented according to several examples. The IBACS split-rendering architecture is based on the IMS architecture and enhances support for data channel services such as AR services. This architecture includes SRC 502, SRS 504, and Data Channel (DC)-AS 506, which support the split-rendering process. DC-AS is an example of AS 314, and AR-AS is a specific DC-AS that supports AR services. The IBACS split-rendering architecture includes IMS 508, and IMS includes, for example, an Interrogation / Service / Proxy Call Session Control Function (I / S / P-CSCF) 510, IMS AS 512, DC Media Function (MF) 514 (or DC Media Resource Function (MRF)), and Data Channel Signaling Function (DCSF) 516.

[0059] SRC 502 can be provided by a DC multimedia telephony service residing in UE 110 for an IMS (DC-MTSI) client 518 (e.g., an AR-MTSI client), and SRS 504 can be provided by MF 514 (or MRF) or DC-AS 506 (e.g., AR-AS). MF is an NF that interacts with IMS AS 512 via the service-based interface DC2, and it provides media capabilities supporting IMS DC and AR. SRC can be responsible for acquiring UE media capabilities and interacting with SRS during split rendering. SRS can be responsible for interacting with SRC during split rendering, monitoring resource usage, and managing / running the split rendering process. DC-AS can be responsible for service control related to split rendering, including session media control and media capability negotiation with the UE via MF.

[0060] To support data channel services, DCSF 516 is a signaling control function that provides data channel control logic. DCSF can receive event reports from IMS AS 512 and determine whether AR communication services are permitted during an IMS session. DCSF can support HTTP web server functionality to download data channel applications (e.g., IMS-aware application 520) to UE 110 via MF 514 based on UE subscription. DCSF can download data channel applications from DC application repository 522. DCSF can interact with DC-AS 506 for DC resource control via DC4 / DC3 and traffic forwarding via MDC3 / MDC2.

[0061] Figure 6The illustration shows an SR architecture 600 implemented according to some examples, which can be implemented by an SR MSE architecture 400 or an IBACS split rendering architecture 500. As shown, the SR architecture includes an SRC 602 (residing in the UE 110) and an SRS 604 (residing in the 5G network). The SRC and SRS can communicate with each other via the 5G network 606 (5GC 202 and NG-RAN 204) and receive graphics assets and application logic from an application provider 608. For example, in the SR MSE architecture, the SRC 402 is defined as an entity within the UE that handles SR-related operations and communications and renders frames for display, while the SRS 404 is provided by an AS 314 capable of conducting SR sessions. The SRC and SRS are specified to communicate using a 5G RTC architecture. The IBACS split rendering architecture specifies communication over an IMS architecture; and therefore, the SRC 502 is provided by a DC-MTSI client 518, and the SRS is provided by an MF (or MRF) with SR functionality.

[0062] Within 3GPP, research was conducted on artificial intelligence (AI) and machine learning (ML) in 5G media services. Beyond media service architectures for AI / ML, media-based AI / ML use cases and scenarios were investigated, including object recognition in images and videos, video quality enhancement in streaming media, crowdsourced media capture, and natural language processing (NLP) in speech. However, AI / ML for enhancing video streams in split-rendering scenarios has not yet been considered.

[0063] Split rendering improves the efficiency of video games and XR applications on devices with limited rendering capabilities, but it also puts pressure on the network devices hosting SRS 604 and the network's computing infrastructure. Increased rendering quality generally necessitates increased computing and network resources to deliver higher rendering quality. In this respect, split rendering currently suffers from scaling issues in both split SRS and the network. Theoretically, SRS can benefit from horizontal scalability, where more compute nodes are continuously added to accommodate new users, but horizontal scalability is costly in practice and can lead to resource overprovisioning. Network capacity also has clear limitations and cannot be scaled, especially in the case of a PLMN using limited spectrum resources.

[0064] There is an industry trend to accelerate AI / ML applications using dedicated hardware accelerators, such as AI accelerators, deep learning processors, neural processing units (NPUs), and tensor processing units (TPUs). However, in the split-rendering scenarios developed by 3GPP, the resources on these devices are still not fully utilized.

[0065] In light of the foregoing, the example implementation of this disclosure provides a solution to the scalability problem of split rendering. The example implementation's solution generally reduces server computation and network capacity requirements by rendering at a lower quality at SRS 604 on a server or other network device and providing AI-based recovery of the lower-quality rendered media at SRC on the UE 110. The quality of the rendered media can be indicated by one or more of many different factors, such as resolution, bit depth, frame rate, etc. Some examples of suitable video quality metrics that can be used to evaluate the quality of the rendered media include Peak Signal-to-Noise Ratio (PSNR), Structural Similarity Index (SSIM), Video Multi-Method Evaluation Function (VMAF), etc. In some examples, AI-based recovery of the lower-quality rendered media can leverage AI accelerators or other dedicated AI acceleration chipsets on the UE.

[0066] The example implementation of this disclosure can be applied to XR applications that utilize split rendering, sending at least a portion of the rendering process to a remote server. This can have several benefits, such as reducing the power consumption of the UE 110 or reducing the form factor of the UE.

[0067] In a split-rendering scenario where the UE 110 will display the scene at least at the first quality level, the SRC 602 on the UE can send rendering and session-related metadata to the SRS 604 on the network device. The metadata may include control information, session and application state information, etc. Control information may include, for example (e.g., HMD) pose, pose prediction, user input, etc.

[0068] SRS 604 can render objects in a scene to generate rendering media based on control information, such as using graphics assets corresponding to the objects. In this respect, SRS can render objects in a scene at a lower second quality level to generate rendering media, which SRS can then send back to SRC 602 (e.g., after any appropriate encoding and encapsulation). SRC can then feed the rendering media into an ML model (referred to in some examples as an AI-based restoration model) to restore the rendering media to at least a first quality level, and then composite a view of the scene (including the rendering media at least at the first quality level) for display.

[0069] Beyond XR applications, the example implementation's solution can also be applied to video games, such as cloud-based games, where game logic and rendering are offloaded to cloud servers or other network devices that may include the SRS 604. In some of these use cases, the SRC 602 on the UE 110 may only send game-related control information to the SRS and receive rendered media (e.g., video frames) from the SRS to use an ML model to restore it to at least a first quality level, and then perform compositing and display. In video games, XR, and other suitable applications, the SRC uses the ML model to perform inference to restore the rendered media, and one or more computations of the inference may be performed by an AI accelerator on the UE.

[0070] The example implementations disclosed herein primarily involve inputting rendering media into an ML model on an SRC 602. However, it should be understood that the ML model can be deployed on an SRC or SRS 604. Alternatively, in some example implementations, the SRS functionality can be distributed to more than one network device. In these other example implementations, the ML model can be deployed on an SRC or one of the network devices that includes SRS functionality.

[0071] In some example implementations of this disclosure, the SRC 602 on the UE 110 and the SRS 604 on the network device (e.g., AS 314, RTCAS, MF / MRF, DC-AS, AR-AS) can establish a split rendering session for rendering a scene, in which objects will be displayed by the UE at least at a first quality level. When the split rendering session is established, the SRC and SRS can negotiate rendering output, metadata format, state and timing synchronization, rendering splits, available rendering splits, and initial graphics asset delivery to the SRC. The SRC and SRS can also negotiate limitations that the UE can process during the session, such as minimum and / or maximum rendering quality, the complexity of inference supported by the SRC, the frequency at which the ML model or format can be changed or updated, and the multiple input formats of the ML model. Additionally or alternatively, the SRC and SRS can negotiate decoding characteristics of the decoder circuitry on the UE and decoding information (e.g., motion vectors, partitions) exposed by the decoder circuitry.

[0072] The UE 110 can be pre-configured with an ML model, and the decoder circuitry can be instantiated and prepared. In some examples, the content-specific ML model can be downloaded from the network during or after session setup. The ML model can be trained based on the content of the objects to be rendered in segments and the expected display frame rate and resolution of the scene on the UE.

[0073] An SRS 604 or a network device that hosts an SRS may include a rendering engine, encoder, and / or wrapper, which can be initialized. The SRS or network device may also include one or more versions of an ML model for training and updating the ML model.

[0074] During the duration of a split rendering session, a rendering loop can be executed continuously for each frame (or a series of frames) of the scene. The SRC 602 can acquire control information (e.g., pose, pose prediction, user input) and send this control information to the SRS 604. In some examples, the control information can be predicted, such as pose prediction based on pose motion vectors. The SRC can also correlate predicted display times with pose predictions, events, etc.

[0075] The SRS 604 can receive control information and use it to render objects in the scene at a lower second quality level to produce rendering media (e.g., video frames). In some examples, the second quality level can be negotiated between the SRC 602 and the SRS during the split rendering session setup. The SRS can encode and encapsulate (e.g., compress) the rendering media to be sent to the SRC.

[0076] SRS 604 can also perform ML model updates based on the rendered media at a second quality level (before restoration). In some examples, the updates can also be based on the restored rendered media by the ML model (e.g., a version of the ML model at the SRS), depth information such as a depth map, and / or decoding information (e.g., motion vectors, partitions) accessed by the decoder circuitry of UE 110. For example, the SRS can train the ML model to improve the perceived quality of the rendered media from a second quality level to a first quality level. In some examples, the SRS can determine a second quality level such that the ML model can improve the quality level of the rendered media by at least the quality gap between the second and first quality levels.

[0077] In some examples, SRS 604 can access the highest quality version of the rendering media, and SRS can train an ML model to process the rendering media to get closer to the highest quality version. In some examples, this enhancement can be based on pixel-domain information (e.g., samples in an image). In other examples, enhancement can use the depth domain to identify which regions should be enhanced, or enhancement can be based on coded domain information to indicate the location of partitions and the location of possible block artifacts.

[0078] In some examples, continuous learning or retraining of the ML model version at SRS 604 can be used to update the ML model. The SRS can generate updates (e.g., patches) that will be applied to the ML model and send these updates to the SRC. In some examples, the SRS can send updates to the SRC when an update results in an increase in the quality level of the restored rendering media greater than a threshold. The SRS can encode and encapsulate updates to the ML model. In this regard, when the ML model is trained, the results can be the structure (e.g., the number and size of convolutional layers) and weight values. The trained ML model can be sent as is, or the ML model can be compressed to reduce the transmission cost. For transmission, the ML model can be sent in-band (e.g., in Dedicated Supplemental Enhancement Information (SEI)), out-of-band, at the file format level, or at the system level.

[0079] SRS 604 can send encoded and packaged rendering media to SRC 602, optionally updating the encoding and packaging of the ML model. In some examples where the rendering media includes a rendered view, SRS can encode the depth information of the rendered view, package the depth information, and send the packaged depth information to SRC along with the encoded and packaged rendering media.

[0080] SRC 602 may receive encoded and encapsulated rendering media from SRS 604, and optionally receive updates to the encoding and encapsulation of the ML model, and / or packaged depth information. The rendering media and updates can be unpacked and decoded. In some examples, the decoder circuitry on UE 110 may in particular decode the rendering media. The rendering media, along with optional depth channels (carrying depth information, such as depth maps) and other decoding information (e.g., motion vectors, partitions) opened by the decoder circuitry, can be input into the AI ​​recovery model to restore the rendering media from a second quality level to at least a first quality level. Before or after the rendering media is restored, updates to the ML model can be applied to the AI ​​recovery model to create an updated ML model. A view of the scene can be composited for display, including the rendering media restored to at least a first quality level by the (updated) ML model.

[0081] A rendering loop can repeat for the next frame of the scene. In some examples, the rendering loop for a frame and the rendering loop for the next frame can be implemented at least partially in parallel, where the rendering loop for a frame may not be completed before the rendering loop for the next frame begins.

[0082] To further illustrate some example implementations of this disclosure, Figure 7This is signaling diagram 700 for AI-assisted split rendering, implemented according to some examples of this disclosure. As shown in step 701, a split rendering session can be established between SRC 602 and SRS 604. This may include negotiating rendering output, metadata format, status and timing synchronization, rendering splits, available rendering splits, and initial delivery of graphics assets to the SRC.

[0083] During session establishment, SRC 602 can discover the AI ​​processing capabilities of UE 110, such as AI processing capabilities based on the AI ​​accelerator on the UE or other dedicated AI acceleration chipsets. It can negotiate with SRS 604 limitations that the UE can process during the session, such as minimum and / or maximum rendering quality, the complexity of inference supported by the SRC, the frequency at which the ML model or format can be changed or updated, and the input formats(s) of the ML model. SRC can also negotiate with SRS the decoding characteristics of the decoder circuitry on the UE, and the decoding information (e.g., motion vectors, partitions) exposed by the decoder circuitry. Although not shown separately, other entities, such as 5G NF and servers, and application provider servers, may be involved during session establishment. In the example shown, SRS can receive application logic and graphics assets during session establishment.

[0084] After the session is established, the main loop begins. The main loop can include the processes and operations required for the consistent operation of the split rendering session and the application running within it, and it can be executed once or multiple times per frame. Conceptually, the main loop can include multiple repetitive computational processes and operations, and can include one or more sub-loops, such as those for physics updates, lightmap updates, application logic execution, rendering, split adaptation, and proposed graphics asset management operations. These sub-loops (each of which can be referred to as a loop) can be executed sequentially or in parallel with each other, synchronously or asynchronously. Some of these sub-loops will be discussed below, albeit in a less detailed manner, to illustrate an example of AI-assisted split rendering.

[0085] Within the main loop, the rendering sub-loop may include performing graphics rendering operations (potentially at different quality levels) on a given frame to be displayed. Within the rendering sub-loop, SRC 602 may acquire control information (e.g., pose, pose prediction, user input) for rendering objects in the scene. SRC may send rendering and session-related metadata to SRS 604 at step 702a. Metadata may include control information, session and application state information, etc. SRS may render objects in the scene at step 702b, or perform rendering operations assigned to SRS based on a split of valid rendering operations. In various examples, objects in the scene rendered by SRS may be a portion of the scene, while other objects in other parts of the scene are rendered by SRC based on the split; or objects in the scene rendered by SRS may be used for the entire scene.

[0086] The scene may be intended to be displayed by UE 110 at least at a first quality level, and SRS 604 may render objects in the scene at a lower second quality level to generate rendering media. SRC 602 may then feed the rendering media into the ML model to restore the rendering media from the second quality level to at least the first quality level, or otherwise make it perceptibly equivalent to the first quality level. At step 702c, SRS 604 may generate updates to be applied to the ML model based on the rendering media. In some examples, the updates may include updates to the structure of the ML model. Additionally or alternatively, in some examples, the updates may include updates to the weights or other parameters of the ML model.

[0087] SRS 604 can encode and encapsulate the rendering media, and optionally, updates to the ML model. The SRS can then send the rendering media, along with, optionally, the updates to the ML model, to the SRC at step 702d. Updates can be delivered as a properly compressed complete model, or incrementally, such as weight updates or computation graph updates. In other examples, updates can be delivered as patches to the ML model. In some examples, the SRS can send the rendering media, along with metadata associated with the rendering media, to the SRC, as explained in more detail below.

[0088] When SRS 604 sends an update to the ML model, SRC 602 may apply the update to the ML model at step 702e (e.g., after any appropriate unpacking and decoding) to create an updated ML model, such as by updating the structure and / or weights or other parameters of the ML model. SRC may decode the rendering media at step 702f (e.g., after any appropriate unpacking) and input the (decoded) rendering media into the ML model (or into the updated ML model if updated) to restore the rendering media to at least a first quality level. In some examples where the rendering media and metadata associated with the rendering media are received from SRS, SRC may decode the rendering media and metadata (e.g., after any appropriate unpacking) and input the (decoded) rendering media and metadata into the ML model (or into the updated ML model if updated) to restore the rendering media to at least a first quality level, or otherwise perceptibly equivalent to a first quality level.

[0089] At step 702g, SRC 602 can composite a view of the scene for display, including rendering media of at least a first quality level. This step may include one or more processes, such as pose correction, reprojection, etc.

[0090] exist Figure 7 In the AI-assisted split rendering shown in signaling diagram 700, media rendering and ML model training / update operations can be performed jointly or synchronously. However, in other example implementations, these operations can be performed separately or asynchronously to account for the possibility that the ML model training / update operations may not be fast enough to maintain real-time operation.

[0091] Figure 8 Signaling diagram 800 for AI-assisted split rendering in a separate media rendering and ML model training / update scenario is implemented according to other examples of this disclosure. As shown in step 801, a split rendering session can be established between SRC 602 and SRS 604, such as in a manner similar to step 701. Also similar to... Figure 7 The description states that after the session is established, there is a main loop. For example, as shown in the figure, the main loop may include a rendering sub-loop and a model update sub-loop. The rendering sub-loop may include performing graphics rendering operations on a given frame that will be displayed.

[0092] In the rendering sub-loop, SRC 602 can acquire control information (e.g., pose, pose prediction, user input) for rendering objects in the scene. SRC can send rendering and session-related metadata to SRS 604 at step 802a. Metadata may include control information, session and application state information, etc. SRS can render objects in the scene at step 802b, or perform rendering operations assigned to SRS based on a split of valid rendering operations. In various examples, objects in the scene rendered by SRS may be a part of the scene, while other objects in other parts of the scene are rendered by SRC according to the split; or objects in the scene rendered by SRS may be used for the entire scene.

[0093] The scene may be intended to be displayed by the UE 110 at least at a first quality level, and the SRS 604 may render objects in the scene at a lower second quality level to generate rendering media. The SRS may encode and encapsulate the rendering media. The SRS may then send the rendering media to the SRC 602 at step 802c. In some examples, the SRS may send the rendering media, along with metadata associated with the rendering media, to the SRC.

[0094] At step 802d, SRC 602 may decode the rendering media (e.g., after any appropriate unpacking) and input the (decoded) rendering media into the ML model to restore the rendering media to at least a first quality level. In some examples where the rendering media and metadata associated with the rendering media are received from SRS, SRC may decode the rendering media and metadata (e.g., after any appropriate unpacking) and input the (decoded) rendering media and metadata into the ML model to restore the rendering media to at least a first quality level. Then, at step 802e, SRC may composite a view of the scene for display, including the rendering media at at least the first quality level. This step may include one or more processes, such as pose correction, reprojection, etc.

[0095] In the model update sub-loop, SRS 604 can generate, at step 803a, an update to be applied to the ML model based on the rendering media. In some examples, the update may include an update to the structure of the ML model. Alternatively, in some examples, the update may include an update to the weights or other parameters of the ML model.

[0096] SRS 604 can encode and encapsulate updates to the ML model. The SRS can then send the ML model updates to SRC 602 at step 803b. In some examples, the model update sub-loop can run in parallel with the rendering sub-loop to update the ML model. In some of these examples, the ML model can be trained or retrained during runtime, such as for every N new frames that have been rendered. SRC can apply the updates to the ML model at step 803c (e.g., after any appropriate unpacking and decoding) to create an updated ML model, such as by updating the structure and / or weights or other parameters of the ML model.

[0097] In both joint and separate media rendering and ML model training / update scenarios, in some examples, the negotiation during the establishment of the split rendering session at steps 701 and 801 may include negotiation regarding additional parameters that the decoder circuitry of UE 110 can provide to the ML model. These parameters may include decoded motion fields, prediction modes and partitioning information, loop filter parameters, etc. These parameters can be input into the ML model along with the rendered media to recover the rendered media, which may result in better recovery or enhancement of the rendered media by the ML model, but may also increase complexity.

[0098] When negotiating additional parameters, the procedures used for training and inference on SRC 602 and SRS 604 can be modified to accommodate them. On the SRC side, additional parameters can be received from the SRS, such as via metadata or supplemental enhancement information along with the (encoded) rendering media, and output by the decoder circuitry on the UE 110 hosting the SRC. On the SRS side, additional parameters can be output from the encoder to feed the training process along with the encoded rendering media.

[0099] To conserve inference or power resources at SRC 602, in some examples, the ML model can be used to reconstruct frames (rendering media) based on a spatial quality weight map, which allows for varying the reconstruction levels applied to different regions of the frame. For example, it might be desirable to enhance only the frame regions visible in the user's field of view (FoV), the areas the user's gaze is focused on, or visually prominent areas.

[0100] In some other examples, in addition to visual restoration of frames, ML models can also perform pose correction on frames. In some of these examples, the input to the ML model can include the rendering pose and current pose information or display pose information.

[0101] Furthermore, in both joint and separate media rendering and ML model training / update scenarios, the SRC 602 hosted on UE 110 can decide to modify the split rendering session during model updates, such as to achieve better QoE. This decision can be made within... Figure 7 The steps in step 702e of the joint media rendering and ML model training / update scenario shown are performed, or in Figure 8 The steps for separate media rendering and ML model training / updating the scene, as shown in step 803c, are performed.

[0102] During a model update, SRC 602 can determine that the updated ML model cannot be rendered in the near future, such as when the maximum rendering quality is exceeded. Therefore, SRC can decide to modify the current split rendering session. In some examples, SRC may seek a better server or SRS 604 to render objects in the scene with the desired rendering capabilities. In some of these examples, SRC may send a session modification request to SRS with updated negotiation information.

[0103] In some examples, the ML model can be trained offline and delivered to the SRC 602. Delivery of the ML model can be out-of-band, such as when a rendering application (e.g., a video game, XR application) is installed on the UE 110. In another example, an offline-trained ML model can be delivered in-band during the initialization of a split rendering session.

[0104] In some examples, the ML model can be trained offline using offline training data. For instance, this offline training data could include playback based on automatic or historical user trajectories from one or more appropriate rendering applications. In some examples, the ML model can be retrained based on training data from the target rendering applications, fine-tuning or retraining the ML model for each target rendering application. Fine-tuning or retraining can be actuated online during the split rendering session as described above, or offline before the split rendering session begins.

[0105] In some examples, the training of the ML model on the SRS side can take reports from the SRC 602 as input, such as QoE aspects, local viewing conditions (e.g., lighting, light sources), etc. In more advanced examples, the SRC can stream video captures of the scene's environment to the SRS, which can be particularly useful for XR applications.

[0106] In some examples, during a split rendering session, SRS 604 and SRC 602 can modify the recovery performed on the rendering media during the split rendering session, such as based on operating conditions at the SRS, SRC, or network. For example, the modification could include switching to a different ML model. Alternatively or additionally, for example, the modification could include changes to the input / output data of the ML model, such as changes to the training / inference data input size (frame size in pixels), visual quality or type (e.g., RGB / RGBD), output data size, type, or target visual quality, etc. Furthermore, alternatively or additionally, the modification could include updating the currently deployed ML model to accept or output the desired data type, size, and quality, such as by updating the input or output layers of the ML model.

[0107] In some examples, SRC 602 and SRS 604 can negotiate the available ML model and acceptable recovery at the start of a split rendering session and modify the recovery during the split rendering session by appropriately exchanging control messages. In some of these examples, control messages can be exchanged via WebRTC or IMS data channels. In some examples, control messages can conform to any of a variety of metadata message formats, and control messages can be exchanged via the "3GPP-SR" data channel sub-protocol.

[0108] In some examples, multiple ML models can be identified and preloaded onto SRC 602 during the split rendering session setup. In some of these examples, updates to one or more ML models (multiple models) may include model indexes that can be sent to and applied by the SRC. Furthermore, in some examples involving multiple ML models, the optimal ML model among the multiple ML models used to restore the rendering media can be determined in real-time or asynchronously. In some of these examples, the model index of the optimal ML model can be sent to and applied by the SRC, which avoids potentially large data update transfers.

[0109] Figures 9A-9C This is a flowchart illustrating various steps in method 900, implemented according to various examples and performed by a split rendering client on a user device. The method includes: establishing a split rendering session for rendering a scene, wherein objects in the scene will be rendered by a split rendering server on a network device. In some of these examples, the scene will be displayed by the user device at least at a first quality level, such as... Figure 9AAs shown in box 902. The method includes: receiving rendering media generated by a split rendering server from an object rendered at a second quality level. In some of these examples, the second quality level is lower than the first quality level, as shown in box 904. The method includes: feeding the rendering media into a machine learning (ML) model to restore the rendering media to at least the first quality level, as shown in box 906. Furthermore, the method includes: compositing a view of the scene for display. In some of these examples, the view of the scene includes: rendering media at least at the first quality level, as shown in box 908.

[0110] In some examples, method 900 also includes: obtaining control information for rendering the object, such as... Figure 9B As shown in box 910. In some of these examples, the method further includes sending control information to a split rendering server, as shown in box 912. Also in some of these examples, the rendering media is generated by the split rendering server from an object rendered at a second quality level based on the control information.

[0111] In some examples, inputting the rendering media into the ML model at box 906 includes performing inference using the ML model to restore the rendering media to at least a first quality level. In some of these examples, the inference includes one or more computations performed by an AI accelerator on the user's device.

[0112] In some examples, establishing a split rendering session involves negotiating the complexity of inference supported by the split rendering client with the split rendering server at box 902. In some of these examples, the object is rendered by the split rendering server based on the complexity of inference supported by the split rendering client.

[0113] In some examples, the rendering media received from the split rendering server at box 904 is decoded by the decoder circuitry. In some of these examples, establishing a split rendering session at box 902 involves negotiating decoding information disclosed by the decoder circuitry with the split rendering server. Similarly, in some of these examples, the version of the ML model at the split rendering server is trained based on the decoding information disclosed by the decoder circuitry.

[0114] In some examples, the rendering media received from the split rendering server at box 904 is encoded rendering media. In some of these examples, method 900 also includes decoding the encoded rendering media into decoded rendering media. Similarly, in some of these examples, inputting the rendering media into the ML model at box 906 includes inputting the decoded rendering media into the ML model.

[0115] In some examples, receiving the rendering media at box 904 includes receiving the rendering media and metadata associated with the rendering media. In some of these examples, feeding the rendering media into the ML model at box 906 includes feeding the rendering media and metadata into the ML model to restore the rendering media to at least a first quality level.

[0116] In some examples, method 900 also includes: receiving updates from the split rendering server that will be applied to the ML model, such as Figure 9C As shown in box 914. In some of these examples, the method also includes applying updates to the ML model to create an updated ML model, as shown in box 916.

[0117] In some examples, the updates to be applied to the ML model are received along with the rendering media. In some of these examples, inputting the rendering media into the ML model at box 906 includes inputting the rendering media into the updated ML model.

[0118] In some examples, after the rendering media is fed into the ML model to restore the rendering media, an update is applied to the ML model at box 916 to create an updated ML model.

[0119] Figure 10A and Figure 10B This is a flowchart illustrating various steps in a method 1000 performed by a split rendering server on a network device, implemented according to various examples. The method includes: establishing a split rendering session for rendering a scene, wherein objects in the scene will be rendered by the split rendering server. In some of these examples, the scene will be displayed by a user device at least at a first quality level, such as... Figure 10A As shown in box 1002. The method includes: rendering an object at a second quality level to generate rendering media. In some of these examples, the second quality level is lower than the first quality level, as shown in box 1004. The method includes: sending the rendering media to a split rendering client on a user device, where the rendering media will be fed into a machine learning (ML) model to restore the rendering media to at least the first quality level, as shown in box 1006. The method includes: performing an update of the ML model based on the rendering media, as shown in box 1008. The method includes: generating an update to be applied to the ML model based on the rendering media, as shown in box 1010. And the method includes: sending the update to a split rendering client, where the update will be applied to the ML model to create an updated ML model, as shown in box 1012.

[0120] In some examples, method 1000 also includes receiving control information for rendering objects from a split rendering client. In some of these examples, the object is rendered at a second quality level at box 1004 based on the control information.

[0121] In some examples, the split rendering client will use an ML model to perform inference to restore the rendering media to at least a first quality level. In some of these examples, establishing a split rendering session at box 1002 includes negotiating the complexity of the inference supported by the split rendering client.

[0122] In some examples, objects are rendered based on the complexity of inference supported by the split rendering client.

[0123] In some examples, method 1000 further includes encoding the rendering media into encoded rendering media. In some of these examples, sending the rendering media at box 1012 includes sending the encoded rendering media to a split rendering client, where the encoded rendering media is decoded into decoded rendering media, and the decoded rendering media is input into the ML model.

[0124] In some examples, metadata associated with the rendering media is sent along with the rendering media to a split rendering client on the user's device. At the split rendering client, the rendering media and metadata are applied to the ML model to restore the rendering media. In some of these examples, updates to the ML model are performed at box 1008 based on the rendering media and the metadata associated with it.

[0125] In some examples, such as Figure 10B As shown in box 1014, performing an update of the ML model at box 1008 includes generating an update based on the rendering media to be applied to the ML model. In some of these examples, the method includes sending an update to a split rendering client, where the update is applied to the ML model to create an updated ML model, as shown in box 1016.

[0126] In some examples, the updates to be applied to the ML model are sent to the split rendering client along with the rendering media, and the rendering media is fed into the updated ML model.

[0127] In some examples, after the rendering media is fed into the ML model to restore the rendering media, an update is sent to the split rendering client at box 1016, where the update is applied to the ML model to create an updated ML model.

[0128] In some examples, method 1000 further includes: determining a second quality level such that the ML model can improve the quality level of the rendered media by at least the quality gap between the second quality level and the first quality level.

[0129] According to the example implementations of this disclosure, the telecommunications system 100 or PLMN 102 and its components (such as UE 110, CN 106, RAN 108, 5GC 202, NG-RAN 204, gNB 206, UE 110, AF 312, AS 314, SRC 402, SRS 404, SRC502, SRS 504, MF 514, DC-MTSI client 518, SRC 602, SRS 604 and / or application provider 608) can be implemented in various ways. The means of implementing the system and its components can include hardware, firmware, software, or a combination thereof. In some examples, one or more devices can be configured to serve as or otherwise implement the system and its components shown and described herein. In examples involving more than one device, the respective devices can be connected to or otherwise communicate with each other in a variety of different ways, such as directly or indirectly via a wired or wireless network.

[0130] Based on some example implementations, regarding Figures 9A-9C At least some of the described methods 900 can be performed by means including components that function to perform corresponding steps of the method. Similarly, regarding Figure 10A and Figure 10B At least some of the described methods 1000 can be performed by means of components including functions for performing the corresponding steps of the method. Examples of suitable means may include network functions, application servers, media functions, media resource functions, or any suitable means such as servers, hosts, or nodes. Other examples of suitable means may include user equipment, user devices, user terminals, etc.

[0131] Figure 11 The illustration shows an apparatus 1100 implemented according to some examples of the present disclosure, wherein components for performing various functions include hardware, either alone or under the guidance of one or more computer programs from a computer-readable storage medium or other memory, such as computer memory. Generally, the apparatus of the example implementations of the present disclosure may include, be incorporated into, or be embodied in one or more fixed or portable electronic devices. Examples of suitable electronic devices include wearable computers, mobile phones, portable computers, desktop computers, workstation computers, servers (server computers), etc. The apparatus may include one or more of each of a plurality of components, such as a processing circuitry system 1102 connected to a computer-readable storage medium or other memory 1104.

[0132] The processing circuit system 1102 may be composed of one or more processors, either alone or in combination with one or more computer-readable storage media. The processing circuit system is generally any computer hardware capable of processing information (e.g., data, computer programs, and / or other suitable electronic information). The processing circuit system consists of a set of electronic circuits, some of which may be packaged as integrated circuits or multiple interconnected integrated circuits (integrated circuits are sometimes more commonly referred to as "chips"). The processing circuit system may be configured to execute computer programs, which may be stored on the processing circuit system or otherwise stored in memory 1104 (of the same or another device).

[0133] Depending on the specific implementation, the processing circuitry system 1102 may be multiple processors, a multi-core processor, or some other type of processor. Furthermore, the processing circuitry system may be implemented using multiple heterogeneous processor systems, where a main processor and one or more auxiliary processors reside on a single chip. As another illustrative example, the processing circuitry system may be a symmetric multiprocessor system comprising multiple processors of the same type. In another example, the processing circuitry system may be embodied as one or more ASICs, FPGAs, etc., or otherwise include one or more ASICs, FPGAs, etc. Therefore, while the processing circuitry system may be able to execute a computer program to perform one or more functions, the various examples of processing circuitry systems may be able to perform one or more functions without the assistance of a computer program. In either case, the processing circuitry system may be appropriately programmed to perform functions or operations implemented according to the examples of this disclosure.

[0134] Memory 1104 is generally any computer hardware capable of temporarily and / or permanently storing information (e.g., data, computer programs, instructions 1106 (e.g., computer-readable program code), and / or other suitable information). Memory may include volatile and / or non-volatile memory and may be fixed or removable. Examples of suitable memory include recording media, random access memory (RAM), read-only memory (ROM), hard disk drives, flash memory, thumb drives, removable computer disks, optical disks, or some combination thereof.

[0135] Memory 1104 is a non-transitory device capable of storing information. An example of a suitable memory is a computer-readable storage medium, which differs from a computer-readable transport medium capable of carrying information from one location to another. Examples of suitable computer-readable transport media include electronic carrier signals, telecommunication signals, software distribution packages, or some combination thereof. As used herein, the term "non-transitory" is a limitation on the medium itself (i.e., tangible, not signaling), not a limitation on the persistence of data storage (e.g., RAM and ROM). Computer-readable media as described herein generally refers to either computer-readable storage media or computer-readable transport media. A computer-readable medium is any entity or device capable of storing and carrying information, such as one or more computer programs or portions thereof.

[0136] In addition to memory 1104 (e.g., a computer-readable storage medium), processing circuitry 1102 may also be connected to one or more interfaces for displaying, sending, and / or receiving information. These interfaces may include communication interface 1108 and / or one or more user interfaces. The communication interface may be configured to send and / or receive information to and from other devices, networks, etc. The communication interface may be configured to send and / or receive information via physical (wired) and / or wireless communication links. Examples of suitable communication interfaces include network interface controllers (NICs), wireless NICs (WNICs), etc.

[0137] The user interface may include a display 1110 and / or one or more user input interfaces 1112. The display may be configured to present or otherwise display information to a user; suitable examples include liquid crystal displays (LCDs), light-emitting diode (LED) displays, organic LED (OLED) displays, active-matrix OLEDs (AMOLEDs), etc. The user input interface may be wired or wireless and may be configured to receive information from the user into the device, such as for processing, storage, and / or display. Suitable examples of the user input interface include a microphone, image or video capture device, keyboard or keypad, joystick, touch-sensitive surface (separate from or integrated into the touchscreen), biometric sensors, etc. The user interface may also include one or more interfaces for communicating with peripheral devices such as printers, scanners, etc.

[0138] The execution of instruction 1106 by processing circuitry 1102, or the storage of instructions in memory 1104, can support combinations of operations for implementing exemplary implementations of this disclosure. In this way, apparatus 1100 may include at least one processing circuitry and at least one memory coupled to the at least one processing circuitry, wherein the at least one processing circuitry is configured to execute instructions stored in the at least one memory. It will also be understood that one or more functions and combinations of functions can be implemented by a dedicated hardware-based computer system and / or processing circuitry that performs the specified function, or by a combination of dedicated hardware and program code instructions.

[0139] Some example implementations of this disclosure can also be executed as a computer process defined by one or more computer programs or portions thereof. Example implementations of this disclosure can be executed by executing at least a portion of a computer program including instructions. The computer program can be in source code form, object code form, or some intermediate form. The computer program can be stored on a computer-readable medium that can be read by a computer, processing circuitry system, or other suitable means. As indicated above, for example, the computer program can be stored in memory, such as a computer-readable storage medium. Alternatively or alternatively, for example, the computer program can be stored on a computer-readable transmission medium. The coding of software used to perform example implementations of this disclosure is entirely within the scope of those skilled in the art.

[0140] It should be understood that any suitable instructions can be loaded from memory or a computer-readable medium (e.g., a computer-readable storage medium, a computer-readable transmission medium) onto a computer, processing circuitry, or other programmable means to produce a particular machine, such that the particular machine becomes a component for implementing the functions specified herein. The instructions can also be stored in a computer-readable medium that can instruct a computer, processing circuitry, or other programmable means to operate in a particular manner to produce a particular machine or article of manufacture. In some examples, instructions stored in a computer-readable medium can produce an article of manufacture, wherein the article of manufacture becomes a component for implementing the functions described herein. The instructions can be retrieved from a computer-readable medium and loaded onto a computer, processing circuitry, or other programmable means to configure the computer, processing circuitry, or other programmable means to perform operations that will be performed on or by the computer, processing circuitry, or other programmable means.

[0141] The retrieval, loading, and execution of instructions, including program code instructions, can be performed sequentially, such that one instruction is retrieved, loaded, and executed at a time. In some example implementations, retrieval, loading, and / or execution can be performed in parallel, such that multiple instructions can be retrieved, loaded, and / or executed together. The execution of program code instructions can produce computer-implemented processes, such that the instructions, executed by a computer, processing circuitry system, or other programmable device, provide operations for implementing the functions described herein.

[0142] As stated above and reiterated below, this disclosure includes, but is not limited to, the following example implementations.

[0143] Clause 1. A method executed by a split rendering client on a user device, the method comprising: Establish a split rendering session for rendering the scene, where objects in the scene will be rendered by a split rendering server on a network device, and the scene will be displayed by the user device at least at a first quality level; Receive rendering media generated by the split rendering server from an object rendered at a second quality level, wherein the second quality level is lower than the first quality level; The rendering media is fed into a machine learning (ML) model to restore it to at least a first-level quality; and A view of the composite scene is provided for display, wherein the view of the scene includes: rendering media of at least a first quality level.

[0144] Clause 2. The method pursuant to Clause 1, wherein the method further includes: Obtain control information used for rendering objects; and Send control information to the split rendering server, and The rendering media is generated by the split rendering server from objects rendered at a second quality level based on control information.

[0145] Clause 3. The method according to Clause 1 or Clause 2, wherein inputting rendering media into an ML model comprises: performing inference using the ML model to restore the rendering media to at least a first quality level, and wherein the inference comprises: one or more computations performed by an AI accelerator on the user device.

[0146] Clause 4. According to the method of Clause 3, establishing a split rendering session includes negotiating with the split rendering server the complexity of inference supported by the split rendering client, and The objects are rendered by the split rendering server based on the complexity of inference supported by the split rendering client.

[0147] Clause 5. The method according to Clause 4, wherein the rendering media received from the split rendering server is decoded by the decoder circuitry system, wherein establishing a split rendering session includes negotiating with the split rendering server the decoding information opened by the decoder circuitry system, and The version of the ML model at the split rendering server is trained based on the decoding information exposed by the decoder circuitry.

[0148] Clause 6. The method according to any one of Clauses 1 to 5, wherein the rendering media received from the split rendering server is encoded rendering media, and the method further includes: decoding the encoded rendering media into decoded rendering media, and Inputting rendering media into the ML model includes inputting the decoded rendering media into the ML model.

[0149] Clause 7. The method according to any one of Clauses 1 to 6, wherein receiving the rendering media includes: receiving the rendering media and metadata associated with the rendering media, and Inputting rendering media into the ML model includes inputting rendering media and metadata into the ML model to restore the rendering media to at least a first quality level.

[0150] Clause 8. The method pursuant to any one of Clauses 1 to 7 also includes: Receive updates from the split rendering server that will be applied to the ML model; and Apply the update to the ML model to create an updated ML model.

[0151] Clause 9. According to the method of Clause 8, the updates to be applied to the ML model are received along with the rendering media, and Inputting rendering media into the ML model includes inputting rendering media into the updated ML model.

[0152] Clause 10. The method of Clause 8 or Clause 9, wherein after the rendering media is input into the ML model to restore the rendering media, an update is applied to the ML model to create an updated ML model.

[0153] Clause 11. An apparatus comprising: at least one memory configured to store instructions; and at least one processing circuitry configured to access the at least one memory and execute the instructions to cause the apparatus to perform a method according to any one of Clauses 1 to 10.

[0154] Clause 12. An apparatus comprising components for performing the method according to any one of Clauses 1 to 10.

[0155] Clause 13. A computer-readable medium comprising instructions that, in response to execution by at least one processing circuitry system, cause a device to perform a method according to any one of Clauses 1 to 10.

[0156] Clause 14. A computer-readable storage medium comprising instructions that, in response to execution by at least one processing circuitry system, cause a device to perform a method according to any one of Clauses 1 to 10.

[0157] Clause 15. A computer program comprising instructions that, in response to execution by at least one processing circuitry system, cause a device to perform a method according to any one of Clauses 1 to 10.

[0158] Clause 16. A method performed by a split rendering server on a network device, the method comprising: Establish a split rendering session for rendering the scene, where objects in the scene will be rendered by a split rendering server, and the scene will be displayed by the user device at least at a first quality level; Render the object at a second quality level to generate rendering media, wherein the second quality level is lower than the first quality level; The rendering media is sent to the split rendering client on the user's device, where it is fed into a machine learning (ML) model to restore the rendering media to at least a first quality level. Based on the rendering media, perform updates to the ML model; Based on the rendering media, generate updates that will be applied to the ML model; and Send the update to the split rendering client, where the update will be applied to the ML model to create the updated ML model.

[0159] Clause 17. The method according to Clause 16, wherein the method further includes: receiving control information for rendering objects from a split rendering client, and The objects are rendered at a second quality level based on control information.

[0160] Clause 18. According to the method of Clause 16 or Clause 17, wherein the split rendering client will use an ML model to perform inference to restore the rendering media to at least a first quality level, and Establishing a split rendering session includes negotiating with the split rendering client the complexity of inference supported by the split rendering client.

[0161] Clause 19. The method of Clause 18, wherein objects are rendered based on the complexity of inference supported by a split rendering client.

[0162] Clause 20. The method according to any one of Clauses 16 to 19, wherein the method further comprises: encoding the rendering media into an encoded rendering media, and Sending rendering media includes sending encoded rendering media to the split rendering client, where the encoded rendering media is decoded into decoded rendering media, and the decoded rendering media is input into the ML model.

[0163] Clause 21. According to any one of Clauses 16 to 20, wherein metadata related to the rendering media is sent along with the rendering media to a split rendering client on the user's device, at the split rendering client, the rendering media and metadata are applied to the ML model to recover the rendering media, and The updates to the ML model are performed based on the rendering media and the metadata associated with the rendering media.

[0164] Clause 22. The method according to any one of Clauses 16 to 21, wherein performing an update of the ML model includes: Based on the rendering media, generate updates that will be applied to the ML model; and Send the update to the split rendering client, where the update will be applied to the ML model to create the updated ML model.

[0165] Clause 23. The method of Clause 22, wherein the update to be applied to the ML model is sent to the split rendering client along with the rendering media, and the rendering media is input into the updated ML model.

[0166] Clause 24. The method according to Clause 22 or Clause 23, wherein after the rendering media is input into the ML model to restore the rendering media, an update is sent to the split rendering client, where the update is applied to the ML model to create an updated ML model.

[0167] Clause 25. The method according to any one of Clauses 16 to 24 further includes determining a second quality level such that the ML model is able to improve the quality level of the rendered media by at least the quality gap between the second quality level and the first quality level.

[0168] Clause 26. An apparatus comprising: at least one memory configured to store instructions; and at least one processing circuitry configured to access the at least one memory and execute the instructions to cause the apparatus to perform a method according to any one of Clauses 16 to 25.

[0169] Clause 27. An apparatus comprising components for performing the method pursuant to any one of Clauses 16 to 25.

[0170] Clause 28. A computer-readable medium comprising instructions that, in response to execution by at least one processing circuitry system, cause a device to perform a method according to any one of Clauses 16 to 25.

[0171] Clause 29. A computer-readable storage medium comprising instructions that, in response to execution by at least one processing circuitry system, cause a device to perform a method according to any one of Clauses 16 to 25.

[0172] Clause 30. A computer program comprising instructions that, in response to execution by at least one processing circuitry system, cause a device to perform a method according to any one of Clauses 16 to 25.

[0173] Many modifications and other implementations of the disclosure will occur to those skilled in the art upon which this disclosure pertains, taking advantage of the teachings presented in the foregoing description and associated drawings. Therefore, it should be understood that this disclosure is not limited to the specific implementations disclosed, and that modifications and other implementations are intended to be included within the scope of the appended claims. Furthermore, although the foregoing description and associated drawings describe exemplary implementations in the context of certain exemplary combinations of elements and / or functions, it should be understood that different combinations of elements and / or functions can be provided by alternative implementations without departing from the scope of the appended claims. In this regard, for example, combinations of elements and / or functions different from those explicitly described above may also be considered, as may be set forth in some of the appended claims. Although specific terms are used herein, they are used only in a general and descriptive sense and not for limiting purposes.

Claims

1. An apparatus for implementing a split-rendering client on a user device, the apparatus comprising: At least one memory is configured to store instructions; as well as At least one processing circuitry system is configured to access the at least one memory and execute the instructions such that the means at least: Establish a split rendering session for rendering a scene, wherein objects in the scene will be rendered by a split rendering server on a network device, and wherein the scene will be displayed by the user device at least at a first quality level; Receive rendering media generated by the split rendering server from the object rendered at a second quality level, wherein the second quality level is lower than the first quality level; The rendering media is fed into a machine learning (ML) model to restore the rendering media to at least the first quality level; as well as A view of the scene is synthesized for display, wherein the view of the scene includes: at least the rendering media of the first quality level.

2. The apparatus of claim 1, wherein the apparatus is further configured to at least: Obtain control information for rendering the object; and Send the control information to the split rendering server, and The rendering media is generated by the split rendering server from the object rendered at the second quality level based on the control information; or The rendering media received from the split rendering server is encoded rendering media, and the at least one processing circuitry is configured to execute the instructions to cause the apparatus to further decode the encoded rendering media into decoded rendering media. The apparatus is configured to input the rendering media into the ML model including: The apparatus is configured to input the decoded rendering media into the ML model; or The means of receiving the rendering media includes: the means of receiving the rendering media and metadata associated with the rendering media, and The means of wherein inputting the rendering media into the ML model comprises: the means of inputting the rendering media and the metadata into the ML model to restore the rendering media to at least the first quality level.

3. The apparatus of claim 1, wherein inputting the rendering media into the ML model comprises: The device is configured to perform inference using the ML model to restore the rendered media to at least the first quality level, and wherein the inference includes one or more computations performed by an AI accelerator on the user device.

4. The apparatus of claim 3, wherein establishing the split rendering session comprises: The device is configured to negotiate with the split rendering server the complexity of the inference supported by the split rendering client, and The object is rendered by the split rendering server based on the complexity of the inference supported by the split rendering client; and The rendering media received from the split rendering server is decoded by a decoder circuitry system, wherein establishing the split rendering session by means of the means includes: negotiating with the split rendering server the decoding information opened by the decoder circuitry system, and The version of the ML model at the split rendering server is trained based on the decoding information provided by the decoder circuitry.

5. The apparatus of claim 1, wherein the at least one processing circuit system is configured to execute the instructions such that the apparatus further at least: Receive updates from the split rendering server that will be applied to the ML model; and The update is applied to the ML model to create an updated ML model.

6. The apparatus of claim 5, wherein the update to be applied to the ML model is received together with the rendering medium, and The apparatus is configured to input the rendering media into the ML model including: The device is configured to input the rendering media into the updated ML model; or After the rendering media is input into the ML model to restore the rendering media, the update is applied to the ML model to create the updated ML model.

7. An apparatus for implementing a split rendering server on a network device, the apparatus comprising: At least one memory is configured to store instructions; as well as At least one processing circuitry system is configured to access the at least one memory and execute the instructions such that the means at least: Establish a split rendering session for rendering a scene, wherein objects in the scene will be rendered by the split rendering server, and wherein the scene will be displayed by a user device at least at a first quality level; The object is rendered at a second quality level to generate rendering media, wherein the second quality level is lower than the first quality level; The rendering media is sent to a split rendering client on the user device, whereby the rendering media is fed into a machine learning (ML) model to restore the rendering media to at least the first quality level; Based on the rendering media, the ML model is updated; Based on the rendering media, an update will be generated that will be applied to the ML model; as well as The update is sent to the split rendering client, where it is applied to the ML model to create an updated ML model.

8. The apparatus of claim 7, wherein the at least one processing circuitry is configured to execute the instructions such that the apparatus further receives control information from the split rendering client for rendering the object, and The object is rendered at the second quality level based on the control information; or The at least one processing circuitry is configured to execute the instructions to cause the apparatus to further encode the rendering media into an encoded rendering media, and The device is configured to transmit the rendering media including: The apparatus is configured to send the encoded rendering media to the split rendering client, whereby the encoded rendering media is decoded into decoded rendering media, and the decoded rendering media is input into the ML model. or The metadata associated with the rendering media is sent along with the rendering media to the split rendering client on the user device. At the split rendering client, the rendering media and the metadata are applied to the ML model to reconstruct the rendering media. The update of the ML model is performed based on the rendering media and metadata associated with the rendering media; or The at least one processing circuit system is configured to execute the instructions such that the apparatus further determines the second quality level, enabling the ML model to improve the quality level of the rendering media by at least the quality gap between the second quality level and the first quality level.

9. The apparatus of claim 7, wherein the split rendering client performs inference using the ML model to restore the rendering media to at least the first quality level, and The device is configured to establish the split rendering session by: The device is configured to negotiate with the split rendering client the complexity of the inference supported by the split rendering client, and The objects are rendered based on the complexity of the inference supported by the split rendering client.

10. The apparatus of claim 7, wherein the apparatus is configured to perform the update of the ML model comprising: The device is made to perform the following: Based on the rendering media, an update will be generated that will be applied to the ML model; as well as The update is sent to the split rendering client, where it is applied to the ML model to create an updated ML model; The update to be applied to the ML model is sent to the split rendering client along with the rendering media, and the rendering media is input into the updated ML model; as well as After the rendering media is input into the ML model to restore the rendering media, the update is sent to the split rendering client, where the update is applied to the ML model to create the updated ML model.

Images