Data rendering method, media processing network element, terminal equipment and storage medium

Abstract

The invention discloses a data rendering method, a media processing network element, terminal equipment and a storage medium. The data rendering method is applied to a media processing network element, and the method comprises: establishing a transmission channel between the media processing network element and a terminal device, and allocating a corresponding identifier to the transmission channel, the transmission channel at least comprising a first real-time transport protocol (RTP) channel, and the identifier corresponding to the first RTP channel being used for indicating the first RTP channel to transmit data to be rendered; obtaining to-be-rendered data sent by the terminal device through the first RTP channel; and rendering the to-be-rendered data to obtain rendered data.

Description

Technical Field

[0001] This application relates to the field of communication technology, such as a data rendering method, media processing network element, terminal device, and storage medium. Background Technology

[0002] With the deployment of 5G networks and immersive services based on 5G, emerging media services based on the Internet Protocol Multimedia Subsystem (IMS) have developed rapidly. Due to the characteristics of IMS-based media services, such as large data volume, low latency, complex processing, and interactivity, operators prefer that the network handle data processing to reduce the demands on terminal devices and increase user stickiness. However, the 3rd Generation Partnership Project (3GPP) has not yet proposed specific solutions for different media data types, service types, and the data processing capabilities of different terminal devices and networks, thus affecting the media service experience. Summary of the Invention

[0003] This application provides a data rendering method applied to a media processing network element, the method comprising:

[0004] A transmission channel is established between the media processing network element and the terminal device, and a corresponding identifier is assigned to the transmission channel. The transmission channel includes at least a first real-time transport protocol (RTP) channel. The identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits data to be rendered.

[0005] Obtain the data to be rendered sent by the terminal device through the first RTP channel;

[0006] Render the data to be rendered to obtain the rendered data.

[0007] This application provides a data rendering method applied to a terminal device, the method including:

[0008] A transmission channel is established between the terminal device and the media processing network element, and the identifier corresponding to the transmission channel is determined. The transmission channel includes at least a first real-time transport protocol (RTP) channel. The identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits the data to be rendered.

[0009] The data to be rendered is sent to the media processing network element through the first RTP channel.

[0010] This application provides a media processing network element, including: a processor; the processor is used to implement the data rendering method of any of the above embodiments when executing a computer program.

[0011] This application provides a terminal device, including: a processor; the processor is used to implement the data rendering method of any of the above embodiments when executing a computer program.

[0012] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method of any of the above embodiments.

[0013] Further details regarding the above embodiments and other aspects of this application, as well as their implementations, are provided in the accompanying drawings, detailed description, and claims. Attached Figure Description

[0014] Figure 1 This is a network diagram of a wireless communication system provided in one embodiment;

[0015] Figure 2 This is a flowchart illustrating a data rendering method provided in one embodiment;

[0016] Figure 3 This is a flowchart illustrating another data rendering method provided in one embodiment;

[0017] Figure 4 Example 1 provides an interactive diagram of data rendering;

[0018] Figure 5 Example 2 provides an interactive diagram of data rendering;

[0019] Figure 6 This is an interactive schematic diagram of data rendering based on an IMS network provided in one embodiment;

[0020] Figure 7 This is a schematic diagram of the structure of a data rendering device provided in one embodiment;

[0021] Figure 8 This is a schematic diagram of the structure of another data rendering device provided in one embodiment;

[0022] Figure 9 This is a schematic diagram of the structure of a media processing network element provided in one embodiment;

[0023] Figure 10 This is a schematic diagram of the structure of a UE provided in one embodiment. Detailed Implementation

[0024] It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0025] Media services based on IMS are characterized by large data volume, low latency, complex processing, and interactivity. For example, call services are no longer limited to traditional audio and video calls; they can be overlaid with augmented reality (AR) features and operations on virtual objects to enrich call services and improve the call experience.

[0026] Due to the aforementioned characteristics of IMS-based media services, the transmission, processing, and presentation of media data all require enhancement. From the operator's perspective, it is preferable for the network to handle media data processing, thereby reducing the demands on terminal devices and increasing user stickiness to the network. IMS-based media services place high demands on operations such as graphics and image compositing and rendering, which in turn places higher demands on the capabilities of terminal devices (such as Central Processing Units (CPUs) and Graphics Processing Units (GPUs)) and the network.

[0027] Currently, 3GPP has proposed a rendering method based on IMS user entities and network assistance. However, 3GPP has not proposed specific solutions for different media data types, different service types, and different terminal device and network capabilities, resulting in the inability to meet the needs of IMS-based media services and affecting the media service experience.

[0028] The data rendering method provided in this application can be applied to various wireless communication systems, such as 5G, 5G New Radio (NR), and new communication technologies that will emerge in the future development of communication, such as the IMS system based on 6th-generation (6G) mobile communication technology. Figure 1 This is a network diagram of a wireless communication system provided in one embodiment. For example... Figure 1 As shown, the wireless communication system includes a terminal device 110 and a media processing network element 120. This wireless communication system can support data channel (DC) applications, such as AR calling services.

[0029] Terminal device 110 can be a device with wireless transceiver capabilities, which can be deployed on land (such as indoors or outdoors, handheld, wearable or vehicle-mounted); on water (such as ships); or in the air (such as airplanes, balloons and satellites). Examples of terminal devices 110 include: wireless terminals, user equipment (UE), mobile phones, mobile stations, tablets, laptops, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, personal digital assistants (PDAs), and other network-connected user devices; virtual reality (VR) terminals, augmented reality (AR) terminals, wireless terminals in industrial control, wireless terminals in self-driving, wireless terminals in remote medical care, wireless terminals in smart grids, wireless terminals in transportation safety, wireless terminals in smart cities, wireless terminals in smart homes, etc.; IoT nodes in the Internet of Things (IoT); in-vehicle communication devices in the Internet of Vehicles (IoV); entertainment and gaming devices or systems; and GPS devices, etc. The embodiments of this application do not limit the specific technologies or device forms used in the terminal devices. Furthermore, the term "terminal device" can be abbreviated as "terminal."

[0030] The media processing network element 120 is a network-side device that allows the terminal device 110 to access the wireless communication system wirelessly. The media processing network element 120 may include the following structure.

[0031] DC Application Server 121: Responsible for DC-based traffic processing and service control, including DC session media control and media capability negotiation with UE and other media processing network elements 120. DC Application Server 121 can be deployed within the IMS network; it can also be deployed in a third-party non-IMS network, using network open capabilities to interact with UE through the IMS network.

[0032] Data Channel Signalling Function (DCSF) 122: Provides data channel control capabilities.

[0033] Media Function / Media Resource Function (MF / MRF) 123: Supports media processing for IMS-based media services, receives and stores media processing logic from the IP Multimedia System Application Server (IMS AS) 124, and provides AR media capabilities for media data received from the UE.

[0034] IMS AS 124: Supports control plane functions and negotiation. It interacts with DCSF 122 for event notification, receives data channel control commands from DCSF 122, and accordingly interacts with MF / MRF 123 for data channel media resource management. Simultaneously, it receives media control commands from DCSF 122 and accordingly interacts with terminal device 110 to connect the audio / video media terminals of terminal device 110 to MF / MRF 123.

[0035] Call Session Control Function (CSCF): Responsible for session control and routing, including control of functions closely related to user services such as registration and authentication, session control, routing management, network management, and billing. CSCF mainly includes Query CSCF (I-CSCF)125, Service CSCF (S-CSCF)126, and Proxy CSCF (P-CSCF)127.

[0036] I-CSCF 125: A gateway node similar to Media Processing Element 120, used for allocating S-CSCF 126, route lookup, and IMS inter-domain topology hiding.

[0037] S-CSCF 126: It occupies a core control position in the IMS core network and is responsible for the registration and authentication of terminal devices 110, conference control, and user data management.

[0038] P-CSCF 127: The first connection point between the media processing network element 120 and the terminal device 110, providing proxy functionality and also user agent functionality.

[0039] IP Multimedia System Access Gateway (IMS-Access Gateway, IMS-AGW) 128.

[0040] Combination Figure 1 It can be seen that there can also be connections between media processing network elements 120, that is... Figure 1 The media processing network element 120 on the right side can be a remote media processing network element. There is a connection between the media processing network element 120 connected to the terminal device and the remote media processing network element.

[0041] In this application embodiment, a data rendering method, media processing network element, terminal device, and storage medium that can run on the above-mentioned wireless communication system are provided. The method can negotiate the media processing capabilities of the terminal device and the media processing capabilities of the IMS network, so that the terminal device and the media processing network element can provide flexible rendering processing, thereby improving the processing efficiency of media data and optimizing the experience of media services.

[0042] The following describes the data rendering methods, media processing network elements, terminal devices, and their technical effects.

[0043] Figure 2 This is a flowchart illustrating a data rendering method provided in one embodiment. For example... Figure 2 As shown, the method provided in this embodiment is applicable to media processing network elements. For example, it can be the above-described... Figure 1 The Chinese media processing network element 120 can also be used for the above. Figure 1 The method includes the following steps: MF / MRF 123 in the media processing network element 120.

[0044] S210. Establish a transmission channel between the media processing network element and the terminal device, and assign a corresponding identifier to the transmission channel. The transmission channel includes at least a first real-time transmission protocol (RTP) channel. The identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits data to be rendered.

[0045] An IMS session typically involves at least two terminal devices. In this embodiment, the terminal device can be either the device that initiates the IMS session or a device that does not. These terminal devices participating in the IMS session can be in the same network or in different networks.

[0046] When a terminal device needs to establish an IMS session with one or more other terminal devices, in order for the media processing network element to participate in the rendering process, the terminal device needs to send the data to be rendered to the media processing network element. Therefore, it is first necessary to establish a transmission channel between the media processing network element and the terminal device, and assign a corresponding identifier to the transmission channel to indicate the data type transmitted by each transmission channel.

[0047] In one embodiment, the transmission channel includes at least a first Real-time Transport Protocol (RTP) channel, and the identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits the data to be rendered.

[0048] The identifier corresponding to the first RTP channel can be either the first identifier or the second identifier.

[0049] When the identifier corresponding to the first RTP channel is the first identifier, the first identifier is used to indicate that the first RTP channel only transmits data to be rendered. That is, all data transmitted by the first RTP channel needs to be rendered by the media processing network element.

[0050] When the identifier corresponding to the first RTP channel is the second identifier, the second identifier is used to indicate that the first RTP channel transmits data to be rendered and non-rendered data. Non-rendered data refers to data that does not require rendering by the media processing network element. That is, in addition to transmitting data that needs to be rendered by the media processing network element, the first RTP channel can also transmit data that does not need to be rendered by the media processing network element. Furthermore, non-rendered data may or may not have been rendered by the terminal device.

[0051] In one embodiment, the transmission channel may further include at least one of a second RTP channel and a data channel. The data channel may include a bootstrap data channel (bootstrap DC) and an application data channel (application DC).

[0052] The identifier corresponding to the second RTP channel indicates that the second RTP channel only transmits non-rendered data. That is, all data transmitted by the second RTP channel does not require rendering by the media processing network elements.

[0053] The identifier corresponding to the application data channel is used to indicate the transmission of media metadata through the guiding data channel. Media metadata can be data that accompanies the data to be rendered and / or the non-rendered data, such as scene data or descriptive data (used to describe the data to be rendered and / or the non-rendered data). In some scenarios, non-rendered data may also include media metadata.

[0054] In one embodiment, when there are multiple transmission channels, the media processing network element can also assign a corresponding group identifier to the transmission channel when establishing a transmission channel with the terminal device, and there is a binding relationship between transmission channels with the same group identifier.

[0055] For example, assume that five transmission channels are established between the media processing network element and the terminal device: two first RTP channels (denoted as first RTP channel 1 and first RTP channel 2, respectively), one second RTP channel, and two application data channels (denoted as application data channel 1 and application data channel 2, respectively). The group identifiers corresponding to first RTP channel 1, second RTP channel, and application data channel 2 are all identifier 1, while the group identifiers corresponding to first RTP channel 2 and application data channel 1 are both identifier 2. That is, first RTP channel 1, second RTP channel, and application data channel 2 belong to the same group and have a binding relationship; first RTP channel 2 and application data channel 1 belong to the same group and have a binding relationship.

[0056] In one embodiment, the process by which the media processing network element assigns corresponding identifiers and group identifiers to the transmission channel can be implemented during the Session Description Protocol (SDP) negotiation between the media processing network element and the terminal device. The SDP negotiation process can take into account the media processing capabilities of the terminal device and the media processing capabilities of the IMS network.

[0057] For example, when the terminal device has strong media processing capabilities, it can perform a small amount of data rendering on the IMS network side, such as establishing a relatively small number of initial RTP channels. Conversely, when the terminal device has weak media processing capabilities, it can perform a large amount or all of the data rendering on the IMS network side, such as establishing a relatively large number of initial RTP channels. The same principle applies to the media processing capabilities of the IMS network.

[0058] S220. Obtain the data to be rendered sent by the terminal device through the first RTP channel.

[0059] After the transmission channel between the media processing network element and the terminal device is established, SDP negotiation between the two devices is complete, and data transmission can begin. The media processing network element can then obtain the data to be rendered sent by the terminal device through the first RTP channel.

[0060] Since the first RTP channel may be used only to transmit data to be rendered, or it may be used to transmit both data to be rendered and non-rendered data, the media processing network element first needs to determine the identifier corresponding to the first RTP channel. When the identifier corresponding to the first RTP channel is the first identifier, the media processing network element can directly use all the data transmitted by the first RTP channel as the data to be rendered. When the identifier corresponding to the first RTP channel is the second identifier, the media processing network element needs to further distinguish all the data transmitted by the first RTP channel.

[0061] Specifically, when the identifier corresponding to the first RTP channel is the second identifier, the media processing network element can further distinguish all the data transmitted through the first RTP channel by: receiving the RTP packet sent by the terminal device through the first RTP channel; and determining the data to be rendered from the data carried by the RTP packet according to the RTP extension header of the RTP packet.

[0062] For example, the media processing network element and the terminal device can negotiate that when the identifier corresponding to the first RTP channel is the second identifier, the RTP extension headers from the 4th to the 10th in the RTP packet all carry data to be rendered. In this way, the distinction between data to be rendered and non-rendered data can be achieved.

[0063] In one embodiment, if the transmission channel established in step S210 includes a second RTP channel, the media processing network element can also acquire non-rendered data sent by the terminal device through the second RTP channel; if the transmission channel established in step S210 includes a boot data channel, the media processing network element can also acquire media metadata sent by the terminal device through the boot data channel.

[0064] S230. Render the data to be rendered to obtain the rendered data.

[0065] In one embodiment, after obtaining the rendered data, the media processing network element can also send media data to the terminal device or other terminal devices, the media data including at least the rendered data.

[0066] In one embodiment, if the transmission channel established in step S210 includes a second RTP channel, and the media processing network element obtains the non-rendered data sent by the terminal device through the second RTP channel, then the media data may further include the non-rendered data sent by the terminal device through the second RTP channel. If the transmission channel established in step S210 includes a boot data channel, and the media processing network element obtains the media metadata sent by the terminal device through the boot data channel, then the media data may further include the media metadata sent by the terminal device through the boot data channel.

[0067] In one embodiment, the media data may also include data sent by other terminal devices.

[0068] In one embodiment, if the media processing network element assigns a corresponding group identifier to the transmission channel in step S210 above, the media processing network element can also perform at least one of the following processes on the data transmitted by the transmission channel with the same group identifier: scene synthesis processing, data association processing.

[0069] Scene compositing and data association processing can be performed before or after the data to be rendered; this application does not impose any specific restrictions on this.

[0070] In this way, by negotiating the media processing capabilities of terminal devices and the IMS network, the terminal devices and media processing network elements can provide flexible rendering processing, thereby improving the efficiency of media data processing and optimizing the media service experience.

[0071] Figure 3 This is a flowchart illustrating another data rendering method provided in one embodiment. For example... Figure 3 As shown, the method provided in this embodiment is applicable to terminal devices. The method includes the following steps.

[0072] S310. Establish a transmission channel between the terminal device and the media processing network element, and determine the identifier corresponding to the transmission channel. The transmission channel includes at least a first real-time transmission protocol (RTP) channel. The identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits the data to be rendered.

[0073] An IMS session typically involves at least two terminal devices. In this embodiment, the terminal device can be either the device that initiates the IMS session or a device that does not. These terminal devices participating in the IMS session can be in the same network or in different networks.

[0074] When a terminal device needs to establish an IMS session with one or more other terminal devices, in order for the media processing network element to participate in the rendering process, the terminal device needs to send the data to be rendered to the media processing network element. Therefore, it is first necessary to establish a transmission channel between the media processing network element and the terminal device, and determine the identifier corresponding to the transmission channel in order to determine the data type to be transmitted on each transmission channel.

[0075] In one embodiment, the transmission channel includes at least a first RTP channel, and the identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits data to be rendered.

[0076] The identifier corresponding to the first RTP channel can be either the first identifier or the second identifier.

[0077] When the identifier corresponding to the first RTP channel is the first identifier, the first identifier is used to indicate that the first RTP channel only transmits data to be rendered. That is, all data transmitted by the first RTP channel needs to be rendered by the media processing network element.

[0078] When the identifier corresponding to the first RTP channel is the second identifier, the second identifier is used to indicate that the first RTP channel transmits data to be rendered and non-rendered data. Non-rendered data refers to data that does not require rendering by the media processing network element. That is, in addition to transmitting data that needs to be rendered by the media processing network element, the first RTP channel can also transmit data that does not need to be rendered by the media processing network element. Furthermore, non-rendered data may or may not have been rendered by the terminal device.

[0079] In one embodiment, the transmission channel may further include at least one of a second RTP channel and a data channel. The data channel may include a bootstrap data channel and an application data channel.

[0080] The identifier corresponding to the second RTP channel indicates that the second RTP channel only transmits non-rendered data. That is, all data transmitted by the second RTP channel does not require rendering by the media processing network elements.

[0081] The identifier corresponding to the application data channel is used to indicate the transmission of media metadata through the guiding data channel. Media metadata can be data that accompanies the data to be rendered and / or the non-rendered data, such as scene data or descriptive data (used to describe the data to be rendered and / or the non-rendered data). In some scenarios, non-rendered data may also include media metadata.

[0082] In one embodiment, when there are multiple transmission channels, the terminal device can also determine the group identifier corresponding to the transmission channel when establishing a transmission channel with the media processing network element, and there is a binding relationship between transmission channels with the same group identifier.

[0083] For example, assume that five transmission channels are established between the media processing network element and the terminal device: two first RTP channels (denoted as first RTP channel 1 and first RTP channel 2, respectively), one second RTP channel, and two application data channels (denoted as application data channel 1 and application data channel 2, respectively). The group identifiers corresponding to first RTP channel 1, second RTP channel, and application data channel 2 are all identifier 1, while the group identifiers corresponding to first RTP channel 2 and application data channel 1 are both identifier 2. That is, first RTP channel 1, second RTP channel, and application data channel 2 belong to the same group and have a binding relationship; first RTP channel 2 and application data channel 1 belong to the same group and have a binding relationship.

[0084] In one embodiment, the process by which the terminal device determines the identifier and group identifier corresponding to the transmission channel can be implemented during the SDP negotiation process between the media processing network element and the terminal device. The SDP negotiation process can take into account the media processing capabilities of the terminal device and the IMS network.

[0085] For example, when the terminal device has strong media processing capabilities, it can perform a small amount of data rendering on the IMS network side, such as establishing a relatively small number of initial RTP channels. Conversely, when the terminal device has weak media processing capabilities, it can perform a large amount or all of the data rendering on the IMS network side, such as establishing a relatively large number of initial RTP channels. The same principle applies to the media processing capabilities of the IMS network.

[0086] S320: Send the data to be rendered to the media processing network element through the first RTP channel.

[0087] After the transmission channel between the media processing network element and the terminal device is established, SDP negotiation between the two has been completed, and data transmission can begin. The terminal device sends the data to be rendered to the media processing network element through the first RTP channel.

[0088] It should be noted that the first RTP channel may be used only to transmit data to be rendered, or it may be used to transmit both data to be rendered and non-rendered data. When the identifier corresponding to the first RTP channel is the second identifier, the terminal device can also send non-rendered data to the media processing network element through the first RTP channel.

[0089] In one embodiment, if the transmission channel established in step S310 includes a second RTP channel, the terminal device can also send non-rendered data to the media processing network element through the second RTP channel; if the transmission channel established in step S310 includes a boot data channel, the terminal device can also send media metadata to the media processing network element through the boot data channel.

[0090] In one embodiment, after sending the data to be rendered to the media processing network element through the first RTP channel, the terminal device can also receive media data sent by the media processing network element, the media data including at least the rendered data.

[0091] In one embodiment, the media data may also include data sent by other terminal devices.

[0092] In this way, by negotiating the media processing capabilities of terminal devices and the IMS network, the terminal devices and media processing network elements can provide flexible rendering processing, thereby improving the efficiency of media data processing and optimizing the media service experience.

[0093] Below are some examples illustrating the data rendering method provided in this application. In the following examples, UE-A is used as an example of the terminal device.

[0094] Example 1: UE-A has strong capabilities. UE-A can render locally acquired data and send it to the media processing network element, without the media processing network element needing to participate in the rendering process. UE-A can also render only a portion of the locally acquired data, leaving the remaining data to be rendered by the media processing network element. Furthermore, in DC applications involving multiple UEs, the media processing network element can perform other media processing (such as spatial calculation, scene combination, etc.) on the data from multiple UEs (including rendered and unrendered data) to generate a scene containing multi-user data.

[0095] Figure 4 This is an interactive diagram illustrating data rendering provided in Example 1. For example... Figure 4 As shown, the steps include the following.

[0096] S401, UE-A conducts IMS sessions with other UEs and establishes a transmission channel, which has a corresponding identifier.

[0097] In Example 1, because UE-A has strong capabilities, it can choose whether to allow the media processing network element to participate in rendering, but scene compositing may be required (especially in multi-party service scenarios). In multi-party service scenarios, some UEs have weaker capabilities, and these UEs can also request media processing network elements to perform media processing, including rendering and scene compositing. IMS sessions are established between UE-A and other UEs, and between UE-A and the media processing network element. The SDP session description protocol is used to initiate (offer) and respond (answer) SDP processes, initialize the media stream (including media capability negotiation) and renegotiation, and establish a transmission channel.

[0098] This IMS session can be initiated by UE-A, which can be in the same network or in different networks with other UEs. Establishing a transport channel refers to establishing a transport channel between UE-A and the media processing network element, and between other UEs and the media processing network element. The transport channel includes the RTP channel, the bootstrap data channel, and the application data channel. The RTP channel is used to transmit video / image data. The bootstrap data channel is mainly used for downloading DC applications, but can also transmit media metadata (such as scene descriptions) or other data. The application data channel is mainly used to transmit media metadata, but can also transmit video / image data, as well as service negotiation data.

[0099] Specifically, if UE-A or other UEs (hereinafter referred to as UE) do not have a pre-installed DC application, then the DC application needs to be obtained through the boot data channel; if the UE has a pre-installed DC application, then the boot data channel is not required to obtain it. DC applications are usually Hyper Text Markup Language (HTML) applications.

[0100] When acquiring DC applications through the data channel, the UE can also acquire pre-set / pre-stored related data, such as scene description data.

[0101] In one embodiment, the data in the DC application can be any type of data, such as scene description files, digital human images stored on the IMS network or a third-party server. The IMS network can transmit this data to the UE after establishing a transmission channel (such as an RTP channel or a DC channel) via a bootstrap data channel or an application data channel.

[0102] In one embodiment, taking the establishment of a transmission channel between UE-A and the media processing network element as an example, if the established RTP channel is a channel between UE-A and other UEs, that is, not anchored to the MF / MRF of the IMS network, and directly interacts with other UEs through IMS-AGW, then UE-A can renegotiate to anchor the RTP channel to the MF / MRF in the IMS system so that the MF / MRF can perform media processing on the data (such as audio and video data) of the RTP channel.

[0103] Application data channels are typically anchored to the MF / MRF during establishment. Therefore, when establishing an application data channel, the DC application server can directly distribute DC application-related data using the application data channel via the MF / MRF. The MF / MRF can process the data transmitted by the UE through the application data channel. If the application data channel is not anchored to the MF / MRF during establishment (e.g., an application data channel is established between UE-A and other UEs), it can be renegotiated to the MF / MRF via SDP.

[0104] In one embodiment, the transmission channel has a corresponding identifier. The identifier corresponding to the transmission channel can be negotiated using SDP description rules.

[0105] For example, in a media-level description, a new line “a=" can be added below the “m=" line (representing a media stream channel) to describe the characteristics of the data transmitted by that media stream channel.

[0106] 1) For RTP channels, add a field a = data_processing_mode to the attributes of the media stream channel to distinguish whether the RTP channel is the first RTP channel or the second RTP channel.

[0107] When a = data_processing_mode: 1 rendered, data_processing_mode being 1 indicates that all data transmitted through this RTP channel does not require rendering by the media processing network element. That is, this RTP channel is the second RTP channel. "Rendered" indicates that all data transmitted through this RTP channel has been rendered by the terminal device.

[0108] When a = data_processing_mode: 1 unrendered, data_processing_mode being 1 indicates that all data transmitted through this RTP channel does not require rendering by the media processing network element. That is, this RTP channel is the second RTP channel. "Unrendered" means that all data transmitted through this RTP channel is not rendered by the terminal device.

[0109] When a = data_processing_mode: 1 rendered / unrendered, data_processing_mode being 1 indicates that all data transmitted through this RTP channel does not require rendering by the media processing network element. That is, this RTP channel is the second RTP channel. rendered / unrendered indicates that some of the data transmitted through this RTP channel has been rendered by the terminal device, while some has not.

[0110] When a = data_processing_mode: 2render, data_processing_mode being 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. render indicates that all data transmitted by this RTP channel requires rendering by media processing network elements. That is, in this case, this RTP channel is the first RTP channel.

[0111] When a = data_processing_mode: 2rendered, data_processing_mode of 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. "rendered" means that all data transmitted by this RTP channel does not require rendering by media processing network elements and has already been rendered by the terminal device. In this case, this RTP channel is the second RTP channel.

[0112] When a = data_processing_mode: 2unrendered, data_processing_mode being 2 indicates that the data transmitted through this RTP channel may or may not require rendering by media processing network elements. "Unrendered" means that all data transmitted through this RTP channel is not rendered by the terminal device but does not require rendering by media processing network elements. In this case, the RTP channel is a second RTP channel.

[0113] When a = data_processing_mode: 2render / rendered, data_processing_mode of 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. render / rendered indicates that some of the data transmitted by this RTP channel requires rendering by media processing network elements, while some does not require rendering by media processing network elements and has already been rendered by the terminal device. That is, in this case, this RTP channel is the first RTP channel.

[0114] When a = data_processing_mode: 2render / unrendered, data_processing_mode of 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. render / unrendered means that some of the data transmitted by this RTP channel requires rendering by media processing network elements, while some is not rendered by the terminal device but does not require rendering by media processing network elements. In this case, this RTP channel is the first RTP channel.

[0115] When a = data_processing_mode: 2rendered / unrendered, data_processing_mode of 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. rendered / unrendered means that some of the data transmitted by this RTP channel does not require rendering by media processing network elements and has already been rendered by the terminal device, while some data has not been rendered by the terminal device but does not require rendering by media processing network elements. In this case, this RTP channel is a second RTP channel.

[0116] When a = data_processing_mode: 2render / rendered / unrendered, a data_processing_mode of 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. render / rendered / unrendered means that some of the data transmitted by this RTP channel requires rendering by media processing network elements, some does not require rendering by media processing network elements and has already been rendered by the terminal device, and some has not been rendered by the terminal device but does not require rendering by media processing network elements. In this case, the RTP channel is the first RTP channel.

[0117] Of course, if the a=data_processing_mode field is not used, the media processing network element can default to not processing the received data.

[0118] Furthermore, when the RTP channel is the first RTP channel, and some of the data transmitted on this RTP channel needs to be rendered by media processing network elements while others do not, the RTP header extension method provided by the RTP protocol can be used to distinguish these data. The extended RTP header can transmit media metadata associated with the rendered data, such as the spatial location information of the rendered data. During SDP negotiation, the a=extmap attribute is used to describe the location of the RTP extended header and the corresponding information.

[0119] For example, a = extmap:4urn:3gpp:pose means that the location information (including coordinate system information and specific coordinate information) is carried starting from the 4th RTP extension header. Then, the 4th to 10th extension headers all carry location information. The specific number of extension headers required needs to be determined according to the coordinate system.

[0120] 2) For boot data channels, if the boot data channel is used to transmit media metadata, a new a=media_metadata field is added to the properties of the media stream channel to indicate that the boot data channel can transmit one or more types of media metadata.

[0121] When a = media_metadata:1SD, and the metadata value is 1, it indicates that the boot data channel transmits a type of media metadata, and SD indicates that the type is scene description.

[0122] 3) For application data channels, add a field a = data_resource to the attributes of media stream channels to indicate that the application data channel can transmit at least one of media metadata, video / image data, and other business data.

[0123] When a = data_resource:1video, and the value of data_resource is 1, it means that the application's data channel transmits a type of data. video indicates that the type is video, and some data characteristics of the video, such as encoding format and frequency, can also be added to video.

[0124] Optionally, new fields such as a=media_metadata and a=media can be added to the properties of the media stream channel to indicate the data type of the data stream in the channel.

[0125] In one embodiment, if data is transmitted using both RTP channels and application data channels simultaneously, and this data is media-processing related, such as rendered data and its location information during rendering, unrendered data and its location information when it is collected, and related processing is required, SDP negotiation can also assign corresponding group identifiers to the transmission channels, and transmission channels with the same group identifier have a binding relationship.

[0126] For example, in the session level description, adding a new line with "a=" further describes the current multiple media streams. For instance, adding "a=" would be equivalent to adding "media_processing_group. <group-1> / ... / <group-n>Fields are used to represent the relationships between multiple media streams (such as RTP channels and data channels).

[0127] When a = media_processing_group:composition 1 2 3 / 4 5 6, composition indicates the media processing that requires the media processing network element to perform scene composition. "1 2 3" and "4 5 6" represent two groups respectively. The media streams in the two groups must undergo associated media processing. Each number is the same as the value of a = mid in each media stream (media level). The specific media stream can be identified through this value.

[0128] It should also be noted that if the data received by the UE is already rendered, it can be further rendered according to presentation requirements. This is determined by the UE's own needs and does not require negotiation.

[0129] S402, UE-A obtains local data.

[0130] UE-A acquires local data through one or more local devices (such as cameras, sensors, etc., which can be integrated into settings), such as images of AR (augmented reality) objects (objects, people, etc.), user posture information, traditional audio and video data, etc.

[0131] S403 and UE-A process local data.

[0132] UE-A processes local data, such as media rendering, based on its own processing capabilities, network conditions, and media service types.

[0133] S404a and UE-A send the rendered data to the media processing network element.

[0134] S404b and other UEs send locally collected data to the media processing network element.

[0135] Among them, the data collected locally by other UEs can be data that has undergone local media processing (rendering) or data that has not undergone local media processing (rendering).

[0136] Specifically, UE-A / other UEs can upload rendered (which may not include scene combinations) or non-rendered data to the media processing network element via the RTP channel. The media processing network element needs to know that the data has already been processed locally by the UE or does not need to be processed by the media processing network element. Optionally, data uploaded by UE-A / other UEs can also be uploaded to the media processing network element via the application data channel.

[0137] S405. The media processing network element performs secondary media processing on the data uploaded by two or more UEs, based on the type of DC application or through negotiation.

[0138] The media processing network element performs secondary media processing on the data uploaded by two or more UEs, based on the type of DC application or negotiation (not limited to SDP negotiation, but also through data channels). This includes scene composition (performing spatial calculations based on the spatial information of each media object to generate a scene graph containing multiple media objects).

[0139] This step is optional. For example, if the RTP channel is not anchored to MF / MRF, and the UE obtains the scene description through the bootstrap data channel or application data channel, the receiving UE can synthesize the scene based on the data transmitted by the peer UE to generate a complete scene for presentation. Other unknown situations may also occur, where secondary media processing by the media processing network element is not required, and the processing is done directly by the sending / receiving UE.

[0140] Optionally, if some UEs (non-session-initiating UEs) upload data to the media processing network element that has not undergone local media processing by the UE and requires media processing by the media processing network element, the media processing network element should process the data uploaded by the UEs first, and then perform scene synthesis and other media processing on the data of all UEs.

[0141] Optionally, the media processing network element can perform differentiated processing based on the state information of each UE (such as pose) and the rendering method requested by the UE, such as performing partial rendering based on the UE's viewport during rendering.

[0142] S406. The media processing network element sends the data after secondary media processing to UE-A / other UEs.

[0143] If step S405 above is not executed, that is, if the media processing network element does not need to perform non-rendering media processing such as scene combination, the media processing network element can directly transmit data from the sending UE to the receiving UE, such as from other UEs to UE-A, or from UE-A to other UEs.

[0144] Optionally, the media processing network element can transmit all or part of the data after secondary media processing, depending on the UE's status. For example, when UE status information (such as location information) is transmitted to the media processing network element along with audio and video data via the RTP channel, or transmitted separately to the media processing network element via the application data channel, the media processing network element can transmit the rendered or unrendered data portion to the UE based on the UE status, network conditions, etc.

[0145] Example 2: UE-A has limited capabilities. UE-A can request assistance from media processing network elements for rendering. For example, before sending data, UE-A can perform rendering and other processing first, then send it to the media processing network element for further rendering. Finally, the media processing network element will send it to other UEs and / or return it to UE-A. This method can reduce the amount of data transmitted and the server's data cache loading time. Alternatively, UE-A can directly send the data to be rendered to the media processing network element for rendering. After rendering, the media processing network element will send it to other UEs and / or return it to UE-A. In this method, the UE can perform parallel rendering or wait for the network side to complete rendering before performing terminal-side rendering. Parallel rendering can reduce rendering latency. Of course, after rendering is complete, the media processing network element can perform further media processing such as scene compositing as needed.

[0146] Figure 5 This is an interactive diagram illustrating data rendering provided in Example 2. For example... Figure 5 As shown, the steps include the following.

[0147] S501, UE-A conducts IMS sessions with other UEs and establishes a transmission channel, which has a corresponding identifier.

[0148] In Example 2, due to the weak capabilities of UE-A or other reasons (such as the user's subjective desire for network-assisted rendering, power status, signal status, computing power status, internal storage status, etc.), media processing network elements are required to assist in media processing for rendering. UE-A establishes IMS sessions with other UEs and with the media processing network element, using the SDP session description protocol to perform SDP offer and answer procedures, initializing the media stream (including negotiating and renegotiating media capabilities), and establishing a transmission channel.

[0149] This IMS session can be initiated by UE-A, which can be in the same network or in different networks with other UEs. Establishing a transport channel refers to establishing a transport channel between UE-A and the media processing network element, and between other UEs and the media processing network element. The transport channel includes the RTP channel, the bootstrap data channel, and the application data channel. The RTP channel is used to transmit video / image data. The bootstrap data channel is mainly used for downloading DC applications, but can also transmit media metadata (such as scene descriptions) or other data. The application data channel is mainly used to transmit media metadata, but can also transmit video / image data, as well as service negotiation data.

[0150] Specifically, if UE-A or other UEs (hereinafter referred to as UE) do not have a pre-installed DC application, then the DC application needs to be obtained through the boot data channel; if the UE has a pre-installed DC application, then it does not need to be obtained through the boot data channel. The DC application is usually an HTML application.

[0151] In one embodiment, the transmission channel has a corresponding identifier. The identifier corresponding to the transmission channel can be negotiated using SDP description rules.

[0152] For example, in a media-level description, a new line "a=" can be added below the "m=" line (representing a media stream channel) to describe the characteristics of the data transmitted by that media stream channel.

[0153] 1) For RTP channels, add a field a = data_processing_mode to the attributes of the media stream channel to distinguish whether the RTP channel is the first RTP channel or the second RTP channel.

[0154] When a = data_processing_mode: 1 rendered, data_processing_mode being 1 indicates that all data transmitted through this RTP channel does not require rendering by the media processing network element. That is, this RTP channel is the second RTP channel. "Rendered" indicates that all data transmitted through this RTP channel has been rendered by the terminal device.

[0155] When a = data_processing_mode: 1 unrendered, data_processing_mode being 1 indicates that all data transmitted through this RTP channel does not require rendering by the media processing network element. That is, this RTP channel is the second RTP channel. "Unrendered" means that all data transmitted through this RTP channel is not rendered by the terminal device.

[0156] When a = data_processing_mode: 1 rendered / unrendered, data_processing_mode being 1 indicates that all data transmitted through this RTP channel does not require rendering by the media processing network element. That is, this RTP channel is the second RTP channel. rendered / unrendered indicates that some of the data transmitted through this RTP channel has been rendered by the terminal device, while some has not.

[0157] When a = data_processing_mode: 2render, data_processing_mode being 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. render indicates that all data transmitted by this RTP channel requires rendering by media processing network elements. That is, in this case, this RTP channel is the first RTP channel.

[0158] When a = data_processing_mode: 2rendered, data_processing_mode of 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. "rendered" means that all data transmitted by this RTP channel does not require rendering by media processing network elements and has already been rendered by the terminal device. In this case, this RTP channel is the second RTP channel.

[0159] When a = data_processing_mode: 2unrendered, data_processing_mode being 2 indicates that the data transmitted through this RTP channel may or may not require rendering by media processing network elements. "Unrendered" means that all data transmitted through this RTP channel is not rendered by the terminal device but does not require rendering by media processing network elements. In this case, the RTP channel is a second RTP channel.

[0160] When a = data_processing_mode: 2render / rendered, data_processing_mode of 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. render / rendered indicates that some of the data transmitted by this RTP channel requires rendering by media processing network elements, while some does not require rendering by media processing network elements and has already been rendered by the terminal device. That is, in this case, this RTP channel is the first RTP channel.

[0161] When a = data_processing_mode: 2render / unrendered, data_processing_mode of 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. render / unrendered means that some of the data transmitted by this RTP channel requires rendering by media processing network elements, while some is not rendered by the terminal device but does not require rendering by media processing network elements. In this case, this RTP channel is the first RTP channel.

[0162] When a = data_processing_mode: 2rendered / unrendered, data_processing_mode of 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. rendered / unrendered means that some of the data transmitted by this RTP channel does not require rendering by media processing network elements and has already been rendered by the terminal device, while some data has not been rendered by the terminal device but does not require rendering by media processing network elements. In this case, this RTP channel is a second RTP channel.

[0163] When a = data_processing_mode: 2render / rendered / unrendered, a data_processing_mode of 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. render / rendered / unrendered means that some of the data transmitted by this RTP channel requires rendering by media processing network elements, some does not require rendering by media processing network elements and has already been rendered by the terminal device, and some has not been rendered by the terminal device but does not require rendering by media processing network elements. In this case, the RTP channel is the first RTP channel.

[0164] Of course, if the `a=data_processing_mode` field is not used, the media processing network element can default to not processing the received data. For the first RTP channel, its network capabilities can also be determined to support UE rendering requests (such as the rendering format, rendering computing resources, etc.).

[0165] Furthermore, when the RTP channel is the first RTP channel, and some of the data transmitted on this RTP channel needs to be rendered by media processing network elements while others do not, the RTP header extension method provided by the RTP protocol can be used to distinguish these data. The extended RTP header can transmit media metadata associated with the rendered data, such as the spatial location information of the rendered data. During SDP negotiation, the a=extmap attribute is used to describe the location of the RTP extended header and the corresponding information.

[0166] For example, `a = extmap:1urn:3gpp:media_processing` indicates the type of data carried by the first RTP extension header. In the first RTP extension header of a specific RTP packet, a value of 1 indicates that the data carried by the RTP packet has been rendered; a value of 2 indicates that the data carried by the RTP packet has not been rendered but does not need to be rendered; a value of 3 indicates that the data carried by the RTP packet has not been rendered but needs to be rendered. If the extension header has sufficient bytes, more values ​​can be set to represent different meanings.

[0167] For example, a = extmap:4urn:3gpp:pose means that the location information (including coordinate system information and specific coordinate information) will be carried starting from the 4th RTP extension header. Then, the 4th to the 10th extension headers will all carry location information. The specific number of extension headers required needs to be determined according to the coordinate system.

[0168] 2) For boot data channels, if the boot data channel is used to transmit media metadata, a new a=media_metadata field is added to the properties of the media stream channel to indicate that the boot data channel can transmit one or more types of media metadata.

[0169] When a = media_metadata:1SD, and the metadata value is 1, it indicates that the boot data channel transmits a type of media metadata, and SD indicates that the type is scene description.

[0170] 3) For application data channels, add a field a = data_resource to the attributes of media stream channels to indicate that the application data channel can transmit at least one of media metadata, video / image data, and other business data.

[0171] When a = data_resource:1video, and the value of data_resource is 1, it means that the application's data channel transmits a type of data. video indicates that the type is video, and some data characteristics of the video, such as encoding format and frequency, can also be added to video.

[0172] When a = data_resource:2media_metadata pose / orientation video (which can be a URN), and the data_resource value is 2, it indicates that this channel transmits two types of data: video and media_metadata. Additional data characteristics of the video, such as encoding format and frequency, can be added after video. Similarly, data characteristics such as position and orientation can be added after media_metadata.

[0173] The application layer protocol of the application data channel can be indicated by "subprotocol=" in "a=dcmap", such as subprotocol=http.

[0174] Optionally, new fields such as a=media_metadata and a=media can be added to the properties of the media stream channel to indicate the data type of the data stream in the channel.

[0175] In one embodiment, if data is transmitted using both RTP channels and application data channels simultaneously, and this data is media-processing related, such as rendered data and its location information during rendering, unrendered data and its location information when it is collected, and related processing is required, SDP negotiation can also assign corresponding group identifiers to the transmission channels, and transmission channels with the same group identifier have a binding relationship.

[0176] For example, in the session level description, adding a new line with "a=" further describes the current multiple media streams. For instance, adding "a=" would be equivalent to adding "media_processing_group. <group-1> / ... / <group-n>Fields are used to represent the relationships between multiple media streams (such as RTP channels and data channels).

[0177] When a = media_processing_group:composition 1 2 3 / 4 5 6, composition indicates the media processing that requires the media processing network element to perform scene composition. "1 2 3" and "4 5 6" represent two groups respectively. The media streams in the two groups must undergo associated media processing. Each number is the same as the value of a = mid in each media stream (media level). The specific media stream can be identified through this value.

[0178] When a = media_processing_group: coding 1 2 3 / 4 5 6, coding indicates that when decoding data in a media stream, association is required. For example, multi-view encoded video requires the RTP channel to transmit video encoded data and the application data channel to transmit atlas encoded data. The RTP channel and the application data channel form a group.

[0179] It should also be noted that if a DC application uses the application data channel to transmit audio / video data (including data that needs to be rendered and / or data that does not need to be rendered), it can still simultaneously use the RTP channel to transmit audio / video data. That is, audio / video data can be transmitted through the RTP channel and / or the application data channel, depending on the application itself. For example, multiple audio / video data collected by the UE can be transmitted through multiple RTP channels and / or application data channels.

[0180] S502 and UE-A negotiate rendering with the media processing network element to anchor and redirect the audio and video streams to MF / MRF.

[0181] UE-A negotiates media rendering with media processing network elements based on an application data channel or SDP session. Typically, UE-A sends a media rendering negotiation request, which negotiates the computing power required for rendering, the type of rendering, and the data formats for rendering input and output. IMS AS forwards the rendering request to the DC application server, which then sends a media resource request to the MF / MRF, requesting the MF / MRF to allocate rendering resources based on the information carried in the UE-A rendering request.

[0182] If rendering negotiation is complete and UE-A's RTP stream is not anchored to MF / MRF (e.g., anchored to another UE), then UE-A's RTP stream will be re-anchored to MF / MRF.

[0183] S503 and other UEs will anchor and redirect audio and video streams to MF / MRF.

[0184] If the IMS network accessed by other UEs is different from the IMS network accessed by UE-A, and the IMS network accessed by other UEs also provides rendering resources and rendering functions, then after the rendering negotiation is completed, the audio and video streams of the IMS network accessed by UE-A can be directly anchored to other UEs, or they can be anchored to the MF / MRF of the IMS network accessed by other UEs. That is, the IMS AS and MF / MRF of the IMS network accessed by other UEs renegotiate to anchor the audio and video streams transmitted by the IMS network accessed by UE-A to the MF / MRF.

[0185] S504 and UE-A process some local data according to rendering negotiation.

[0186] UE-A processes some local data, such as media rendering, based on rendering negotiation.

[0187] S505 and UE-A send the data to be rendered to the media processing network element.

[0188] Optionally, if scene combination processing is required in the media processing network element, the UE-A can send the rendered data and the data to be rendered at the same time, or steps S504 and S505 can be performed simultaneously. That is, the data to be rendered is sent to the media processing network element first, and after the UE-A completes local rendering, the locally rendered data is also sent to the media processing network element.

[0189] S506, the media processing network element performs network-side media processing based on rendering negotiation.

[0190] Among them, media processing on the network side may include at least one of rendering processing and scene compositing.

[0191] Optionally, if there is a need for scene combination during rendering negotiation, the media processing network element can send a scene combination request through the application data channel after rendering is completed, and continue to perform scene combination processing.

[0192] S507, the media processing network element sends the data processed by the network side to other UEs.

[0193] This application also provides a data rendering method based on an IMS network. Based on the DC application service requested by the UE, the media processing network element can directly provide media data, such as a digital human model. This data can be provided by a third party or stored in the media storage server of the IMS system. That is, the IMS system can directly render the media data provided by the network side on the IMS network side and then send it to the UE.

[0194] Figure 6 This is an interactive schematic diagram illustrating data rendering based on an IMS network, provided in one embodiment. For example... Figure 6 As shown, the steps include the following.

[0195] S601, UE and DC application conduct IMS session and establish transmission channel, the transmission channel has corresponding identifier.

[0196] This IMS session can be initiated by the UE. Establishing a transport channel refers to establishing a transport channel between the UE and the media processing network element, and between the DC application and the media processing network element. The transport channel includes the RTP channel, the bootstrap data channel, and the application data channel. The RTP channel is used to transmit video / image data. The bootstrap data channel is mainly used for downloading data for DC applications, but can also transmit media metadata (such as scene descriptions) or other data. The application data channel is mainly used to transmit media metadata, but can also transmit video / image data, as well as service negotiation data.

[0197] Specifically, if the UE does not have a pre-installed DC application, it needs to obtain the DC application through the boot data channel; if the UE has a pre-installed DC application, it does not need to obtain it through the boot data channel. The DC application is usually an HTML application.

[0198] In one embodiment, the transmission channel has a corresponding identifier. The identifier corresponding to the transmission channel can be negotiated using SDP description rules.

[0199] For example, in a media-level description, a new line "a=" can be added below the "m=" line (representing a media stream channel) to describe the characteristics of the data transmitted by that media stream channel.

[0200] 1) For RTP channels, add a field a = data_processing_mode to the attributes of the media stream channel to distinguish whether the RTP channel is the first RTP channel or the second RTP channel.

[0201] When a = data_processing_mode: 1 rendered, data_processing_mode being 1 indicates that all data transmitted through this RTP channel does not require rendering by the media processing network element. That is, this RTP channel is the second RTP channel. "Rendered" indicates that all data transmitted through this RTP channel has been rendered by the terminal device.

[0202] When a = data_processing_mode: 1 unrendered, data_processing_mode being 1 indicates that all data transmitted through this RTP channel does not require rendering by the media processing network element. That is, this RTP channel is the second RTP channel. "Unrendered" means that all data transmitted through this RTP channel is not rendered by the terminal device.

[0203] When a = data_processing_mode: 1 rendered / unrendered, data_processing_mode being 1 indicates that all data transmitted through this RTP channel does not require rendering by the media processing network element. That is, this RTP channel is the second RTP channel. rendered / unrendered indicates that some of the data transmitted through this RTP channel has been rendered by the terminal device, while some has not.

[0204] When a = data_processing_mode: 2render, data_processing_mode being 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. render indicates that all data transmitted by this RTP channel requires rendering by media processing network elements. That is, in this case, this RTP channel is the first RTP channel.

[0205] When a = data_processing_mode: 2rendered, data_processing_mode of 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. "rendered" means that all data transmitted by this RTP channel does not require rendering by media processing network elements and has already been rendered by the terminal device. In this case, this RTP channel is the second RTP channel.

[0206] When a = data_processing_mode: 2unrendered, data_processing_mode being 2 indicates that the data transmitted through this RTP channel may or may not require rendering by media processing network elements. "Unrendered" means that all data transmitted through this RTP channel is not rendered by the terminal device but does not require rendering by media processing network elements. In this case, the RTP channel is a second RTP channel.

[0207] When a = data_processing_mode: 2render / rendered, data_processing_mode of 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. render / rendered indicates that some of the data transmitted by this RTP channel requires rendering by media processing network elements, while some does not require rendering by media processing network elements and has already been rendered by the terminal device. That is, in this case, this RTP channel is the first RTP channel.

[0208] When a = data_processing_mode: 2render / unrendered, data_processing_mode of 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. render / unrendered means that some of the data transmitted by this RTP channel requires rendering by media processing network elements, while some is not rendered by the terminal device but does not require rendering by media processing network elements. In this case, this RTP channel is the first RTP channel.

[0209] When a = data_processing_mode: 2rendered / unrendered, data_processing_mode of 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. rendered / unrendered means that some of the data transmitted by this RTP channel does not require rendering by media processing network elements and has already been rendered by the terminal device, while some data has not been rendered by the terminal device but does not require rendering by media processing network elements. In this case, this RTP channel is a second RTP channel.

[0210] When a = data_processing_mode: 2render / rendered / unrendered, a data_processing_mode of 2 indicates that the data transmitted by this RTP channel may or may not require rendering by media processing network elements. render / rendered / unrendered means that some of the data transmitted by this RTP channel requires rendering by media processing network elements, some does not require rendering by media processing network elements and has already been rendered by the terminal device, and some has not been rendered by the terminal device but does not require rendering by media processing network elements. In this case, the RTP channel is the first RTP channel.

[0211] Furthermore, when the RTP channel is the first RTP channel, and some of the data transmitted on this RTP channel needs to be rendered by media processing network elements while others do not, the RTP header extension method provided by the RTP protocol can be used to distinguish these data. The extended RTP header can transmit media metadata associated with the rendered data, such as the spatial location information of the rendered data. During SDP negotiation, the a=extmap attribute is used to describe the location of the RTP extended header and the corresponding information.

[0212] 2) For application data channels, add a field a = data_resource to the attributes of media stream channels to indicate that the application data channel can transmit at least one of media metadata, video / image data, and other business data.

[0213] When a = data_resource:1video, and the value of data_resource is 1, it means that the application's data channel transmits a type of data. video indicates that the type is video, and some data characteristics of the video, such as encoding format and frequency, can also be added to video.

[0214] When a = data_resource:1rendered_video, and the value of data_resource is 1, it indicates that the channel transmits a type of data. `rendered_video` indicates that the data type is rendered video, and some data characteristics of the video, such as encoding format and frequency, can be added after `video`. Alternatively, data_source can simply indicate that the data type being transmitted is video, and whether the data is rendered or not can be indicated directly during data transmission.

[0215] S602 and DC applications provide stored media data based on the service requests made when the UE session is established.

[0216] Among them, media data can be used to create digital human models.

[0217] Media data can also be provided by a storage server that stores media data on the network side, but the DC application needs to request the storage server to provide the corresponding data.

[0218] S603, the media processing network element processes media data provided by DC applications or other network elements.

[0219] The media processing network element performs at least one type of processing on media data: rendering.

[0220] S604. The media processing network element transmits the processed media data to the UE.

[0221] Media processing network elements can use RTP channels or application data channels to transmit rendered media data.

[0222] S605, UE obtains local media data.

[0223] Local media data includes video / audio data.

[0224] S606 and UE perform rendering processing on local media data.

[0225] Optionally, the UE can also perform secondary rendering, scene combination, and other media processing on the rendered media data provided by the DC application.

[0226] S607, the UE transmits the locally collected data to the DC application.

[0227] Locally collected data is typically used for network-side rendering or to provide media information to media processing network elements. Data used for network-side rendering is usually media metadata (such as location information); the media information provided to media processing network elements can be audio / video data, but it is usually different from the data provided to the UE by the media processing network elements. For example, the UE collects audio data and does not need the media processing network elements to perform media processing such as rendering and scene compositing.

[0228] Figure 7 This is a schematic diagram of a data rendering apparatus provided in one embodiment. This apparatus can be configured in a media processing network element, such as... Figure 7 As shown, the device includes: a negotiation module 701, a communication module 702, and a rendering module 703.

[0229] The negotiation module 701 is configured to establish a transmission channel between the media processing network element and the terminal device, and to assign a corresponding identifier to the transmission channel. The transmission channel includes at least a first real-time transport protocol (RTP) channel. The identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits data to be rendered.

[0230] The communication module 702 is configured to acquire the data to be rendered sent by the terminal device through the first RTP channel;

[0231] Rendering module 703 is configured to render the data to be rendered and obtain the rendered data.

[0232] The data rendering device provided in this embodiment is for implementing... Figure 2 The data rendering method of the embodiment shown is similar in principle and technical effect to the data rendering device provided in this embodiment, and will not be repeated here.

[0233] In one embodiment, when the identifier corresponding to the first RTP channel is a first identifier, the first identifier is used to indicate that the first RTP channel only transmits data to be rendered;

[0234] When the identifier corresponding to the first RTP channel is the second identifier, the second identifier is used to indicate that the first RTP channel transmits data to be rendered and non-rendered data.

[0235] In one embodiment, when the identifier corresponding to the first RTP channel is the second identifier, the communication module 702 is configured to receive RTP packets sent by the terminal device through the first RTP channel; and determine the data to be rendered from the data carried by the RTP packet according to the RTP extension header of the RTP packet.

[0236] In one embodiment, the transmission channel further includes at least one of a second RTP channel and a boot data channel;

[0237] The identifier corresponding to the second RTP channel is used to indicate that the second RTP channel only transmits non-rendered data;

[0238] The identifier corresponding to the boot data channel is used to indicate the media metadata transmitted by the boot data channel.

[0239] In one embodiment, the communication module 702 is further configured to acquire non-rendered data sent by the terminal device through the second RTP channel; and acquire media metadata sent by the terminal device through the boot data channel.

[0240] In one embodiment, the negotiation module 701 is further configured to assign a corresponding group identifier to the transmission channel, and there is a binding relationship between transmission channels with the same group identifier.

[0241] In one embodiment, the rendering module 703 is further configured to perform at least one of the following processes on data transmitted through a transmission channel having the same set of identifiers: scene compositing processing, data association processing.

[0242] In one embodiment, the communication module 702 is further configured to send media data, which includes at least rendered data.

[0243] Figure 8 This is a schematic diagram of another data rendering apparatus provided in one embodiment. This apparatus can be configured in a terminal device, such as... Figure 8 As shown, the device includes a negotiation module 801 and a communication module 802.

[0244] The negotiation module 801 is configured to establish a transmission channel between the terminal device and the media processing network element, and determine the identifier corresponding to the transmission channel. The transmission channel includes at least a first real-time transport protocol (RTP) channel, and the identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits data to be rendered.

[0245] The communication module 802 is configured to send the data to be rendered to the media processing network element through the first RTP channel.

[0246] The data rendering device provided in this embodiment is for implementing... Figure 3 The data rendering method of the embodiment shown is similar in principle and technical effect to the data rendering device provided in this embodiment, and will not be repeated here.

[0247] In one embodiment, when the identifier corresponding to the first RTP channel is a first identifier, the first identifier is used to indicate that the first RTP channel only transmits data to be rendered;

[0248] When the identifier corresponding to the first RTP channel is the second identifier, the second identifier is used to indicate that the first RTP channel transmits data to be rendered and non-rendered data.

[0249] In one embodiment, the transmission channel further includes at least one of a second RTP channel and a boot data channel;

[0250] The identifier corresponding to the second RTP channel is used to indicate that the second RTP channel only transmits non-rendered data;

[0251] The identifier corresponding to the boot data channel is used to indicate the media metadata transmitted by the boot data channel.

[0252] In one embodiment, the communication module 802 is further configured to send non-rendered data to the media processing network element via the second RTP channel and send media metadata to the media processing network element via the boot data channel.

[0253] In one embodiment, the negotiation module 801 is further configured to determine the group identifier corresponding to the transmission channel, and there is a binding relationship between transmission channels with the same group identifier.

[0254] In one embodiment, the communication module 802 is further configured to receive media data sent by the media processing network element, the media data including at least rendered data.

[0255] This application also provides a media processing network element, including: a processor, which is configured to implement the methods provided in any embodiment of this application when executing a computer program.

[0256] Figure 9 This is a schematic diagram of the structure of a media processing network element provided in one embodiment, such as... Figure 9 As shown, the media processing network element includes a processor 60, a memory 61, and a communication interface 62; the number of processors 60 in the media processing network element can be one or more. Figure 9 Taking a processor 60 as an example; the processor 60, memory 61, and communication interface 62 in the media processing network element can be connected via a bus or other means. Figure 9 Taking the bus connection as an example, a bus can refer to one or more of several bus architectures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration port, a processor, or a local bus that uses any of the various bus architectures.

[0257] The memory 61, as a computer-readable storage medium, can be configured to store software programs, computer-executable programs, and modules, such as the program instructions / modules corresponding to the methods in the embodiments of this application. The processor 60 executes at least one functional application and data processing of the media processing network element by running the software programs, instructions, and modules stored in the memory 61, thereby implementing the methods described above.

[0258] The memory 61 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created based on terminal usage. Furthermore, the memory 61 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 61 may include memory remotely located relative to the processor 60, which can be connected to a media processing network element via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, networks, mobile communication networks, and combinations thereof.

[0259] Communication interface 62 can be configured to receive and send data.

[0260] This application also provides a terminal device, including a processor, which is configured to implement the methods provided in any embodiment of this application when executing a computer program. Exemplarily, the following embodiment provides a schematic diagram of the structure of a UE (User Equipment) terminal device.

[0261] Figure 10 This is a schematic diagram of the structure of a UE provided in one embodiment. The UE can be implemented in various forms. The UE in this application can include, but is not limited to, mobile terminal devices such as mobile phones, smartphones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (PADs), portable media players (PMPs), navigation devices, in-vehicle terminal devices, in-vehicle display terminals, in-vehicle electronic rearview mirrors, etc., as well as fixed terminal devices such as digital television (TV), desktop computers, etc.

[0262] like Figure 10 As shown, UE 50 may include a wireless communication unit 51, an audio / video (A / V) input unit 52, a user input unit 53, a sensing unit 54, an output unit 55, a memory 56, an interface unit 57, a processor 58, and a power supply unit 59, etc. Figure 10 The UE shown includes a variety of components; however, it should be understood that it is not required to implement all of the components shown. More or fewer components may be implemented alternatively.

[0263] In this embodiment, the wireless communication unit 51 allows the UE 50 to communicate wirelessly with a base station or network. The A / V input unit 52 is configured to receive audio or video signals. The user input unit 53 can generate key input data to control various operations of the UE 50 based on user-input commands. The sensing unit 54 detects the current state of the UE 50, the position of the UE 50, the presence or absence of user touch input to the UE 50, the orientation of the UE 50, the acceleration or deceleration of the UE 50, and its direction, etc., and generates commands or signals for controlling the operation of the UE 50. The interface unit 57 serves as an interface through which at least one external device can connect to the UE 50. The output unit 55 is configured to provide output signals in a visual, audio, and / or tactile manner. The memory 56 can store software programs, etc., that perform processing and control operations executed by the processor 58, or can temporarily store data that has been output or will be output. The memory 56 can include at least one type of storage medium. Moreover, the UE 50 can cooperate with a network storage device that performs the storage function of the memory 56 via a network connection. Processor 58 typically controls the overall operation of UE 50. Power supply unit 59, under the control of processor 58, receives external or internal power and provides the appropriate power required to operate various components and assemblies.

[0264] The processor 58 executes at least one functional application and data processing by running a program stored in the memory 56, such as implementing the method provided in the embodiments of this application.

[0265] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the methods provided in any embodiment of this application.

[0266] The computer storage medium in this application embodiment can be any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. For example, a computer-readable storage medium can be, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. Computer-readable storage media include (a non-exhaustive list): electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), electrically erasable, programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this application, the computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device.

[0267] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, the data signals carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0268] Program code contained on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical fiber, radio frequency (RF), or any suitable combination thereof.

[0269] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination of programming languages, including object-oriented programming languages ​​(such as Java, Smalltalk, C++, Ruby, and Go) and conventional procedural programming languages ​​(such as the "C" language or similar programming languages). The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a Local Area Network, LAN, or Wide Area Network, WAN), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0270] This invention also provides a computer program product, including a computer program that, when executed by a processor, implements the method provided in any embodiment of this invention.

[0271] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0272] Those skilled in the art will understand that the term user terminal encompasses any suitable type of wireless user equipment, such as mobile phones, portable data processing devices, portable web browsers, or vehicle-mounted mobile stations.

[0273] Generally, the various embodiments of this application can be implemented in hardware or dedicated circuitry, software, logic, or any combination thereof. For example, some aspects can be implemented in hardware, while others can be implemented in firmware or software that can be executed by a controller, microprocessor, or other computing device, although this application is not limited thereto.

[0274] Embodiments of this application can be implemented by executing computer program instructions through the data processor of a mobile device, for example, in a processor entity, or through hardware, or through a combination of software and hardware. The computer program instructions can be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages.

[0275] Any block diagram of logical flow in the accompanying drawings of this application may represent program steps, or may represent interconnected logic circuits, modules, and functions, or may represent a combination of program steps and logic circuits, modules, and functions. The computer program may be stored in memory. The memory may be of any type suitable to the local technical environment and may be implemented using any suitable data storage technology, such as, but not limited to, read-only memory (ROM), random access memory (RAM), optical storage devices and systems (Digital Multifunction Discs, DVDs, or CDs), etc. Computer-readable media may include non-transitory storage media. The data processor may be of any type suitable to the local technical environment, such as, but not limited to, general-purpose computers, special-purpose computers, microprocessors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), and processors based on multi-core processor architectures. < / group-1> < / group-1>

Claims

1. A data rendering method, characterized in that, Applied to media processing network elements, the method includes: A transmission channel is established between the media processing network element and the terminal device, and a corresponding identifier is assigned to the transmission channel. The transmission channel includes at least a first Real-Time Transport Protocol (RTP) channel. The identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits data to be rendered. Obtain the data to be rendered sent by the terminal device through the first RTP channel; The data to be rendered is rendered to obtain the rendered data.

2. The data rendering method according to claim 1, characterized in that, When the identifier corresponding to the first RTP channel is the first identifier, the first identifier is used to indicate that the first RTP channel only transmits the data to be rendered; When the identifier corresponding to the first RTP channel is the second identifier, the second identifier is used to indicate that the first RTP channel transmits the data to be rendered and the non-rendered data.

3. The data rendering method according to claim 2, characterized in that, When the identifier corresponding to the first RTP channel is the second identifier, obtaining the data to be rendered sent by the terminal device through the first RTP channel includes: Receive RTP packets sent by the terminal device through the first RTP channel; The data to be rendered is determined from the data carried by the RTP packet based on the RTP extension header of the RTP packet.

4. The data rendering method according to claim 1, characterized in that, The transmission channel further includes at least one of a second RTP channel and a boot data channel; The identifier corresponding to the second RTP channel is used to indicate that the second RTP channel only transmits non-rendered data; The identifier corresponding to the boot data channel is used to indicate that the boot data channel transmits media metadata.

5. The data rendering method according to claim 4, characterized in that, It also includes at least one of the following: Obtain the non-rendered data sent by the terminal device through the second RTP channel; Obtain the media metadata sent by the terminal device through the boot data channel.

6. The data rendering method according to claim 1 or 4, characterized in that, When establishing a transmission channel between the media processing network element and the terminal device, the method further includes: A corresponding group identifier is assigned to the transmission channel, and transmission channels with the same group identifier are bound to each other.

7. The data rendering method according to claim 6, characterized in that, Also includes: Perform at least one of the following processes on data transmitted through a transmission channel with the same set of identifiers: scene compositing, data association.

8. The data rendering method according to claim 1, characterized in that, After obtaining the rendered data, it also includes: Send media data, which includes at least the rendered data.

9. A data rendering method, characterized in that, Applied to a terminal device, the method includes: A transmission channel is established between the terminal device and the media processing network element, and an identifier corresponding to the transmission channel is determined. The transmission channel includes at least a first Real-Time Transport Protocol (RTP) channel, and the identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits data to be rendered. The data to be rendered is sent to the media processing network element through the first RTP channel.

10. The data rendering method according to claim 9, characterized in that, When the identifier corresponding to the first RTP channel is the first identifier, the first identifier is used to indicate that the first RTP channel only transmits the data to be rendered; When the identifier corresponding to the first RTP channel is the second identifier, the second identifier is used to indicate that the first RTP channel transmits the data to be rendered and the non-rendered data.

11. The data rendering method according to claim 9, characterized in that, The transmission channel further includes at least one of a second RTP channel and a boot data channel; The identifier corresponding to the second RTP channel is used to indicate that the second RTP channel only transmits non-rendered data; The identifier corresponding to the boot data channel is used to indicate that the boot data channel transmits media metadata.

12. The data rendering method according to claim 11, characterized in that, It also includes at least one of the following: The non-rendered data is sent to the media processing network element through the second RTP channel; The media metadata is sent to the media processing network element through the guidance data channel.

13. The data rendering method according to claim 9 or 11, characterized in that, When establishing a transmission channel between the terminal device and the media processing network element, the method further includes: Determine the group identifier corresponding to the transmission channel, and there is a binding relationship between transmission channels with the same group identifier.

14. The data rendering method according to claim 9, characterized in that, After sending the data to be rendered to the media processing network element through the first RTP channel, the method further includes: The media data sent by the media processing network element is received, and the media data includes at least the rendered data.

15. A media processing network element, characterized in that, include: processor; The processor is used to implement the data rendering method as described in any one of claims 1-8 when executing a computer program.

16. A terminal device, characterized in that, include: processor; The processor is used to implement the data rendering method as described in any one of claims 9-14 when executing a computer program.

17. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the data rendering method as described in any one of claims 1-14.

Images