Data rendering method, media processing network element, terminal device and storage medium

Abstract

Disclosed in the present application are a data rendering method, a media processing network element, a terminal device, and a storage medium. The data rendering method is applied to a media processing network element. The method comprises: establishing a transmission channel between the media processing network element and a terminal device, and allocating a corresponding identifier to the transmission channel, the transmission channel at least comprising a first real-time transport protocol (RTP) channel, and the identifier corresponding to the first RTP channel being used for instructing the first RTP channel to transmit data to be rendered; acquiring said data sent by the terminal device by means of the first RTP channel; and rendering said data to obtain rendered data.

Description

Data rendering method, media processing network element, terminal device and storage medium TECHNICAL FIELD

[0001] The present application relates to the technical field of communication, for example, to a data rendering method, a media processing network element, a terminal device and a storage medium. BACKGROUND

[0002] With the deployment of the 5th-generation (5G) network and the deployment of immersive services based on the 5G network, emerging media services based on the Internet Protocol Multimedia Subsystem (IMS) have developed rapidly. Because the media services based on the IMS have the characteristics of large data volume, low latency, complex processing, and interactivity, from the perspective of operators, it is more desirable to complete data processing by the network to reduce the requirements on terminal devices and improve user stickiness to the network. However, for different media data types, different service types, and different data processing capabilities of terminal devices and networks, the 3rd Generation Partnership Project (3GPP) has not proposed a targeted solution, which affects the experience of media services. SUMMARY

[0003] Embodiments of the present application provide a data rendering method applied to a media processing network element, the method comprising: establishing a transmission channel between the media processing network element and a terminal device, and allocating a corresponding identifier to the transmission channel, the transmission channel comprising at least a first Real-time Transport Protocol (RTP) channel, the identifier corresponding to the first RTP channel being used to indicate that the first RTP channel transmits to-be-rendered data; obtaining the to-be-rendered data sent by the terminal device through the first RTP channel; and rendering the to-be-rendered data to obtain rendered data.

[0004] Embodiments of the present application provide a data rendering method applied to a terminal device, the method comprising: establishing a transmission channel between the terminal device and a media processing network element, and determining a corresponding identifier of the transmission channel, the transmission channel comprising at least a first Real-time Transport Protocol (RTP) channel, the identifier corresponding to the first RTP channel being used to indicate that the first RTP channel transmits to-be-rendered data; and sending the to-be-rendered data to the media processing network element through the first RTP channel.

[0005] Embodiments of the present application provide a media processing network element, comprising: a processor; the processor is used to implement the above-mentioned data rendering method when executing a computer program.

[0006] Embodiments of the present application provide a terminal device, comprising: a processor; the processor is used to implement the above-mentioned data rendering method when executing a computer program.

[0007] The embodiment of the present application further provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the data rendering method. BRIEF DESCRIPTION OF DRAWINGS

[0008] FIG. 1 is a networking schematic diagram of a wireless communication system according to an embodiment;

[0009] FIG. 2 is a flow schematic diagram of a data rendering method according to an embodiment;

[0010] FIG. 3 is a flow schematic diagram of another data rendering method according to an embodiment;

[0011] FIG. 4 is an interaction schematic diagram of data rendering according to example 1;

[0012] FIG. 5 is an interaction schematic diagram of data rendering according to example 2;

[0013] FIG. 6 is an interaction schematic diagram of data rendering based on an IMS network according to an embodiment;

[0014] FIG. 7 is a structural schematic diagram of a data rendering apparatus according to an embodiment;

[0015] FIG. 8 is a structural schematic diagram of another data rendering apparatus according to an embodiment;

[0016] FIG. 9 is a structural schematic diagram of a media processing network element according to an embodiment;

[0017] FIG. 10 is a structural schematic diagram of a UE according to an embodiment. DETAILED DESCRIPTION

[0018] The IMS-based media service has the characteristics of large data volume, low latency, complex processing, and interaction. For example, the call service is no longer limited to the traditional audio and video call, and can be superimposed with the augmented reality (AR) feature, the operation on virtual objects, etc., to enrich the call service and improve the call experience.

[0019] Due to the above characteristics of the IMS-based media service, the transmission, processing and presentation of media data need to be enhanced. From the perspective of the operator, it is more desirable to complete the processing of media data by the network to reduce the requirements on the terminal device and improve the user's stickiness to the network. The IMS-based media service has high requirements on the synthesis rendering of graphics and images, which has higher requirements on the terminal device (such as Central Processing Unit (CPU), Graphics Processing Unit (GPU)) and network capability.

[0020] 3GPP has proposed a rendering mode based on IMS user entity and network assistance. However, 3GPP does not propose a targeted solution for different media data types, different service types, and different terminal device capabilities and network capabilities, which cannot meet the requirements of the IMS-based media service and affects the experience of the media service.

[0021] The data rendering method provided in the present application can be applied to various wireless communication systems, such as IMS systems based on 5G, 5G New Radio (NR) and new communication technologies in future communication development, such as 6th-generation (6G) mobile communication technology. FIG. 1 is a networking schematic diagram of a wireless communication system according to an embodiment. As shown in FIG. 1, the wireless communication system includes a terminal device 110 and a media processing network element 120. The wireless communication system can support a data channel (Data Channel, DC) application, such as an AR call service.

[0022] The terminal device 110 can be a device with wireless transceiving function, which can be deployed on land (such as indoor or outdoor, handheld, wearable or vehicle-mounted, etc.), on water (such as a ship, etc.), or in the air (such as an airplane, a balloon, a satellite, etc.). Some examples of the terminal device 110 include a wireless terminal, a user equipment (UE), a mobile phone, a mobile station, a tablet computer, a notebook computer, an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a personal digital assistant (PDA), etc., which can be connected to a network, or a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical treatment, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc., or an Internet of Things (IoT) node in the Internet of Things, or a vehicle communication device in the Internet of Vehicles, or an entertainment, game device or system, or a global positioning system device, etc. Embodiments of the present application do not limit the specific technology and specific device form of the terminal device 110, and in addition, the terminal device 110 can be referred to simply as a terminal.

[0023] The media processing network element 120 is a network side device through which the terminal device 110 accesses the wireless communication system in a wireless manner. The media processing network element 120 can include the following structure.

[0024] A DC application server (DC Application Server) 121 is responsible for DC-based traffic processing and service control, including DC session media control and media capability negotiation with the UE and other media processing network elements 120. The DC application server 121 can be deployed in an IMS network; or can be deployed in a third-party non-IMS network to interact with the UE through network exposure capabilities and the like.

[0025] A data channel signaling function (Data Channel Signalling Function, DCSF) 122 provides data channel control capabilities.

[0026] Media Function / Media Resource Function (MF / MRF) 123: supports media processing of IMS-based media services, receives and stores media processing logic from an IP Multimedia System (IMS) Application Server (IMS AS) 124, and provides AR media capabilities for media data received from a UE.

[0027] IMS AS 124: supports functions and negotiation of a control plane. Interacts with the DCSF 122 for event notification, receives data channel control instructions from the DCSF 122, and interacts with the MF / MRF 123 accordingly for data channel media resource management. Meanwhile, receives media control instructions from the DCSF 122, and interacts with the terminal device 110 accordingly to connect an audio / video media terminal of the terminal device 110 to the MF / MRF 123.

[0028] Call Session Control Function (CSCF): responsible for session control and routing, including functions such as registration and authentication related to user services, session control, routing management, network management, and billing control. The CSCF mainly includes an interrogating CSCF (I-CSCF) 125, a serving CSCF (S-CSCF) 126, and a proxy CSCF (P-CSCF) 127.

[0029] I-CSCF 125: a gateway node similar to the media processing network element 120, used for allocating the S-CSCF 126, routing queries, and IMS inter-domain topology hiding.

[0030] S-CSCF 126: in a core control position in the IMS core network, responsible for registration and authentication of the terminal device 110, conference control, and user data management, etc.

[0031] P-CSCF 127: the first connection point in the media processing network element 120 with the terminal device 110, providing proxy functions, and can also provide user agent functions.

[0032] IP Multimedia System (IMS) Access Gateway (IMS-AGW) 128.

[0033] As can be seen from FIG. 1, there can also be a connection relationship between the media processing network elements 120, i.e., the media processing network element 120 on the right side of FIG. 1 can be a remote media processing network element, and there is a connection relationship between the media processing network element 120 connected with the terminal device 110 and the remote media processing network element.

[0034] In the embodiments of the present application, a data rendering method, a media processing network element, a terminal device and a storage medium capable of running in the wireless communication system are provided. The media processing capability of the terminal device and the media processing capability of the IMS network are negotiated, so that the terminal device and the media processing network element provide flexible rendering processing, thereby improving the processing efficiency of the media data and optimizing the experience of the media service.

[0035] Next, the data rendering method, the media processing network element, the terminal device and the technical effects thereof are described.

[0036] FIG. 2 is a flowchart of a data rendering method according to an embodiment. As shown in FIG. 2, the method provided in the embodiment is applicable to a media processing network element. For example, the method can be applied to the media processing network element 120 in FIG. 1 or the MF / MRF 123 in the media processing network element 120 in FIG. 1. The method includes S210-S230.

[0037] S210, a transmission channel is established between the media processing network element and the terminal device, and a corresponding identifier is allocated to the transmission channel. The transmission channel includes at least a first real-time transport protocol (RTP) channel, and the identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits data to be rendered.

[0038] An IMS session usually involves at least two terminal devices. In the embodiments, the terminal device can be a device initiating an IMS session or a device not initiating an IMS session. The terminal devices participating in the IMS session can be in the same network or in different networks.

[0039] When the terminal device wants to have an IMS session with one or more other terminal devices, in order to enable the media processing network element to participate in the rendering processing, the terminal device needs to send data to be rendered to the media processing network element. Therefore, first, a transmission channel needs to be established between the media processing network element and the terminal device, and a corresponding identifier needs to be allocated to the transmission channel to indicate the type of data transmitted by each transmission channel.

[0040] In an embodiment, the transmission channel includes at least a first real-time transport protocol (RTP) channel, and the identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits data to be rendered.

[0041] The identifier corresponding to the first RTP channel can be a first identifier or a second identifier.

[0042] When the identifier corresponding to the first RTP channel is the first identifier, the first identifier is used to indicate that the first RTP channel only transmits data to be rendered. That is, all the data transmitted by the first RTP channel needs to be rendered by the media processing network element.

[0043] When the identity corresponding to the first RTP channel is the second identity, the second identity is used to indicate that the first RTP channel transmits the data to be rendered and non-rendered data, where the non-rendered data refers to data that does not need to be rendered by the media processing network element. That is, the first RTP channel can transmit data that does not need to be rendered by the media processing network element in addition to data that needs to be rendered by the media processing network element. In addition, the non-rendered data can or can not be rendered by the terminal device.

[0044] In an embodiment, the transmission channel can further include at least one of a second RTP channel and a data channel. The data channel can include a bootstrap data channel (bootstrap DC) and an application data channel (application DC).

[0045] The identity corresponding to the second RTP channel is used to indicate that the second RTP channel only transmits non-rendered data. That is, all data transmitted by the second RTP channel does not need to be rendered by the media processing network element.

[0046] The identity corresponding to the application data channel is used to indicate that the bootstrap data channel transmits media metadata. The media metadata can be data that is associated with the data to be rendered and / or non-rendered data, such as scene data or description data (used to describe the data to be rendered and / or non-rendered data). In some scenarios, the non-rendered data can also include media metadata.

[0047] In an embodiment, when the number of transmission channels is multiple, the media processing network element can further assign a corresponding group identity to the transmission channels when establishing the transmission channels between the media processing network element and the terminal device, and the transmission channels having the same group identity have a binding relationship.

[0048] For example, assume that five transmission channels are established between the media processing network element and the terminal device, which are two first RTP channels (denoted as first RTP channel 1 and first RTP channel 2, respectively), one second RTP channel, and two application data channels (denoted as application data channel 1 and application data channel 2, respectively). The group identities corresponding to the first RTP channel 1, the second RTP channel, and the application data channel 2 are all identity 1, and the group identities corresponding to the first RTP channel 2 and the application data channel 1 are both identity 2. That is, the first RTP channel 1, the second RTP channel, and the application data channel 2 belong to the same group and have a binding relationship, and the first RTP channel 2 and the application data channel 1 belong to the same group and have a binding relationship.

[0049] In an embodiment, the media processing network element assigns the corresponding identity and group identity to the transmission channel in the process of session description protocol (SDP) negotiation between the media processing network element and the terminal device. The SDP negotiation process can take into account the media processing capability of the terminal device and the media processing capability of the IMS network.

[0050] For example, when the media processing capability of the terminal device is strong, the terminal device can place less data rendering work on the IMS network side, such as establishing a smaller number of first RTP channels; when the media processing capability of the terminal device is weak, the terminal device can place a large amount or all data rendering work on the IMS network side, such as establishing a larger number of first RTP channels. The media processing capability of the IMS network is the same.

[0051] S220, obtaining the to-be-rendered data sent by the terminal device through the first RTP channel.

[0052] After the establishment of the transmission channel between the media processing network element and the terminal device is completed, the SDP negotiation between the media processing network element and the terminal device is completed, and data transmission can be performed. The media processing network element can obtain the to-be-rendered data sent by the terminal device through the first RTP channel.

[0053] Since the first RTP channel can be used to transmit only to-be-rendered data or to transmit to-be-rendered data and non-rendered data, the media processing network element first needs to determine the identity corresponding to the first RTP channel. When the identity corresponding to the first RTP channel is the first identity, the media processing network element can directly transmit all data transmitted by the first RTP channel as to-be-rendered data. When the identity corresponding to the first RTP channel is the second identity, the media processing network element needs to further distinguish all data transmitted by the first RTP channel.

[0054] When the identity corresponding to the first RTP channel is the second identity, the method for the media processing network element to further distinguish all data transmitted by the first RTP channel can be: receiving an RTP packet sent by the terminal device through the first RTP channel; determining to-be-rendered data from data carried by the RTP packet according to an RTP extension header of the RTP packet.

[0055] For example, the media processing network element and the terminal device can negotiate that when the identity corresponding to the first RTP channel is the second identity, the fourth to tenth RTP extension headers of the RTP packet carry to-be-rendered data. In this way, the to-be-rendered data and the non-rendered data can be distinguished.

[0056] In an embodiment, if the transmission channel established in S210 includes the second RTP channel, the media processing network element can further acquire non-rendering data sent by the terminal device through the second RTP channel; and if the transmission channel established in S210 includes the guide data channel, the media processing network element can further acquire media metadata sent by the terminal device through the guide data channel.

[0057] S230, rendering the to-be-rendered data to obtain rendered data.

[0058] In an embodiment, after obtaining the rendered data, the media processing network element can further send media data to the terminal device or other terminal devices, wherein the media data at least includes the rendered data.

[0059] In an embodiment, if the transmission channel established in S210 includes the second RTP channel and the media processing network element acquires the non-rendering data sent by the terminal device through the second RTP channel, the media data can further include the non-rendering data sent by the terminal device through the second RTP channel. If the transmission channel established in S210 includes the guide data channel and the media processing network element acquires the media metadata sent by the terminal device through the guide data channel, the media data can further include the media metadata sent by the terminal device through the guide data channel.

[0060] In an embodiment, the media data can further include data sent by other terminal devices.

[0061] In an embodiment, if the media processing network element assigns corresponding group identifiers to the transmission channels in S210, the media processing network element can further perform at least one of the following processing on data transmitted through transmission channels with the same group identifier: scene synthesis processing, data correlation processing.

[0062] The scene synthesis processing and the data correlation processing can be performed before rendering the to-be-rendered data or after rendering the to-be-rendered data, and the present application does not make a specific limitation in this regard.

[0063] In this way, the media processing capability of the terminal device and the media processing capability of the IMS network are negotiated, so that the terminal device and the media processing network element provide flexible rendering processing, thereby improving the processing efficiency of media data and optimizing the experience of media services.

[0064] FIG. 3 is a flow diagram of another data rendering method provided by an embodiment. As shown in FIG. 3, the method provided by the present embodiment is applicable to a terminal device. The method includes S310-S320.

[0065] S310, a transmission channel is established between the terminal device and the media processing network element, and an identifier corresponding to the transmission channel is determined, the transmission channel at least including a first real-time transport protocol (RTP) channel, the identifier corresponding to the first RTP channel being used to indicate that the first RTP channel transmits the data to be rendered.

[0066] The IMS session usually has at least two terminal devices participating, in this embodiment, the terminal device can be a device initiating the IMS session, or a device not initiating the IMS session. The terminal devices participating in the IMS session can be in the same network, or in different networks.

[0067] When the terminal device is to have an IMS session with one or more other terminal devices, in order to let the media processing network element participate in the rendering process, the terminal device needs to send the data to be rendered to the media processing network element. Therefore, first, a transmission channel needs to be established between the media processing network element and the terminal device, and an identifier corresponding to the transmission channel needs to be determined, to determine the type of data to be transmitted in each transmission channel.

[0068] In an embodiment, the transmission channel at least includes a first RTP channel, and the identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits the data to be rendered.

[0069] The identifier corresponding to the first RTP channel can be a first identifier or a second identifier.

[0070] When the identifier corresponding to the first RTP channel is the first identifier, the first identifier is used to indicate that the first RTP channel only transmits the data to be rendered. That is, all the data transmitted by the first RTP channel needs to be rendered by the media processing network element.

[0071] When the identifier corresponding to the first RTP channel is the second identifier, the second identifier is used to indicate that the first RTP channel transmits the data to be rendered and non-rendering data, the non-rendering data being data that does not need to be rendered by the media processing network element. That is, in addition to transmitting data that needs to be rendered by the media processing network element, the first RTP channel can also transmit data that does not need to be rendered by the media processing network element. In addition, the non-rendering data can have been rendered by the terminal device, or can not have been rendered by the terminal device.

[0072] In an embodiment, the transmission channel can also include at least one of a second RTP channel and a data channel. The data channel can include a guide data channel and an application data channel.

[0073] The identifier corresponding to the second RTP channel is used to indicate that the second RTP channel only transmits non-rendering data. That is, all the data transmitted by the second RTP channel does not need to be rendered by the media processing network element.

[0074] The identifier corresponding to the application data channel is used to indicate that the guide data channel transmits media metadata. The media metadata can be data accompanying the to-be-rendered data and / or non-rendered data, such as scene data or description data (used to describe the to-be-rendered data and / or non-rendered data). In some scenarios, the non-rendered data can also include media metadata.

[0075] In an embodiment, when the number of transmission channels is multiple, the terminal device can also determine the group identifier corresponding to the transmission channel when establishing the transmission channel with the media processing network element, and the transmission channels having the same group identifier have a binding relationship.

[0076] For example, it is assumed that five transmission channels are established between the media processing network element and the terminal device, which are two first RTP channels (denoted as first RTP channel 1 and first RTP channel 2 respectively), one second RTP channel, and two application data channels (denoted as application data channel 1 and application data channel 2 respectively). The group identifiers corresponding to the first RTP channel 1, the second RTP channel, and the application data channel 2 are all identifier 1, and the group identifiers corresponding to the first RTP channel 2 and the application data channel 1 are all identifier 2. That is, the first RTP channel 1, the second RTP channel, and the application data channel 2 belong to the same group and have a binding relationship; the first RTP channel 2 and the application data channel 1 belong to the same group and have a binding relationship.

[0077] In an embodiment, the process of determining the identifier and the group identifier corresponding to the transmission channel by the terminal device can be implemented in the process of SDP negotiation between the media processing network element and the terminal device. The negotiation process of SDP can consider the media processing capability of the terminal device and the media processing capability of the IMS network.

[0078] For example, when the media processing capability of the terminal device is strong, the terminal device can place a small amount of data rendering work on the IMS network side, such as a small number of first RTP channels established; when the media processing capability of the terminal device is weak, the terminal device can place a large amount or all data rendering work on the IMS network side, such as a large number of first RTP channels established. The media processing capability of the IMS network is the same.

[0079] S320, sending to-be-rendered data to the media processing network element through the first RTP channel.

[0080] After the establishment of the transmission channel between the media processing network element and the terminal device is completed, the SDP negotiation between the media processing network element and the terminal device is completed at this time, and data transmission can be performed. The terminal device sends to-be-rendered data to the media processing network element through the first RTP channel.

[0081] The first RTP channel can be used to transmit the data to be rendered or the data to be rendered and non-rendered data. When the first RTP channel corresponds to the second identifier, the terminal device can further send non-rendered data to the media processing network element through the first RTP channel.

[0082] In an embodiment, if the transmission channel established in S310 includes the second RTP channel, the terminal device can further send non-rendered data to the media processing network element through the second RTP channel; if the transmission channel established in S310 includes the bootstrap data channel, the terminal device can further send media metadata to the media processing network element through the bootstrap data channel.

[0083] In an embodiment, after sending the data to be rendered to the media processing network element through the first RTP channel, the terminal device can further receive media data sent by the media processing network element, wherein the media data at least includes rendered data.

[0084] In an embodiment, the media data can further include data sent by other terminal devices.

[0085] In this way, the media processing capability of the terminal device and the media processing capability of the IMS network are negotiated, so that the terminal device and the media processing network element provide flexible rendering processing, thereby improving the processing efficiency of the media data and optimizing the experience of the media service.

[0086] The following provides some examples for describing the data rendering method provided in the present application. In the following examples, the terminal device is taken as UE-A for description.

[0087] Example 1: The capability of UE-A is strong, and UE-A can render the locally acquired data and send the rendered data to the media processing network element, in which case the media processing network element does not need to participate in the rendering processing; UE-A can also render part of the locally acquired data, and let the media processing network element render the remaining data. Further, if the DC application involves multiple UEs, the media processing network element can perform other media processing (such as spatial calculation, scene combination, etc.) on the data of the multiple UEs (including rendered data and non-rendered data) to generate a scene containing multi-user data.

[0088] FIG. 4 is an interaction diagram of data rendering provided in example 1. As shown in FIG. 4, S401-S406 are included.

[0089] S401: UE-A performs an IMS session with other UEs, establishes a transmission channel, and the transmission channel has a corresponding identifier.

[0090] In example 1, UE-A can autonomously select whether to involve the media processing network element in rendering due to its stronger capability, but can need scene composition processing (especially in the case of multi-party participation service). In the case of multi-party participation service, some UEs have weaker capability, and these UEs can also request the media processing network element to perform media processing including rendering, scene composition, etc. UE-A and other UEs perform IMS session between UE-A and other UEs, and between UE-A and the media processing network element, and perform SDP offer and answer procedures through SDP session description protocol to initialize (including media capability negotiation) and re-negotiate media stream, and establish transmission channel.

[0091] The IMS session can be initiated by UE-A, and UE-A and other UEs can be in the same network or in different networks. Establishing the transmission channel means establishing the transmission channel between UE-A and the media processing network element, and establishing the transmission channel between other UEs and the media processing network element. The transmission channel includes an RTP channel, a bootstrap data channel, and an application data channel. The RTP channel is used to transmit video / image data. The bootstrap data channel is mainly used for downloading the DC application, and can also transmit media metadata (such as scene description) or other data. The application data channel is mainly used for transmitting media metadata, and can also transmit video / image data, negotiation data of the service, etc.

[0092] If UE-A or other UEs (hereinafter referred to as UE) does not preinstall the DC application, the DC application needs to be acquired through the bootstrap data channel; if the UE preinstalls the DC application, the bootstrap data channel can not be needed. The DC application is usually a Hyper Text Markup Language (HTML) application.

[0093] When the DC application is acquired through the bootstrap data channel, the UE can also acquire preinstalled / prestored related data related to the DC application, such as scene description data.

[0094] In an embodiment, the data in the DC application can be any type of data, such as a scene description file, a digital human image stored in an IMS network or a third-party server, etc. The IMS network can transmit these data to the UE through the bootstrap data channel or the application data channel after establishing the transmission channel (such as the RTP channel and the DC channel).

[0095] In an embodiment, taking the establishment of a transmission channel between UE-A and a media processing network element as an example, if the established RTP channel is a channel between UE-A and another UE, i.e., not anchored to the MF / MRF of the IMS network, the UE-A can directly interact with the other UE through the IMS-AGW, and then re-negotiate the RTP channel to be anchored to the MF / MRF in the IMS system, so that the MF / MRF can perform media processing on the data (such as audio / video data) of the RTP channel in the future.

[0096] The application data channel is usually anchored to the MF / MRF when established, and therefore, when the application data channel is established, the DC application server can directly distribute data related to the DC application through the MF / MRF using the application data channel. The MF / MRF can process the data transmitted by the UE through the application data channel. If the application data channel is not anchored to the MF / MRF when established (for example, the established application data channel is between UE-A and another UE), it can be anchored to the MF / MRF through SDP re-negotiation.

[0097] In an embodiment, the transmission channel has a corresponding identifier. The identifier corresponding to the transmission channel can be negotiated through the SDP description rule.

[0098] For example, in the media level description, an "a=" line is added under the "m=" line (indicating a media stream channel) to describe the characteristics of the data transmitted by the media stream channel.

[0099] 1) For the RTP channel, an a=data_processing_mode field is added in the attribute of the media stream channel to distinguish whether the RTP channel is the first RTP channel or the second RTP channel.

[0100] When a=data_processing_mode:1 rendered, data_processing_mode is 1, indicating that all data transmitted by the RTP channel does not need to be rendered by the media processing network element. That is, the RTP channel is the second RTP channel. rendered indicates that all data transmitted by the RTP channel has been rendered by the terminal device.

[0101] When a=data_processing_mode:1 unrendered, data_processing_mode is 1, indicating that all data transmitted by the RTP channel does not need to be rendered by the media processing network element. That is, the RTP channel is the second RTP channel. unrendered indicates that all data transmitted by the RTP channel has not been rendered by the terminal device.

[0102] When a = data_processing_mode: 1 rendered / unrendered, data_processing_mode is 1 means that all data transmitted by the RTP channel does not need to be rendered by the media processing network element. That is, the RTP channel is a second RTP channel. Rendered / unrendered means that some of the data transmitted by the RTP channel has been rendered by the terminal device and some has not been rendered by the terminal device.

[0103] When a = data_processing_mode: 2 render, data_processing_mode is 2 means that the data transmitted by the RTP channel can or can not need to be rendered by the media processing network element. Render means that all data transmitted by the RTP channel needs to be rendered by the media processing network element. That is, in this case, the RTP channel is a first RTP channel.

[0104] When a = data_processing_mode: 2 rendered, data_processing_mode is 2 means that the data transmitted by the RTP channel can or can not need to be rendered by the media processing network element. Rendered means that all data transmitted by the RTP channel does not need to be rendered by the media processing network element and has been rendered by the terminal device. That is, in this case, the RTP channel is a second RTP channel.

[0105] When a = data_processing_mode: 2 unrendered, data_processing_mode is 2 means that the data transmitted by the RTP channel can or can not need to be rendered by the media processing network element. Unrendered means that all data transmitted by the RTP channel has not been rendered by the terminal device but does not need to be rendered by the media processing network element. That is, in this case, the RTP channel is a second RTP channel.

[0106] When a = data_processing_mode: 2 render / rendered, data_processing_mode is 2 means that the data transmitted by the RTP channel can or can not need to be rendered by the media processing network element. Render / rendered means that some of the data transmitted by the RTP channel needs to be rendered by the media processing network element and some does not need to be rendered by the media processing network element and has been rendered by the terminal device. That is, in this case, the RTP channel is a first RTP channel.

[0107] When a = data_processing_mode: 2 render / unrendered, data_processing_mode is 2, which means that the data transmitted by the RTP channel can need media processing network element to render or can not need media processing network element to render. render / unrendered means that the data transmitted by the RTP channel is some of which needs media processing network element to render and some of which is not rendered by terminal device but does not need media processing network element to render. That is, in this case, the RTP channel is the first RTP channel.

[0108] When a = data_processing_mode: 2 rendered / unrendered, data_processing_mode is 2, which means that the data transmitted by the RTP channel can need media processing network element to render or can not need media processing network element to render. rendered / unrendered means that the data transmitted by the RTP channel is some of which does not need media processing network element to render and has been rendered by terminal device and some of which is not rendered by terminal device but does not need media processing network element to render. That is, in this case, the RTP channel is the second RTP channel.

[0109] When a = data_processing_mode: 2 render / rendered / unrendered, data_processing_mode is 2, which means that the data transmitted by the RTP channel can need media processing network element to render or can not need media processing network element to render. render / rendered / unrendered means that the data transmitted by the RTP channel is some of which needs media processing network element to render, some of which does not need media processing network element to render and has been rendered by terminal device, and some of which is not rendered by terminal device but does not need media processing network element to render. That is, in this case, the RTP channel is the first RTP channel.

[0110] If the a = data_processing_mode field is not used, the media processing network element can not process the received data by default.

[0111] When the RTP channel is the first RTP channel and the data transmitted by the RTP channel is some of which needs media processing network element to render and some of which does not need media processing network element to render, the data can be distinguished by the RTP header extension method provided by the RTP protocol. The extended RTP header can transmit media metadata matched with the rendered data, such as spatial position information of the rendered data. The position of the RTP extension header and the corresponding information are described by the a = extmap attribute in the SDP negotiation.

[0112] For example, a = extmap:4 urn:3gpp:pose, means that the position information (including coordinate system information and specific coordinate information) is carried from the 4th RTP extension header, and the position information is carried from the 4th extension header to the 10th extension header. The number of required extension headers needs to be determined according to the coordinate system.

[0113] 2) For the guide data channel, if the guide data channel is used to transmit media metadata, an a = media_metadata field is added in the attribute of the media stream channel to indicate that the guide data channel can transmit one or more types of media metadata.

[0114] When the a = media_metadata:1 SD and the metadata value is 1, it means that the guide data channel transmits one type of media metadata, and SD indicates that the type is scene description.

[0115] 3) For the application data channel, an a = data_resource field is added in the attribute of the media stream channel to indicate that the application data channel can transmit at least one of media metadata, video / image data, and other service data.

[0116] When the a = data_resource:1 video and the data_resource value is 1, it means that the application data channel transmits one type of data, and video indicates that the type is video. In addition, some data characteristics of the video, such as encoding format, frequency, etc., can be added to the video.

[0117] The a = media_metadata and a = media fields are added in the attribute of the media stream channel to indicate the data type of the data stream in the channel.

[0118] In an embodiment, if the RTP channel and the application data channel are used to transmit data at the same time, and the data has media processing relevance, such as rendered data and position information during rendering, unrendered data and position information during collection of the data, etc., the corresponding group identifier can be assigned to the transmission channel during SDP negotiation when the data needs to be associated and processed. Transmission channels with the same group identifier have a binding relationship.

[0119] For example, in the session level description, the current multiple media streams are further described by adding an "a =" line. For example, an a = media_processing_group <group-1> / ... / <group-n>The field is used to represent the association of multiple media streams (such as RTP channels, data channels).

[0120] When a = media_processing_group: composition 1 2 3 / 4 5 6, composition represents the media processing required for the scene composition by the media processing network element, "1 2 3" and "4 5 6" respectively represent two groups, and the media streams in the two groups must be associated with media processing, wherein each number is the same as the value of a = mid in each media stream (media level), and the specific media stream can be identified by the value.

[0121] If the data received by the UE is already rendered data, it can also be further rendered according to the requirements of the presentation. It is determined according to the requirements of the UE itself, and there is no need to negotiate.

[0122] S402, the UE-A acquires local data.

[0123] The UE-A acquires local data through one or more local devices (such as a camera, a sensor, etc., which can be integrated), such as images of AR (augmented reality) objects (objects, people, etc.), posture information of a user, traditional audio and video data, etc.

[0124] S403, the UE-A processes the local data.

[0125] The UE-A processes the local data according to its processing capability, network condition, media service type, etc., such as media rendering processing.

[0126] S404a, the UE-A sends the rendered data to the media processing network element.

[0127] S404b, the other UE sends the locally collected data to the media processing network element.

[0128] The data collected by the other UE locally can be data that has been processed (rendered) locally, or data that has not been processed (rendered) locally.

[0129] The UE-A / other UE can upload the data that has been rendered (may not include scene combination) or does not need to be rendered to the media processing network element through an RTP channel, and the media processing network element needs to know that the data has been processed locally by the UE or does not need to be processed by the media processing network element. Optionally, the data uploaded by the UE-A / other UE can also be uploaded to the media processing network element through an application data channel.

[0130] S405, the media processing network element performs secondary media processing on the data uploaded by two or more UEs according to the type of DC application or negotiation.

[0131] The media processing network element performs secondary media processing on the data uploaded by the two or more UEs according to the type of the DC application or negotiation (not limited to the SDP negotiation method, but also can be negotiated through a data channel). For example, scene synthesis (spatial calculation according to the spatial information of each media object to generate a scene graph containing multiple media objects).

[0132] S405 is optional. For example, the RTP channel is not anchored to the MF / MRF, and the UE obtains the scene description through the guide data channel or the application data channel. The receiving UE can perform scene synthesis according to the data transmitted by the opposite UE to generate a complete scene for presentation. Other unknown situations can also occur, and the media processing network element does not need to perform secondary media processing, but is directly processed by the sending UE / receiving UE.

[0133] If the data uploaded by part of the UEs (non-session initiation UEs) to the media processing network element is data that has not been processed by the UE locally and needs to be processed by the media processing network element, the media processing network element should first process the data uploaded by the part of the UEs, and then perform scene synthesis and other media processing on the data of all UEs.

[0134] The media processing network element can perform differential processing according to the state information (such as the pose) of each UE and the rendering method requested by the UE, such as partial rendering according to the view window of the UE during rendering.

[0135] S406, the media processing network element transmits the secondary media processing data to UE-A / other UEs.

[0136] If the above S405 is not executed, that is, the media processing network element does not need to perform scene combination and other non-rendering media processing, the media processing network element can directly transmit the data from the sending UE to the receiving UE, such as from other UEs to UE-A and from UE-A to other UEs.

[0137] The media processing network element can transmit the secondary media processing data in whole or in part according to the state of the UE. For example, when the UE state information (such as the position information) is transmitted to the media processing network element through the RTP channel together with the audio and video data, or is separately transmitted to the media processing network element through the application data channel, the media processing network element can transmit the rendered or unrendered data to the UE based on the UE state, network conditions, and the like.

[0138] Example 2: UE-A has weak capability, UE-A can request media processing network element to assist in rendering processing. For example, before sending data, UE-A first performs rendering and other processing, and then sends it to the media processing network element for further rendering processing, and finally sends it to other UEs and / or returns it to UE-A by the media processing network element. This way can reduce the amount of data transmission, reduce the data cache loading time of the server, etc. For another example, UE-A directly sends the data that needs to be rendered to the media processing network element for rendering processing, and the media processing network element sends it to other UEs and / or returns it to UE-A after completing rendering. This way, UE can perform parallel rendering, or wait for the network side to complete rendering before performing terminal side rendering. When parallel rendering, the time delay caused by rendering can be reduced. Of course, after rendering is completed, the media processing network element can further perform scene synthesis and other media processing according to requirements.

[0139] FIG. 5 is an interaction diagram of data rendering provided by example 2. As shown in FIG. 5, it includes S501-S507.

[0140] S501, UE-A performs IMS session with other UEs, establishes a transmission channel, and the transmission channel has a corresponding identifier.

[0141] In example 2, due to the weak capability of UE-A or other reasons (such as user subjective desire for network assistance in rendering, power state, signal state, computing power state, internal storage state, etc.), media processing network element is needed to assist in rendering media processing. UE-A and other UEs perform IMS session, and through SDP session description protocol, SDP offer and answer process is performed to initialize (including media capability negotiation) and re-negotiate media stream, and establish transmission channel.

[0142] The IMS session can be initiated by UE-A, and UE-A and other UEs can be in the same network or in different networks. Establishing a transmission channel means establishing a transmission channel between UE-A and the media processing network element, and establishing a transmission channel between other UEs and the media processing network element. The transmission channel includes an RTP channel, a guide data channel, and an application data channel. The RTP channel is used to transmit video / image data. The guide data channel is mainly used for downloading of DC application, and can also transmit media metadata (such as scene description) or other data. The application data channel is mainly used for transmitting media metadata, and can also transmit video / image data and service negotiation data, etc.

[0143] If the UE-A or other UE (hereinafter, the UE-A or other UE is referred to as UE) does not preinstall the DC application, the DC application needs to be acquired through the bootstrap data channel; if the UE preinstalls the DC application, the bootstrap data channel can not be needed to acquire. The DC application is usually an HTML application.

[0144] In an embodiment, the transmission channel has a corresponding identity. The corresponding identity of the transmission channel can be negotiated through the SDP description rule.

[0145] For example, in the media level description, an "a=" line is added under the "m=" line (representing a media stream channel) to describe the characteristics of the media stream channel transmitting data.

[0146] 1) For the RTP channel, an a=data_processing_mode field is added in the attribute of the media stream channel to distinguish whether the RTP channel is the first RTP channel or the second RTP channel.

[0147] When a=data_processing_mode:1 rendered, data_processing_mode is 1, indicating that all data transmitted by the RTP channel does not need to be rendered by the media processing network element. That is, the RTP channel is the second RTP channel. Rendered indicates that all data transmitted by the RTP channel has been rendered by the terminal device.

[0148] When a=data_processing_mode:1 unrendered, data_processing_mode is 1, indicating that all data transmitted by the RTP channel does not need to be rendered by the media processing network element. That is, the RTP channel is the second RTP channel. Unrendered indicates that all data transmitted by the RTP channel has not been rendered by the terminal device.

[0149] When a=data_processing_mode:1 rendered / unrendered, data_processing_mode is 1, indicating that all data transmitted by the RTP channel does not need to be rendered by the media processing network element. That is, the RTP channel is the second RTP channel. Rendered / unrendered indicates that some of the data transmitted by the RTP channel has been rendered by the terminal device, and some has not been rendered by the terminal device.

[0150] When a = data_processing_mode: 2 render, data_processing_mode is 2 indicates that the data transmitted by the RTP channel can need media processing network element to render or can not need media processing network element to render. The render indicates that all the data transmitted by the RTP channel needs media processing network element to render. That is, in this case, the RTP channel is the first RTP channel.

[0151] When a = data_processing_mode: 2 rendered, data_processing_mode is 2 indicates that the data transmitted by the RTP channel can need media processing network element to render or can not need media processing network element to render. The rendered indicates that all the data transmitted by the RTP channel does not need media processing network element to render and has been rendered by the terminal device. That is, in this case, the RTP channel is the second RTP channel.

[0152] When a = data_processing_mode: 2 unrendered, data_processing_mode is 2 indicates that the data transmitted by the RTP channel can need media processing network element to render or can not need media processing network element to render. The unrendered indicates that all the data transmitted by the RTP channel has not been rendered by the terminal device but does not need media processing network element to render. That is, in this case, the RTP channel is the second RTP channel.

[0153] When a = data_processing_mode: 2 render / rendered, data_processing_mode is 2 indicates that the data transmitted by the RTP channel can need media processing network element to render or can not need media processing network element to render. The render / rendered indicates that some of the data transmitted by the RTP channel needs media processing network element to render and some does not need media processing network element to render and has been rendered by the terminal device. That is, in this case, the RTP channel is the first RTP channel.

[0154] When a = data_processing_mode: 2 render / unrendered, data_processing_mode is 2, which means that the data transmitted by the RTP channel can need media processing network elements for rendering, or can not need media processing network elements for rendering. render / unrendered means that the data transmitted by the RTP channel is some that needs media processing network elements for rendering, and some that has not been rendered by the terminal device but does not need media processing network elements for rendering. That is, in this case, the RTP channel is the first RTP channel.

[0155] When a = data_processing_mode: 2 rendered / unrendered, data_processing_mode is 2, which means that the data transmitted by the RTP channel can need media processing network elements for rendering, or can not need media processing network elements for rendering. rendered / unrendered means that the data transmitted by the RTP channel is some that does not need media processing network elements for rendering and has been rendered by the terminal device, and some that has not been rendered by the terminal device but does not need media processing network elements for rendering. That is, in this case, the RTP channel is the second RTP channel.

[0156] When a = data_processing_mode: 2 render / rendered / unrendered, data_processing_mode is 2, which means that the data transmitted by the RTP channel can need media processing network elements for rendering, or can not need media processing network elements for rendering. render / rendered / unrendered means that the data transmitted by the RTP channel is some that needs media processing network elements for rendering, some that does not need media processing network elements for rendering and has been rendered by the terminal device, and some that has not been rendered by the terminal device but does not need media processing network elements for rendering. That is, in this case, the RTP channel is the first RTP channel.

[0157] If the a = data_processing_mode field is not used, the media processing network element can not process the received data by default. For the first RTP channel, its network capability can also be determined to support UE rendering requests (such as rendering format, rendering computing resource, etc.).

[0158] When the RTP channel is the first RTP channel, and the data transmitted by the RTP channel includes some data that needs to be rendered by the media processing network element and some data that does not need to be rendered by the media processing network element, the data can be distinguished by using the RTP header extension method provided by the RTP protocol. The extended RTP header can transmit media metadata associated with the rendered data, such as spatial position information of the rendered data. The position of the RTP extension header and the corresponding information are described by using the a=extmap attribute in the SDP negotiation.

[0159] For example, a=extmap:1 urn:3gpp:media_processing indicates that the first RTP extension header carries data of a type. In the first RTP extension header of a specific RTP packet, the value can be set to 1, indicating that the RTP packet carries rendered data; the value can be set to 2, indicating that the RTP packet carries unrendered data that does not need to be rendered; the value can be set to 3, indicating that the RTP packet carries unrendered data that needs to be rendered. If the byte length of the extension header is sufficient, more values can be set to represent different meanings.

[0160] For example, a=extmap:4 urn:3gpp:pose indicates that the position information (including coordinate system information and specific coordinate information) is carried from the fourth RTP extension header. The position information is carried from the fourth extension header to the tenth extension header. The number of required extension headers needs to be determined according to the coordinate system.

[0161] 2) For the bootstrap data channel, if the bootstrap data channel is used to transmit media metadata, an a=media_metadata field is added to the attribute of the media stream channel, to indicate that the bootstrap data channel can transmit media metadata of one or more types.

[0162] When the a=media_metadata:1 SD value is 1, it indicates that the bootstrap data channel transmits media metadata of one type, and SD indicates that the type is scene description.

[0163] 3) For the application data channel, an a=data_resource field is added to the attribute of the media stream channel, to indicate that the application data channel can transmit at least one of media metadata, video / image data, and other service data.

[0164] When the a=data_resource:1 video value is 1, it indicates that the application data channel transmits data of one type, and video indicates that the type is video. The video can further include data characteristics of the video, such as an encoding format, a frequency, and the like.

[0165] When a = data_resource:2 media_metadata pose / orientation video (URN), the data_resource value is 2, indicating that the channel transmits two types of data, video and media_metadata. Some data characteristics of the video can be added, such as encoding format, frequency, etc. Some data characteristics can also be added after media_metadata, such as position and orientation information.

[0166] The application layer protocol of the application data channel can be indicated by "subprotocol=" in "a=dcmap", such as subprotocol=http.

[0167] The fields of a=media_metadata and a=media are added in the attribute of the media stream channel, respectively indicating the data type of the data stream in the channel.

[0168] In an embodiment, if data is transmitted by using the RTP channel and the application data channel at the same time, and the data has media processing correlation, such as rendered data and position information when rendering, unrendered data and position information when collecting the data, etc., the corresponding group identifier can be assigned to the transmission channel during SDP negotiation when the data needs to be associated and processed, and the transmission channels with the same group identifier have a binding relationship.

[0169] For example, in the session level description, the current multiple media streams are further described by adding an "a=" line. For example, a=media_processing_group is added. <group-1> / ... / <group-n>The field is used to represent the association relationship of multiple media streams (such as RTP channels, data channels).

[0170] When a = media_processing_group: composition 1 2 3 / 4 5 6, composition represents the media processing required for scene composition by the media processing network element, "1 2 3" and "4 5 6" respectively represent two groups, and the media streams in the two groups must be associated with media processing, wherein each number is the same as the value of a = mid in each media stream (media level), and the specific media stream can be identified by the value.

[0171] When a = media_processing_group: coding 1 2 3 / 4 5 6, coding represents that when decoding the data in the media stream, association is required, such as multi-view coded video, which requires an RTP channel to transmit video coded data and an application data channel to transmit atlas coded data, and the RTP channel and the application data channel form a group.

[0172] If a DC application uses an application data channel to transmit audio / video data (including data that needs to be rendered and / or data that does not need to be rendered), it can still use an RTP channel to transmit audio / video data at the same time, that is, audio / video data can be transmitted through an RTP channel and / or an application data channel, which is determined by the application itself. For example, multi-channel audio / video data collected by a UE is transmitted through multiple RTP channels and / or application data channels.

[0173] S502, UE-A performs rendering negotiation with the media processing network element, and anchors and redirects the audio / video stream to the MF / MRF.

[0174] The UE-A performs media rendering negotiation with the media processing network element based on an application data channel or an SDP session, and usually sends a media rendering negotiation request by the UE-A. The negotiation includes the computing power required for rendering, the type of rendering, the data format of the rendering input and output, etc. The IMS AS forwards the rendering request to the DC application server, and the DC application server sends a media resource request to the MF / MRF, requesting the MF / MRF to allocate rendering resources based on the information carried in the UE-A rendering request.

[0175] If the rendering negotiation is completed and the RTP stream of the UE-A is not anchored to the MF / MRF (such as being anchored to another UE), the RTP stream of the UE-A is re-anchored to the MF / MRF.

[0176] S503, the other UE anchors and redirects the audio / video stream to the MF / MRF.

[0177] If the IMS network accessed by the other UE is different from the IMS network accessed by the UE-A, and the IMS network accessed by the other UE also provides rendering resources and rendering functions, after the UE-A accesses the IMS network and completes rendering negotiation, the audio / video stream can be directly anchored to the other UE or anchored to the MF / MRF of the IMS network accessed by the other UE, that is, the IMS AS of the IMS network accessed by the other UE re-negotiates with the MF / MRF to anchor the audio / video stream transmitted by the IMS network accessed by the UE-A to the MF / MRF.

[0178] S504, the UE-A processes part of the local data according to the rendering negotiation.

[0179] The UE-A processes part of the local data according to the rendering negotiation, such as media rendering processing.

[0180] S505, the UE-A sends the data to be rendered to the media processing network element.

[0181] If scene composition processing is required in the media processing network element, the UE-A can send the rendered data and the data to be rendered at the same time, or S504 and S505 are performed at the same time, that is, the data to be rendered is first sent to the media processing network element, and then the local rendering data is also transmitted to the media processing network element after the UE-A local rendering is completed.

[0182] S506, the media processing network element performs network-side media processing according to the rendering negotiation.

[0183] The network-side media processing can include at least one of rendering processing and scene composition.

[0184] If there is a requirement for scene composition in the rendering negotiation, the media processing network element can continue to perform scene composition processing by sending a scene composition request through the application data channel after the rendering is completed.

[0185] S507, the media processing network element sends the data processed by the network-side media processing to the other UE.

[0186] The embodiment of the application also provides a data rendering method based on an IMS network. Based on the DC application service requested by the UE, the media processing network element can directly provide media data, such as a digital human model, which can be provided by a third party or stored in a media storage server of the IMS system. That is, the IMS system can directly render the media data provided by the network side on the network side and then send it to the UE.

[0187] FIG. 6 is an interaction diagram of a data rendering method based on an IMS network according to an embodiment. As shown in FIG. 6, it includes S601-S607.

[0188] S601, the UE and the DC application perform an IMS session, a transmission channel is established, and the transmission channel has a corresponding identifier.

[0189] The IMS session can be initiated by the UE. Establishing the transmission channel means that the UE and the media processing network element establish a transmission channel, and the DC application and the media processing network element establish a transmission channel. The transmission channel includes an RTP channel, a bootstrap data channel, and an application data channel. The RTP channel is used to transmit video / image data. The bootstrap data channel is mainly used for downloading of the DC application, and can also transmit media metadata (such as scene description) or other data. The application data channel is mainly used for transmission of media metadata, and can also transmit video / image data and negotiation data of services and the like.

[0190] If the UE does not preinstall the DC application, the DC application needs to be acquired through the bootstrap data channel; if the UE preinstalls the DC application, the bootstrap data channel can not be needed for acquisition. The DC application is usually an HTML application.

[0191] In an embodiment, the transmission channel has a corresponding identifier. The corresponding identifier of the transmission channel can be negotiated through an SDP description rule.

[0192] For example, in a media level description, an "a=" line is added under an "m=" line (indicating a media stream channel) to realize description of characteristics of data transmitted by the media stream channel.

[0193] 1) For the RTP channel, an a=data_processing_mode field is added in the attribute of the media stream channel, to distinguish whether the RTP channel is a first RTP channel or a second RTP channel.

[0194] When a=data_processing_mode:1 rendered, data_processing_mode is 1, indicating that all data transmitted by the RTP channel does not need to be rendered by the media processing network element. That is, the RTP channel is a second RTP channel. Rendered indicates that all data transmitted by the RTP channel has been rendered by the terminal device.

[0195] When a=data_processing_mode:1 unrendered, data_processing_mode is 1, indicating that all data transmitted by the RTP channel does not need to be rendered by the media processing network element. That is, the RTP channel is a second RTP channel. Unrendered indicates that all data transmitted by the RTP channel has not been rendered by the terminal device.

[0196] When a = data_processing_mode: 1 rendered / unrendered, data_processing_mode is 1 means that all the data transmitted by the RTP channel does not need to be rendered by the media processing network element. That is, the RTP channel is a second RTP channel. Rendered / unrendered means that some of the data transmitted by the RTP channel has been rendered by the terminal device and some has not been rendered by the terminal device.

[0197] When a = data_processing_mode: 2 render, data_processing_mode is 2 means that the data transmitted by the RTP channel can need to be rendered by the media processing network element or can not need to be rendered by the media processing network element. Render means that all the data transmitted by the RTP channel needs to be rendered by the media processing network element. That is, in this case, the RTP channel is a first RTP channel.

[0198] When a = data_processing_mode: 2 rendered, data_processing_mode is 2 means that the data transmitted by the RTP channel can need to be rendered by the media processing network element or can not need to be rendered by the media processing network element. Rendered means that all the data transmitted by the RTP channel does not need to be rendered by the media processing network element and has been rendered by the terminal device. That is, in this case, the RTP channel is a second RTP channel.

[0199] When a = data_processing_mode: 2 unrendered, data_processing_mode is 2 means that the data transmitted by the RTP channel can need to be rendered by the media processing network element or can not need to be rendered by the media processing network element. Unrendered means that all the data transmitted by the RTP channel has not been rendered by the terminal device but does not need to be rendered by the media processing network element. That is, in this case, the RTP channel is a second RTP channel.

[0200] When a = data_processing_mode: 2 render / rendered, data_processing_mode is 2 means that the data transmitted by the RTP channel can need to be rendered by the media processing network element or can not need to be rendered by the media processing network element. Render / rendered means that some of the data transmitted by the RTP channel needs to be rendered by the media processing network element and some does not need to be rendered by the media processing network element and has been rendered by the terminal device. That is, in this case, the RTP channel is a first RTP channel.

[0201] When a = data_processing_mode: 2 render / unrendered, data_processing_mode is 2, which means that the data transmitted by the RTP channel can need media processing network element to render or can not need media processing network element to render. render / unrendered means that the data transmitted by the RTP channel is some of which needs media processing network element to render, and some of which is not rendered by terminal device but does not need media processing network element to render. That is, in this case, the RTP channel is the first RTP channel.

[0202] When a = data_processing_mode: 2 rendered / unrendered, data_processing_mode is 2, which means that the data transmitted by the RTP channel can need media processing network element to render or can not need media processing network element to render. rendered / unrendered means that the data transmitted by the RTP channel is some of which does not need media processing network element to render and has been rendered by terminal device, and some of which is not rendered by terminal device but does not need media processing network element to render. That is, in this case, the RTP channel is the second RTP channel.

[0203] When a = data_processing_mode: 2 render / rendered / unrendered, data_processing_mode is 2, which means that the data transmitted by the RTP channel can need media processing network element to render or can not need media processing network element to render. render / rendered / unrendered means that the data transmitted by the RTP channel is some of which needs media processing network element to render, some of which does not need media processing network element to render and has been rendered by terminal device, and some of which is not rendered by terminal device but does not need media processing network element to render. That is, in this case, the RTP channel is the first RTP channel.

[0204] When the RTP channel is the first RTP channel, and the data transmitted by the RTP channel is some of which needs media processing network element to render and some of which does not need media processing network element to render, the data can be distinguished by the RTP header extension method provided by the RTP protocol. The extended RTP header can transmit media metadata matched with the rendered data, such as spatial position information of the rendered data. The position of the RTP extension header and the corresponding information are described by a = extmap attribute in SDP negotiation.

[0205] 2) For application data channel, add a = data_resource field in the attribute of media stream channel to indicate that the application data channel can transmit at least one of media metadata, video / image data, and other service data.

[0206] When a = data_resource: 1 video, the data_resource value is 1, indicating that the application data channel transmits one type of data, and video indicates that the type is video, and some data characteristics of the video such as encoding format, frequency, etc. can be added after video.

[0207] When a = data_resource: 1 rendered_video, the data_resource value is 1, indicating that the channel transmits one type of data, and rendered_video indicates that the type is rendered video, and some data characteristics of the video such as encoding format, frequency, etc. can be added after video. Alternatively, only the data type video is indicated by data_source, and the data is rendered or not rendered, which can be directly indicated by indicating the transmitted data.

[0208] S602, the DC application provides stored media data according to the service request when the UE session is established.

[0209] The media data can be a digital human model.

[0210] The media data provider can also be a storage server that stores media data on the network side, but the DC application needs to request the storage server to provide the corresponding data.

[0211] S603, the media processing network element processes the media data provided by the DC application or other network elements.

[0212] The processing of the media data by the media processing network element at least includes rendering processing.

[0213] S604, the media processing network element transmits the processed media data to the UE.

[0214] The media processing network element can transmit the rendered media data using an RTP channel or an application data channel.

[0215] S605, the UE acquires local media data.

[0216] The local media data includes video / audio data.

[0217] S606, the UE performs rendering processing on the local media data.

[0218] The UE can also perform secondary rendering, scene composition, and other media processing on the rendered media data provided by the DC application.

[0219] S607. The UE transmits the locally collected data to the DC application.

[0220] The locally collected data is usually used for network-side rendering or providing media information to a media processing network element. The data used for network-side rendering is usually media metadata (such as location information); the media information provided to the media processing network element can be audio / video data, but is usually different from the data provided by the media processing network element to the UE, for example, the UE collects audio data and does not need to be rendered by the media processing network element or perform media processing such as scene composition.

[0221] FIG. 7 is a structural schematic diagram of a data rendering apparatus provided in an embodiment, which can be configured in a media processing network element. As shown in FIG. 7, the apparatus includes a negotiation module 701, a communication module 702, and a rendering module 703.

[0222] The negotiation module 701 is configured to establish a transmission channel between the media processing network element and the terminal device and assign a corresponding identifier to the transmission channel. The transmission channel includes at least a first real-time transport protocol (RTP) channel. The identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits to-be-rendered data. The communication module 702 is configured to obtain the to-be-rendered data sent by the terminal device through the first RTP channel. The rendering module 703 is configured to render the to-be-rendered data to obtain rendered data.

[0223] The data rendering apparatus provided in the embodiment is used to implement the data rendering method shown in FIG. 2. The implementation principle and technical effects of the data rendering apparatus provided in the embodiment are similar to those of the above-described embodiments, which will not be described herein again.

[0224] In an embodiment, when the identifier corresponding to the first RTP channel is a first identifier, the first identifier is used to indicate that the first RTP channel only transmits to-be-rendered data. When the identifier corresponding to the first RTP channel is a second identifier, the second identifier is used to indicate that the first RTP channel transmits to-be-rendered data and non-rendered data.

[0225] In an embodiment, when the identifier corresponding to the first RTP channel is the second identifier, the communication module 702 is configured to receive an RTP packet sent by the terminal device through the first RTP channel. The to-be-rendered data is determined from the data carried in the RTP packet according to an RTP extension header of the RTP packet.

[0226] In an embodiment, the transmission channel further comprises at least one of a second RTP channel and a guide data channel; the corresponding identifier of the second RTP channel is used to indicate that the second RTP channel only transmits non-rendering data; and the corresponding identifier of the guide data channel is used to indicate that the guide data channel transmits media metadata.

[0227] In an embodiment, the communication module 702 is further configured to acquire the non-rendering data transmitted by the terminal device through the second RTP channel, and acquire the media metadata transmitted by the terminal device through the guide data channel.

[0228] In an embodiment, the negotiation module 701 is further configured to assign a corresponding group identifier to the transmission channel, and the transmission channels having the same group identifier have a binding relationship.

[0229] In an embodiment, the rendering module 703 is further configured to perform at least one of the following processes on the data transmitted by the transmission channels having the same group identifier: scene synthesis processing, data correlation processing.

[0230] In an embodiment, the communication module 702 is further configured to transmit media data, and the media data at least comprises rendered data.

[0231] FIG. 8 is a structural schematic diagram of another data rendering device provided in an embodiment, which can be configured in a terminal device, as shown in FIG. 8, the device comprises a negotiation module 801 and a communication module 802.

[0232] The negotiation module 801 is configured to establish a transmission channel between the terminal device and a media processing network element, and determine a corresponding identifier of the transmission channel, wherein the transmission channel at least comprises a first real-time transport protocol (RTP) channel, and the corresponding identifier of the first RTP channel is used to indicate that the first RTP channel transmits to-be-rendered data; and the communication module 802 is configured to transmit the to-be-rendered data to the media processing network element through the first RTP channel.

[0233] The data rendering device provided in the embodiment is used to implement the data rendering method of the embodiment shown in FIG. 3, and the implementation principle and technical effects of the data rendering device provided in the embodiment are similar to those of the above-mentioned embodiments, which will not be described herein again.

[0234] In an embodiment, when the corresponding identifier of the first RTP channel is a first identifier, the first identifier is used to indicate that the first RTP channel only transmits to-be-rendered data; and when the corresponding identifier of the first RTP channel is a second identifier, the second identifier is used to indicate that the first RTP channel transmits to-be-rendered data and non-rendering data.

[0235] In an embodiment, the transmission channel further includes at least one of a second RTP channel and a guide data channel; the identifier corresponding to the second RTP channel is used to indicate that the second RTP channel only transmits non-rendering data; and the identifier corresponding to the guide data channel is used to indicate that the guide data channel transmits media metadata.

[0236] In an embodiment, the communication module 802 is further configured to send non-rendering data to the media processing network element via the second RTP channel, and send media metadata to the media processing network element via the guide data channel.

[0237] In an embodiment, the negotiation module 801 is further configured to determine a group identifier corresponding to the transmission channel, and the transmission channels having the same group identifier have a binding relationship.

[0238] In an embodiment, the communication module 802 is further configured to receive media data sent by the media processing network element, wherein the media data at least includes rendered data.

[0239] Embodiments of the present application also provide a media processing network element, which comprises a processor configured to implement the method provided in any of the embodiments of the present application when the processor executes a computer program.

[0240] FIG. 9 is a structural schematic diagram of a media processing network element according to an embodiment of the present application. As shown in FIG. 9, the media processing network element comprises a processor 60, a memory 61 and a communication interface 62; the number of processors 60 in the media processing network element can be one or more, and one processor 60 is taken as an example in FIG. 9; the processor 60, the memory 61 and the communication interface 62 in the media processing network element can be connected through a bus or other means, and the connection through the bus is taken as an example in FIG. 9. The bus represents one or more of several bus structures, including a memory bus or a memory controller, a peripheral bus, a graphics acceleration port, a processor or a local bus using any of the bus structures.

[0241] The memory 61 is a computer readable storage medium, which can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiments of the present application. The processor 60 executes at least one function application and data processing of the media processing network element by running the software programs, instructions and modules stored in the memory 61, i.e. implements the above method.

[0242] The memory 61 can include a program region and a data region. The program region can store an operating system, an application required for at least one function, and the like. The data region can store data created according to a use of the terminal, and the like. The memory 61 can further include a high-speed random access memory, and can further include a non-volatile memory such as at least one of a magnetic disk storage device, a flash memory device, or other non-volatile solid state storage device. In some examples, the memory 61 can include a memory remotely located with respect to the processor 60, and the remote memory can be connected to the media processing network element through a network. Examples of the network include, but are not limited to, the Internet, an intranet, an extranet, a local area network, a wide area network, a mobile communication network, and a combination thereof.

[0243] The communication interface 62 can be configured to receive and transmit data.

[0244] The embodiments of the present application also provide a terminal device, including: a processor, the processor is used for implementing the method provided by any of the embodiments of the present application when executing a computer program. For example, the following embodiment provides a structural schematic diagram of a terminal device which is a UE.

[0245] FIG. 10 is a structural schematic diagram of a UE according to an embodiment of the present application. The UE can be implemented in various forms, and the UE in the present application can include, but is not limited to, a mobile terminal device such as a mobile phone, a smart phone, a notebook computer, a digital broadcast receiver, a Personal Digital Assistant (PDA), a Portable Device (PAD), a Portable Media Player (PMP), a navigation device, a vehicle-mounted terminal device, a vehicle-mounted display terminal, a vehicle-mounted electronic rearview mirror, and the like, and a fixed terminal device such as a digital television (TV), a desktop computer, and the like.

[0246] As shown in FIG. 10, the UE 50 can include a wireless communication unit 51, an Audio / Video (A / V) input unit 52, a user input unit 53, a sensing unit 54, an output unit 55, a memory 56, an interface unit 57, a processor 58, and a power supply unit 59, and the like. FIG. 10 shows the UE including various components, but it is understood that implementation of all of the components shown in FIG. 10 is not required. More or less components can alternatively be implemented.

[0247] In the present embodiment, the wireless communication unit 51 allows radio communication between the UE 50 and a base station or a network. The A / V input unit 52 is configured to receive audio or video signals. The user input unit 53 can generate key input data to control various operations of the UE 50 according to a user's input command. The sensing unit 54 detects a current state of the UE 50, a location of the UE 50, a presence or absence of a user's touch input with respect to the UE 50, an orientation of the UE 50, acceleration or deceleration movement and direction of the UE 50, and the like, and generates a command or signal for controlling the operation of the UE 50. The interface unit 57 serves as an interface through which at least one external device can be connected to the UE 50. The output unit 55 is configured to provide an output signal in a form of visual, audio, and / or tactile. The memory 56 can store software programs and the like for processing and control operations performed by the processor 58, or can temporarily store data that has been output or is to be output. The memory 56 can include at least one type of storage medium. Also, the UE 50 can cooperate with a network storage device that performs a storage function of the memory 56 through a network connection. The processor 58 generally controls the overall operation of the UE 50. The power supply unit 59 receives external power or internal power under the control of the processor 58 and provides appropriate power required for operating various elements and components.

[0248] The processor 58 performs at least one functional application and data processing by running a program stored in the memory 56, for example, a method provided by the present embodiment.

[0249] The present embodiment also provides a computer readable storage medium, and the computer readable storage medium stores a computer program. The computer program is executed by a processor to implement a method provided by any of the embodiments of the present application.

[0250] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer readable media. The computer readable medium can be a computer readable signal medium or a computer readable storage medium. The computer readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. The computer readable storage medium includes, but is not limited to, a non-exhaustive list: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present application, the computer readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, device or apparatus.

[0251] The computer readable signal medium can include a data signal propagating in a baseband or as part of a carrier wave propagating through a transmission medium, and bearing computer readable program code. Such a propagating data signal can take many forms, including but not limited to electro-magnetic, optical, or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate or transport program for use by or in connection with an instruction execution system, apparatus or device.

[0252] The program code contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wire, cable, radio frequency (RF), or any suitable combination thereof.

[0253] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++, Ruby, Go, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0254] The embodiments of the present application also provide a computer program product, comprising a computer program which, when executed by a processor, implements the method provided by any of the embodiments of the present application.

[0255] The computer program product, in its implementation, can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++, or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0256] Those skilled in the art will appreciate that the term user terminal encompasses any appropriate type of wireless user equipment, such as a mobile phone, a portable data processing apparatus, a portable web browser, or a vehicle mounted mobile station.

[0257] In general, the various embodiments of the application can be implemented in hardware or special purpose circuits, software, logic or any combination thereof. For example, some aspects can be implemented in hardware, while other aspects can be implemented in

[0258] Embodiments of the application can be implemented by computer program instructions on a mobile device's data processor, for example in the processor entity, or by hardware, or by a combination of software and hardware. Computer program instructions can be in assemblies, Instruction Set Architecture (ISA), machine, machine-related, microcode, firmware, state setting data, or source code or object code written in any combination of one or more programming languages to be ultimately mapped to machine language.

[0259] Any block diagrams of logical flows of the application can represent program operations, or a logical flow of operations, or a combination of program operations and logical flows of operations. Computer program instructions can be stored on a memory. The memory can be of any suitable type and can be implemented using any suitable data storage technology, such as, but not limited to, read only memory (ROM), random access memory (RAM), optical storage devices, and systems, digital versatile disc (DVD) or CD discs, and the like. The computer readable media can include non-transitory storage media. The data processor can be of any suitable type and can be implemented using any suitable combination of hardware and software. < / group-1> < / group-1>

Claims

1. A data rendering method applied to a media processing network element, comprising: establishing a transmission channel between the media processing network element and a terminal device, and assigning a corresponding identifier to the transmission channel, wherein the transmission channel comprises at least a first real-time transport protocol (RTP) channel, and the identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits to-be-rendered data; obtaining the to-be-rendered data transmitted by the terminal device through the first RTP channel; rendering the to-be-rendered data to obtain rendered data. 2.The data rendering method of claim 1, wherein: when the identifier corresponding to the first RTP channel is a first identifier, the first identifier is used to indicate that the first RTP channel only transmits the to-be-rendered data; when the identifier corresponding to the first RTP channel is a second identifier, the second identifier is used to indicate that the first RTP channel transmits the to-be-rendered data and non-rendered data.

3. The data rendering method of claim 2, wherein, when the identifier corresponding to the first RTP channel is the second identifier, the obtaining the to-be-rendered data transmitted by the terminal device through the first RTP channel comprises: receiving an RTP packet transmitted by the terminal device through the first RTP channel; and determining the to-be-rendered data from data carried by the RTP packet according to an RTP extension header of the RTP packet.

4. The data rendering method of claim 1, wherein, the transmission channel further comprises at least one of a second RTP channel and a guide data channel; the identifier corresponding to the second RTP channel is used to indicate that the second RTP channel only transmits non-rendered data; the identifier corresponding to the guide data channel is used to indicate that the guide data channel transmits media metadata. 5.The data rendering method of claim 4, further comprising at least one of: obtaining the non-rendered data transmitted by the terminal device through the second RTP channel; and obtaining the media metadata transmitted by the terminal device through the guide data channel.

6. The data rendering method of claim 1 or 4, wherein, when establishing the transmission channel between the media processing network element and the terminal device, the method further comprises: assigning a corresponding group identifier to the transmission channel, and a plurality of transmission channels having the same group identifier have a binding relationship. 7.The data rendering method of claim 6, further comprising: performing at least one of the following processes on data transmitted by the plurality of transmission channels having the same group identifier: scene synthesis processing, data correlation processing. 8.The data rendering method of claim 1, after obtaining the rendered data, the method further comprises: sending media data, wherein the media data at least comprises the rendered data. 9.A data rendering method applied to a terminal device, comprising: establishing a transmission channel between the terminal device and a media processing network element, and determining an identifier corresponding to the transmission channel, wherein the transmission channel comprises at least a first real-time transport protocol (RTP) channel, and the identifier corresponding to the first RTP channel is used to indicate that the first RTP channel transmits to-be-rendered data; transmitting the to-be-rendered data to the media processing network element through the first RTP channel. 10.The data rendering method of claim 9, wherein: when the first RTP channel corresponds to a first identifier, the first identifier is used to indicate that the first RTP channel only transmits the data to be rendered; when the first RTP channel corresponds to a second identifier, the second identifier is used to indicate that the first RTP channel transmits the data to be rendered and non-rendered data.

11. The data rendering method of claim 9, wherein, the transmission channel further comprises at least one of a second RTP channel and a guide data channel; an identifier corresponding to the second RTP channel is used to indicate that the second RTP channel only transmits non-rendered data; an identifier corresponding to the guide data channel is used to indicate that the guide data channel transmits media metadata.

12. The data rendering method of claim 11, further comprising at least one of: sending the non-rendered data to the media processing network element through the second RTP channel; and sending the media metadata to the media processing network element through the guide data channel.

13. The data rendering method of claim 9 or 11, wherein, when establishing the transmission channel between the terminal device and the media processing network element, further comprising: determining a group identifier corresponding to the transmission channel, and a plurality of transmission channels having a same group identifier have a binding relationship.

14. The data rendering method of claim 9, after sending the data to be rendered to the media processing network element through the first RTP channel, further comprising: receiving media data transmitted by the media processing network element, the media data at least comprising rendered data.

15. A media handling network element comprising: a processor; the processor is configured to implement the data rendering method of any one of claims 1-8 when executing the computer program.

16. A terminal device comprising: a processor; the processor is configured to implement the data rendering method of any one of claims 9-14 when executing the computer program.

17. A computer readable storage medium storing a computer program, the computer program being executed by a processor to implement the data rendering method of any one of claims 1-14.

Images