Rendering method, communication node and storage medium

Abstract

The invention discloses a rendering method, a communication node and a storage medium, and the method comprises the steps: negotiating a rendering resource for rendering call service media data; transmitting the call service media data to a media function network element to which the rendering resource belongs; and rendering the call service media data based on the rendering resource.

Description

Technical Field

[0001] The present application relates to the field of communication technology, for example, to a rendering method, a communication node and a storage medium. Background Art

[0002] With the deployment of the fifth generation mobile communication technology (5G) network, calls are no longer limited to traditional audio and video calls. 5G calls can add features to traditional calls. For example, augmented reality (AR) features can be added to traditional calls to improve the experience of 5G call services. Richer call services can be provided, such as fun calls and remote collaboration.

[0003] The increase in the amount of media data, the increase in data types, the complexity of media processing, and the need to improve experience in 5G-based call services have put forward higher requirements on the media capabilities of terminals, and the rendering of media data is an indispensable processing link. Therefore, how to improve the rendering efficiency of call service media data is currently an urgent problem to be solved. Summary of the invention

[0004] The present application provides a rendering method, a communication node and a storage medium.

[0005] In a first aspect, an embodiment of the present application provides a rendering method, including:

[0006] Negotiate rendering resources for rendering call service media data;

[0007] Transmitting the call service media data to the media function network element to which the rendering resource belongs;

[0008] The call service media data is rendered based on the rendering resource.

[0009] In a second aspect, an embodiment of the present application provides a rendering method, including:

[0010] Negotiate rendering resources for rendering call service media data;

[0011] Associating the call service media data with the media function network element to which the media resource belongs;

[0012] Transmit the call service media data to the media function network element.

[0013] In a third aspect, an embodiment of the present application provides an Internet protocol multimedia subsystem, characterized in that it includes:

[0014] one or more processors;

[0015] A storage device for storing one or more programs;

[0016] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the embodiments of the present application.

[0017] In a fourth aspect, an embodiment of the present application provides a communication node, characterized in that it includes:

[0018] one or more processors;

[0019] A storage device for storing one or more programs;

[0020] When the one or more programs are executed by the one or more processors, the one or more processors implement the method described in the embodiments of the present application.

[0021] In a fifth aspect, an embodiment of the present application provides a storage medium, characterized in that the storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the embodiment of the present application is implemented.

[0022] With regard to the above embodiments and other aspects of the present application and their implementation, further description is provided in the accompanying drawings, detailed description and claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] Figure 1 It is a flowchart of a rendering method provided in an embodiment of the present application;

[0024] Figure 2 It is a structural diagram of a communication system provided in an embodiment of the present application;

[0025] Figure 3a It is a schematic diagram of a network architecture of an Internet protocol multimedia subsystem provided by an embodiment of the present application;

[0026] Figure 3b This is a schematic diagram of a rendering process of a terminal device provided in an embodiment of the present application;

[0027] Figure 3c This is a flow chart of a network-side initiated split rendering process provided in an embodiment of the present application;

[0028] Figure 3d This is a schematic diagram of a process of initiating separate rendering on the terminal side provided in an embodiment of the present application;

[0029] Figure 3e This is a schematic diagram of a process of UE initiating re-negotiation provided in an embodiment of the present application;

[0030] Figure 3f This is a schematic diagram of a process of initiating re-negotiation on the network side provided in an embodiment of the present application;

[0031] Figure 4 is a flowchart of another rendering method provided in an embodiment of the present application;

[0032] Figure 5 is a structural schematic diagram of a rendering device provided in an embodiment of the present application;

[0033] Figure 6 is a structural schematic diagram of a rendering device provided in an embodiment of the present application;

[0034] Figure 7 It is a structural diagram of an Internet protocol multimedia subsystem provided by an embodiment of the present application;

[0035] Figure 8 It is a structural diagram of a communication node provided in an embodiment of the present application. DETAILED DESCRIPTION

[0036] In order to make the purpose, technical solution and advantages of the present application more clear, the embodiments of the present application will be described in detail with reference to the accompanying drawings. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other arbitrarily without conflict.

[0037] The steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions. Also, although a logical sequence is shown in the flowchart, in some cases, the steps shown or described can be performed in a sequence different from that shown here.

[0038] In the present application, the terms "first", "second", etc. are used to distinguish similar objects and are not necessarily used to describe a specific sequence or order.

[0039] With the characteristics of high-definition, immersive, and real-time interactive media data in 5G calls, the transmission, processing, and presentation of media data need to be enhanced to adapt to various services and application scenarios in 5G calls. The increase in the amount of data, the increase in data types, the complexity of media processing, and the improvement of experience in 5G AR call services have put forward higher requirements on the media capabilities of the terminal. The rendering of media data is an indispensable processing link. AR-related data processing has higher requirements on the central processing unit (CPU) and the graphics processing unit (GPU). The 3rd Generation Partnership Project (3GPP) has proposed rendering based on terminal equipment (UE) and network-based rendering, and process examples have been proposed based on the two rendering methods. However, based on different media data types, different service types, and different terminal and network media processing capabilities, the rendering method selection is diverse, and a reasonable rendering scheme is needed to meet the business needs of AR calls and improve the user's business experience. For this reason, this application proposes a rendering method.

[0040] In an exemplary embodiment, Figure 1 1 is a flowchart of a rendering method provided by an embodiment of the present application, which can be applied to the case of rendering call service media data. The method can be executed by a rendering device, which can be implemented by software and / or hardware and integrated in an Internet Protocol Multimedia Subsystem.

[0041] Figure 2 1 is a schematic diagram of the structure of a communication system provided in an embodiment of the present application; the communication system may be a communication node for implementing AR calls based on IMS. The communication system is composed of a terminal device 10 and an IP multimedia subsystem 20. The terminal device 10 includes the following modules:

[0042] The AR call application 101 can be considered as an application for implementing AR calls. The AR call application provides a 5G-based augmented reality media call application or media call service for the terminal device 10. The AR call can be considered as a call with AR features superimposed.

[0043] Runtime 102 obtains the call service media data collected locally by the terminal device 10, also known as call service media information. Other modules of the terminal device 10 can access the data / information obtained by runtime through an open interface.

[0044] The rendering module 103 provides local rendering processing of the terminal device 10 for the locally collected call service media data / call service media information and the call service media data / information transmitted by the IP multimedia subsystem 20;

[0045] The sending / receiving module 104 establishes a session between the terminal device and the IP multimedia subsystem 20 and performs session management, and at the same time, sends and receives call service media data / information.

[0046] The IP multimedia subsystem 20 includes the following modules:

[0047] AR application server 201, controls the business logic of AR calls;

[0048] The signaling function network element 202 establishes session management with the terminal device and manages or forwards signaling between the functional network elements within the IP multimedia subsystem 20;

[0049] The media function network element 203 provides media processing functions for call service media data in the AR call service, such as rendering.

[0050] Figure 3a This is a network architecture diagram of an Internet protocol multimedia subsystem provided in an embodiment of the present application; this embodiment provides an IMS network architecture based on 5G, an architecture example diagram that supports AR call services.

[0051] AR Application Server: responsible for the service control of AR calls, including AR session media control and negotiation of call service media capabilities and media resources with UE. AR application server can be deployed in IMS network or in third-party non-IMS network. The second deployment mode interacts with UE through IMS network through network openness, controls the service logic of AR calls and provides AR call applications.

[0052] Media Function / Media Resource Function (MF / MRF): Supports media processing of AR call service media data. Receives and stores media processing logic of AR call service from IMS application server (AS), and provides corresponding media capabilities for AR call service media data (including AR-related 2D audio and video data, 3D audio and video data, 3D models, scene descriptions, AR metadata, etc.) received from UE.

[0053] Telephone application server and application server of IP multimedia system (IMS TAS+IMS AS): Support the functions and negotiation of control plane. Interact with DCSF for event notification, receive data channel control instructions from DCSF, and interact with MF / MRF accordingly to manage data channel media resources. At the same time, receive media control instructions from DCSF, and interact with UE accordingly to connect UE to MF / MRF to transmit AR call service media data generated by UE to MF / MRF.

[0054] Data Channel Signaling Function (DCSF): provides data channel control capabilities.

[0055] IMS Access Gateway (IMS-AGW), network address translation, exchanging media plane data.

[0056] Call Session Control Function (CSCF): Responsible for session control and routing, including registration and authentication, session control, routing management, network management and billing functions closely related to user services.

[0057] Proxy CSCF (P-CSCF): The first connection point with the user in the IMS, providing proxy functions and can also provide user agent functions;

[0058] Inter-CSCF (I-CSCF): Similar to the gateway node of IMS, it allocates S-CSCF, performs routing query and hides the topology between IMS domains.

[0059] Serving CSCF (S-CSCF): It plays a core control role in the IMS core network and is responsible for UE registration and authentication, conference control, and user data management.

[0060] Among them, MF / MRF can be called a media function network element. AR AS can be called an application service network element. Other network elements can be session control network elements of the control plane.

[0061] like Figure 1 As shown, a rendering method provided by the present application comprises the following steps:

[0062] S110 . Negotiate rendering resources for rendering call service media data.

[0063] The call service media data may be considered as media data in a call service scenario. The call service may be a service for implementing a call. The rendering resource may be considered as a resource for rendering the call service media data.

[0064] This embodiment can be executed by the network side, such as the Internet Protocol Multimedia Subsystem. The Internet Protocol Multimedia Subsystem can negotiate with the sender of the call service media data, that is, the communication node that sends the call service media data, about the rendering resources in the Internet Protocol Multimedia Subsystem for rendering the call service media data, so as to realize the separate rendering of the call service media data.

[0065] In this embodiment, the negotiation of rendering resources may be initiated by the Internet protocol multimedia subsystem or by the communication node.

[0066] When initiated by the Internet Protocol Multimedia Subsystem, the Internet Protocol Multimedia Subsystem may initiate a request to the communication node after determining the allocation of rendering resources to achieve negotiation with the communication node. The Internet Protocol Multimedia Subsystem may also first send a first rendering negotiation request to the communication node, and allocate rendering resources to the communication node after the communication node agrees to separate rendering.

[0067] S120: Transmit the call service media data to the media function network element to which the rendering resource belongs.

[0068] This operation transmits the call service media data to the media function network element to which the rendering resources belong, thereby associating the call service media data with the media function network element to which the rendering resources belong, that is, anchoring the call service media data and the media function network element, so that the media function network element renders the call service media data.

[0069] The media function network element to which the rendering resource belongs may be considered as a media function network element that allocates the rendering resource. The media function network element may be a network element with media functions, such as MF or MRF.

[0070] S130: Render the call service media data based on the rendering resources.

[0071] This operation renders the call service media data from the communication node through the rendering resource, thereby realizing the separate rendering of the call service media data. The call service media data may be the media data rendered by the communication node, or the media data not rendered by the communication node.

[0072] The embodiment of the present application provides a rendering method, in which the Internet protocol multimedia subsystem negotiates rendering resources, realizes the determination of rendering resources for call service media data of the communication node, and after determining the rendering resources, the call service media data of the communication node can be rendered by the rendering resources in the Internet protocol multimedia subsystem. The negotiation of rendering resources and the configuration of rendering resources are realized on the network side. The call service media data is rendered separately, which improves the rendering efficiency of the call service media data.

[0073] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0074] In one embodiment, the negotiating rendering resources for rendering call service media data includes:

[0075] Initiate the first rendering negotiation request;

[0076] Configure rendering resources;

[0077] Transmitting configuration information of the rendering resource.

[0078] The first rendering negotiation request may be considered as a request for implementing rendering resource negotiation. The Internet protocol multimedia subsystem may configure rendering resources based on the first rendering negotiation request, i.e., allocate rendering resources. After the rendering resources are configured, the configuration information of the rendering resources may be transmitted, such as transmitting the configuration information to the communication node to which the call service media data belongs.

[0079] The configuration information may be considered as information for configuring rendering resources, such as rendering resource identifier, computing power value, data format, rendering object, rendering type, etc. The rendering resource identifier may be considered as information for identifying a rendering resource.

[0080] In this embodiment, the first rendering negotiation request may be a request transmitted within the Internet Protocol Multimedia Subsystem, such as initiating the first rendering negotiation request through an application service network element in the Internet Protocol Multimedia Subsystem, or may be a request initiated to a media function network element.

[0081] In this embodiment, the first rendering negotiation request may also be a request sent by the Internet Protocol Multimedia Subsystem to the communication node to which the call service media data belongs.

[0082] The communication node to which the call service media data belongs may be the sender of the call service media data.

[0083] In one embodiment, initiating a first rendering negotiation request includes:

[0084] Initiating the first rendering negotiation request through an application service network element in the Internet protocol multimedia subsystem; or

[0085] Transmitting a first rendering negotiation request to the communication node to which the call service media data belongs.

[0086] In this embodiment, the application service network element may initiate a first rendering negotiation request to the media function network element to request the media function network element to allocate rendering resources.

[0087] In this embodiment, the Internet protocol multimedia subsystem may also transmit a first rendering negotiation request to the communication node to which the call service media data belongs, so as to implement negotiation with the communication node to which the call service media data belongs regarding rendering resources.

[0088] Among them, the application service network element can be considered as a network element that provides application service functions, such as an AR application server.

[0089] In one embodiment, configuring rendering resources includes:

[0090] The rendering resources are configured through the media function network element in the Internet protocol multimedia subsystem.

[0091] When sending the first rendering negotiation request to the media function network element, the media function network element may configure rendering resources.

[0092] When the communication node to which the call service media data belongs responds to the first rendering negotiation request and indicates that it agrees to separate rendering, the media function network element can configure rendering resources.

[0093] In one embodiment, the negotiating rendering resources for rendering call service media data includes:

[0094] Get the second rendering negotiation request;

[0095] Determining the rendering resource of the call service media data based on the second rendering negotiation request;

[0096] Transmitting configuration information of the rendering resource.

[0097] The second rendering negotiation request may be considered as a rendering negotiation request initiated by the communication node to which the call service media data belongs.

[0098] After the Internet Protocol Multimedia Subsystem obtains the second rendering negotiation request, it can determine the rendering resources used to render the call service media data. After determining the rendering resources, the configuration information of the rendering resources can be transmitted to the communication node to which the call service media data belongs. The configuration information can be considered as information indicating the rendering resources configured for the communication node to which the call service media data belongs and the rendering input and output format description information (optional).

[0099] After the communication node to which the call service media data belongs obtains the configuration information, it can directly use the rendering resources based on the instructions of the configuration information to render the call service media data; it can also determine whether to use the configured rendering resources, and if not, it can initiate negotiation again.

[0100] In one embodiment, obtaining the second rendering negotiation request includes:

[0101] The second rendering negotiation request is obtained through the application service network element.

[0102] The Internet protocol multimedia subsystem can obtain the second rendering negotiation request transmitted by the communication node to which the call service media data belongs through the application service network element. Then, the request is forwarded through the application service network element. The forwarded request can be the processed second rendering negotiation request, or the second rendering negotiation request is directly forwarded.

[0103] In one embodiment, determining the rendering resource of the call service media data based on the second rendering negotiation request includes:

[0104] forwarding the second rendering negotiation request to the media function network element through the application service network element;

[0105] The rendering resource of the call service media data is determined by the media function network element.

[0106] The application service network element may process the second rendering negotiation request and forward it to the media function network element. The processing means are not limited here. For example, the application service network element may first preliminarily determine whether to provide the media rendering function based on the second rendering negotiation request. The application service network element sends the determined result and the second rendering negotiation request to the media function network element separately or in combination. The application service network element may also send the negotiation request to the media function network element according to the message format of the resource negotiation request specified by the media rendering function.

[0107] The media function network element may allocate rendering resources for the call service media data based on the data received from the application service network element, such as allocating rendering resources for the call service media data based on the information carried in the rendering request forwarded to the media function network element.

[0108] In one embodiment, transmitting the call service media data to the media function network element to which the rendering resource belongs includes:

[0109] An association request for the call service media data is transmitted, wherein the association request indicates that the call service media data is associated with the media function network element in the Internet Protocol Multimedia Subsystem.

[0110] The association request is used to anchor the call service media data to the media function network element.

[0111] This embodiment transmits an association request so that the media function network element can render the call service media data.

[0112] In one embodiment, transmitting the association request for the call service media data includes:

[0113] The association request of the call service media data is transmitted to the media function network element through the application service network element.

[0114] In one embodiment, the rendering method further includes:

[0115] The call service media data and / or the first indication information are transmitted via a real-time transport protocol channel and / or an application data channel, wherein the first indication information indicates the rendering status of the call service media data, and the call service media data includes rendered data and / or unrendered data.

[0116] The real-time transport protocol channel can be considered as a channel based on the real-time transport protocol and can be used to transmit call service media data.

[0117] The application data channel can be considered as a channel based on the Stream Control Transmission Protocol (SCTP), which can be used by users to transmit call service media data and data format information (optional).

[0118] The rendering status may be a status indicating information related to rendering, such as whether rendering is performed.

[0119] The rendered data may be data rendered by the communication node to which the call service media data belongs.

[0120] This embodiment obtains the call service media data and the first indication information, can determine the rendering status of the call service media data based on the first indication information, and then render the call service media data based on the rendering request. After rendering, it can be returned to the communication node to which the call service media data belongs, or it can be transmitted to the node to which the communication node to which the call service media data belongs communicates.

[0121] In one embodiment, the rendering method further includes:

[0122] Renegotiate rendering resources.

[0123] This embodiment can realize re-negotiation of rendering resources. As the rendering capability of the terminal side and / or the network side changes, this embodiment can also renegotiate the rendering resources to realize adaptive rendering.

[0124] The content of the renegotiation may include whether the network side is to be used for rendering. If the network side is to be used for rendering, the allocation of rendering resources on the network side, and whether to add or reduce rendering resources.

[0125] In one embodiment, the renegotiating rendering resources includes:

[0126] Get the first negotiation request for rendering resources;

[0127] Transmitting a resource allocation result corresponding to the first re-negotiation request.

[0128] The first renegotiation request may be a renegotiation request initiated by the terminal side. The renegotiation request may be considered as a request to trigger the renegotiation of rendering resources.

[0129] In this embodiment, the terminal side may initiate the renegotiation. After the network side obtains the first renegotiation request, it may allocate rendering resources, and then transmit the resource allocation result of the rendering resources to the terminal side. The resource allocation result may indicate the allocation of the rendering resources after the renegotiation.

[0130] In one embodiment, the renegotiating rendering resources includes:

[0131] Initiate a second negotiation request for rendering resources through a media function network element or an application service network element in an Internet protocol multimedia subsystem;

[0132] The second negotiation request includes a resource allocation result.

[0133] The second re-negotiation request may be initiated by the network side. In the case where the rendering capability of the Internet Protocol Multimedia Subsystem changes, the second re-negotiation request may be initiated by the media function network element and / or the application service network element, such as initiating the second re-negotiation request to the terminal side.

[0134] In this embodiment, when the Internet protocol multimedia subsystem initiates the second re-negotiation request, it may carry the resource allocation result. After reallocating rendering resources to the terminal side, the second re-negotiation request is initiated to the terminal side.

[0135] The application service network element may transmit a second negotiation request to the media function network element, and the resource allocation result carried by the second negotiation request may be a result of a preliminary judgment by the application service network element.

[0136] In one embodiment, the media function network element in the Internet Protocol Multimedia Subsystem allocates resources based on the first re-negotiation request or the second re-negotiation request.

[0137] After the first re-negotiation request is forwarded to the media function network element, it can trigger the media function network element to allocate resources.

[0138] After the second negotiation request is forwarded to the media function network element, the media function network element can allocate rendering resources based on the second negotiation request.

[0139] In one embodiment, rendering the call service media data based on the rendering resource includes:

[0140] Transmitting the call service media data to the rendering resources allocated by the media function network element of the Internet Protocol Multimedia Subsystem through the real-time transport protocol channel and / or the application data channel;

[0141] The call service media data is rendered using rendering resources allocated by the media function network element.

[0142] In one embodiment, the call service media data includes at least one of two-dimensional audio and video data, three-dimensional audio and video data, three-dimensional model, scene description, and augmented reality metadata in an augmented reality call scenario.

[0143] The scene description may be description information of the scene. The augmented reality metadata may be metadata under the augmented reality scene.

[0144] The following is an exemplary description of the present application. The rendering method provided in the present application can be considered as a 5G AR call service method. The present application provides different combinations of local and network rendering of terminal devices for media data such as graphics and images in the service, targeting the media processing capabilities of the communication terminal device and the media processing capabilities of the IMS network, to meet the real-time immersive experience in the AR call service. It mainly solves the negotiation, reasonable allocation and use of local and network media processing resources of the terminal device, as well as media data transmission under different rendering modes. By efficiently using the media processing resources of the terminal device and the network, a good AR call experience such as vision and latency can be obtained. At the same time, richer media services can be obtained through the reasonable allocation of media processing resources.

[0145] Figure 3b This is a schematic diagram of a rendering process of a terminal device provided in an embodiment of the present application, see Figure 3b When the terminal device has strong capabilities, the terminal device can directly render the media data and / or media metadata, and then present or send it to the terminal device at the other end. This method does not require the network edge server or central server to participate in the rendering process, which can reduce the transmission process (no need to transmit to the relevant network elements in the network for rendering and other media processing, and directly transmit to the terminal device at the other end through the IMS access gateway). Among them, the media data and / or media metadata can be considered as the call service media data to be rendered.

[0146] Figure 3b The UE rendering process of an IMS-based AR call is shown:

[0147] Step 301: Establish an IMS session.

[0148] Initiated by UE-A, an IMS session is established between UE-A and UE-B, including the establishment of an RTP channel, a bootstrap data channel, and an application data channel.

[0149] UE-A and UE-B can be considered as communication nodes for making calls. The guidance data channel can be considered as a channel for transmitting guidance data. The application data channel can be considered as a channel for transmitting application data. Guidance data can be considered as data for guiding the implementation of call functions, such as AR call applications and AR call service media data. Application data can be considered as application-related data.

[0150] If UE-A and UE-B do not have the AR call application pre-installed, they need to obtain the AR call application through the guided data channel (HTML method can be used). If UE-A and UE-B have the AR call application pre-installed, there is no need to establish a guided data channel to obtain the AR call application.

[0151] When the AR call application is obtained through the guidance data channel, the preset / pre-stored AR call service media data (such as scene description, 3D model) related to the AR call application can also be obtained.

[0152] Optionally, UE-A may be directly anchored to MF / MRF when establishing the application data channel, and when establishing the application data channel, the AR application server may directly distribute AR call service media data (such as scene description, 3D model) using the application data channel through MF / MRF.

[0153] The AR call service media data can be any type of media data, such as a scene description applicable to the AR call service, a 3D model (object, digital human) stored in the IMS system or AR application server, 2D audio and video data obtained by the terminal device, metadata (such as user posture), etc.

[0154] Optionally, both UE-A-rendered and non-rendered AR call service media data may be transmitted through an RTP channel.

[0155] Step 302: AR data collection / acquisition.

[0156] UE-A collects / obtains AR call service media data, including AR models (objects, people, etc.), user posture information, etc., through one or more devices such as cameras and sensors (which can be integrated into a terminal device).

[0157] Step 303a: AR data rendering and other processing.

[0158] UE-A processes and renders the collected AR call service media data. This step is an optional step. The call service media data collected / obtained by UE-A can be sent directly to UE-B without being processed and rendered, and UE-B can process it by itself.

[0159] Step 303b: Transmitting rendered or unrendered AR data.

[0160] UE-A sends rendered or unrendered AR call service media data (also known as AR data) to UE-B. If the AR call service media data rendered locally by UE-A is only used for local display, that is, there is no need to send the rendered AR data to UE-B, then the order of steps 303a and 303b can be interchanged, or performed in parallel, which will reduce the delay caused by transmitting the unrendered AR call service media data to the receiving UE after local rendering processing.

[0161] For real-time audio and video call service media data (such as 2D audio and video data), RTP channels are usually used for transmission. For non-real-time, overlay, 3D and other data types, application data channels are usually used for transmission.

[0162] Step 304a: AR data collection / acquisition.

[0163] UE-B collects / obtains AR call service media data through one or more devices such as cameras and sensors (which can be integrated into a terminal device). In this step, UE-B's collection behavior can be performed synchronously with UE-A's collection behavior.

[0164] Step 304b: Transmit AR data.

[0165] UE-B transmits the collected AR call service media data to UE-A.

[0166] Optionally, UE-B may also render the image locally and then transmit it to UE-A.

[0167] Step 305: AR data rendering, synthesis, and other processing, and presentation.

[0168] UE-A further processes the AR call service media data received from UE-B, such as rendering a 3D model, 3D audio and video according to the user's posture (position, window), or merging multiple 3D models into one scene. UE-A presents the rendered AR call service media data to the user. If the AR call service media data transmitted by UE-B is already rendered, UE-A may not render it again (such as rendering based on the current window).

[0169] When the transmitting UE sends AR call service media data to the receiving UE, if the transmitting UE has completed rendering, it is necessary to prompt the receiving UE when transmitting the AR data, and the receiving UE decides whether to further render (such as rendering based on the current window). If the transmitting UE has not rendered, it is still necessary to prompt the receiving UE when sending the AR call service media data, and the receiving UE can be further prompted with information such as rendering recommendations (such as recommended windows, rendering object descriptions), and the receiving UE decides how to render.

[0170] When the terminal device has weak capabilities, the terminal device can request the network-side edge server or central server to render, including:

[0171] Before sending the AR call service media data, the sending UE first renders the AR call service media data, and then sends the unrendered and / or rendered AR call service media data to the network side for rendering. After the network side completes the rendering, it can be sent to the receiving UE and returned to the sending UE. This separate rendering method can reduce the amount of transmitted media data, reduce the server's media data cache loading time, etc.

[0172] The sending UE directly sends the AR call service media data that needs to be rendered to the network side for rendering. After rendering is completed, it is returned to the sending UE and can be sent to the receiving UE. In this way, the receiving and sending UEs can perform rendering in parallel, or wait until the network side completes the rendering before performing terminal side rendering. Parallel rendering can reduce the delay caused by rendering.

[0173] Figure 3c This is a flow chart of a network-side initiated split rendering process provided in an embodiment of the present application, see Figure 3c The negotiation process of the separated rendering for AR calls based on IMS (separate rendering sent by the network side) is as follows:

[0174] Step 4001: Establish an IMS session.

[0175] Initiated by UE-A, an IMS session is established between UE-A and UE-B through the IMS system, including an RTP channel and an application data channel.

[0176] If UE-A and UE-B do not have the AR call application pre-installed, they need to obtain the AR call application through the guided data channel (HTML method can be used). If UE-A and UE-B have the AR call application pre-installed, there is no need to establish a guided data channel to obtain the AR call application.

[0177] When the AR call application is obtained through the guidance data channel, the preset / pre-stored AR call service media data (such as scene description, 3D model) related to the AR call application can also be obtained.

[0178] Optionally, UE-A may be directly anchored to MF / MRF when establishing the application data channel, and when establishing the application data channel, the AR application server may directly distribute AR call service media data (such as scene description, 3D model) using the application data channel through MF / MRF.

[0179] The AR call service media data can be any type of media data, such as a scene description applicable to the AR call service, a 3D model (object, digital human) stored in the IMS system or AR application server, 2D audio and video data obtained by the terminal, metadata (such as user posture), etc.

[0180] Optionally, the application data channel can be established after the rendering resource negotiation is completed, or it does not need to be established. In the second method, all AR call service media data is transmitted through the RTP channel.

[0181] Step 4002: The AR application server initiates IMS rendering, and UE-A feeds back a response.

[0182] The AR application server sends a separate rendering negotiation request to UE-A, informing UE-A that separate rendering can be provided (i.e., transmitting a first rendering negotiation request to the communication node to which the call service media data belongs). UE-A returns whether it agrees or disagrees with the negotiation.

[0183] Separate rendering can be considered as collaborative rendering by multiple devices, such as local rendering and collaborative network rendering.

[0184] Optionally, the AR application server may directly request MF / MRF to allocate resources, and then negotiate separate rendering with UE-A according to the allocated resources or inform UE-A of the rendering resource configuration of separate rendering. That is, the first rendering negotiation request is initiated through the application service network element in the Internet protocol multimedia subsystem, the rendering resources are configured through the media function network element in the Internet protocol multimedia subsystem, and the configuration information of the rendering resources is transmitted to UE-A.

[0185] Step 4003: Request rendering resources.

[0186] If UE-A returns and agrees to negotiate separate rendering, the AR application server requests MF / MRF to allocate rendering resources for the features, service types, etc. of UE-A's AR call application; wherein the request message is forwarded to MF / MRF through DCSF and IMS AS.

[0187] Optionally, if the AR application server directly requests the MF / MRF to allocate resources, the process directly proceeds to step 4004 to perform rendering resource negotiation between the AR application server and the MF / MRF.

[0188] Step 4004: MF / MRF configures available rendering resources according to the rendering resource conditions (such as idle computing power), the characteristics of UE-A's AR call service (such as media data type, real-time requirements), service type, etc., including determining the available rendering resource identifier, computing power value, data format, rendering object, rendering type, etc. That is, the rendering resources are configured through the media function network element in the Internet protocol multimedia subsystem.

[0189] Step 4005: Return the allocation result.

[0190] MF / MRF returns the configuration result of the rendering resources to the AR application server.

[0191] Step 4006: Send resource allocation status.

[0192] The MF / MRF sends the configuration information of the rendering resource to UE-A (i.e., transmits the configuration information of the rendering resource), wherein the sending of the configuration information of the rendering resource can be initiated by the AR application server, or can be actively sent by the MF / MRF after completing the configuration of the rendering resource. The configuration information can be sent by request (such as SIP INVITE message).

[0193] Optionally, if there is no negotiation with UE-A in step 4002 as to whether separate rendering is required, a negotiation request (such as a SIP INVITE message) is directly initiated by sending configuration information in step 4006, and the configuration information of the rendering resources is provided in the message. That is, the first rendering negotiation request is initiated by the application service network element in the Internet protocol multimedia subsystem; the rendering resources are configured by the media function network element in the Internet protocol multimedia subsystem; and the configuration information of the rendering resources is transmitted to UE-A.

[0194] Step 4007a: Send audio and video stream anchor redirection.

[0195] The IMS AS initiates re-negotiation and requests that the streaming media transmitted by the RTP channel of UE-A be anchored to the MF / MRF.

[0196] Optionally, when forwarding MF / MRF to complete the rendering resource configuration, the IMS AS may send it in the form of re-negotiation, including the configuration information of the rendering resources, and a request to anchor the streaming media (such as call service media data) transmitted by the RTP channel of UE-A to the MF / MRF, that is, step 4006 is merged with step 4007a.

[0197] Step 4007b: UE-A's audio and video streams are anchored and redirected to MF / MRF, and an RTP channel is established.

[0198] According to UE-A's response to the rendering resource configuration, the IMS AS anchors UE-A's RTP channel to the MF / MRF (ie, the association request for transmitting the call service media data), and UE-A and the MF / MRF establish an RTP channel.

[0199] If the RTP channel transmits call service media data that needs to be rendered, information can be used to indicate that the data transmitted by the RTP channel supports MF / MRF rendering when the channel is established.

[0200] Step 4008: UE-B audio and video streams are anchored and redirected to MF / MRF, and an RTP channel is established.

[0201] The IMS AS initiates re-negotiation to anchor the streaming media transmitted by UE-B's RTP channel to the MF / MRF. UE-B and the MF / MRF establish an RTP channel.

[0202] Figure 3d This is a schematic diagram of a process of initiating separate rendering on the terminal side provided in an embodiment of the present application. The separate rendering negotiation process of an AR call based on IMS (separate rendering sent by the terminal side) is as follows:

[0203] Step 4101: Establish an IMS session.

[0204] Initiated by UE-A, an IMS session is established between UE-A and UE-B through the IMS system, including an RTP channel.

[0205] If UE-A and UE-B do not have the AR call application pre-installed, they need to obtain the AR call application through the guided data channel (HTML method can be used). If UE-A and UE-B have the AR call application pre-installed, there is no need to establish a guided data channel to obtain the AR call application.

[0206] When the AR call application is obtained through the guidance data channel, the preset / pre-stored AR call service media data (such as scene description, 3D model) related to the AR call application can also be obtained.

[0207] Optionally, UE-A may be directly anchored to MF / MRF when establishing the application data channel, and when establishing the application data channel, the AR application server may directly distribute AR call service media data (such as scene description, 3D model) using the application data channel through MF / MRF.

[0208] The AR call service media data can be any type of media data, such as a scene description applicable to the AR call service, a 3D model (object, digital human) stored in the IMS system or AR application server, 2D audio and video data obtained by the terminal device, metadata (such as user posture), etc.

[0209] Optionally, the application data channel can be established after the rendering resource negotiation is completed, or it does not need to be established. In the second method, all AR call service media data is transmitted through the RTP channel.

[0210] Step 4102a: UE-A initiates establishment of a data channel, which includes a rendering request.

[0211] UE-A initiates a request to establish an application data channel between UE-A and MF / MRF and sends a media rendering negotiation request, ie, a second rendering negotiation request, to the IMS system based on the request.

[0212] If the application data channel transmits call service media data that needs to be rendered, when establishing the channel, information can be used to indicate that the data transmitted by the application data channel supports MF / MRF rendering.

[0213] The AR application on UE-A cannot meet the rendering requirements of the application in the AR call based on the client's media capabilities (current status, such as power status, signal status, computing power status, and internal storage status), so the AR application decides to start a separate rendering negotiation process.

[0214] Optionally, UE-A may directly initiate a media rendering negotiation request to the IMS system without initiating an application data channel establishment request, and the request is transmitted to the IMS AS in the IMS system, that is, the second rendering negotiation request is obtained through the application service network element.

[0215] Step 4102b: Forward the rendering request.

[0216] The IMS AS forwards the rendering request (i.e., media rendering negotiation request) to the AR application server. The negotiation request may include information such as the computing power required for rendering, media data description (data type, frequency, size), and rendering type. That is, the second rendering negotiation request is forwarded to the media function network element through the application service network element, and the forwarding can be transferred through the AR application server.

[0217] Step 4103: Send rendering request.

[0218] The AR application server sends a media resource request (i.e., rendering request) to the MF / MRF, requesting the MF / MRF to allocate rendering resources for the AR call service media data or AR call service. The request message is forwarded to the MF / MRF through the DCSF and IMS AS. In addition, it can carry the application data channel address information, audio and video address information of UE-A, and the address information of the receiving end UE-B.

[0219] Optionally, the AR application server may preliminarily decide whether to provide the media rendering function according to the request of UE-A in step 4013 .

[0220] Step 4014: Apply for and confirm that rendering resources are available.

[0221] The MF / MRF configures rendering resources for UE-A based on the received rendering request and the information carried in the request, that is, determines the rendering resources of the call service media data through the media function network element.

[0222] Step 4015: Return the request result.

[0223] When the MF / MRF resource configuration is successful, a success response is returned to the AR application server through the IMS AS, carrying the address information allocated on the MF / MRF. The address information is the address to which the media data to be rendered is transmitted, so as to realize the transmission of the configuration information of the rendering resource.

[0224] Step 4016: Return the rendering request result.

[0225] The AR application server sends a media rendering negotiation response to UE-A, indicating whether the media rendering negotiation is successful or not. The response is forwarded by network elements in the IMS system such as IMS AS and DCSF. The request result will eventually be notified to the renderer and AR application in UE-A. This enables the transmission of the configuration information of the rendering resources.

[0226] Optionally, the request result may be returned via an application data channel between UE-A and MF / MRF.

[0227] If UE-A's request contains specific computing power values, rendering capabilities and other information, and because the rendering resources applied for by MF / MRF are limited, the final resource configuration result (which may be different from the applied resources) may be included in the response returned to UE-A.

[0228] The returned configuration information may include the data format that UE-A needs to input and the media format that MF / MRF may feedback.

[0229] Step 4017: UE-A's audio and video streams are anchored and redirected to MF / MRF, and an RTP channel is established.

[0230] The IMS AS initiates a media renegotiation process to anchor the media stream of UE-A's AR call service media data to the MF / MRF (i.e., an association request for transmitting the call service media data). UE-A sends a response, carrying the address information of UE-A, to complete the establishment of the RTP channel between UE-A and MF / MRF.

[0231] If the IMS network accessed by UE-B is different from the IMS network accessed by UE-A, and if the IMS network accessed by UE-B also provides rendering resources and rendering functions, then after the IMS network accessed by UE-A completes the rendering requirements of UE-A, the media stream of the AR call service media data can be directly transmitted to UE-B; it can also be transmitted to the MF / MRF of the IMS network accessed by UE-B. This requires the network element (MF / MRF) in the IMS network accessed by UE-A and the network element (MF / MRF) in the IMS network accessed by UE-B to anchor the media stream of the AR call service media data. The IMS AS of the IMS network accessed by UE-B can renegotiate with the MF / MRF to anchor the media stream of the AR call service media data transmitted by the IMS network accessed by UE-A to the MF / MRF of the IMS network accessed by UE-B.

[0232] If the MF / MRF of the IMS network accessed by UE-B has received the AR call service media data and completed rendering, the MF / MRF can decide whether to render again based on the negotiated rendering resources, rendering type, and the AR call service media data collected locally and transmitted by UE-B (such as metadata such as posture information).

[0233] If UE-A or the IMS network accessed by UE-A has completed rendering, then when transmitting the AR call service media data to the MF / MRF of the IMS network accessed by UE-B, it is necessary to indicate that the transmitted AR call service media data has completed rendering. Conversely, if the rendering is not completed, it is also necessary to indicate when transmitting the data.

[0234] The AR application server on the network initiates the negotiation of rendering resources for separate rendering. If the rendering resources on the network change, the AR application server on the network or the UE on the terminal can initiate the re-negotiation of resources. The AR application server can also choose to keep the rendering resources fixed.

[0235] During an AR call service, the rendering resources or rendering capabilities on the terminal side or network side may change, or other reasons may require renegotiation of rendering resources, such as changes in the network, changes in rendering charges, image security issues, etc. The rendering resource negotiation for separate rendering initiated by the network-side AR application server or the UE-side terminal, if the rendering resources change, can be initiated by the network-side AR application server or the terminal-side UE to renegotiate the resources. Normally, for rendering resource allocation initiated by the network-side AR application server, the rendering resources remain fixed.

[0236] Figure 3e is a flow chart of a UE initiating re-negotiation provided in an embodiment of the present application. Figure 3e The adaptive rendering re-negotiation process (UE initiated) of AR calls based on IMS is as follows:

[0237] Step 5001: UE-A sends a rendering resource renegotiation.

[0238] UE-A sends a new rendering requirement (ie, the first re-negotiation request) to the AR application server.

[0239] Step 5002: Re-request rendering resources based on the decision.

[0240] The AR application server decides whether to request to add new or reduce rendering resources, and requests the MF / MRF to increase or reduce rendering resources based on the decision.

[0241] Step 5003: Return the request result.

[0242] The MF / MRF returns a response to the AR application server indicating whether the resource allocation is successful or not.

[0243] Step 5004: Return the re-negotiation request result.

[0244] The AR application server feeds back the configuration result of the rendering resources to UE-A, that is, transmits the resource allocation result corresponding to the first re-negotiation request.

[0245] Figure 3f is a flow chart of a network-side initiated renegotiation process provided by an embodiment of the present application. Figure 3f The adaptive rendering re-negotiation process of AR calls based on IMS (initiated by the AR application server) is as follows:

[0246] Step 5101: Rendering resources change, and decide to initiate re-negotiation.

[0247] Based on the acquired MF / MRF resource changes, the AR application server may not be able to meet the current UE-A's rendering requirements, and decides to send a rendering resource renegotiation, that is, to initiate a second renegotiation request for rendering resources through the media function network element or application service network element in the Internet protocol multimedia subsystem.

[0248] Step 5102: The AR application server sends rendering resource renegotiation.

[0249] The AR application server sends a rendering resource renegotiation process, and sends the rendering resource changes and new rendering resources to UE-A, that is, the second renegotiation request includes the resource allocation result.

[0250] Step 5103: Return re-negotiation request information.

[0251] UE-A receives the rendering resource changes and new rendering resources, and returns a response to the AR application server.

[0252] If the rendering resources re-provided by the AR server cannot meet the needs of UE-A, UE-A can request reallocation of rendering resources again based on the current device capabilities; if UE-A's current device capabilities are strong and can complete the rendering by itself, it can cancel the rendering resource configuration provided by the IMS network through response, or it can maintain anchoring with the media processing network element but not transmit data.

[0253] If UE-A does not use the rendering resources provided by the IMS network, the RTP channel or application data channel between UE-A and MF / MRF can be canceled through re-negotiation. UE-A can transmit the rendered or unrendered AR call service media data directly to the receiving end UE-B through IMS-AGW.

[0254] During an AR call service, the UE requests the IMS network to provide rendering resources due to insufficient media capabilities. However, the media capabilities may change dynamically, and because the content or object of the rendering changes, the UE can adaptively choose whether to use the network's rendering resources based on its current state and the content of the service.

[0255] Method 1: At least two media data transmission channels are established between UE-A and UE-B, at least one of which is an RTP channel. The call service media data transmitted through the RTP channel does not pass through the MF / MRF, but is directly transmitted to UE-B (or the network element of the IMS network to which UE-B belongs) through the IMS-AGW of the IMS network. The RTP channel transmits media data of call services that do not require rendering (such as 2D audio and video data); one media data channel is an RTP channel or an application data channel, which is connected from UE-A to the MF / MRF, and the call service media data of UE-A is transmitted to the MF / MRF. After being rendered by the MF / MRF, the MF / MRF data is sent to UE-B (or the network element of the IMS network to which UE-B belongs). The channel transmits media data of call services that need to be rendered (such as 3D models, 2D video data).

[0256] When establishing a media data transmission channel, it is necessary to instruct the media data transmission channel to transmit call service media data that needs to be rendered.

[0257] Method 2: At least one media data transmission channel is established between UE-A and MF / MRF, and the media data channel is an RTP channel or an application data channel. The RTP channel or the application data channel is connected from UE-A to MF / MRF, and the call service media data of UE-A is transmitted to MF / MRF. The rendering indication information in each RTP packet or the rendering indication information carried in other data packets is used to determine whether MF / MRF rendering is required. If rendering is required, it is sent to UE-B (or the network element of the IMS network to which UE-B belongs) after MF / MRF rendering; if rendering is not required, it can be sent directly to UE-B by MF / MRF, or sent together with the call service media data waiting for rendering to UE-B (or the network element of the IMS network to which UE-B belongs).

[0258] The recommended media data transmission channel between UE-B and MF / MRF is usually the RTP channel.

[0259] Method 3: At least two media data transmission channels are established between UE-A and MF / MRF, namely an RTP channel and an application data channel, or both are RTP channels. The RTP channel can only transmit call service media data that needs to be rendered, or transmit any call service media data; the application data channel transmits call service media data that needs to be rendered or is used for rendering.

[0260] At least one media data transmission channel is established between the MF / MRF and UE-B to transmit communication service media data rendered and not rendered by the MF / MRF.

[0261] When establishing a media data transmission channel, it is necessary to instruct the media data transmission channel to transmit call service media data that needs to be rendered.

[0262] This application proposes a method for 5G AR call service. According to the media processing capabilities of the communication terminal and the media processing capabilities of the IMS network, the terminal side or the network side can initiate negotiation of rendering resources to provide rendering for AR communication services. At the same time, through information indication in media data or control signaling, the terminal and the network provide flexible rendering processing, and adaptively and reasonably use rendering resources based on current resource changes. Through this application, resource capabilities can be used efficiently to provide the best AR call experience.

[0263] This application is applicable to real-time transmission business scenarios based on AVS3 video encoding stream, such as video conferencing, video calls, interactive live broadcasts, remote monitoring, etc.

[0264] In an exemplary embodiment, the present application also provides a rendering method. Figure 4 It is a flowchart of another rendering method provided by an embodiment of the present application; the method can be applicable to the case of rendering call service media data. The method can be executed by a rendering device, and the device can be implemented by software and / or hardware and integrated on a communication node. The communication node can be a node with communication function. Such as a terminal device. The contents that are not yet detailed in this embodiment can be referred to the above embodiment, and will not be repeated here.

[0265] like Figure 4 As shown, the rendering method provided in the embodiment of the present application includes the following steps:

[0266] S410 , negotiating rendering resources for rendering call service media data.

[0267] The communication node may negotiate with the Internet Protocol Multimedia Subsystem to determine rendering resources for rendering call service media data.

[0268] In this embodiment, the negotiation may be triggered by a communication node or by an Internet Protocol multimedia subsystem.

[0269] S420: Associate the call service media data with the media function network element to which the media resource belongs.

[0270] This embodiment can associate call media service data with the media function network element, thereby anchoring the call service media data to the media function network element.

[0271] S430: Transmit the call service media data to the media function network element.

[0272] This embodiment transmits the call service media data to the media function network element, so that the media function network element of the Internet Protocol Multimedia Subsystem can render the call service media data.

[0273] The present application provides a rendering method, in which a communication node negotiates rendering resources, and determines the rendering resources of the communication node's call service media data. After the rendering resources are determined, the call service media data of the communication node can be rendered through the rendering resources in the Internet protocol multimedia subsystem. The network side negotiates rendering resources and configures rendering resources. The call service media data is rendered separately, which improves the rendering efficiency of the call service media data.

[0274] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0275] In one embodiment, the negotiating rendering resources for rendering call service media data includes:

[0276] Get configuration information of the rendering resource.

[0277] The configuration information may be transmitted by the Internet Protocol Multimedia Subsystem. After obtaining the configuration information, the communication node may determine whether to use the configured rendering resources to render the call service media data.

[0278] In one embodiment, the negotiating rendering resources for rendering call service media data includes:

[0279] Transmitting a second rendering negotiation request.

[0280] In this embodiment, the second rendering negotiation request may be transmitted to the Internet protocol multimedia subsystem.

[0281] In one embodiment, the rendering method further includes:

[0282] Media data and / or first indication information are transmitted via a real-time transport protocol channel and / or an application data channel with an Internet protocol multimedia subsystem, wherein the first indication information indicates a rendering status of the media data, and the media data includes rendered data and / or unrendered data.

[0283] In one embodiment, the rendering method further includes:

[0284] Renegotiate rendering resources.

[0285] The renegotiation of rendering resources may be initiated by the communication node or by the Internet Protocol Multimedia Subsystem.

[0286] In one embodiment, the renegotiating rendering resources includes:

[0287] Transmit the first negotiation request for rendering resources;

[0288] Obtain a resource allocation result of the first re-negotiation request.

[0289] This embodiment enables the communication node to initiate a first re-negotiation request to obtain a resource allocation result of the Internet Protocol Multimedia Subsystem.

[0290] In one embodiment, the renegotiating rendering resources includes:

[0291] Get the second negotiation request for rendering resources;

[0292] The second negotiation request includes a resource allocation result.

[0293] This embodiment implements the Internet protocol multimedia subsystem initiating a second negotiation request. Determine whether to use the allocated rendering resources based on the resource allocation result.

[0294] In an exemplary embodiment, the present application provides a rendering device, which can be integrated into an Internet protocol multimedia subsystem. Figure 5 is a schematic diagram of the structure of a rendering device provided in an embodiment of the present application; Figure 5 As shown, a rendering device provided in an embodiment of the present application includes:

[0295] The negotiation module 510 is configured to negotiate the rendering resources for rendering call service media data; the transmission module 520 is configured to transmit the call service media data to the media function network element to which the rendering resources belong; the rendering module 530 is configured to render the call service media data based on the rendering resources.

[0296] The rendering device provided in this embodiment is used to implement Figure 1 The rendering method of the embodiment shown in the figure, the rendering device provided in this embodiment realizes the same principle and technical effect as Figure 1 The rendering method of the illustrated embodiment is similar and will not be described in detail here.

[0297] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0298] In one embodiment, the negotiation module includes:

[0299] An initiating unit, configured to initiate a first rendering negotiation request;

[0300] Configuration unit, set to configure rendering resources;

[0301] A transmission unit is configured to transmit the configuration information of the rendering resource.

[0302] In one embodiment, the initiating unit is specifically configured as:

[0303] Initiating the first rendering negotiation request through an application service network element in the Internet protocol multimedia subsystem; or

[0304] Transmitting a first rendering negotiation request to the communication node to which the call service media data belongs.

[0305] In one embodiment, the configuration unit is specifically configured as:

[0306] The rendering resources are configured through the media function network element in the Internet protocol multimedia subsystem.

[0307] In one embodiment, the negotiation module includes:

[0308] An acquiring unit, configured to acquire a second rendering negotiation request;

[0309] A determining unit, configured to determine the rendering resource of the call service media data based on the second rendering negotiation request;

[0310] A transmission unit is configured to transmit the configuration information of the rendering resource.

[0311] In one embodiment, the acquisition unit is specifically configured as:

[0312] The second rendering negotiation request is obtained through the application service network element.

[0313] In one embodiment, the determining unit is specifically configured as:

[0314] forwarding the second rendering negotiation request to the media function network element through the application service network element;

[0315] The rendering resource of the call service media data is determined by the media function network element.

[0316] In one embodiment, the transmission module is specifically configured as follows:

[0317] An association request for the call service media data is transmitted, wherein the association request indicates that the call service media data is associated with the media function network element in the Internet Protocol Multimedia Subsystem.

[0318] In one embodiment, the transmission module is specifically configured as follows:

[0319] The association request of the call service media data is transmitted to the media function network element through the application service network element.

[0320] In one embodiment, the rendering device further includes: a data transmission module configured to:

[0321] The call service media data and / or the first indication information are transmitted via a real-time transport protocol channel and / or an application data channel, wherein the first indication information indicates the rendering status of the call service media data, and the call service media data includes rendered data and / or unrendered data.

[0322] In one embodiment, the rendering device further includes: a renegotiation module configured to:

[0323] Renegotiate rendering resources.

[0324] In one embodiment, the renegotiation module is specifically configured as follows:

[0325] Get the first negotiation request for rendering resources;

[0326] Transmitting a resource allocation result corresponding to the first re-negotiation request.

[0327] In one embodiment, the renegotiation module is specifically configured as follows:

[0328] Initiate a second negotiation request for rendering resources through a media function network element or an application service network element in an Internet protocol multimedia subsystem;

[0329] The second negotiation request includes a resource allocation result.

[0330] In one embodiment, the media function network element in the Internet Protocol Multimedia Subsystem allocates resources based on the first re-negotiation request or the second re-negotiation request.

[0331] In one embodiment, the rendering module is specifically configured as follows:

[0332] Transmitting the call service media data to the rendering resources allocated by the media function network element of the Internet Protocol Multimedia Subsystem through the real-time transport protocol channel and / or the application data channel;

[0333] The call service media data is rendered using rendering resources allocated by the media function network element.

[0334] In one embodiment, the call service media data includes at least one of two-dimensional audio and video data, three-dimensional audio and video data, three-dimensional model, scene description, and augmented reality metadata in an augmented reality call scenario.

[0335] In an exemplary embodiment, the present application also provides a rendering device, which can be integrated in a communication node. Figure 6 : is a schematic diagram of a rendering device provided in an embodiment of the present application. The rendering device includes:

[0336] A negotiation module 610, configured to negotiate rendering resources for rendering call service media data;

[0337] An association module 620 is configured to associate the call service media data with a media function network element to which the media resource belongs;

[0338] The transmission module 630 is configured to transmit the call service media data to the media function network element.

[0339] The rendering device provided in this embodiment is used to implement Figure 4 The rendering method of the embodiment shown in the figure, the information determination device provided in this embodiment implements the principle and technical effect of Figure 4 The information determination method of the illustrated embodiment is similar and will not be described again here.

[0340] Based on the above embodiment, a variant embodiment of the above embodiment is proposed. It should be noted that in order to make the description concise, only the differences from the above embodiment are described in the variant embodiment.

[0341] In one embodiment, the negotiation module is specifically configured as follows:

[0342] Get configuration information of the rendering resource.

[0343] In one embodiment, the negotiation module is specifically configured as follows:

[0344] Transmitting a second rendering negotiation request.

[0345] In one embodiment, the rendering device further includes: a data transmission module configured to:

[0346] Media data and / or first indication information are transmitted via a real-time transport protocol channel and / or an application data channel with an Internet protocol multimedia subsystem, wherein the first indication information indicates a rendering status of the media data, and the media data includes rendered data and / or unrendered data.

[0347] In one embodiment, the rendering device further includes: a renegotiation module configured to:

[0348] Renegotiate rendering resources.

[0349] In one embodiment, the renegotiation module is specifically configured as follows:

[0350] Transmit the first negotiation request for rendering resources;

[0351] Obtain a resource allocation result of the first re-negotiation request.

[0352] In one embodiment, the renegotiation module is specifically configured as follows:

[0353] Get the second negotiation request for rendering resources;

[0354] The second negotiation request includes a resource allocation result.

[0355] In an exemplary implementation, the present application also provides an Internet protocol multimedia subsystem. Figure 7 : is a schematic diagram of the structure of an Internet protocol multimedia subsystem provided by an embodiment of the present application; as shown in the figure, the Internet protocol multimedia subsystem provided by the present application includes:

[0356] One or more processors 71 and storage device 72; the processor 71 in the Internet protocol multimedia subsystem can be one or more, Figure 7 In the example, a processor 71 is used; the storage device 72 is used to store one or more programs; the one or more programs are executed by the one or more processors 71, so that the one or more processors 71 implement the rendering method as described in the embodiment of the present application. The processor 71 can be a network element that executes the corresponding method in the Internet protocol multimedia subsystem or a processor in the network element. No specific limitation is made here.

[0357] The Internet Protocol multimedia subsystem also includes: a communication device 73, an input device 74 and an output device 75.

[0358] The processor 71, storage device 72, communication device 73, input device 74 and output device 75 in the Internet protocol multimedia subsystem can be connected through a bus or other means. Figure 7 The example of connecting through bus is taken in the following.

[0359] The input device 74 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the Internet protocol multimedia subsystem. The output device 75 may include display devices such as a display screen.

[0360] The communication device 73 may include a receiver and a transmitter. The communication device 73 is configured to perform information transmission and reception communication according to the control of the processor 71.

[0361] The storage device 72, as a computer-readable storage medium, can be configured to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the rendering method described in the embodiment of the present application (for example, the negotiation module 510, the transmission module 520 and the rendering module 530 in the rendering device). The storage device 72 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application required for at least one function; the storage data area may store data created according to the use of the Internet protocol multimedia subsystem, etc. In addition, the storage device 72 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 72 may further include a memory remotely arranged relative to the processor 71, and these remote memories may be connected to the Internet protocol multimedia subsystem via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0362] In an exemplary embodiment, the present application also provides a communication node, Figure 8 1 is a schematic diagram of a communication node provided in an embodiment of the present application. As shown in the figure, the communication node includes:

[0363] One or more processors 81 and storage device 82; the processor 81 in the communication node may be one or more, Figure 8 A processor 81 is taken as an example; the storage device 82 is used to store one or more programs; the one or more programs are executed by the one or more processors 81, so that the one or more processors 81 implement the rendering method as described in the embodiment of the present application.

[0364] The communication node further comprises: a communication device 83 , an input device 84 and an output device 85 .

[0365] The processor 81, storage device 82, communication device 83, input device 84 and output device 85 in the communication node may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0366] The input device 84 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the communication node. The output device 85 may include display devices such as a display screen.

[0367] The communication device 83 may include a receiver and a transmitter. The communication device 83 is configured to perform information transmission and reception communication according to the control of the processor 81.

[0368] As a computer-readable storage medium, the storage device 82 may be configured to store software programs, computer executable programs, and modules, such as program instructions / modules corresponding to the rendering method described in the embodiment of the present application (e.g., the negotiation module 610, the association module 620, and the transmission module 630 in the rendering device). The storage device 82 may include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application required for at least one function; the storage data area may store data created according to the use of the communication node, etc. In addition, the storage device 82 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the storage device 82 may further include a memory remotely arranged relative to the processor 81, and these remote memories may be connected to the communication node via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0369] In an exemplary embodiment, the present application also provides a storage medium, the storage medium stores a computer program, the computer program is executed by a processor to implement any of the rendering methods described in the present application, and the storage medium stores a computer program, the computer program is executed by a processor to implement any of the rendering methods described in the present application. Such as a rendering method applied to an Internet protocol multimedia subsystem and a rendering method applied to a communication node, wherein the rendering method applied to an Internet protocol multimedia subsystem includes: negotiating rendering resources for rendering call service media data;

[0370] Transmitting the call service media data to the media function network element to which the rendering resource belongs;

[0371] The call service media data is rendered based on the rendering resource.

[0372] The rendering method applied to the communication node includes: negotiating rendering resources for rendering call service media data;

[0373] Associating the call service media data with the media function network element to which the media resource belongs;

[0374] Transmit the call service media data to the media function network element.

[0375] The computer storage medium of the embodiment of the present application can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (Random Access Memory, RAM), a read-only memory (Read Only Memory, ROM), an erasable programmable read-only memory (ErasableProgrammable Read Only Memory, EPROM), a flash memory, an optical fiber, a portable CD-ROM, an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, a device or a device or used in combination with it.

[0376] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to: electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0377] The program code contained on the computer-readable medium may be transmitted using any appropriate medium, including but not limited to: wireless, wire, optical cable, radio frequency (RF), etc., or any suitable combination of the above.

[0378] The computer program code for performing the operation of the present application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and also conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on the remote computer, or completely on the remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0379] The above description is merely an exemplary embodiment of the present application and is not intended to limit the protection scope of the present application.

[0380] It will be appreciated by those skilled in the art that the term terminal equipment covers any suitable type of wireless user equipment, such as a mobile phone, a portable data processing device, a portable web browser or a car-mounted mobile station.

[0381] In general, various embodiments of the present application can be implemented in hardware or dedicated circuits, software, logic or any combination thereof. For example, some aspects can be implemented in hardware, while other aspects can be implemented in firmware or software that can be executed by a controller, microprocessor or other computing device, although the present application is not limited thereto.

[0382] Embodiments of the present application may be implemented by executing computer program instructions by a data processor of a mobile device, for example in a processor entity, or by hardware, or by a combination of software and hardware. The computer program instructions may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages.

[0383] The block diagram of any logic flow in the drawings of the present application may represent program steps, or may represent interconnected logic circuits, modules and functions, or may represent a combination of program steps and logic circuits, modules and functions. A computer program may be stored on a memory. The memory may have any type suitable for the local technical environment and may be implemented using any suitable data storage technology, such as but not limited to read-only memory (ROM), random access memory (RAM), optical memory devices and systems (digital versatile discs (DVD) or compact disks (CD)), etc. Computer-readable media may include non-transient storage media. A data processor may be any type suitable for the local technical environment, such as but not limited to a general-purpose computer, a special-purpose computer, a microprocessor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FGPAs) and a processor based on a multi-core processor architecture.

[0384] By way of exemplary and non-limiting examples, a detailed description of exemplary embodiments of the present application has been provided above. However, various modifications and adjustments to the above embodiments will be apparent to those skilled in the art when considered in conjunction with the accompanying drawings and claims, but will not depart from the scope of the present application. Therefore, the proper scope of the present application will be determined according to the claims.

Claims

1. A rendering method, characterized in that: include: Negotiate rendering resources for rendering call service media data; Transmitting the call service media data to the media function network element to which the rendering resource belongs; The call service media data is rendered based on the rendering resource.

2. The method according to claim 1, characterized in that The negotiation of rendering resources for rendering call service media data includes: Initiate the first rendering negotiation request; Configure rendering resources; Transmitting configuration information of the rendering resource.

3. The method according to claim 2, characterized in that The initiating a first rendering negotiation request includes: Initiating the first rendering negotiation request through an application service network element in the Internet protocol multimedia subsystem; or Transmitting a first rendering negotiation request to the communication node to which the call service media data belongs.

4. The method according to claim 2, characterized in that: The configuration rendering resource includes: The rendering resources are configured through the media function network element in the Internet protocol multimedia subsystem.

5. The method according to claim 1, characterized in that The negotiation of rendering resources for rendering call service media data includes: Get the second rendering negotiation request; Determining the rendering resource of the call service media data based on the second rendering negotiation request; Transmitting configuration information of the rendering resource.

6. The method according to claim 5, characterized in that The obtaining of the second rendering negotiation request includes: The second rendering negotiation request is obtained through the application service network element.

7. The method according to claim 5, characterized in that Determining the rendering resource of the call service media data based on the second rendering negotiation request includes: forwarding the second rendering negotiation request to the media function network element through the application service network element; The rendering resource of the call service media data is determined by the media function network element.

8. The method according to claim 1, characterized in that The transmitting the call service media data to the media function network element to which the rendering resource belongs includes: An association request for the call service media data is transmitted, wherein the association request indicates that the call service media data is associated with the media function network element in the Internet Protocol Multimedia Subsystem.

9. The method according to claim 8, characterized in that The association request for transmitting the call service media data includes: The association request of the call service media data is transmitted to the media function network element through the application service network element.

10. The method according to claim 1, characterized in that Also includes: The call service media data and / or the first indication information are transmitted via a real-time transport protocol channel and / or an application data channel, wherein the first indication information indicates the rendering status of the call service media data, and the call service media data includes rendered data and / or unrendered data.

11. The method according to claim 1, characterized in that: Also includes: Renegotiate rendering resources.

12. The method according to claim 11, characterized in that The re-negotiation rendering resources include: Get the first negotiation request for rendering resources; Transmitting a resource allocation result corresponding to the first re-negotiation request.

13. The method according to claim 11, characterized in that The re-negotiation rendering resources include: Initiate a second negotiation request for rendering resources through a media function network element or an application service network element in an Internet protocol multimedia subsystem; The second negotiation request includes a resource allocation result.

14. The method according to claim 12 or 13, characterized in that The media function network element in the Internet Protocol Multimedia Subsystem allocates resources based on the first re-negotiation request or the second re-negotiation request.

15. The method according to claim 1, characterized in that Rendering the call service media data based on the rendering resource includes: Transmitting the call service media data to the rendering resources allocated by the media function network element of the Internet Protocol Multimedia Subsystem through the real-time transport protocol channel and / or the application data channel; The call service media data is rendered using rendering resources allocated by the media function network element.

16. The method according to any one of claims 1 to 15, characterized in that: The call service media data includes at least one of two-dimensional audio and video data, three-dimensional audio and video data, three-dimensional model, scene description, and augmented reality metadata in an augmented reality call scenario.

17. A rendering method, characterized in that: include: Negotiate rendering resources for rendering call service media data; Associating the call service media data with the media function network element to which the media resource belongs; Transmit the call service media data to the media function network element.

18. The method according to claim 17, characterized in that The negotiation of rendering resources for rendering call service media data includes: Get configuration information of the rendering resource.

19. The method according to claim 17, characterized in that The negotiation of rendering resources for rendering call service media data includes: Transmitting a second rendering negotiation request.

20. The method according to claim 17, characterized in that Also includes: Media data and / or first indication information are transmitted via a real-time transport protocol channel and / or an application data channel with an Internet Protocol multimedia subsystem, wherein the first indication information indicates a rendering status of the media data, and the media data includes rendered data and / or unrendered data.

21. The method according to claim 17, characterized in that Also includes: Renegotiate rendering resources.

22. The method according to claim 21, characterized in that The re-negotiation rendering resources include: Transmit the first negotiation request for rendering resources; Obtain a resource allocation result of the first re-negotiation request.

23. The method according to claim 21, characterized in that The re-negotiation rendering resources include: Get the second negotiation request for rendering resources; The second negotiation request includes a resource allocation result.

24. An Internet protocol multimedia subsystem, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 16.

25. A communication node, characterized in that: include: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 17 to 23.

26. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 23 is implemented.

Images