Collaborative rendering method and device, computer equipment and readable storage medium

Abstract

The invention relates to a collaborative rendering method and device, computer equipment and a readable storage medium, and is applied to a scene of deploying a real-time interaction application based on an IP multimedia subsystem, and the method comprises the steps: obtaining the rendering permission information of a terminal and the business demand type of the real-time interaction application; determining a terminal type and terminal rendering capability information according to a data channel establishment request initiated by a terminal; determining a rendering mode according to the rendering permission information, the service requirement type, the terminal type and the terminal rendering capability information under the condition that a message that the data channel is successfully established is received; and sending a determination result and the rendering media data to a rendering execution body corresponding to the rendering mode, so that the rendering execution body executes rendering operation on the rendering media data and a user video stream collected by the terminal. The method supports the dynamic selection (terminal or network) of the rendering function entity in the execution process of IMS DC real-time interactive application and other services through an application program data channel.

Description

Technical Field

[0001] The present application relates to the field of image rendering technology, and in particular to a collaborative rendering method, apparatus, computer equipment, computer-readable storage medium, and computer program product. Background Art

[0002] Providing digital human and extended reality (XR) services based on the IP Multimedia Sybsystem Data Channel (IMSDC) is a key development direction for IMS DC applications. However, there is no solution for implementing collaborative rendering during the execution of real-time interactive applications such as XR and digital humans when deploying them based on IMS DC. Summary of the Invention

[0003] Based on this, it is necessary to provide a collaborative rendering method, apparatus, computer device, computer-readable storage medium and computer program product to address the above technical issues.

[0004] In a first aspect, the present application provides a collaborative rendering method. This method is applied to a scenario where a real-time interactive application is deployed based on an IP multimedia subsystem. In the architecture of the IP multimedia subsystem, a data channel signaling function network element manages the establishment of a data channel, and multimedia communication services are implemented through a first application server and a media gateway. The real-time interactive application provides application services through a second application server. The method comprises:

[0005] Obtaining rendering permission information of the terminal and the business requirement type of the real-time interactive application; determining the terminal type and terminal rendering capability information of the terminal based on a data channel establishment request initiated by the terminal; the data channel establishment request is a request to establish an application data channel with the media gateway;

[0006] Upon receiving a message that the data channel is successfully established, determining a rendering mode according to the rendering permission information, the service requirement type, the terminal type, and the terminal rendering capability information; the rendering mode includes terminal-side rendering and network-side rendering;

[0007] The determination result and the rendering media data are sent to a rendering execution body corresponding to the rendering mode; the rendering execution body is used to perform a rendering operation on the rendering media data and the user video stream collected by the terminal.

[0008] In one embodiment, determining the terminal type and terminal rendering capability information of the terminal according to the data channel establishment request initiated by the terminal includes:

[0009] If the data channel establishment request carries terminal rendering capability information, determining that the terminal type is the first type, and extracting the terminal rendering capability information;

[0010] If the data channel establishment request does not carry the terminal rendering capability information, the terminal type is determined to be the second type, and the terminal rendering capability information is determined to be no rendering capability.

[0011] In one embodiment, determining the rendering mode according to the rendering permission information, the business requirement type, the terminal type, and the terminal rendering capability information includes:

[0012] determining whether it is necessary to send a rendering capability query request to the terminal according to the rendering permission information, the business requirement type, the terminal type, and the terminal rendering capability information;

[0013] If not required, determining that the rendering mode is network-side rendering;

[0014] If necessary, a rendering capability query request is sent to the terminal, and the rendering mode is determined according to the rendering capability query result returned by the terminal.

[0015] In one embodiment, determining whether it is necessary to send a rendering capability query request to the terminal based on the rendering permission information, the business requirement type, the terminal type, and the terminal rendering capability information includes:

[0016] When any one of the rendering permission information, the business requirement type, the terminal type, and the terminal rendering capability information does not meet corresponding conditions, determining that there is no need to send a rendering capability query request to the terminal;

[0017] When each of the rendering permission information, the business requirement type, the terminal type, and the terminal rendering capability information satisfies corresponding conditions, it is determined that a rendering capability query request needs to be sent to the terminal.

[0018] In one embodiment, the rendering capability query request sent to the terminal carries a rendering resource requirement; and the terminal is configured to determine a rendering capability query result according to the rendering resource requirement and the real-time performance of the terminal.

[0019] In one embodiment, the method further comprises:

[0020] If the rendering mode is network-side rendering, determining that the corresponding rendering execution body is a media gateway;

[0021] Sending the determination result to the terminal to instruct the terminal to collect a user video stream and send the user video stream to the media gateway;

[0022] And, sending the determination result and the rendered media data to the media gateway; the media gateway renders the user video stream according to the rendered media data to obtain a rendered video stream.

[0023] In one embodiment, the method further comprises:

[0024] If the rendering mode is terminal-side rendering, determining that the corresponding rendering execution entity is the terminal;

[0025] The determination result and the rendered media data are sent to the terminal; the terminal renders the user video stream according to the rendered media data to obtain a rendered video stream, and sends the rendered video stream to the media gateway.

[0026] In one embodiment, after receiving the data channel establishment request initiated by the terminal, the method further includes:

[0027] forwarding the data channel establishment request to the second application server; the second application server is configured to determine a data channel establishment strategy according to the data channel establishment request;

[0028] receiving a data channel establishment policy returned by the second application server and forwarding the data channel establishment policy to the first application server; the first application server is configured to initiate a resource reservation request to the media gateway according to the data channel establishment policy, and after receiving a response from the media gateway indicating successful resource reservation, send a message indicating successful data channel establishment;

[0029] Upon receiving a message from the first application server indicating that the data channel is successfully established, a rendering mode is determined according to the rendering permission information, the service requirement type, the terminal type, and the terminal rendering capability information.

[0030] In a second aspect, the present application further provides a collaborative rendering device, which is applied to a scenario where a real-time interactive application is deployed based on an IP multimedia subsystem. In the architecture of the IP multimedia subsystem, the establishment of a data channel is managed by a data channel signaling function network element, and multimedia communication services are implemented through a first application server and a media gateway; the real-time interactive application provides application services through a second application server; the device includes:

[0031] an information acquisition unit configured to acquire rendering permission information of a terminal and a service requirement type of the real-time interactive application; and determine the terminal type and terminal rendering capability information of the terminal according to a data channel establishment request initiated by the terminal; the data channel establishment request being a request to establish an application data channel with the media gateway;

[0032] a mode determination unit, configured to, upon receiving a message indicating successful establishment of a data channel, determine a rendering mode based on the rendering permission information, the service requirement type, the terminal type, and the terminal rendering capability information; the rendering mode includes terminal-side rendering and network-side rendering;

[0033] The result sending unit is used to send the determination result and the rendering media data to the rendering execution body corresponding to the rendering mode; the rendering execution body is used to perform a rendering operation on the rendering media data and the user video stream collected by the terminal.

[0034] In a third aspect, the present application further provides a computer device, wherein the computer device comprises a memory and a processor, wherein the memory stores a computer program, and the processor implements any of the above methods when executing the computer program.

[0035] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above methods.

[0036] In a fifth aspect, the present application further provides a computer program product, wherein the computer program product comprises a computer program, and when the computer program is executed by a processor, the method described in any one of the above items is implemented.

[0037] The above-mentioned collaborative rendering method, device, computer equipment, storage medium and computer program product support the dynamic selection of rendering functional entities (terminal or network) during the execution of services such as IP Multimedia Subsystem (IMS), Data Channel (DC), real-time interactive applications (such as XR) through application data channels, solving the problem that existing technical solutions do not support collaborative rendering on the network side and terminal side, and can meet the business needs of providing IMS DC users with real-time interactive application services such as XR, improve user experience and promote business development. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 A diagram illustrating an application environment of a collaborative rendering method in one embodiment;

[0039] Figure 2 1 is a flow chart of a collaborative rendering method according to an embodiment;

[0040] Figure 3 A schematic flow chart of a rendering mode determination step in one embodiment;

[0041] Figure 4 This is a schematic diagram of the interaction process of end-network collaborative rendering in an embodiment, taking the real-time interactive application as an XR application as an example;

[0042] Figure 5 is a structural block diagram of a collaborative rendering device in one embodiment;

[0043] Figure 6 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0045] It should be noted that the terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0046] The collaborative rendering method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown. The application scenario of this application is: based on the data channel (DataChannel) of the IP Multimedia Sybsystem (IMS), real-time interactive applications such as Extended Reality (XR) and digital human communication are deployed on the basis of existing audio and video services to improve user experience and promote business development. In the architecture of the IP Multimedia Subsystem (IMS), the establishment of the data channel is managed by the data channel signaling function network element, and the multimedia communication service is realized through the first application server and the media gateway; the real-time interactive application provides application services through the second application server.

[0047] The Data Channel Signaling Function (DCSF) is used to manage the establishment of IMS Data Channels, trigger IMS Data Channel services, and provide a northbound interface for IMS DC services to call.

[0048] The first application server is a Multimedia Telephony Application Server (MMTel-AS), which is used to provide multimedia telephony services.

[0049] The second application server is a real-time interactive application server, such as an XR application server (XR-AS).

[0050] In the application scenario of this application, terminal 102 communicates with first application server 104 via a network. First application server 104 communicates with data channel signaling function 106 (DCSF) and media gateway 108 via the network. DCSF 106 communicates with media gateway 108 and second application server 110 via the network. DCSF 106 and terminal 102 do not directly exchange information; instead, media gateway 108 acts as an intermediary to forward information and achieve interaction.

[0051] A rendering execution module is provided in the terminal 102 , and the operations performed by the terminal in the embodiment of the present application are performed by the rendering execution module.

[0052] A rendering decision module is provided in the data channel signaling function network element 106 (DCSF). In the embodiment of the present application, the operations performed by the data channel signaling function network element 106 are performed through the rendering decision module.

[0053] The terminal 102 may be referred to as a DC terminal, and may be, but is not limited to, various personal computers, laptops, smartphones, tablet computers, IoT devices, and portable wearable devices. IoT devices may include smart speakers, smart TVs, smart air conditioners, and smart car devices. Portable wearable devices may include smart watches, smart bracelets, and head-mounted devices.

[0054] In one embodiment, Figure 2 As shown, a collaborative rendering method is provided, which is applied to Figure 1 Taking the rendering decision module in the data channel signaling function network element 106 (DCSF) in FIG. 1 as an example, the following steps are included:

[0055] Step S210 , obtaining rendering permission information of the terminal and the service requirement type of the real-time interactive application; and determining the terminal type and terminal rendering capability information of the terminal according to the data channel establishment request initiated by the terminal.

[0056] The data channel establishment request is a request for establishing an application data channel with a media gateway.

[0057] Rendering permission information indicates whether the user allows terminal rendering, including both "allowed" and "not allowed." Business requirement type indicates whether the business requires rendering, including both "yes" and "no." Terminal type indicates whether the terminal is a traditional terminal, including both "traditional" and "non-traditional." Terminal rendering capability information indicates whether the terminal supports rendering, including both "yes" and "no."

[0058] In specific implementations, users can set whether to allow the terminal to render based on their needs. If not, rendering operations cannot be performed through the terminal. Different real-time interactive applications (such as XR, digital human communication, etc.) have different business requirements, some requiring rendering operations and some not. Different terminals also have different rendering capabilities. Therefore, before confirming the rendering mode, you can first obtain the terminal's rendering permission information and the business requirement type of the deployed real-time interactive application. At the same time, the terminal type and terminal rendering capabilities can be determined based on whether the data channel establishment request initiated by the terminal carries terminal rendering capability information.

[0059] Step S220: upon receiving a message indicating successful establishment of the data channel, determining a rendering mode based on rendering permission information, business requirement type, terminal type, and terminal rendering capability information; the rendering modes include terminal-side rendering and network-side rendering.

[0060] In specific implementations, different rendering modes correspond to different requirements. Therefore, the conditions for terminal-side rendering and the requirements for network-side rendering can be determined in advance. When the rendering permission information, business requirement type, terminal type, and terminal rendering capability information meet the requirements for terminal-side rendering, the rendering mode is determined to be terminal-side rendering; when the rendering permission information, business requirement type, terminal type, and terminal rendering capability information determine that the rendering mode meets the requirements for network-side rendering, the rendering mode is determined to be network-side rendering. Among them, whether it is terminal-side rendering or network-side rendering, the rendering permission information, business requirement type, terminal type, and terminal rendering capability information can each have corresponding requirements.

[0061] Step S230 , sending the determination result and the rendered media data to a rendering execution body corresponding to the rendering mode; the rendering execution body is used to perform a rendering operation on the rendered media data and the user video stream collected by the terminal.

[0062] The rendered media data refers to media data used for rendering, that is, media data added to the video stream.

[0063] In a specific implementation, when the rendering mode is terminal-side rendering, the rendering executor is the terminal; when the rendering mode is network-side rendering, the rendering executor is the media gateway. When the rendering mode is determined, the determination result and the rendered media data can be sent to the rendering executor corresponding to the rendering mode, so that the rendering executor performs rendering operations on the user video stream collected by the terminal based on the rendered media data.

[0064] In the above-mentioned collaborative rendering method, the rendering permission information of the terminal and the business requirement type of the real-time interactive application are obtained; the terminal type and terminal rendering capability of the terminal are determined based on the data channel establishment request initiated by the terminal; when a message that the data channel is successfully established is received, the rendering mode is determined to be terminal-side rendering or network-side rendering based on the rendering permission information, business requirement type, terminal type and terminal rendering capability information; the determination result and rendering media data are sent to the rendering executor corresponding to the rendering mode, and the rendering executor performs rendering operations on the user video stream collected by the terminal based on the rendering media data. The end-network collaborative rendering method provided in this embodiment supports the dynamic selection of rendering functional entities (terminal or network) during the execution of services such as IP Multimedia Subsystem (IMS), Data Channel (DC), and real-time interactive applications (such as XR) through application data channels. This solves the problem that existing technical solutions do not support network-side and terminal-side collaborative rendering, and can meet the business needs of providing real-time interactive application services such as XR for IMS DC users, thereby improving user experience and promoting business development.

[0065] In an exemplary embodiment, the above-mentioned step S210 determines the terminal type and terminal rendering capability information of the terminal based on the data channel establishment request initiated by the terminal, including: if the data channel establishment request carries the terminal rendering capability information, then the terminal type is determined to be the first type, and the terminal rendering capability information is extracted; if the data channel establishment request does not carry the terminal rendering capability information, then the terminal type is determined to be the second type, and the terminal rendering capability information is determined to be no rendering capability.

[0066] In a specific implementation, when the terminal initiates an IMS application data channel (ADC) establishment request, the newly added rendering execution module in the terminal can insert information such as the terminal rendering capability into the data channel establishment request. After the data channel establishment request is transmitted to the data channel signaling function network element (DCSF), the rendering decision module in the data channel signaling function network element (DCSF) checks whether the data channel establishment request carries the terminal rendering capability information. If the data channel establishment request carries the terminal rendering capability information, the terminal type is marked as the first type, that is, a non-traditional terminal, and the terminal rendering capability information is extracted from the data channel establishment request. If the data channel establishment request does not carry the terminal rendering capability information, the terminal type is marked as the second type, that is, a traditional terminal, and the terminal rendering capability information is marked as having no rendering capability.

[0067] In this embodiment, a rendering execution module newly added to the terminal is used to add the terminal rendering capability information to the data channel establishment request, so that the data channel signaling function network element (DCSF) can determine the terminal type and terminal rendering capability information based on whether the data channel establishment request carries the terminal rendering capability information, thereby serving as a basis for determining the subsequent rendering mode.

[0068] In an exemplary embodiment, Figure 3 As shown, the above step S220 determines the rendering mode according to the rendering permission information, the business requirement type, the terminal type and the terminal rendering capability information, including:

[0069] Step S221: determining whether it is necessary to send a rendering capability query request to the terminal based on the rendering permission information, the business requirement type, the terminal type, and the terminal rendering capability information;

[0070] Step S222: If not required, determine the rendering mode as network-side rendering;

[0071] Step S223: If necessary, a rendering capability query request is sent to the terminal, and a rendering mode is determined according to the rendering capability query result returned by the terminal.

[0072] In the specific implementation, when determining the rendering mode, in addition to considering whether rendering is possible at the terminal, it is also necessary to consider whether the terminal can meet the rendering requirements. When it is determined that rendering can be performed at the terminal based on the rendering permission information, business requirement type, terminal type, and terminal rendering capability information, it is necessary to further send a rendering capability query request to the terminal to determine whether the terminal can meet the rendering requirements. However, if it is determined that rendering is not possible at the terminal based on the rendering permission information, business requirement type, terminal type, and terminal rendering capability information, it is not necessary to send a rendering capability query request to the terminal, and the rendering mode can be directly determined to be network-side rendering.

[0073] Further, in an exemplary embodiment, step S221 determines whether it is necessary to send a rendering capability query request to the terminal based on the rendering permission information, business requirement type, terminal type and terminal rendering capability information, including: when any one of the rendering permission information, business requirement type, terminal type and terminal rendering capability information does not meet the corresponding conditions, determining that it is not necessary to send a rendering capability query request to the terminal; when each information in the rendering permission information, business requirement type, terminal type and terminal rendering capability information meets the corresponding conditions, determining that it is necessary to send a rendering capability query request to the terminal.

[0074] Specifically, when any one of the following conditions is triggered: the rendering permission information does not allow terminal rendering, the business requirement type does not require rendering, the terminal type is a traditional terminal, and the terminal rendering capability information indicates that the terminal does not support rendering operations, it is determined that there is no need to send a rendering capability query request to the terminal.

[0075] When the rendering permission information indicates that terminal rendering is allowed, the business requirement type indicates that rendering needs to be performed, the terminal type is a non-traditional terminal, and the terminal rendering capability information indicates that the terminal supports rendering operations, all of the conditions are triggered, it is determined that a rendering capability query request needs to be sent to the terminal.

[0076] In the above embodiment, the rendering mode is determined based on multiple dimensions such as rendering permission information, business requirement type, terminal type, and terminal rendering capability information, which can ensure the accuracy and reliability of the determined rendering mode. Furthermore, based on information from multiple dimensions such as rendering permission information, business requirement type, terminal type, and terminal rendering capability information, it is determined whether a rendering capability query request needs to be sent to the terminal. If not, the rendering mode is directly determined to be network-side rendering, which can avoid wasting communication resources.

[0077] In an exemplary embodiment, the rendering capability query request sent to the terminal carries a rendering resource requirement; the terminal is configured to determine a rendering capability query result according to the rendering resource requirement and the real-time performance of the terminal.

[0078] The rendering resource requirement can be understood as the resource requirement for rendering.

[0079] Among them, the real-time performance of the terminal includes CPU usage, memory usage, and power consumption.

[0080] In specific implementations, the rendering capability query request sent by the Data Channel Signaling Function (DCSF) to the terminal carries the rendering resource requirements. Upon receiving the rendering capability query request, the terminal extracts the rendering resource requirements from the request and determines the real-time performance to determine whether the rendering resource requirements are met. If not, the rendering capability query result indicates that the terminal does not meet the rendering resource requirements, and the rendering mode is network-side rendering. If so, the rendering capability query result indicates that the terminal meets the rendering resource requirements, and the rendering mode is terminal-side rendering.

[0081] In this embodiment, by carrying the rendering resource requirements in the rendering capability query request, the terminal can match the rendering resource requirements with the real-time performance of the terminal, thereby determining whether the terminal can support rendering, and then determining whether the rendering mode is terminal-side rendering or network-side rendering.

[0082] In an exemplary embodiment, the method further includes: if the rendering mode is network-side rendering, determining that the corresponding rendering executor is a media gateway; sending the determination result to the terminal to instruct the terminal to collect the user video stream and send the user video stream to the media gateway; and, sending the determination result and rendered media data to the media gateway; the media gateway renders the user video stream according to the rendered media data to obtain a rendered video stream.

[0083] In a specific implementation, if the rendering mode is determined to be network-side rendering, the corresponding rendering executor is the media gateway. The rendering decision module in the data channel signaling function network element (DCSF) sends the determination result and rendered media data to the media gateway through the established data channel, namely the application data channel (ADC channel). On the other hand, the determination result is sent to the terminal (rendering is performed by the network side), so that the terminal collects the user video stream according to the determination result and sends the collected user video stream to the media gateway through the IMS video channel. In this way, the media gateway can obtain the determination result of the rendering mode, the rendered media data, and the user video stream collected by the terminal, and can thus perform the rendering operation according to the determination result: rendering the received rendered media data and the user video stream collected by the terminal, specifically rendering the collected user video stream according to the rendered media data to obtain a rendered video stream.

[0084] In this embodiment, when the rendering mode is network-side rendering, the determination result and the rendered media data are sent to the media gateway, and the determination result is sent to the terminal at the same time to instruct the terminal to collect the user video stream and send the user video stream to the media gateway, so that the media gateway can render the user video stream according to the rendered media data to obtain the rendered video stream, thereby realizing the rendering operation on the network side.

[0085] In an exemplary embodiment, the method further includes: if the rendering mode is terminal-side rendering, determining that the corresponding rendering execution body is the terminal; sending the determination result and rendering media data to the terminal; the terminal renders the user video stream according to the rendering media data, obtains the rendered video stream, and sends the rendered video stream to the media gateway.

[0086] In the specific implementation, if the rendering mode is determined to be terminal-side rendering, the corresponding rendering execution body is the terminal. When sending the determination result and rendering media data, the rendering decision module in the data channel signaling function network element (DCSF) sends the determination result and rendering media data to the terminal through the established data channel, that is, the application data channel (ADC channel). The newly added rendering execution module in the terminal performs the rendering operation based on the determination result (the terminal is responsible for rendering): the received rendering media data and the user video stream collected by the terminal are rendered. Specifically, the collected user video stream is rendered according to the rendering media data to obtain the rendered video stream, and the rendered video stream is sent to the media gateway through the IMS video channel.

[0087] The data channel signaling function network element (DCSF) does not communicate directly with the terminal, but sends the determination result and rendered media data to the media gateway, which then forwards the determination result and rendered media data to the terminal.

[0088] In this embodiment, when the rendering mode is terminal-side rendering, the determination result and the rendered media data are sent to the terminal, so that the terminal renders the user video stream according to the rendered media data, obtains the rendered video stream, and sends the rendered video stream to the media gateway, thereby realizing the rendering operation on the terminal side.

[0089] In an exemplary embodiment, after receiving a data channel establishment request initiated by a terminal, it also includes: forwarding the data channel establishment request to a second application server; the second application server is used to determine a data channel establishment strategy based on the data channel establishment request; receiving the data channel establishment strategy returned by the second application server, and forwarding the data channel establishment strategy to the first application server; the first application server is used to initiate a resource reservation request to the media gateway according to the data channel establishment strategy, and after receiving a successful resource reservation response returned by the media gateway, send a message of successful data channel establishment; in the case of receiving a successful data channel establishment message sent by the first application server, determine the rendering mode according to the rendering permission information, business requirement type, terminal type and terminal rendering capability information.

[0090] Specifically, the terminal initiates an IMS application data channel (ADC) establishment request. The rendering execution module in the terminal adds the terminal's rendering capability information to the ADC establishment request, which is then transmitted to the data channel signaling function (DCSF). Based on the ADC establishment request, the DCSF determines the terminal's type and rendering capability information. It then forwards the ADC establishment request to the second application server for the real-time interactive application, which responds with a data channel establishment policy. The DCSF forwards the policy to the first application server, which then initiates a resource reservation request to the media gateway (MF). If the resource reservation is successful, the data channel between the terminal and the MG is successfully established. The MG sends a successful resource request message to the first application server, which then responds to the DCSF and the second application server.

[0091] In this embodiment, the application data channel (ADC channel) is established through interaction between the data channel signaling function network element (DCSF), the media gateway, and the second application server. This facilitates the dynamic selection of rendering functional entities (terminal or network) during the execution of services such as the IP Multimedia Subsystem (IMS), data channel (DC), and real-time interactive applications (such as XR) through the established data channel, thereby realizing end-to-end collaborative rendering.

[0092] In one embodiment, in order to more clearly illustrate the embodiment of the present application, the following will be described with reference to specific examples of the accompanying drawings. Figure 4 The following is a diagram of the interactive process for end-to-end collaborative rendering, taking XR applications as an example, including the following steps:

[0093] Step 1-3: The DC terminal initiates an IMS application data channel (ADC) establishment request. The rendering execution module can insert information such as the terminal rendering capability into the ADC establishment request. The ADC establishment request carrying the terminal rendering capability information is transmitted to the data channel signaling function network element (DCSF) through the first application server (MMTel-AS).

[0094] Step 4: The rendering decision module in the Data Channel Signaling Function (DCSF) determines the terminal type and rendering capability information based on whether the ADC setup request contains the terminal rendering capability information. Specifically, if the ADC setup request does not contain the terminal rendering capability information, the terminal type is marked as a traditional terminal.

[0095] In step 5-11, the Data Channel Signaling Function (DCSF) forwards the ADC establishment request to the second application server (XR application AS). The second application server issues an ADC establishment policy. Based on the ADC establishment policy, the first application server initiates a resource reservation request to the media gateway (MF). MF resources are successfully reserved, and the ADC channel between the DC terminal and the media gateway (MF) is successfully established. The first application server returns an ADC establishment success message to the DCSF and the second application server.

[0096] Step 12: The rendering decision module in the data channel signaling function (DCSF) receives the ADC establishment success message and decides whether to initiate a rendering capability query request to the terminal for this call based on the terminal type (whether it is a traditional terminal), rendering permission information (such as whether the user allows terminal rendering), terminal rendering capability information (such as whether the terminal supports rendering), and service requirement type (such as whether the service requires rendering operations).

[0097] If it is determined that this call needs to initiate a rendering capability query request to the terminal, go to step 13.

[0098] If it is determined that this call does not need to initiate a rendering capability query request to the terminal, directly enter the network side rendering process of steps 23-29.

[0099] Steps 13-14: The rendering decision module in the data channel signaling function (DCSF) initiates a rendering capability query request to the DC terminal through the ADC channel. The rendering capability query request message carries information such as rendering resource requirements.

[0100] Step 15: The rendering execution module in the DC terminal decides whether to execute the rendering operation on the terminal side based on the received rendering resource requirements and the terminal's real-time performance (including CPU usage, memory usage, power consumption, etc.).

[0101] Steps 16-17: The rendering execution module in the DC terminal returns the rendering capability query result to the rendering decision module in the DCSF through the ADC channel. The rendering capability query result message carries the rendering capability query result (whether the terminal performs the rendering operation).

[0102] Case 1: If the rendering capability query result indicates that rendering is performed by the terminal side, the terminal side rendering process of steps 18-22 is entered.

[0103] Step 18: The rendering decision module in the data channel signaling function network element (DCSF) determines, based on the rendering capability query result, that the terminal is responsible for rendering this call.

[0104] Steps 19-20: The rendering decision module of DCSF sends and renders media data to the DC terminal through the ADC channel.

[0105] Steps 21-22: The rendering execution module in the DC terminal performs rendering operations based on the (terminal is responsible for rendering) rendering media data received from the DCSF rendering decision module and the user video stream collected by the terminal. The rendered video stream is then sent to the media gateway (MF) via the IMS video channel.

[0106] Scenario 2: If the rendering capability query result indicates that the network side performs rendering, the network side rendering process of steps 23-29 is entered.

[0107] Step 23: The rendering decision module in the data channel signaling function network element (DCSF) determines, based on the rendering capability query result, that the network side is responsible for rendering this call.

[0108] Steps 24-25: The rendering decision module in the DCSF sends the information to the DC terminal through the ADC channel.

[0109] Step 26: The rendering decision module in the DCSF sends the rendering mode and rendered media data to the media gateway (MF).

[0110] Steps 27-28: The rendering execution module in the DC terminal collects the user video stream according to the rendering mode (the network is responsible for rendering) and sends the user video stream to the media gateway (MF) through the IMS video channel.

[0111] Step 29: The media gateway (MF) performs rendering operations according to the rendering mode, the rendering media data, and the user video stream collected by the terminal.

[0112] This method adds a rendering decision module to the Data Channel Signaling Function (DCSF). The module is responsible for receiving ADC establishment requests initiated by the terminal, recording the terminal's rendering capabilities carried in the ADC establishment request, and marking the terminal type as a traditional terminal if the ADC establishment request does not carry information such as the terminal's rendering capabilities. Upon receiving the ADC establishment success message, the module decides whether to initiate a rendering capability query request to the terminal or use network rendering based on the terminal type, terminal rendering capabilities, service requirement type, and the terminal's rendering permission information. If necessary, a rendering capability query request is initiated to the terminal. Based on the rendering capability query result returned by the terminal, the rendering mode for this call is determined, and the rendering mode is sent to the terminal and media gateway (MF) via the ADC channel. If the rendering is terminal-side, the rendered media data is pushed to the terminal via the ADC channel. If the rendering is network-side, the rendered media data is pushed to the media gateway (MF), controlling the media gateway (MF) to perform the rendering operation.

[0113] Add a rendering execution module to the terminal: responsible for inserting parameters such as terminal rendering capabilities, evaluating terminal rendering capabilities, and executing rendering modes. Insert information such as terminal rendering capabilities in the initiated application data channel (ADC) establishment request message. Receive the rendering capability query request initiated by the rendering decision module, and decide whether to execute the rendering operation on the terminal side based on the received rendering resource requirements and the real-time performance of the terminal (including CPU occupancy, memory occupancy, power consumption, etc.), and return the decision result to the rendering decision module. Receive and execute the rendering mode issued by the rendering decision module: If it is terminal rendering, receive the rendering media data pushed by the rendering decision module through the ADC channel, and send the rendered video stream to the media gateway (MF) after the rendering operation is completed; if it is network-side rendering, directly send the collected user video stream to the media gateway (MF), and the media gateway (MF) will implement the rendering operation.

[0114] The end-network collaborative rendering method provided in this embodiment supports the dynamic selection of rendering functional entities (terminal or network) during the execution of services such as IP Multimedia Subsystem (IMS), Data Channel (DC), and real-time interactive applications (such as XR) through application data channels. This solves the problem that existing technical solutions do not support collaborative rendering on the network side and the terminal side. It can meet the business needs of providing real-time interactive application services such as XR for IMS DC users, improve user experience, and promote business development.

[0115] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0116] Based on the same inventive concept, embodiments of the present application also provide a collaborative rendering device for implementing the collaborative rendering method described above. The solution provided by this device is similar to the solution described in the method described above. Therefore, the specific limitations of one or more collaborative rendering device embodiments provided below can be found in the limitations of the collaborative rendering method described above and will not be repeated here.

[0117] In one embodiment, Figure 5 As shown, a collaborative rendering device is provided, which is applied to the scenario of deploying real-time interactive applications based on an IP multimedia subsystem. In the architecture of the IP multimedia subsystem, the establishment of a data channel is managed by a data channel signaling function network element, and multimedia communication services are implemented through a first application server and a media gateway. The real-time interactive application provides application services through a second application server. The device includes:

[0118] The information acquisition unit 510 is configured to acquire the rendering permission information of the terminal and the service requirement type of the real-time interactive application; determine the terminal type and terminal rendering capability information of the terminal based on the data channel establishment request initiated by the terminal; the data channel establishment request is a request to establish an application data channel with the media gateway;

[0119] The mode determination unit 520 is configured to determine a rendering mode based on the rendering permission information, the service requirement type, the terminal type, and the terminal rendering capability information when a message indicating that the data channel is successfully established is received. The rendering modes include terminal-side rendering and network-side rendering.

[0120] The result sending unit 530 is used to send the determination result and the rendered media data to the rendering execution body corresponding to the rendering mode; the rendering execution body is used to perform a rendering operation on the rendered media data and the user video stream collected by the terminal.

[0121] In one embodiment, the information acquisition unit 510 is further configured to, if the data channel establishment request carries terminal rendering capability information, determine that the terminal type is the first type and extract the terminal rendering capability information;

[0122] If the data channel establishment request does not carry the terminal rendering capability information, the terminal type is determined to be the second type, and the terminal rendering capability information is determined to be no rendering capability.

[0123] In one embodiment, the mode determination unit 520 is further used to determine whether it is necessary to send a rendering capability query request to the terminal based on the rendering permission information, business requirement type, terminal type, and terminal rendering capability information; if not, determining the rendering mode as network-side rendering; if necessary, sending a rendering capability query request to the terminal, and determining the rendering mode based on the rendering capability query result returned by the terminal.

[0124] In one embodiment, the mode determination unit 520 is further configured to determine that it is not necessary to send a rendering capability query request to the terminal when any one of the rendering permission information, business requirement type, terminal type, and terminal rendering capability information does not meet the corresponding conditions; and to determine that it is necessary to send a rendering capability query request to the terminal when each of the rendering permission information, business requirement type, terminal type, and terminal rendering capability information meets the corresponding conditions.

[0125] In one embodiment, the rendering capability query request sent to the terminal carries a rendering resource requirement; the terminal is configured to determine a rendering capability query result according to the rendering resource requirement and the real-time performance of the terminal.

[0126] In one embodiment, the result sending unit 530 is further used to determine that the corresponding rendering execution body is a media gateway if the rendering mode is network-side rendering; send the determination result to the terminal to instruct the terminal to collect the user video stream and send the user video stream to the media gateway; and send the determination result and rendered media data to the media gateway; the media gateway renders the user video stream according to the rendered media data to obtain a rendered video stream.

[0127] In one embodiment, the result sending unit 530 is further used to determine that the corresponding rendering execution body is the terminal if the rendering mode is terminal-side rendering; send the determination result and rendering media data to the terminal; the terminal renders the user video stream according to the rendering media data, obtains the rendered video stream, and sends the rendered video stream to the media gateway.

[0128] In one embodiment, the device also includes a channel establishment module, which is used to forward the data channel establishment request to the second application server; the second application server is used to determine the data channel establishment strategy based on the data channel establishment request; receive the data channel establishment strategy returned by the second application server, and forward the data channel establishment strategy to the first application server; the first application server is used to initiate a resource reservation request to the media gateway according to the data channel establishment strategy, and after receiving the resource reservation success response returned by the media gateway, send a message of successful data channel establishment; when receiving the data channel establishment success message sent by the first application server, determine the rendering mode according to the rendering permission information, business requirement type, terminal type and terminal rendering capability information.

[0129] Each module in the collaborative rendering apparatus described above may be implemented in whole or in part through software, hardware, or a combination thereof. Each module may be embedded in or independent of a processor in a computer device in hardware form, or may be stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0130] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, memory, a communication interface, a display screen, and an input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal via wired or wireless communication. The wireless communication can be achieved via Wi-Fi, a mobile cellular network, NFC (near-field communication), or other technologies. When executed by the processor, the computer program implements a collaborative rendering method. The display screen of the computer device can be a liquid crystal display or an electronic ink display. The input device of the computer device can be a touch layer covering the display screen, or keys, a trackball, or a touchpad provided on the computer device housing, or an external keyboard, touchpad, or mouse.

[0131] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0132] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0133] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0134] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0135] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions.

[0136] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0137] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0138] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A collaborative rendering method, characterized in that: Applicable to scenarios where real-time interactive applications are deployed based on an IP multimedia subsystem. In the architecture of the IP multimedia subsystem, the establishment of a data channel is managed by a data channel signaling function network element, and multimedia communication services are implemented through a first application server and a media gateway. The real-time interactive application provides application services through a second application server; the method includes: Obtaining rendering permission information of the terminal and the business requirement type of the real-time interactive application; determining the terminal type and terminal rendering capability information of the terminal based on a data channel establishment request initiated by the terminal; the data channel establishment request is a request to establish an application data channel with the media gateway; Upon receiving a message that the data channel is successfully established, determining a rendering mode according to the rendering permission information, the service requirement type, the terminal type, and the terminal rendering capability information; the rendering mode includes terminal-side rendering and network-side rendering; The determination result and the rendering media data are sent to a rendering execution body corresponding to the rendering mode; the rendering execution body is used to perform a rendering operation on the rendering media data and the user video stream collected by the terminal.

2. The method according to claim 1, characterized in that The determining, according to the data channel establishment request initiated by the terminal, the terminal type and terminal rendering capability information of the terminal includes: If the data channel establishment request carries terminal rendering capability information, determining that the terminal type is the first type, and extracting the terminal rendering capability information; If the data channel establishment request does not carry the terminal rendering capability information, the terminal type is determined to be the second type, and the terminal rendering capability information is determined to be no rendering capability.

3. The method according to claim 1, characterized in that The determining of the rendering mode according to the rendering permission information, the business requirement type, the terminal type, and the terminal rendering capability information includes: determining whether it is necessary to send a rendering capability query request to the terminal according to the rendering permission information, the business requirement type, the terminal type, and the terminal rendering capability information; If not required, determining that the rendering mode is network-side rendering; If necessary, a rendering capability query request is sent to the terminal, and the rendering mode is determined according to the rendering capability query result returned by the terminal.

4. The method according to claim 3, characterized in that The determining whether it is necessary to send a rendering capability query request to the terminal according to the rendering permission information, the business requirement type, the terminal type, and the terminal rendering capability information includes: When any one of the rendering permission information, the business requirement type, the terminal type, and the terminal rendering capability information does not meet corresponding conditions, determining that there is no need to send a rendering capability query request to the terminal; When each of the rendering permission information, the business requirement type, the terminal type, and the terminal rendering capability information satisfies corresponding conditions, it is determined that a rendering capability query request needs to be sent to the terminal.

5. The method according to claim 3, characterized in that The rendering capability query request sent to the terminal carries a rendering resource requirement; the terminal is configured to determine a rendering capability query result according to the rendering resource requirement and the real-time performance of the terminal.

6. The method according to claim 1, characterized in that The method further comprises: If the rendering mode is network-side rendering, determining that the corresponding rendering execution body is a media gateway; Sending the determination result to the terminal to instruct the terminal to collect a user video stream and send the user video stream to the media gateway; And, sending the determination result and the rendered media data to the media gateway; the media gateway renders the user video stream according to the rendered media data to obtain a rendered video stream.

7. The method according to claim 1, characterized in that The method further comprises: If the rendering mode is terminal-side rendering, determining that the corresponding rendering execution entity is the terminal; The determination result and the rendered media data are sent to the terminal; the terminal renders the collected user video stream according to the rendered media data to obtain a rendered video stream, and sends the rendered video stream to the media gateway.

8. The method according to claim 1, characterized in that After receiving the data channel establishment request initiated by the terminal, the method further includes: forwarding the data channel establishment request to the second application server; the second application server is configured to determine a data channel establishment strategy according to the data channel establishment request; receiving a data channel establishment policy returned by the second application server and forwarding the data channel establishment policy to the first application server; the first application server is configured to initiate a resource reservation request to the media gateway according to the data channel establishment policy, and after receiving a response from the media gateway indicating successful resource reservation, send a message indicating successful data channel establishment; Upon receiving a message from the first application server indicating that the data channel is successfully established, a rendering mode is determined according to the rendering permission information, the service requirement type, the terminal type, and the terminal rendering capability information.

9. A collaborative rendering device, characterized in that: Applicable to scenarios where real-time interactive applications are deployed based on an IP multimedia subsystem. In the architecture of the IP multimedia subsystem, the establishment of a data channel is managed by a data channel signaling function network element, and multimedia communication services are implemented through a first application server and a media gateway. The real-time interactive application provides application services through a second application server; the device includes: an information acquisition unit configured to acquire rendering permission information of a terminal and a service requirement type of the real-time interactive application; and determine the terminal type and terminal rendering capability information of the terminal according to a data channel establishment request initiated by the terminal; the data channel establishment request being a request to establish an application data channel with the media gateway; a mode determination unit, configured to, upon receiving a message indicating successful establishment of a data channel, determine a rendering mode based on the rendering permission information, the service requirement type, the terminal type, and the terminal rendering capability information; the rendering mode includes terminal-side rendering and network-side rendering; The result sending unit is used to send the determination result and the rendering media data to the rendering execution body corresponding to the rendering mode; the rendering execution body is used to perform a rendering operation on the rendering media data and the user video stream collected by the terminal.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the collaborative rendering method according to any one of claims 1 to 8 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the collaborative rendering method according to any one of claims 1 to 8 are implemented.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the collaborative rendering method according to any one of claims 1 to 8 are implemented.

Images