Cloud rendering method and device for audio and video

Abstract

The invention relates to an audio and video cloud rendering method and device. According to the method, long connection between a cloud rendering server and a user side is firstly established; and after the long connection is established, the cloud rendering server starts rendering software. Then, signaling information of the user side and the rendering software is exchanged through the cloud rendering server, and a first transmission channel and a second transmission channel between the user side and the rendering software are established; and then, the user side sends interaction data used for triggering rendering operation to the rendering software through the first transmission channel. And then, executing a corresponding rendering operation in response to the interaction data by using rendering software to obtain a rendered audio and video stream, and sending the audio and video stream to the user side through the second transmission channel. Thus, the rendering software of the cloud is remotely called through the point-to-point transmission channel to execute the corresponding rendering task, the burden brought by rendering of the user side can be relieved, and smoother and more stable 3D visual experience is provided for the user.

Description

Technical Field

[0001] This disclosure relates to the field of digital twin technology, and in particular to a cloud rendering method and apparatus for audio and video. Background Technology

[0002] In the current development of Web technologies, Web3D rendering technology, as an important means to achieve three-dimensional visual effects on web pages, has been widely used in various fields such as games, education, and product demonstrations. However, current Web3D rendering technology mainly relies on local rendering on the user's end, that is, using the graphics card resources of the user's local computer to complete the 3D graphics rendering task in the web page.

[0003] However, client-side rendering often places a heavy burden on the local computer. During complex 3D rendering, the CPU and graphics card resources on the client side are heavily consumed, which may not only cause slow webpage loading and sluggish interactive response, but may even cause serious malfunctions such as computer system lag, blue screens, or even crashes.

[0004] Therefore, there is an urgent need for a rendering solution that can reduce the dependence on users' local hardware, thereby providing users with a smoother and more stable 3D visual experience. Summary of the Invention

[0005] The purpose of this disclosure is to provide a cloud rendering method and apparatus for audio and video to provide a smoother and more stable 3D visual experience.

[0006] In a first aspect, embodiments of this disclosure provide a cloud rendering method for audio and video, applied to a user terminal, the method comprising:

[0007] Send a connection request to the cloud rendering server; the connection request is used to request the establishment of a long connection between the cloud rendering server and the user client, and to start the rendering software running on the cloud rendering server.

[0008] After establishing a long connection with the cloud rendering server, the client and the rendering software exchange signaling information through the cloud rendering server, and establish the first transmission channel and the second transmission channel between the client and the rendering software.

[0009] Interactive data for triggering rendering operations is sent to the rendering software through the first transmission channel, so that the rendering software responds to the interactive data and performs the corresponding rendering operation to obtain the rendered audio and video streams.

[0010] The second transmission channel receives audio and video streams sent by the rendering software.

[0011] Optionally, interactive data for triggering rendering operations is sent to the rendering software via the first transmission channel, so that the rendering software responds to the interactive data to perform corresponding rendering operations and obtain the rendered audio and video streams, including:

[0012] Upon receiving user input to trigger a screen switching operation, generate the corresponding interaction data for the screen switching operation;

[0013] Interactive data is sent to the rendering software through the first transmission channel, so that the rendering software can obtain the rendering data corresponding to the target switching interface based on the interactive data, and render and generate the audio and video stream corresponding to the target switching interface based on the rendering data.

[0014] Optionally, interactive data for triggering rendering operations is sent to the rendering software via the first transmission channel, so that the rendering software responds to the interactive data to perform corresponding rendering operations and obtain the rendered audio and video streams, including:

[0015] When the currently displayed scene is a virtual reality scene, collect inertial sensor data from the user's device;

[0016] Interactive data containing inertial sensor data is sent to the rendering software through the first transmission channel, so that the rendering software can determine the rendering perspective based on the inertial sensor data and render and generate audio and video streams applicable to the virtual reality scene based on the rendering perspective.

[0017] Optionally, a second transmission channel may be established between the user client and the rendering software, including:

[0018] Create a video tag and add an event listener to the video tag; the video tag is used to display the audio and video streams generated by the rendering software, and the event listener is used to monitor the playback and / or transmission status of the audio and video streams;

[0019] Receive connection configuration information sent by the cloud rendering server;

[0020] Based on connection configuration information and signaling information, a streaming media connection instance is established between the user terminal and the rendering software;

[0021] By using streaming media connection instances, a second transmission channel is established between the user client and the rendering software.

[0022] Optionally, the method also includes:

[0023] Multiple audio and video stream slices sent by the rendering software are received through the first transmission channel; wherein the data volume of each audio and video stream slice is less than the data volume threshold.

[0024] Multiple audio and video stream slices are spliced ​​together into a complete audio and video stream.

[0025] Secondly, embodiments of this disclosure provide a cloud rendering method for audio and video, applied to a cloud rendering server, the method comprising:

[0026] Receive connection requests sent by the user client;

[0027] Upon receiving a connection request, a long-lived connection is established between the cloud rendering server and the user client, and the rendering software running on the cloud rendering server is started.

[0028] Exchange signaling information between the rendering software and the user terminal, and establish a first transmission channel and a second transmission channel between the rendering software and the user terminal;

[0029] The system receives interactive data sent by the user client to trigger rendering operations through the first transmission channel.

[0030] The rendering software responds to the interactive data and performs corresponding rendering operations to obtain the rendered audio and video streams;

[0031] Audio and video streams are sent to the user terminal through the second transmission channel.

[0032] Thirdly, embodiments of this disclosure provide an audio / video cloud rendering device applied to a user terminal, the device comprising:

[0033] The first sending module is used to send a connection request to the cloud rendering server. The connection request is used to request the establishment of a long connection between the cloud rendering server and the user terminal, and to start the rendering software running on the cloud rendering server.

[0034] The transmission channel establishment module is used to exchange signaling information between the user terminal and the rendering software through the cloud rendering server after establishing a long connection with the cloud rendering server, and to establish a first transmission channel and a second transmission channel between the user terminal and the rendering software.

[0035] The second sending module is used to send interactive data to the rendering software through the first transmission channel to trigger the rendering operation, so that the rendering software responds to the interactive data to perform the corresponding rendering operation and obtain the rendered audio and video stream.

[0036] The receiving module is used to receive audio and video streams sent by the rendering software through the second transmission channel.

[0037] Optionally, the second sending module may be specifically used to: generate interactive data corresponding to the interface switching operation when receiving user input for triggering interface switching; send the interactive data to the rendering software through the first transmission channel so that the rendering software can obtain the rendering data corresponding to the target switching interface based on the interactive data, and render and generate the audio and video stream corresponding to the target switching interface based on the rendering data.

[0038] Optionally, the second sending module can be specifically used to: collect inertial sensor data from the user terminal when the currently displayed scene is a virtual reality scene; send interactive data containing inertial sensor data to the rendering software through the first transmission channel, so that the rendering software can determine the rendering perspective based on the inertial sensor data, and render and generate an audio and video stream applied to the virtual reality scene based on the rendering perspective.

[0039] Optionally, the transmission channel establishment module can be specifically used for: creating video tags and adding event listeners to the video tags; the video tags are used to display the audio and video streams generated by the rendering software, and the event listeners are used to monitor the playback and / or transmission status of the audio and video streams; receiving connection configuration information sent by the cloud rendering server; establishing a streaming media connection instance between the user terminal and the rendering software based on the connection configuration information and signaling information; and establishing a second transmission channel between the user terminal and the rendering software through the streaming media connection instance.

[0040] Optionally, the receiving module can also be used to receive multiple audio and video stream slices sent by the rendering software through the first transmission channel; wherein the data volume of each audio and video stream slice is less than the data volume threshold; and multiple audio and video stream slices are spliced ​​into a complete audio and video stream.

[0041] Fourthly, embodiments of this disclosure provide an audio / video cloud rendering apparatus applied to a cloud rendering server, the apparatus comprising:

[0042] The first receiving module is used to receive connection requests sent by the user terminal;

[0043] The connection establishment module is used to respond to a received connection request, establish a long connection between the cloud rendering server and the user client, and start the rendering software running on the cloud rendering server.

[0044] The transmission channel establishment module is used to exchange signaling information between the rendering software and the user terminal, and to establish the first transmission channel and the second transmission channel between the rendering software and the user terminal.

[0045] The second receiving module is used to receive interactive data sent by the user terminal to trigger rendering operations through the first transmission channel;

[0046] The rendering module is used to perform corresponding rendering operations in response to interactive data using rendering software, and to obtain the rendered audio and video streams.

[0047] The sending module is used to send audio and video streams to the user terminal through the second transmission channel.

[0048] Fifthly, embodiments of this disclosure provide an electronic device, including a memory and a processor; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, and when the computer program is executed, it implements the method of the first or second aspect described above.

[0049] Sixthly, embodiments of this disclosure provide a computer-readable storage medium storing a computer program that, when executed, implements the methods of the first or second aspect described above.

[0050] The cloud rendering method and apparatus for audio and video provided in this disclosure can establish a point-to-point first transmission channel and a second transmission channel between a user terminal and rendering software. Then, the user terminal can transmit real-time interactive data to the rendering software through the first transmission channel, enabling the rendering software to respond to the interactive data in real time and render the corresponding audio and video stream. Finally, the rendering software can transmit the rendered audio and video stream to the user terminal through the second transmission channel.

[0051] In this way, the user terminal can remotely call the rendering software in the cloud to execute the corresponding rendering task through the point-to-point transmission channel, which can relieve the user terminal from the burden of rendering, realize a smooth remote interactive experience, and provide users with a smoother and more stable 3D visual experience. Attached Figure Description

[0052] To more clearly illustrate the technical solutions of the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0053] Figure 1 A flowchart illustrating a cloud rendering method for audio and video provided in an embodiment of this disclosure is shown.

[0054] Figure 2 This illustration shows a flowchart of establishing a point-to-point transmission channel via a relay server, according to an embodiment of this disclosure.

[0055] Figure 3 A schematic diagram of a WebRTC software architecture provided by an embodiment of this disclosure is shown;

[0056] Figure 4 This diagram illustrates a structural block diagram of a cloud rendering apparatus for audio and video provided in an embodiment of the present disclosure.

[0057] Figure 5 This illustration shows a structural block diagram of yet another audio / video cloud rendering apparatus provided in an embodiment of the present disclosure;

[0058] Figure 6 A structural block diagram of an electronic device provided in an embodiment of this disclosure is shown;

[0059] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure is shown. Detailed Implementation

[0060] The present disclosure will be further described in detail below with reference to the accompanying drawings and embodiments. Through these descriptions, the features and advantages of the present disclosure will become clearer and more apparent.

[0061] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments. Although various aspects of embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless specifically indicated otherwise.

[0062] Furthermore, the technical features involved in the different embodiments of this disclosure described below can be combined with each other as long as they do not conflict with each other.

[0063] The digital 3D scene described in this disclosure refers to a highly realistic terrain environment used in various simulation, visualization, and game development applications. The digital 3D scene integrates graphics rendering technology to achieve the desired visual effects and interactivity, and has the following core characteristics:

[0064] Realistic terrain simulation: Digital 3D scenes include terrain generated from real-world data such as satellite imagery and topographic survey data. Terrain features include mountains, valleys, plains, and rivers to present a realistic feel of natural terrain.

[0065] Dynamic environmental elements: Environmental elements in digital 3D scenes, such as weather, lighting, and seasonal changes, are dynamic. For example, simulating day and night cycles, cloud movement, rain erosion, and seasonal vegetation changes, these elements can respond to user interactions in real time.

[0066] Advanced graphics effects: Utilizes graphics technologies such as ray tracing, physically-based rendering (PBR), high dynamic range imaging (HDR), and shadow mapping to provide visual detail and depth.

[0067] Interactivity and explorability: Users can freely move within the digital 3D scene, exploring different terrains and environments. Various interaction methods are supported, including walking, driving, or flying modes.

[0068] Ecosystem simulation: The digital 3D scene includes simulations of plant and animal ecosystems. The behavior of animals and the growth of vegetation are based on real-world models and data.

[0069] Customizability: Digital 3D scenes offer customizability, allowing users to adjust terrain, vegetation, environmental conditions, etc., according to specific needs, making them suitable for customized simulation requirements.

[0070] Application areas:

[0071] - Education and training: Can be used for educational purposes in geography, environmental science, and military training.

[0072] - Game Development: Provides game developers with rich and realistic environments to enhance the immersive experience of games.

[0073] - Urban planning and landscape design: Helping designers and planners visualize new building projects or landscape renovations.

[0074] - Film and television production: Providing high-quality background scenes for film and television production.

[0075] In digital 3D scenes, Web3D rendering technology is involved. Current Web3D rendering technology mainly relies on the local rendering method on the user's end, that is, using the graphics card resources of the user's local computer to complete the 3D graphics rendering task in the webpage.

[0076] However, client-side rendering often places a heavy burden on the local computer. During complex 3D rendering, the CPU and graphics card resources on the client side are heavily consumed, which may not only cause slow webpage loading and sluggish interactive response, but may even cause serious malfunctions such as computer system lag, blue screens, or even crashes.

[0077] Therefore, there is an urgent need for a rendering solution that can reduce the dependence on users' local hardware, thereby providing users with a smoother and more stable 3D visual experience.

[0078] To provide users with a smoother and more stable 3D visual experience, this disclosure provides a cloud rendering method and apparatus for audio and video.

[0079] The cloud rendering method and apparatus for audio and video provided in this disclosure will be described in detail below with reference to the accompanying drawings and through specific embodiments and application scenarios.

[0080] See Figure 1 , Figure 1 This is a flowchart illustrating a cloud rendering method for audio and video provided in an embodiment of this disclosure. Figure 1 As shown, the method may include the following steps:

[0081] Step S101: The user sends a connection request to the cloud rendering server.

[0082] The connection request is used to request the establishment of a persistent connection between the cloud rendering server and the client. A persistent connection is a continuous connection maintained between the client and the cloud rendering server, allowing the client and the cloud rendering server to send and receive data at any time without having to establish and close the connection each time.

[0083] It should be noted that in the embodiments of this disclosure, the user terminal can be a browser or an application on the user terminal, and this disclosure does not limit it.

[0084] Step S102: Upon receiving the connection request, the cloud rendering server establishes a long connection between the cloud rendering server and the user client.

[0085] Step S103: After establishing a long connection, the cloud rendering server starts the rendering software running on the cloud rendering server.

[0086] Rendering software is software that runs on a cloud rendering server and is used to perform rendering operations.

[0087] When a cloud rendering server receives a connection request, it will also allocate a certain amount of GPU resources for the rendering software to run and perform rendering operations.

[0088] Step S104: Exchange signaling information between the user terminal and the rendering software through the cloud rendering server.

[0089] In this embodiment of the disclosure, by exchanging signaling information between the user terminal and the rendering software, the communication protocol, audio and video stream format, encoding and decoding format, etc. between the user terminal and the rendering software can be negotiated and agreed upon.

[0090] For example, after establishing a long connection, the client sends client signaling information to the cloud rendering server. This client signaling information includes, for instance, the communication protocols supported by the client, the audio and video formats supported by the client, the codec formats, image resolution, screen size, etc. Then, the cloud rendering server sends the received client signaling information to the rendering software. The rendering software saves the client signaling information and sends its own signaling information to the cloud rendering server. This signaling information includes, for instance, the communication protocols supported by the rendering software, the audio and video formats supported by the rendering software, the codec formats, image resolution, etc. Then, the cloud rendering server sends the received rendering software signaling information to the client. The client saves the rendering software signaling information. At this point, both the client and the rendering software possess their own signaling information and each other's information; therefore, the client and the rendering software can reach a unified agreement on communication protocols, audio and video formats, decoding formats, etc.

[0091] In this way, the user terminal and the rendering software can communicate and perform audio and video stream encoding and decoding based on the agreed format.

[0092] Step S105: Establish the first and second transmission channels between the user terminal and the rendering software.

[0093] After an agreement is reached based on signaling information, a first and a second point-to-point transmission channel can be established between the user terminal and the rendering software. The first transmission channel serves as a low-latency channel, used to transmit interactive data that triggers rendering operations, enabling the rendering software to respond promptly to user interactions and ensuring real-time interactive effects. The second transmission channel can be used to transmit audio and video streams.

[0094] In one possible implementation, establishing a second transmission channel between the user client and the rendering software can be achieved as follows: First, a video tag is created on the user client, and an event listener is added to the video tag. The video tag is used to display the audio and video streams generated by the rendering software, and the event listener is used to monitor the playback and / or transmission status of the audio and video streams. Then, the user client receives connection configuration information sent by the cloud rendering server, and based on the connection configuration information and signaling information, establishes a streaming media connection instance between the user client and the rendering software. Finally, through the streaming media connection instance, the second transmission channel between the user client and the rendering software is established.

[0095] For example, an HTML video tag can be created in the user's browser to receive and play the audio and video streams transmitted from the rendering software. This video tag will serve as the carrier for the audio and video stream output, displaying the rendered content generated by the rendering software to the user in real-time.

[0096] Add event listeners to the video tag to perform corresponding actions under different connection or playback states. These listener events may include, but are not limited to, the following:

[0097] onloadstart: Triggered when the audio / video stream starts loading, used to initialize the playback interface or display a loading prompt.

[0098] oncanplay: Triggered when the audio or video stream is playable, often used to remove loading animations or prompts.

[0099] onerror: Triggered when a playback error occurs, used to catch the error and execute recovery or retry mechanisms.

[0100] onended: Triggered when audio or video playback ends, used to handle stream closing and cleanup.

[0101] After a persistent connection is established between the client and the cloud rendering server, the cloud rendering server can send the connection configuration information used to establish the persistent connection to the client. Then, based on the received connection configuration information and signaling information, the client can establish a streaming media connection instance between the client and the rendering software.

[0102] After establishing a streaming media connection instance, the streaming media connection instance can negotiate and exchange signaling information with the rendering software to complete the establishment of the second transmission channel.

[0103] It should be noted that when the user client and the cloud rendering server cannot communicate directly (for example, the user client and the cloud rendering server are on different intranets or are restricted by network devices such as firewalls and network address translation (NAT), a relay server can be used to achieve network penetration and establish a point-to-point transmission channel between the user client and the cloud rendering server.

[0104] For example, such as Figure 2 As shown, in a network architecture that achieves network traversal through a relay server, the cloud rendering server is responsible for allocating resources, managing sessions between the user client and the rendering software, and playing a role in information exchange and signaling management. The relay server is used to resolve the problem of inability to communicate directly due to network environment limitations (such as NAT traversal failure) when a connection is established between the user client and the rendering software, typically using the TURN protocol. NAT is a network technology used to allocate IP addresses and ports between internal network devices and the public network. NAT is a common obstacle to WebRTC connections, requiring other technologies to traverse it. The STUN server helps devices obtain their own public IP address and port information, facilitating NAT traversal and enabling the establishment of point-to-point connections in different network environments.

[0105] based on Figure 2 The network architecture shown establishes a point-to-point transmission channel between the user client and the cloud rendering server, which can be achieved through the following steps:

[0106] Step 1: Signaling Exchange (Client and Cloud Rendering Server)

[0107] The client sends a connection request to the cloud rendering server. After receiving the request, the cloud rendering server allocates the corresponding resources, establishes a session for the client, and generates the signaling information for the session.

[0108] The cloud rendering server sends session information to the client, and the two parties exchange signaling. During this process, the client and the cloud rendering server can communicate via HTTPS or WebSocket.

[0109] After the signaling exchange is completed, the point-to-point communication parameters (such as encoding format, IP, port, etc.) between the user terminal and the rendering software are negotiated, laying the foundation for point-to-point connection.

[0110] Step 2: NAT traversal (using a STUN server)

[0111] In order for the client and rendering software to establish a connection in different network environments, WebRTC usually sends a request to the STUN server to obtain the public IP addresses and ports of the client and rendering software.

[0112] The client and rendering software each send a STUN request (UDP) to the STUN server, which then returns their respective public IP addresses and port information.

[0113] The information from the STUN server helps clients and rendering software understand their public IP address and transmits this information to each other through signaling exchange.

[0114] Step 3: Attempt to establish a peer-to-peer connection

[0115] After obtaining the public IP address information through the STUN server, the client and rendering software will attempt to establish a direct peer-to-peer connection.

[0116] When NAT type allows, the client and rendering software can send audio and video streams and data streams directly in a peer-to-peer manner without relying on other servers for forwarding, thus achieving low-latency and high-efficiency communication.

[0117] If the peer-to-peer connection is successful, audio and video streams and interactive data will be transmitted directly through the first and second transmission channels of both parties.

[0118] Step 4: Relay Service

[0119] When the type of NAT causes point-to-point connections to fail (such as when encountering symmetric NAT), the client and rendering software will be unable to establish a direct connection.

[0120] At this point, both parties will negotiate via signaling to determine whether to use a relay server (usually a TURN server) for data forwarding.

[0121] The client and rendering software each connect to a relay server, which forwards the audio and video streams and interactive data from both sides, ensuring stable communication between the client and rendering software even in complex network environments.

[0122] Step S106: The user terminal sends interactive data to the rendering software through the first transmission channel to trigger the rendering operation.

[0123] In this embodiment of the disclosure, the interactive data may be data related to the rendering operation triggered by the user's operation input on the user terminal or the system settings on the user terminal.

[0124] For example, when the client receives a user input that triggers a screen switching operation, it generates interactive data corresponding to the screen switching operation.

[0125] For example, user actions such as mouse clicks, keyboard input, and touchscreen swipes trigger the switching of the current interface on the user's device. Therefore, this action information needs to be transmitted as interactive data to the rendering software in real time so that the rendering software can update the rendered screen.

[0126] For example, in design or drawing applications, users can select tools, colors, layers, etc. The client can then generate interactive data based on the user's selections, including status information such as tools, colors, and layers. This allows the rendering software to update the rendered content in real time based on the received status information. This status data also includes real-time changes in application settings (such as screen brightness and contrast) to adjust the rendering effect.

[0127] As another example, when the scene currently displayed in the user's browser is a virtual reality scene, inertial sensor data from the user's device can be collected and used as interactive data. In this way, the rendering software can determine the rendering perspective based on the inertial sensor data and generate audio and video streams applicable to the virtual reality scene based on these perspectives. This audio and video stream conforms to the user's changing viewpoints, enhancing the immersive experience.

[0128] In this embodiment of the disclosure, the aforementioned interactive data is sent to the rendering software through a low-latency first transmission channel, which enables the cloud-rendered content to respond to the user's operation in a timely manner, achieving real-time interaction and dynamic updates.

[0129] Step S107: The rendering software responds to the interactive data by performing the corresponding rendering operation to obtain the rendered audio and video stream.

[0130] After receiving interactive data, the rendering software can respond to it. Specifically, the rendering software can perform corresponding rendering operations based on the received interactive data, thereby rendering and generating audio and video streams to respond to the interactive data.

[0131] For example, when the received interaction data corresponds to an interface switching operation, the rendering software can obtain the rendering data corresponding to the target switching interface based on the interaction data, and then render and generate the audio and video stream corresponding to the target switching interface based on the rendering data. Here, the target switching interface refers to the interface to be displayed after the interface switching operation. The rendering data corresponding to the target switching interface may be stored on the user's end or a cloud rendering server. Correspondingly, the user's end can send the corresponding rendering data to the rendering software, or the rendering software can obtain the corresponding rendering data from the cloud rendering server.

[0132] As another example, when the received interactive data includes inertial sensor data, the rendering software can determine the rendering perspective based on the inertial sensor data and render and generate an audio and video stream applicable to the virtual reality scene based on the rendering perspective.

[0133] When the rendering software generates audio and video streams, it generates audio and video streams that meet the requirements based on parameters such as audio and video format and image resolution agreed upon with the user in advance.

[0134] Step S108: The rendering software sends audio and video streams to the user terminal through the second transmission channel.

[0135] After the rendering software generates the rendered audio and video streams, the rendered audio and video streams can be transmitted to the user terminal through the second transmission channel.

[0136] Optionally, the rendering software can encode the audio and video streams before sending them to the user. Then, it sends the encoded audio and video streams to the user to improve output efficiency. The rendering software can encode the audio and video streams based on a pre-agreed encoding / decoding format with the user.

[0137] In an optional embodiment of this disclosure, the rendering software can also send audio and video streams to the user terminal via the first transmission channel. To ensure smooth transmission of the audio and video streams, the amount of data transmitted in a single transmission through the first transmission channel can be limited. Thus, before sending the audio and video streams to the user terminal via the first transmission channel, the rendering software can first determine whether the amount of audio and video streams to be transmitted is greater than or equal to a data volume threshold. If it is greater than or equal to the data volume threshold, the audio and video streams to be transmitted are sliced ​​to obtain multiple audio and video stream slices, each with a data volume less than the data volume threshold. Then, each audio and video stream slice is sequentially transmitted to the user terminal via the first transmission channel.

[0138] Step S109: Play the audio and video streams on the user's end.

[0139] After receiving the rendered audio and video streams, the user can play the rendered audio and video streams on the user's end.

[0140] Optionally, if video tags are created on the user's end, the user can bind the received audio and video streams to the video tags and display the received audio and video streams through the video tags.

[0141] Optionally, if the received video stream consists of multiple audio and video stream slices, the multiple audio and video stream slices can be spliced ​​together into a complete video stream and then the complete video stream can be played.

[0142] The cloud rendering method for audio and video provided in this disclosure establishes a point-to-point first transmission channel and a second transmission channel between the user terminal and the rendering software. The user terminal can then transmit real-time interactive data to the rendering software through the first transmission channel, enabling the rendering software to respond in real-time and render the corresponding audio and video stream. Finally, the rendering software can transmit the rendered audio and video stream back to the user terminal through the second transmission channel.

[0143] In this way, the user terminal can remotely call the rendering software in the cloud to execute the corresponding rendering task through the point-to-point transmission channel, which can relieve the user terminal from the burden of rendering, thereby providing users with a smoother and more stable 3D visual experience.

[0144] The cloud rendering method for audio and video provided in this disclosure can be implemented based on a Web real-time communication (WebRTC) architecture, which can support real-time audio and video streaming between the user terminal and the rendering software.

[0145] See Figure 3 , Figure 3 This is a schematic diagram of a WebRTC software architecture provided in an embodiment of this disclosure. Figure 3 As shown, the WebRTC software architecture includes a multi-platform API, a WebRTC C++ API, a session management module, an abstract signaling module, a voice engine module, a video engine module, a transmission module, an audio acquisition module, a video acquisition module, and a network input / output module.

[0146] The multi-platform API provides support for multiple platforms, including Web API, iOS API, and other platform APIs, making it easy for developers to use the same WebRTC architecture to achieve real-time communication across different platforms. In cloud rendering scenarios, different user terminals (such as Web and iOS devices) can use their respective APIs to call WebRTC functions, establish connections with the rendering software, and transmit audio and video streams.

[0147] The WebRTC C++ API provides core real-time communication interfaces, especially the PeerConnection object. PeerConnection is responsible for establishing peer-to-peer connections and facilitating the exchange of audio and video data streams between the user client and the cloud. In cloud rendering, the user client and rendering software can exchange signaling information through PeerConnection, and after signaling negotiation, establish a stable peer-to-peer connection to achieve real-time data stream transmission.

[0148] Session management is used to manage the session state between the client and the cloud rendering server. The abstract signaling module is used for signaling exchange, that is, the negotiation of audio and video parameters between the client and the cloud rendering server before the connection is established. This module ensures that both parties reach a unified connection parameter such as encoding format, frame rate, and resolution, so that audio and video data can be correctly decoded and displayed, laying the foundation for subsequent audio and video stream transmission.

[0149] The speech engine module is responsible for processing audio data, including audio encoding, decoding, echo cancellation (AEC), and noise suppression (NR). The audio stream is transmitted stably via NetEQ (a network jitter buffer for audio). During cloud rendering, the speech engine ensures that cloud-generated audio (such as application sound effects or voice content) is synchronously transmitted to the user's device, allowing the user to hear sound smoothly.

[0150] The video engine module is responsible for video encoding and decoding, jitter buffering, and image enhancement. The jitter buffer smooths the impact of network fluctuations on the video stream, while image enhancement improves video quality. During cloud rendering, the video engine compresses and encodes the rendered footage, transmitting it to the user's device via a video stream. The user's device decodes the video data using the video engine and displays it in the video tag, ensuring that the user can see the real-time cloud feed.

[0151] The transmission module is responsible for the secure and stable transmission of data, including functions such as SRTP (Secure Real-Time Transport Protocol), multiplexing, and point-to-point connections. This module ensures that audio and video data rendered in the cloud can be securely and with low latency transmitted to the user end. Multiplexing unifies the transmission of different data streams (audio, video, data channels, etc.), while point-to-point connections reduce latency and improve transmission efficiency.

[0152] The audio and video capture modules are responsible for capturing the user's local audio and video streams. For example, in some scenarios, users may need to upload local audio or video streams, which the cloud can then use to enable bidirectional audio and video communication. During unidirectional streaming in cloud rendering, the user's end typically does not involve the capture modules; it mainly receives the rendered stream from the cloud. However, if user interaction requires transmitting audio and video to the cloud, these modules will capture and encode the data.

[0153] The network input / output module manages the input and output of network data, and is responsible for sending and receiving data packets at the network layer. In cloud rendering scenarios, the network input / output module converts audio, video, and control signaling data between the user terminal and the cloud into network data packets for transmission, ensuring real-time network layer data exchange.

[0154] To achieve cloud rendering of audio and video streams, the above modules can collaborate as follows: The user's browser can call the WebRTC interface via multi-platform APIs to create a PeerConnection object and initiate the signaling exchange process. The session management module and the abstract signaling module transmit connection signaling information between the user and the cloud rendering server, negotiate audio and video parameters, and establish a session. After the connection is established, the voice engine module and video engine module begin working, handling audio and video encoding / decoding, jitter buffering, and enhancement, respectively. The transmission module is responsible for securely and with low latency transmitting the audio and video streams to the user. Multiplexing ensures that the audio and video streams are transmitted within the same connection. The network input / output module is responsible for data packet transmission, forwarding the cloud audio and video streams to the user, enabling the user to receive and decode the audio and video streams.

[0155] In this way, the various modules in the WebRTC architecture work together to support data transmission in cloud rendering of audio and video. Each module ensures a complete process from signaling negotiation, audio and video encoding and decoding, secure data transmission to real-time communication, ultimately enabling users to receive and display cloud-rendered audio and video content, achieving a smooth remote interactive experience.

[0156] It is understood that the above embodiments are merely examples, and modifications can be made to the above embodiments in actual implementation. Those skilled in the art will understand that any modifications to the above embodiments that do not require creative effort fall within the protection scope of this disclosure, and will not be described in detail in the embodiments.

[0157] All the above-mentioned optional technical solutions can be referenced or combined with each other to form optional embodiments of this disclosure, and will not be described in detail here.

[0158] Based on the same inventive concept, this disclosure also provides an audio and video cloud rendering device. Since the principle of solving the problem by the audio and video cloud rendering device is similar to that of the aforementioned audio and video cloud rendering method, the implementation of the audio and video cloud rendering device can refer to the implementation of the aforementioned audio and video cloud rendering method, and the repeated parts will not be described again.

[0159] See Figure 4 , Figure 4 This is a structural block diagram of an audio / video cloud rendering device provided in an embodiment of the present disclosure. The audio / video cloud rendering device 400 is applied to a user terminal, such as... Figure 4 As shown, the cloud rendering device 400 for audio and video may include: a first sending module 401, a transmission channel establishment module 402, a second sending module 403, and a receiving module 404. Among them,

[0160] The first sending module 401 is used to send a connection request to the cloud rendering server; the connection request is used to request the establishment of a long connection between the cloud rendering server and the user terminal, and to start the rendering software running on the cloud rendering server.

[0161] The transmission channel establishment module 402 is used to exchange signaling information between the user terminal and the rendering software through the cloud rendering server after establishing a long connection with the cloud rendering server, and to establish a first transmission channel and a second transmission channel between the user terminal and the rendering software.

[0162] The second sending module 403 is used to send interactive data to the rendering software through the first transmission channel to trigger the rendering operation, so that the rendering software responds to the interactive data to perform the corresponding rendering operation and obtain the rendered audio and video stream.

[0163] The receiving module 404 is used to receive audio and video streams sent by the rendering software through the second transmission channel.

[0164] Optionally, the second sending module 403 may be specifically used to: generate interactive data corresponding to the interface switching operation when receiving user input for triggering interface switching; send the interactive data to the rendering software through the first transmission channel so that the rendering software can obtain the rendering data corresponding to the target switching interface based on the interactive data, and render and generate the audio and video stream corresponding to the target switching interface based on the rendering data.

[0165] Optionally, the second sending module 403 can be specifically used to: collect inertial sensor data from the user terminal when the currently displayed scene is a virtual reality scene; send interactive data containing inertial sensor data to the rendering software through the first transmission channel, so that the rendering software determines the rendering perspective based on the inertial sensor data, and renders and generates an audio and video stream applied to the virtual reality scene based on the rendering perspective.

[0166] Optionally, the transmission channel establishment module 402 can be specifically used for: creating video tags and adding event listeners to the video tags; the video tags are used to display the audio and video streams generated by the rendering software, and the event listeners are used to monitor the playback and / or transmission status of the audio and video streams; receiving connection configuration information sent by the cloud rendering server; establishing a streaming media connection instance between the user terminal and the rendering software based on the connection configuration information and signaling information; and establishing a second transmission channel between the user terminal and the rendering software through the streaming media connection instance.

[0167] Optionally, the receiving module 404 can also be used to receive multiple audio and video stream slices sent by the rendering software through the first transmission channel; wherein the data volume of each audio and video stream slice is less than the data volume threshold; and to splice the multiple audio and video stream slices into a complete audio and video stream.

[0168] See Figure 5 , Figure 5 This is a structural block diagram of another audio / video cloud rendering device provided in an embodiment of the present disclosure. The audio / video cloud rendering device 500 is applied to a cloud rendering server, such as... Figure 5 As shown, the cloud rendering device 500 for audio and video may include: a first receiving module 501, a connection establishment module 502, a transmission channel establishment module 503, a second receiving module 504, a rendering module 505, and a sending module 506.

[0169] in,

[0170] The first receiving module 501 is used to receive connection requests sent by the user terminal.

[0171] The connection establishment module 502 is used to establish a long connection between the cloud rendering server and the user terminal in response to a received connection request, and to start the rendering software running on the cloud rendering server.

[0172] The transmission channel establishment module 503 is used to exchange signaling information between the rendering software and the user terminal, and to establish a first transmission channel and a second transmission channel between the rendering software and the user terminal.

[0173] The second receiving module 504 is used to receive interactive data sent by the user terminal to trigger rendering operations through the first transmission channel.

[0174] The rendering module 505 is used to perform corresponding rendering operations in response to interactive data using rendering software to obtain the rendered audio and video streams.

[0175] The sending module 506 is used to send audio and video streams to the user terminal through the second transmission channel.

[0176] The cloud rendering apparatus for audio and video provided in this embodiment can establish a point-to-point first transmission channel and a second transmission channel between the user terminal and the rendering software. The user terminal can then transmit real-time interactive data to the rendering software through the first transmission channel, enabling the rendering software to respond to the interactive data in real time and render the corresponding audio and video stream. Finally, the rendering software can transmit the rendered audio and video stream back to the user terminal through the second transmission channel.

[0177] In this way, the user terminal can remotely call the rendering software in the cloud to execute the corresponding rendering task through the point-to-point transmission channel, which can relieve the user terminal from the burden of rendering, realize a smooth remote interactive experience, and provide users with a smoother and more stable 3D visual experience.

[0178] This disclosure also provides an electronic device, see [link to relevant documentation] Figure 6 , Figure 6 This is a structural block diagram of an electronic device provided in an embodiment of this disclosure. Figure 6 As shown, the electronic device 600 may include a processor 601, a memory 602, and a program or instructions stored in the memory 602 and executable on the processor 601. When the program or instructions are executed by the processor 601, they implement the various processes of the above-described audio and video cloud rendering method embodiments and achieve the same technical effects. To avoid repetition, they will not be described again here. It should be noted that the electronic devices in the embodiments of this disclosure include mobile electronic devices and non-mobile electronic devices.

[0179] Figure 7 A schematic diagram of the hardware structure of an electronic device to implement an embodiment of this disclosure.

[0180] The electronic device 700 includes, but is not limited to, components such as: radio frequency unit 701, network module 702, audio output unit 703, input unit 704, sensor 705, display unit 706, user input unit 707, interface unit 708, memory 709, and processor 710.

[0181] Those skilled in the art will understand that the electronic device 700 may also include a power supply (such as a battery) for supplying power to various components. The power supply may be logically connected to the processor 710 through a power management system, thereby enabling functions such as managing charging, discharging, and power consumption through the power management system. Figure 7 The electronic device structure shown does not constitute a limitation on the electronic device. The electronic device may include more or fewer components than shown, or combine certain components, or have different component arrangements, which will not be elaborated here.

[0182] It should be understood that, in this embodiment of the disclosure, the input unit 704 may include a graphics processing unit (GPU) 7041 and a microphone 7042. The GPU 7041 processes image data of still images or videos obtained by an image capture device (such as a camera) in video capture mode or image capture mode. The display unit 706 may include a display panel 7061, which may be configured in the form of a liquid crystal display, an organic light-emitting diode, etc. The user input unit 707 includes a touch panel 7071 and other input devices 7072. The touch panel 7071 is also called a touch screen. The touch panel 7071 may include a touch detection device and a touch controller. Other input devices 7072 may include, but are not limited to, physical keyboards, function keys (such as volume control buttons, power buttons, etc.), trackballs, mice, joysticks, etc., which will not be described in detail here. The memory 709 can be used to store software programs and various data, including but not limited to applications and operating systems. The processor 710 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understandable that the aforementioned modem processor may not be integrated into the processor 710.

[0183] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the various processes of the above-described audio and video cloud rendering method embodiments and achieves the same technical effects. To avoid repetition, these will not be described again here.

[0184] The processor is the processor in the electronic device described in the above embodiments. The readable storage medium includes computer-readable storage media, such as computer read-only memory (ROM), random access memory (RAM), magnetic disk, or optical disk.

[0185] This disclosure also provides a chip, which includes a processor and a communication interface. The communication interface and the processor are coupled. The processor is used to run programs or instructions to implement the various processes of the above-described audio and video cloud rendering method embodiments and can achieve the same technical effect. To avoid repetition, it will not be described again here.

[0186] It should be understood that the chip mentioned in the embodiments of this disclosure may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0187] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0188] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0189] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0190] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device and system embodiments are relatively simple in description because they are fundamentally similar to the method embodiments; relevant parts can be referred to the descriptions of the method embodiments. In this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. The terms "upper," "lower," etc., indicating orientation or positional relationships based on the orientation or positional relationships shown in the accompanying drawings, are used only for the convenience of describing this disclosure and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of this disclosure. Unless otherwise expressly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to fixed connections, detachable connections, or integral connections; they can refer to mechanical connections or electrical connections; they can refer to direct connections or indirect connections through an intermediate medium; and they can refer to the internal communication between two components. Those skilled in the art can understand the specific meaning of the above terms in this disclosure according to the specific circumstances. It should be noted that, without conflict, the embodiments and features in the embodiments of this disclosure can be combined with each other. This disclosure is not limited to any single aspect, nor to any single embodiment, nor to any combination and / or substitution of these aspects and / or embodiments. Moreover, each aspect and / or embodiment of this disclosure can be used alone or in combination with one or more other aspects and / or embodiments thereof.

[0191] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this disclosure, and not to limit them. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this disclosure, and they should all be covered within the scope of the claims and specification of this disclosure.

Claims

1. A cloud rendering method for audio and video, characterized in that, When applied to a user terminal, the method includes: Send a connection request to the cloud rendering server; the connection request is used to request the establishment of a long connection between the cloud rendering server and the user terminal, and to start the rendering software running on the cloud rendering server; After establishing a long connection with the cloud rendering server, the signaling information between the user terminal and the rendering software is exchanged through the cloud rendering server, and a first transmission channel and a second transmission channel are established between the user terminal and the rendering software. Interactive data for triggering rendering operations is sent to the rendering software through the first transmission channel, so that the rendering software responds to the interactive data to perform the corresponding rendering operation and obtain the rendered audio and video stream; The second transmission channel receives the audio and video streams sent by the rendering software.

2. The method according to claim 1, characterized in that, The step of sending interactive data to the rendering software through the first transmission channel to trigger a rendering operation, so that the rendering software responds to the interactive data to perform a corresponding rendering operation and obtains the rendered audio and video stream, includes: Upon receiving a user input that triggers a screen switching operation, interactive data corresponding to the screen switching operation is generated. The interactive data is sent to the rendering software through the first transmission channel, so that the rendering software can obtain the rendering data corresponding to the target switching interface based on the interactive data, and render and generate the audio and video stream corresponding to the target switching interface based on the rendering data.

3. The method according to claim 1, characterized in that, The step of sending interactive data to the rendering software through the first transmission channel to trigger a rendering operation, so that the rendering software responds to the interactive data to perform a corresponding rendering operation and obtains the rendered audio and video stream, includes: When the currently displayed scene is a virtual reality scene, collect inertial sensor data from the user terminal; Interactive data containing the inertial sensor data is sent to the rendering software through the first transmission channel, so that the rendering software determines the rendering perspective based on the inertial sensor data and renders and generates an audio and video stream applied to the virtual reality scene based on the rendering perspective.

4. The method according to claim 1, characterized in that, Establishing a second transmission channel between the user terminal and the rendering software includes: Create a video tag and add an event listener to the video tag; the video tag is used to display the audio and video stream generated by the rendering software, and the event listener is used to monitor the playback and / or transmission status of the audio and video stream; Receive connection configuration information sent by the cloud rendering server; Based on the connection configuration information and the signaling information, a streaming media connection instance is established between the user terminal and the rendering software; A second transmission channel is established between the user terminal and the rendering software through the streaming media connection instance.

5. The method according to claim 1, characterized in that, The method further includes: Multiple audio and video stream slices sent by the rendering software are received through the first transmission channel; wherein the data volume of each audio and video stream slice is less than a data volume threshold. The multiple audio and video stream slices are spliced ​​together into a complete audio and video stream.

6. A cloud rendering method for audio and video, characterized in that, Applied to cloud rendering servers, the method includes: Receive connection requests sent by the user client; In response to receiving the connection request, a long connection is established between the cloud rendering server and the user terminal, and the rendering software running on the cloud rendering server is started; The rendering software and the user terminal exchange signaling information, and establish a first transmission channel and a second transmission channel between the rendering software and the user terminal. The system receives interactive data sent by the user terminal to trigger the rendering operation through the first transmission channel. The rendering software responds to the interactive data and performs corresponding rendering operations to obtain the rendered audio and video streams. The audio and video streams are sent to the user terminal through the second transmission channel.

7. A cloud rendering device for audio and video, characterized in that, The device, applied to a user terminal, includes: The first sending module is used to send a connection request to the cloud rendering server; the connection request is used to request the establishment of a long connection between the cloud rendering server and the user terminal, and to start the rendering software running on the cloud rendering server. The transmission channel establishment module is used to exchange signaling information between the user terminal and the rendering software through the cloud rendering server after establishing a long connection with the cloud rendering server, and to establish a first transmission channel and a second transmission channel between the user terminal and the rendering software. The second sending module is used to send interactive data for triggering rendering operations to the rendering software through the first transmission channel, so that the rendering software responds to the interactive data to perform corresponding rendering operations and obtain the rendered audio and video streams; The receiving module is used to receive the audio and video streams sent by the rendering software through the second transmission channel.

8. A cloud rendering device for audio and video, characterized in that, The device, applied to a cloud rendering server, includes: The first receiving module is used to receive connection requests sent by the user terminal; A connection establishment module is used to respond to receiving the connection request, establish a long connection between the cloud rendering server and the user terminal, and start the rendering software running on the cloud rendering server. The transmission channel establishment module is used to exchange signaling information between the rendering software and the user terminal, and to establish a first transmission channel and a second transmission channel between the rendering software and the user terminal. The second receiving module is used to receive interactive data sent by the user terminal to trigger rendering operations through the first transmission channel; The rendering module is used to perform corresponding rendering operations in response to the interactive data using the rendering software to obtain the rendered audio and video streams; The sending module is used to send the audio and video streams to the user terminal through the second transmission channel.

9. An electronic device, characterized in that, The electronic device includes: Memory, used to store computer programs; A processor is configured to execute a computer program stored in the memory, wherein when the computer program is executed, it implements the method described in any one of claims 1 to 5 or 6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, implements the method described in any one of claims 1 to 5 or 6.

Images