A video interaction control method based on a WebRTC protocol

Abstract

The application provides a video interaction control method based on a WebRTC protocol, and relates to the technical field of video transmission, and comprises the following steps: a cloud rendering server sends a video track to a client through the WebRTC protocol; the client sequentially checks RTP packets according to sequence numbers and timestamp information, and obtains a checked video track; the client obtains a rendering picture by processing the checked video track; the client performs multi-terminal operation adaptation on a device operation event through the rendering picture, obtains Protobuf data, and sends the Protobuf data to the cloud rendering server; the cloud rendering server receives the Protobuf data, maps the Protobuf data on the video track after analyzing the Protobuf data, obtains a video track after interaction control, and sends the video track after interaction control to the client through the WebRTC protocol. The application realizes the improvement of the interaction control efficiency.

Description

Technical Field

[0001] This invention relates to the field of video transmission technology, and more specifically, to a video interactive control method based on the WebRTC protocol. Background Technology

[0002] With the development of internet technology, video transmission technology is no longer satisfied with one-way information transmission, but places greater emphasis on two-way closed-loop control. The server renders the video data and sends it to the client, which then performs real-time operations on the received video data and sends feedback back to the server.

[0003] Related technologies often achieve video interactive control by embedding browser components in Unity, which can achieve a certain degree of real-time communication. However, since Unity is isolated from the browser runtime environment, the video screen can usually only be displayed in the form of an embedded window, resulting in low interactive control efficiency. Summary of the Invention

[0004] The problem addressed by this invention is how to improve the interactive control efficiency of video transmission.

[0005] This invention provides a video interactive control method based on the WebRTC protocol, comprising:

[0006] The cloud rendering server sends video tracks to the client via the WebRTC protocol, whereby the video tracks represent multiple RTP packets with sequence numbers and timestamp information.

[0007] The client verifies the RTP packets sequentially based on the sequence number and the timestamp information to obtain the verified video track;

[0008] The client obtains the rendered image by processing the verified video track;

[0009] The client performs multi-terminal operation adaptation on the device operation events through the rendered screen, obtains Protobuf data, and sends the Protobuf data to the cloud rendering server. The device operation events are used to represent keyboard and mouse operation events or gamepad and gyroscope operation events.

[0010] The cloud rendering server receives the Protobuf data, parses the Protobuf data, maps it onto the video track to obtain the interactively controlled video track, and sends the interactively controlled video track to the client via the WebRTC protocol.

[0011] Optionally, the step of obtaining Protobuf data by performing multi-terminal operation adaptation of device operation events through the rendered screen includes:

[0012] The rendered screen is used to adapt the keyboard and mouse operation events or the gamepad and gyroscope operation events to multiple terminals to obtain control operation commands.

[0013] The control operation instructions are format-converted to obtain the Protobuf data, wherein the Protobuf data is used to represent binary stream data in Protobuf format.

[0014] Optionally, the step of adapting the keyboard and mouse operation events or the gamepad gyroscope operation events to multiple terminals through the rendered screen to obtain control operation commands includes:

[0015] The user input event and the corresponding world coordinates are obtained through the keyboard and mouse operation events or the gamepad gyroscope operation events;

[0016] The world coordinates are transformed using the rendered image to obtain the pixel positions in the rendered image;

[0017] The pixel positions in the rendered image and the user input events are encapsulated to obtain the control operation instructions.

[0018] Optionally, the step of using the rendered image to perform coordinate transformation on the world coordinates to obtain the pixel positions in the rendered image includes:

[0019] The world coordinates are transformed into screen coordinates through projection from the main camera.

[0020] Mapping the screen coordinates to the rendered image yields the drawing layout coordinate system;

[0021] When the coordinate system of the drawing layout is within the range of the rendered screen, the coordinate system of the drawing layout is normalized to obtain the pixel position in the rendered screen.

[0022] Optionally, obtaining the user input event and corresponding world coordinates through the keyboard and mouse operation event or the gamepad gyroscope operation event includes:

[0023] When the device operation event is the keyboard and mouse operation event, the user input event and the corresponding world coordinates are obtained through the keyboard and mouse operation event;

[0024] When the device operation event is the handle gyroscope operation event, the handle gyroscope operation event is parsed and transformed to obtain the transformed handle gyroscope operation event; the user input event and the corresponding world coordinates are obtained through the transformed handle gyroscope operation event.

[0025] Optionally, the user input event includes an event type, an event state, and a timestamp. The step of encapsulating the pixel position in the rendered image and the user input event to obtain the control operation instruction includes:

[0026] The control operation instructions are obtained by encapsulating the event type, the event state, the timestamp, and the pixel position in the rendered image.

[0027] Optionally, before the cloud rendering server sends the video track to the client via the WebRTC protocol, the following steps are also included:

[0028] The client generates a local SDP session description and sends the local SDP session description to the cloud rendering server via the HTTP protocol;

[0029] The cloud rendering server receives the local SDP session description, parses the local SDP session description to obtain the remote SDP session description, and sends the remote SDP session description to the client via the HTTP protocol. The remote SDP session description is used to establish the WebRTC protocol.

[0030] Optionally, before the client generates a local SDP session description and sends the local SDP session description to the cloud rendering server via the HTTP protocol, the method further includes:

[0031] The client generates the ICE configuration and sends the ICE configuration to the cloud rendering server via the HTTP protocol;

[0032] The cloud rendering server receives the ICE configuration, obtains ICE configuration information based on the ICE configuration, and sends the ICE configuration information to the client via the HTTP protocol.

[0033] Optionally, the step of processing the verified video track to obtain the rendered image includes:

[0034] Obtain the client type, wherein the client type includes Windows device, Android device and PicoXR device;

[0035] The verified video track is processed according to the client type to obtain the rendered image.

[0036] Optionally, sending the Protobuf data to the cloud rendering server includes:

[0037] The Protobuf data is sent to the cloud rendering server via the WebSocket protocol.

[0038] The beneficial effects of the video interactive control method based on the WebRTC protocol of this invention are as follows: The cloud rendering server uses WebRTC as the transmission protocol to send the video track to the client, effectively reducing the latency of audio and video transmission. The client verifies the RTP packets sequentially according to the sequence number and timestamp information, effectively detecting and handling packet loss and other problems in network transmission, ensuring the integrity and timing accuracy of the video stream. The client further processes the verified video track to obtain the rendered image, performs multi-terminal operation adaptation to obtain Protobuf data, and uniformly converts the input from different devices into Protobuf format, reducing the transmission latency of control commands and realizing real-time interaction. The cloud rendering server parses the Protobuf data and maps it onto the video track, performs relevant processing on the video track according to user operations, obtains the interactively controlled video track, and sends the interactively controlled video track to the client, thereby improving the interactive control efficiency of video transmission. Attached Figure Description

[0039] Figure 1 This is a flowchart illustrating a video interactive control method based on the WebRTC protocol according to an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram of a WebRTC protocol establishment process according to an embodiment of the present invention. Detailed Implementation

[0041] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. Although some embodiments of the present invention are shown in the drawings, it should be understood that the present invention can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present invention. It should be understood that the accompanying drawings and embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of protection of the present invention.

[0042] It should be understood that the various steps described in the method embodiments of the present invention may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present invention is not limited in this respect.

[0043] The term "comprising" and its variations as used herein are open-ended, meaning "including but not limited to"; the term "based on" means "at least partially based on"; the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments"; and the term "optionally" means "optional embodiments". Definitions of other terms will be given in the following description. It should be noted that the concepts of "first," "second," etc., mentioned in this invention are used only to distinguish different devices, modules, or units, and are not intended to limit the order of functions performed by these devices, modules, or units or their interdependencies.

[0044] It should be noted that the terms "one" and "more" used in this invention are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0045] The names of the messages or information exchanged between the multiple devices in the embodiments of the present invention are for illustrative purposes only and are not intended to limit the scope of these messages or information.

[0046] In related technologies, a Unity client refers to an application developed based on the Unity engine and running on a terminal device, serving as the front-end interface for users to interact with backend services (such as cloud rendering servers). Traditional Unity streaming methods are mostly based on protocols such as RTMP / HLS, which do not support real-time communication, have high latency, and lack native WebRTC support. While WebRTC is widely used in browsers, direct integration in Unity and XR devices presents compatibility and performance issues. A solution involves embedding a browser component within Unity to load a WebRTC-enabled webpage, thereby indirectly achieving video streaming and playback. This method relies on the browser's own support for the WebRTC protocol, achieving a certain degree of real-time communication and avoiding the complexity of underlying protocol parsing. However, because the Unity and browser runtime environments are isolated, video footage is typically displayed only in an embedded window and cannot be used as Unity textures for scene rendering, object mapping, or spatial interaction, making it difficult to meet the needs of immersive XR scenarios. Furthermore, this approach suffers from poor performance control; the rendering threads between the browser and Unity are independent, resulting in high resource consumption and poor device compatibility, especially on Android XR devices where it is practically unusable.

[0047] like Figure 1 As shown in the figure, an embodiment of the present invention provides a video interactive control method based on the WebRTC protocol, comprising:

[0048] Step 110: The cloud rendering server sends a video track to the client via the WebRTC protocol, wherein the video track is used to represent multiple RTP packets with sequence numbers and timestamp information.

[0049] Specifically, the Unity client starts, loads the network communication module and the rendering adaptation module, detects the current running platform (desktop or XR device) and the supported graphics APIs (DirectX, OpenGL, Vulkan), and prepares for subsequent rendering strategy selection.

[0050] Step 120: The client verifies the RTP packets sequentially according to the sequence number and the timestamp information to obtain the verified video track.

[0051] Specifically, the remote video source sends video tracks to the client via WebRTC's RTP (Real-Time Transport Protocol). Each RTP packet received by the system contains a sequence number and a timestamp. The client reads the sequence number field in the RTP header to determine if the current packet arrived in order and buffers and reassembles multiple packets according to frame identifiers. When a sequence number jump is detected, the system determines it as a packet loss. If the packet cannot be reassembled within the waiting time, the system will, based on the current position and number of lost packets, choose to initiate a NACK (Negative Acknowledgement) request to trigger remote retransmission, or directly skip the frame and use the previous frame for fault-tolerant frame replacement. After frame assembly, the reassembly module hands the complete frame over to the decoding module for processing and stores it in the playback queue according to the timestamp order. To ensure audio and video synchronization, the system uses RTP timestamps as a synchronization reference to align the presentation rhythm of video and audio frames and ensure consistent playback timing.

[0052] After media track negotiation is complete, the Unity client registers an OnTrack event callback to receive audio and video tracks from the server. When a new video track is sent remotely via RTP, the WebRTC library automatically triggers the OnTrack event. The system extracts the VideoStreamTrack object in the callback function and binds it to the subsequent rendering module.

[0053] Step 130: The client processes the verified video track to obtain the rendered image.

[0054] Specifically, the client's local video decoding module decodes the received compressed video, supporting multiple encoding formats. The decoded video frames are converted into Unity-compatible texture formats based on platform requirements. The system uses a platform type determination logic, employing `Application.platform` to identify whether the current operating environment is Windows, Android, or a Pico XR device, and selects the appropriate rendering path. When an XR device is detected as the target platform, the system enables a dual-camera, dual-channel rendering mechanism. Based on the capabilities provided by the XR device SDK, two independent cameras are created to render the same scene from both left and right perspectives. The rendered images are then combined and transmitted to the head-mounted display device to complete the stereoscopic visual output.

[0055] Step 140: The client performs multi-terminal operation adaptation on the device operation events through the rendered screen, obtains Protobuf data, and sends the Protobuf data to the cloud rendering server. The device operation events are used to represent keyboard and mouse operation events or gamepad and gyroscope operation events.

[0056] Specifically, users generate control commands through different input devices. These commands are processed by the multi-device operation adaptation engine, which distributes the commands to the corresponding processing paths based on the device type.

[0057] Step 150: The cloud rendering server receives the Protobuf data, parses the Protobuf data and maps it onto the video track to obtain the interactively controlled video track, and sends the interactively controlled video track to the client through the WebRTC protocol.

[0058] In some more specific embodiments, a frame buffer queue is established, receiving frames are sorted according to timestamps, and frame drops are detected to trigger a compensation mechanism. The buffer size can be dynamically adjusted according to real-time network conditions to reduce stuttering caused by network jitter. Simultaneously, audio and video synchronization is maintained to ensure playback stability. Network quality, decoding status, and rendering performance are monitored in real time. When network fluctuations are severe, the bitrate or resolution is automatically reduced; when decoding fails, the previous valid frame is used to fill the gap; when XR device rendering is abnormal, it switches to monocular preview or reduces the frame rate to ensure stable system operation. The Unity client displays the rendered video frames in scene objects or the UI interface, allowing users to interact simultaneously. Scene status and video content are synchronized in real time, achieving low-latency, high-performance, cross-platform WebRTC video stream access and rendering effects.

[0059] In this embodiment, the cloud rendering server uses WebRTC as the transmission protocol to send the video track to the client, effectively reducing audio and video transmission latency. The client verifies the RTP packets sequentially based on the sequence number and timestamp information, effectively detecting and handling packet loss and other issues in network transmission, ensuring the integrity and timing accuracy of the video stream. The client further processes the verified video track to obtain the rendered image, performs multi-device operation adaptation to obtain Protobuf data, and uniformly converts inputs from different devices into Protobuf format, reducing transmission latency of control commands and enabling real-time interaction. The cloud rendering server parses the Protobuf data and maps it onto the video track, processes the video track according to user operations, obtains the interactively controlled video track, and sends the interactively controlled video track to the client, thereby improving the interactive control efficiency of video transmission.

[0060] Optionally, the step of obtaining Protobuf data by performing multi-terminal operation adaptation of device operation events through the rendered screen includes:

[0061] The rendered screen is used to adapt the keyboard and mouse operation events or the gamepad and gyroscope operation events to multiple terminals to obtain control operation commands.

[0062] The control operation instructions are format-converted to obtain the Protobuf data, wherein the Protobuf data is used to represent binary stream data in Protobuf format.

[0063] Specifically, Protobuf is a serialized structured data format used for efficiently storing or transmitting structured data. By using structured definition and binary serialization, it overcomes the performance bottleneck of traditional text formats in real-time interactive scenarios, providing a low-latency control channel for cloud rendering systems. State needs to be encapsulated into transmittable data objects. Mouse events are constructed into structured objects containing event type, mouse position, button state, and timestamp; keyboard events are constructed into structured objects containing event type, virtual keycode, button state, timestamp, and position information. These are then uniformly packaged into byte arrays of Protobuf structure and sent to the cloud rendering server via the WebSocket (WS) channel. On the server side, the received Protobuf data is parsed into specific control commands, which are then executed by the system, enabling remote devices to interactively control the rendered content.

[0064] Optionally, the step of adapting the keyboard and mouse operation events or the gamepad gyroscope operation events to multiple terminals through the rendered screen to obtain control operation commands includes:

[0065] The user input event and the corresponding world coordinates are obtained through the keyboard and mouse operation events or the gamepad gyroscope operation events;

[0066] The world coordinates are transformed using the rendered image to obtain the pixel positions in the rendered image;

[0067] The pixel positions in the rendered image and the user input events are encapsulated to obtain the control operation instructions.

[0068] Optionally, the step of using the rendered image to perform coordinate transformation on the world coordinates to obtain the pixel positions in the rendered image includes:

[0069] The world coordinates are transformed into screen coordinates through projection from the main camera.

[0070] Mapping the screen coordinates to the rendered image yields the drawing layout coordinate system;

[0071] When the coordinate system of the drawing layout is within the range of the rendered screen, the coordinate system of the drawing layout is normalized to obtain the pixel position in the rendered screen.

[0072] Specifically, main camera projection conversion refers to projecting 3D world points onto a 2D image plane after camera calibration, converting world coordinates into screen coordinates.

[0073] Optionally, obtaining the user input event and corresponding world coordinates through the keyboard and mouse operation event or the gamepad gyroscope operation event includes:

[0074] When the device operation event is the keyboard and mouse operation event, the user input event and the corresponding world coordinates are obtained through the keyboard and mouse operation event;

[0075] When the device operation event is the handle gyroscope operation event, the handle gyroscope operation event is parsed and converted to obtain the converted handle gyroscope operation event;

[0076] The user input event and the corresponding world coordinates are obtained through the converted handle gyroscope operation event.

[0077] Specifically, when the device operation event is a keyboard and mouse operation event, i.e., it applies to Windows, the input events mainly include mouse and keyboard operations. By using Unity's built-in Input class in the Update function (called once per frame), the input state of each frame is captured, including mouse coordinates, click state, scroll wheel action, and keyboard key press and release events.

[0078] To enable precise click control of remotely rendered images by users on their local terminals, the transmission of mouse position information implements a mapping mechanism from 3D world coordinates to remote 2D coordinates. When a user selects a target location in the scene, the system first converts its world coordinates into screen coordinates through the main camera projection. Based on the UI component carrying the remote image, this screen point is mapped to the canvas local coordinate system. If the click does not hit the canvas area, it is considered invalid and the process is terminated. After successfully acquiring the local coordinates, the system normalizes the coordinates according to the canvas size and reverses the Y-axis direction to ensure consistency with the remotely rendered image. Subsequently, based on the resolution of the remote video stream, the normalized coordinates are converted into remote pixel coordinates, and boundary protection is used to prevent out-of-bounds errors. Finally, the pixel coordinates are further mapped into high-precision 16-bit integer coordinates, which serve as the mouse position part of the remote interaction command.

[0079] When the device operation event is a gyroscope operation event on the controller, i.e., applied to XR devices such as Pico, the input information received by the system is more complex. The system continuously detects and collects device sensor data through the native interface provided by the glasses SDK, and performs multi-channel parallel parsing according to the device type. The left-hand controller joystick input is mapped to keyboard events. When the joystick is pushed beyond a set threshold, its directional input (forward, backward, left, right) is mapped to the key actions of a traditional keyboard (W, S, A, D). The right-hand controller joystick input is used to simulate mouse dragging operations. When the user pushes the joystick into the effective range, the system converts it into a mouse press state and converts the joystick direction and displacement amplitude into continuous relative mouse movement events, realizing smooth dragging of the remote screen or viewpoint. When the joystick returns to the center or falls below the activation threshold, the system triggers a drag end signal, releasing the simulated mouse button and forming a complete closed loop. The right-hand trigger button is used to control click behavior. The system calculates spatial rays based on the current position and orientation of the controller and performs hit detection. If the ray hits an object in the remote view, the system converts the world coordinates of the hit point into a pixel position on the remote video canvas, and then normalizes it to the coordinate values ​​required by the standard input protocol. This position then triggers a simulated mouse click (press and release) event, enabling remote click interaction.

[0080] In this optional embodiment, control information generated on the user side is transmitted through the data channel provided by WebRTC. Various control events (such as mouse, keyboard, gamepad, gyroscope, and gestures) are processed by a unified adaptation engine and encapsulated into a structured data format, which is then sent to the server in an efficient binary manner. The server parses the data and synchronously updates the scene state or executes user commands, thereby achieving real-time synchronization between video rendering and scene interaction.

[0081] Optionally, the user input event includes an event type, an event state, and a timestamp. The step of encapsulating the pixel position in the rendered image and the user input event to obtain the control operation instruction includes:

[0082] The control operation instructions are obtained by encapsulating the event type, the event state, the timestamp, and the pixel position in the rendered image.

[0083] Specifically, the system collects user input (including mouse clicks, movements, keyboard presses, XR controller operations, etc.) in each frame and encapsulates it into structured event objects. Control operation instructions include event type, pixel position in the rendered screen (input coordinates or key values), event state (pressed / released), and timestamp.

[0084] Optionally, before the cloud rendering server sends the video track to the client via the WebRTC protocol, the following steps are also included:

[0085] The client generates a local SDP session description and sends the local SDP session description to the cloud rendering server via the HTTP protocol;

[0086] The cloud rendering server receives the local SDP session description, parses the local SDP session description to obtain the remote SDP session description, and sends the remote SDP session description to the client via the HTTP protocol. The remote SDP session description is used to establish the WebRTC protocol.

[0087] Optionally, before the client generates a local SDP session description and sends the local SDP session description to the cloud rendering server via the HTTP protocol, the method further includes:

[0088] The client generates the ICE configuration and sends the ICE configuration to the cloud rendering server via the HTTP protocol;

[0089] The cloud rendering server receives the ICE configuration, obtains ICE configuration information based on the ICE configuration, and sends the ICE configuration information to the client via the HTTP protocol.

[0090] Specifically, such as Figure 2 As shown, Figure 2This diagram illustrates the WebRTC protocol setup process between the client and the cloud rendering server. The client sends the ICE configuration to the cloud rendering server via HTTP to retrieve ICE configuration information. This request carries the device identifier and resolution parameters. The server responds with structured JSON data, including the STUN / TURN server address, authentication credentials, and a session token. After obtaining the configuration, the client creates an RTCPeerConnection instance locally and generates a local SDP session description (SDP Offer). The generated SDP Offer is then transmitted to the cloud rendering server via HTTP. The cloud rendering server parses the Offer, generates the corresponding remote SDP session description (SDP Answer), and returns it to the client. Upon receiving the SDP Answer, the client calls the SetRemoteDescription() method to establish media reception consensus. Subsequently, the ICE candidate exchange phase begins. The client continuously collects local network candidate addresses, automatically eliminates invalid candidates through an intelligent filtering mechanism, encapsulates valid candidates into binary data streams, and incrementally transmits them to the cloud rendering server via the control channel. Both the client and the cloud rendering server send ICE Candidates (candidate addresses). ICE (Interactive Connectivity Establishment) is used to establish P2P (peer-to-peer) network connections in audio and video calls (such as WebRTC).

[0091] In some more specific embodiments, the client first generates a local SDPoffer using the CreateOffer() method and sets it to a local description using the SetLocal Description() method. Then, it connects to the signaling server via HTTP, sending a session request along with parameters such as platform type, resolution, and frame rate. The signaling server parses the request, generates and returns remote SDP (Session Description Protocol) information. The client then uses the ICE (Interactive Connection Establishment) algorithm to collect candidate paths and test connectivity, selecting the optimal peer-to-peer transmission path. Simultaneously, a DTLS (Datagram Transport Layer Security) encrypted channel is established to ensure data security.

[0092] In this optional embodiment, the native integration of the WebRTC protocol in Unity includes SDP negotiation, ICE path establishment, RTP video track reception and texture decoding process, and is adapted for XR device environment to form a video streaming capability with cross-platform compatibility, low latency and high stability.

[0093] Optionally, the step of processing the verified video track to obtain the rendered image includes:

[0094] Obtain the client type, wherein the client type includes Windows device, Android device and PicoXR device;

[0095] The verified video track is processed according to the client type to obtain the rendered image.

[0096] Optionally, sending the Protobuf data to the cloud rendering server includes:

[0097] The Protobuf data is sent to the cloud rendering server via the WebSocket protocol.

[0098] Specifically, WebSocket is a network protocol that enables full-duplex, bidirectional real-time communication over a single TCP connection, designed to address the limitations of the traditional HTTP protocol in real-time communication.

[0099] While the present invention has been disclosed above, its scope of protection is not limited thereto. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention, and all such changes and modifications will fall within the scope of protection of the present invention.

Claims

1. A video interactive control method based on the WebRTC protocol, characterized in that, include: The cloud rendering server sends video tracks to the client via the WebRTC protocol, whereby the video tracks represent multiple RTP packets with sequence numbers and timestamp information. The client verifies the RTP packets sequentially based on the sequence number and the timestamp information to obtain the verified video track; The client obtains the rendered image by processing the verified video track; The client performs multi-terminal operation adaptation on the device operation events through the rendered screen, obtains Protobuf data, and sends the Protobuf data to the cloud rendering server. The device operation events are used to represent keyboard and mouse operation events or gamepad and gyroscope operation events. The cloud rendering server receives the Protobuf data, parses the Protobuf data, maps it onto the video track to obtain the interactively controlled video track, and sends the interactively controlled video track to the client via the WebRTC protocol.

2. The video interactive control method based on the WebRTC protocol according to claim 1, characterized in that, The process of obtaining Protobuf data by adapting device operation events through the rendered screen to multiple terminals includes: The rendered screen is used to adapt the keyboard and mouse operation events or the gamepad and gyroscope operation events to multiple terminals to obtain control operation commands. The control operation instructions are format-converted to obtain the Protobuf data, wherein the Protobuf data is used to represent binary stream data in Protobuf format.

3. The video interactive control method based on the WebRTC protocol according to claim 2, characterized in that, The step of adapting keyboard and mouse operation events or gamepad gyroscope operation events to multiple terminals through the rendered screen to obtain control operation commands includes: The user input event and the corresponding world coordinates are obtained through the keyboard and mouse operation events or the gamepad gyroscope operation events; The world coordinates are transformed using the rendered image to obtain the pixel positions in the rendered image; The pixel positions in the rendered image and the user input events are encapsulated to obtain the control operation instructions.

4. The video interactive control method based on the WebRTC protocol according to claim 3, characterized in that, The step of using the rendered image to perform coordinate transformation on the world coordinates to obtain the pixel positions in the rendered image includes: The world coordinates are transformed into screen coordinates through projection from the main camera. Mapping the screen coordinates to the rendered image yields the drawing layout coordinate system; When the coordinate system of the drawing layout is within the range of the rendered screen, the coordinate system of the drawing layout is normalized to obtain the pixel position in the rendered screen.

5. The video interactive control method based on the WebRTC protocol according to claim 3, characterized in that, The process of obtaining user input events and corresponding world coordinates through the keyboard and mouse operation events or the gamepad gyroscope operation events includes: When the device operation event is the keyboard and mouse operation event, the user input event and the corresponding world coordinates are obtained through the keyboard and mouse operation event; When the device operation event is the handle gyroscope operation event, the handle gyroscope operation event is parsed and transformed to obtain the transformed handle gyroscope operation event; the user input event and the corresponding world coordinates are obtained through the transformed handle gyroscope operation event.

6. The video interactive control method based on the WebRTC protocol according to claim 3, characterized in that, The user input event includes an event type, an event state, and a timestamp. The process of encapsulating the pixel positions in the rendered image and the user input event to obtain the control operation instructions includes: The control operation instructions are obtained by encapsulating the event type, the event state, the timestamp, and the pixel position in the rendered image.

7. The video interactive control method based on the WebRTC protocol according to claim 1, characterized in that, Before the cloud rendering server sends the video track to the client via the WebRTC protocol, the following is also included: The client generates a local SDP session description and sends the local SDP session description to the cloud rendering server via the HTTP protocol; The cloud rendering server receives the local SDP session description, parses the local SDP session description to obtain the remote SDP session description, and sends the remote SDP session description to the client via the HTTP protocol. The remote SDP session description is used to establish the WebRTC protocol.

8. The video interactive control method based on the WebRTC protocol according to claim 7, characterized in that, Before the client generates a local SDP session description and sends the local SDP session description to the cloud rendering server via HTTP protocol, the following steps are also included: The client generates the ICE configuration and sends the ICE configuration to the cloud rendering server via the HTTP protocol; The cloud rendering server receives the ICE configuration, obtains ICE configuration information based on the ICE configuration, and sends the ICE configuration information to the client via the HTTP protocol.

9. The video interactive control method based on the WebRTC protocol according to claim 1, characterized in that, The process of processing the verified video track to obtain the rendered image includes: Obtain the client type, wherein the client type includes Windows devices, Android devices, and Pico XR devices; The verified video track is processed according to the client type to obtain the rendered image.

10. The video interactive control method based on the WebRTC protocol according to claim 1, characterized in that, Sending the Protobuf data to the cloud rendering server includes: The Protobuf data is sent to the cloud rendering server via the WebSocket protocol.

Images