Live broadcast method and system based on Root proxy mechanism

Abstract

The invention relates to the technical field of live broadcast systems, and discloses a live broadcast system and method based on a Root proxy mechanism, and the method comprises the steps: obtaining a local video stream, a remote microphone-connected video stream, local image configuration information and control operation information through various communication protocols, and after obtaining full-amount information, carrying out the remote microphone-connected video stream; compared with the prior art, the method has the advantages that rendering and stream pushing tasks which can only run on a client originally are converted to Function Compute, picture configuration and live broadcast control are creatively reserved locally, complex rendering and stream pushing with the maximum performance consumption are placed in Function Compute, and the live broadcast effect and the live broadcast performance are perfectly balanced.

Description

Technical Field

[0001] The present invention relates to the technical field of live broadcast systems, and in particular to a live broadcast system and method based on a Robot proxy mechanism. Background Art

[0002] Live streaming has now become a very important media form on the Internet, and has shined in the fields of knowledge learning, interactive entertainment, e-commerce, etc.

[0003] The basic principle of live streaming is to shoot video images (for example, using a camera, webcam + computer, mobile phone, etc.) and push the video images to a cloud server in the form of a video stream, which is then sent from the cloud server to the audience's viewing devices (such as mobile phones, computers, TVs, etc.).

[0004] With the rapid development of mobile phone hardware and mobile networks, the performance of mobile phones is fully capable of outdoor live broadcasts, entertainment live broadcasts, or scenes with low requirements for shooting images. In this type of live broadcast, the software used for shooting and streaming is generally the APP provided by various video platforms, and the live broadcast image is mainly based on the video image itself, without too many packaging elements.

[0005] However, in some typical industries, such as finance, medicine, education and other vertical fields, live broadcasts in such industries are generally in the form of knowledge sharing, remote roundtable forums, etc., and have high requirements for live picture packaging, audience interaction, and picture quality. Simply using mobile phones as shooting and streaming carriers cannot meet the needs, so this application proposes a live broadcast system and method based on the Robot agent mechanism. Summary of the invention

[0006] The purpose of the present invention is to provide a live broadcast system and method based on the Robot agent mechanism to solve the above technical problems.

[0007] A live broadcast system based on Robot proxy mechanism, the system comprising: Puppeteer module, rendering and streaming module;

[0008] The Puppeteer module is used to control HeadlessChrome to simulate the behavior of all remote guests participating in the live broadcast joining the microphone room through the Web, in order to obtain the same picture as the main control end, and is also used to serve as a bridge for direct communication between the web page and the nodeserver;

[0009] The rendering and streaming module is used to process the video content obtained by the Puppeteer module, stream it to multiple live servers, and monitor the streaming status in real time;

[0010] The Puppeteer module and the rendering and streaming module run on the same FunctionCompute service.

[0011] As a further description of the solution of the present invention, the specific working process of the Puppeteer module includes:

[0012] Simulation and verification of remote guest behavior:

[0013] The Puppeteer module controls HeadlessChrome to simulate the behavior of all remote guests joining the live broadcast and microphone room;

[0014] During the simulation, Puppeteer triggers the same WebRTC connection process as actual users, including ICE connection establishment, SDP exchange, publishing and receiving audio and video streams, etc. Through this simulation, potential network or device compatibility issues can be detected and resolved in advance.

[0015] Communication bridge between web page and NodeServer:

[0016] Based on Puppeteer's evaluate method, the server-side Node.js code can interact with the JavaScript code in the page environment through Puppeteer to achieve instant data transmission and operation; efficient data operation and page loading optimization:

[0017] When specific metadata needs to be injected into the WebRTC stream, the Node.js server can write the data directly into the JavaScript environment of the page through Puppeteer;

[0018] By preloading critical resources, delaying the loading of non-critical resources, and dynamically adjusting the loading order during page initialization;

[0019] FunctionCompute interaction with the Web client:

[0020] Allows the server to send key control commands and data to the FunctionCompute service through WebSocket or other real-time communication protocols while controlling page loading and operation. After FunctionCompute receives these commands, it can immediately adjust the push stream.

[0021] As a further description of the solution of the present invention, the specific working process of the rendering and streaming module includes:

[0022] Page content capture and video stream generation:

[0023] On the server side, the same screen content as the control side is obtained synchronously through the input device, and the captured content is directly processed in the form of a video stream to generate the original data stream for streaming;

[0024] Video stream encoding and processing:

[0025] Select the appropriate codec for video encoding according to actual needs. During the encoding process, apply different video processing filters to optimize the output effect;

[0026] Push streams to multiple live servers:

[0027] After encoding is completed, the video stream is pushed to multiple specified live servers, and multiple output streams are created at the same time. Each stream corresponds to the RTMP address of a live server. Through parameter configuration, multiple streams can be pushed in the same operation;

[0028] Real-time streaming status monitoring and fault recovery:

[0029] Real-time tracking of streaming status, including bandwidth usage, frame loss rate, and latency parameters. If a network interruption or server failure occurs during streaming, the system can automatically retry the connection or switch to a backup server.

[0030] Audio and video synchronization processing:

[0031] During the streaming process, the built-in synchronization mechanism ensures that the video frames and audio frames are accurately matched according to the timestamps, thus avoiding the misalignment of the picture and sound.

[0032] If synchronization deviation is detected, the system automatically adjusts the delay of the audio or video, recalibrating the timing relationship between the two to maintain high-quality streaming output.

[0033] As a further description of the solution of the present invention, the working process of synchronously acquiring the same screen content as the control end includes:

[0034] The audio and video streams of all users in the connected room are synchronously obtained through the WebRTC mechanism, and the document information is synchronously obtained through the Parse-SDK-JS mechanism.

[0035] As a further description of the solution of the present invention, the specific working process of synchronously acquiring the audio and video streams of all users in the connected microphone room through the WebRTC mechanism includes:

[0036] WebRTC audio and video stream processing:

[0037] The system receives the audio and video streams from the remote user through the RTCPeerConnection interface of WebRTC, and the audio and video streams are bound to the HTMLVideoElement in the web page for further processing;

[0038] Video Stream:

[0039] Whenever HTMLVideoElement updates the video frame, it uses the drawImage method of the CanvasAPI to draw the current frame to <canvas>Elements, <canvas>The element is used to carry and process video frame data;

[0040] Video frame processing further includes scaling operations, which adjust the display ratio of the video frame to adapt to different resolutions and display requirements by calling the scale method of the CanvasAPI;

[0041] Audio Stream:

[0042] The audio stream is processed through WebRTC's RTCPeerConnection, using the Opus codec, and performing noise suppression and echo cancellation.

[0043] As a further description of the solution of the present invention, the working process of synchronously obtaining document information through the Parse-SDK-JS mechanism includes:

[0044] The client uses the HTTP protocol upgrade mechanism to establish a WebSocket connection with the Parse LiveQuery Server. Once the connection is established, the communication between the client and the server can be full-duplex data transmission over the WebSocket connection;

[0045] After establishing a WebSocket connection, the client sends a subscription message in JSON format through WebSocket. After receiving the subscription request, the server will parse the query conditions and create a subscription record for the query internally;

[0046] As a further description of the solution of the present invention, in order to maintain the activity of the WebSocket connection, the client and the server usually detect the connection status by sending heartbeat packets (ping / pong frames) to avoid connection interruptions caused by network fluctuations or firewall policies;

[0047] The sending frequency of the heartbeat packet can be adjusted through configuration to ensure that the connection remains stable when there is no data transmission for a long time.

[0048] A live broadcast method of a live broadcast system based on a Robot proxy mechanism, the method comprising the following steps:

[0049] Step S1, obtaining local video stream, remote video stream, local screen configuration information, and control operation information through a communication protocol;

[0050] Step S2, leaving the screen configuration and live broadcast control to local operation;

[0051] Step S3: After obtaining all the information, convert the rendering and streaming tasks to FunctionCompute operations.

[0052] Beneficial effects of the present invention:

[0053] The present invention obtains local video streams, remote video streams, local screen configuration information, and control operation information through various communication protocols. After obtaining all the information, the rendering and streaming tasks that could only be run on the client are converted to Function Compute, creatively leaving the screen configuration and live broadcast control locally, while placing the complex and most performance-intensive rendering and streaming on Function Compute, thereby perfectly balancing the live broadcast effect and live broadcast performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The present invention will be further described below in conjunction with the accompanying drawings.

[0055] Figure 1 It is a schematic diagram of the principle of a live broadcast system based on the Robot agent mechanism provided by the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] The present invention is a live broadcast system based on the Robot agent mechanism. The core technology of the present invention, which is also the key point different from other similar live broadcast systems, is the introduction of the Robot agent mechanism.

[0058] In simple terms, the Robot agent is a combination of a series of functional modules running on Function Compute. Its main functions are as follows: Figure 1 As shown in the figure. Through various communication protocols, local video streams, remote video streams, local screen configuration information, and control operation information are obtained. After obtaining all the information, the rendering and streaming tasks that were originally only run on the client are converted to Function Compute. Live streaming software can only render and stream locally, and conference software can only perform simple confluence in the cloud.

[0059] The Robot proxy mechanism introduced in the present invention creatively keeps the screen configuration and live broadcast control locally, while placing the complex and most performance-intensive rendering and streaming in Function Compute, which perfectly balances the live broadcast effect and live broadcast performance.

[0060] The system includes: a Puppeteer module, a rendering and streaming module;

[0061] The Puppeteer module is used to control HeadlessChrome to simulate the behavior of all remote guests participating in the live broadcast joining the microphone room through the Web, in order to obtain the same picture as the main control end, and is also used to serve as a bridge for direct communication between the web page and the nodeserver;

[0062] The rendering and streaming module is used to process the video content obtained by the Puppeteer module, stream it to multiple live servers, and monitor the streaming status in real time;

[0063] The Puppeteer module and the rendering and streaming module run on the same FunctionCompute service.

[0064] The specific working process of the Puppeteer module includes:

[0065] Simulation and verification of remote guest behavior:

[0066] The Puppeteer module controls HeadlessChrome to simulate the behavior of all remote guests joining the live broadcast room. In this way, the system can pre-verify whether the audio and video connection of the remote guests is normal during the testing phase, and ensure that in the actual live broadcast, the picture seen by each participant is consistent with the main control end.

[0067] During the simulation, Puppeteer triggers the same WebRTC connection process as actual users, including ICE connection establishment, SDP exchange, publishing and receiving audio and video streams, etc. Through this simulation, potential network or device compatibility issues can be detected and resolved in advance.

[0068] Communication bridge between web page and NodeServer:

[0069] The Puppeteer module is not only responsible for the behavior simulation of the remote guest, but also for the direct communication between the web page and the Node.js server. It is similar to a nativeJSbridge, which can call JavaScript functions in the page on the Node.js server side and vice versa, thus achieving two-way communication.

[0070] Based on Puppeteer's evaluate method, the Node.js code on the server side can interact with the JavaScript code in the page environment through Puppeteer. Through this efficient direct communication, the system can achieve instant data transmission and operation, avoiding the delay problem in traditional communication methods.

[0071] Efficient data operation and page loading optimization:

[0072] In order to speed up the page response speed, the present invention uses the capabilities of Puppeteer to implement direct operations on the data in the page. For example, when it is necessary to inject specific metadata into the WebRTC stream, the Node.js server can directly write the data into the JavaScript environment of the page through Puppeteer without going through the middle layer processing;

[0073] In addition, Puppeteer is also used to optimize page loading performance. For example, the system can significantly improve page loading speed and responsiveness by preloading critical resources, delaying the loading of non-critical resources, and dynamically adjusting the loading order during page initialization.

[0074] FunctionCompute interaction with the Web client:

[0075] The WebRTC application of the present invention needs to interact closely with the server streaming end running on FunctionCompute. The Puppeteer module serves as a key bridge in this interaction process, enabling the Web client to efficiently exchange data with FunctionCompute through the Node.js server.

[0076] Specifically, Puppeteer allows the server to send key control commands and data to the FunctionCompute service through WebSocket or other real-time communication protocols while controlling page loading and operation. After receiving these commands, FunctionCompute can immediately adjust the streaming, such as switching audio and video sources, dynamically adjusting the bit rate, or switching live broadcast scenes.

[0077] Real-time communication and optimization strategies:

[0078] In practical applications, the Puppeteer module ensures real-time synchronization of WebRTC streams and page states through an efficient communication mechanism. In order to further improve communication efficiency, the present invention also designs a data compression and batch processing mechanism to minimize communication overhead while ensuring low latency.

[0079] At the same time, the system optimizes Puppeteer's task scheduling to ensure that when multiple tasks are running in parallel, resources can be reasonably allocated between different tasks to avoid resource contention and performance bottlenecks. Through this optimization, Puppeteer can run stably in high-concurrency scenarios and provide consistent performance.

[0080] The specific process of task scheduling includes:

[0081] Get the number of tasks running in parallel n;

[0082] Get the required memory size R of each task i and the required flow rate Q i , where i∈(1,2,…,n);

[0083] By formula Find the emergency value C of each task i ;

[0084] For each emergency value C i Sort by size, and prioritize server resources based on the size-to-size order.

[0085] Among them, α1 and α2 are proportional coefficients, D i is the distance between the server node and the user.

[0086] The specific working process of the rendering and streaming module includes:

[0087] Page content capture and video stream generation:

[0088] On the server side, input devices such as x11grab (Linux) or gdigrab (Windows) are used to synchronously obtain the same screen content as the control side. The content can include the final video frame rendered by Canvas, animation effects, virtual backgrounds, and other dynamic elements.

[0089] The captured content is directly processed in the form of a video stream to generate a raw data stream for streaming; during the capture process, users can specify an area or window ID to accurately capture a specific part of the page.

[0090] Video stream encoding and processing:

[0091] Through the built-in multiple video codecs, such as H.264, H.265, VP8, VP9, ​​etc., the system can select the appropriate codec for video encoding according to actual needs. During the encoding process, different video processing filters (such as scaling, cropping, overlaying text or watermarks, etc.) are applied to optimize the output effect; for example, the video can be denoised before streaming or the resolution and bit rate of the video can be dynamically adjusted to adapt to different network conditions and live broadcast platform requirements.

[0092] Push streams to multiple live servers:

[0093] After encoding is completed, the video stream is pushed to multiple designated live broadcast servers. The streaming operation is usually implemented through the RTMP (Real-Time Messaging Protocol) protocol, which is a standard protocol currently widely used in live broadcasts.

[0094] To support multi-channel streaming, FFmpeg can create multiple output streams at the same time. Each stream corresponds to the RTMP address of a live server. Through parameter configuration, multiple streams can be pushed in the same operation.

[0095] Real-time streaming status monitoring and fault recovery:

[0096] Track the streaming status in real time, including bandwidth usage, frame loss rate and latency parameters. If a network interruption or server failure occurs during streaming, the system can automatically retry the connection or switch to a backup server to ensure the continuity and stability of the live broadcast.

[0097] The process of real-time tracking of the streaming status includes:

[0098] Real-time monitoring of the data transmission rate V, the data frame loss ratio K, and the time T required for the receiving end to receive the data during the streaming process;

[0099] The data transmission rate V, the data frame loss ratio K, and the time T required for the receiving end to receive the data during the streaming process are compared with the corresponding limit values:

[0100] If there are parameter items that do not meet the limit value, it is judged that the network is disconnected or the server fails, and the system will automatically retry the connection or switch to the backup server immediately.

[0101] If there is no parameter item that does not meet the limit value, the streaming state is judged and a streaming state coefficient calculation model is constructed. The expression is as follows:

[0102]

[0103] In the formula, is the streaming state coefficient, V0, K0 and T0 are the preset data transmission rate, data frame loss ratio and the limit value of the time required for the receiving end to receive data during the streaming process, ρ V , K and ρ T The weight coefficients are the data transmission rate, the proportion of data frame loss, and the time required for the receiving end to receive the data during the streaming process, G V , G K and G T The difference reference values ​​are the data transmission rate, the data frame loss ratio and the time required for the receiving end to receive the data.

[0104] The push flow state coefficient With preset threshold To compare:

[0105] like If the current streaming is judged to have the risk of network interruption or server failure, the system will immediately switch to the backup server and troubleshoot the current server;

[0106] Otherwise, it is determined that there is no risk of network interruption or server failure in the current streaming.

[0107] FFmpeg provides detailed log output, which the system can parse to monitor the health of the streaming and make dynamic adjustments based on preset strategies, such as lowering the encoding bitrate to cope with network jitter.

[0108] Audio and video synchronization processing:

[0109] During the streaming process, the built-in synchronization mechanism ensures that the video frames and audio frames are accurately matched according to the timestamps, thus avoiding the misalignment of the picture and sound.

[0110] If synchronization deviation is detected, the system automatically adjusts the delay of the audio or video, recalibrating the timing relationship between the two to maintain high-quality streaming output.

[0111] Scalability and Customization:

[0112] The FFmpeg module in the present invention is highly scalable, allowing users to customize streaming parameters, encoder settings, and video processing filters according to their needs. The system also supports dynamic loading of configuration files to adapt to different live broadcast scenarios (such as low-latency live broadcast, high-quality recorded broadcast, etc.).

[0113] Through seamless integration with other modules (such as WebRTC and CanvasAPI), the FFmpeg streaming module is able to handle complex multi-level content and stably push processed high-quality video streams to multiple platforms.

[0114] The working process of synchronously acquiring the same screen content as the control end includes:

[0115] The audio and video streams of all users in the connected room are synchronously obtained through the WebRTC mechanism, and the document information is synchronously obtained through the Parse-SDK-JS mechanism.

[0116] The specific working process of synchronously acquiring the audio and video streams of all users in the connected microphone room through the WebRTC mechanism includes:

[0117] WebRTC audio and video stream processing:

[0118] The system receives the audio and video streams from the remote user through the RTCPeerConnection interface of WebRTC, and the audio and video streams are bound to the HTMLVideoElement in the web page for further processing;

[0119] Video Stream:

[0120] Whenever HTMLVideoElement updates the video frame, it uses the drawImage method of the CanvasAPI to draw the current frame to <canvas>Elements, <canvas>The element is used to carry and process video frame data;

[0121] Draw on <canvas>The clipping area is defined by the clip method of the CanvasAPI, and only the area of ​​interest of the video frame is displayed and processed to reduce the amount of data processing and improve rendering efficiency.

[0122] Video frame processing further includes scaling operations, which adjust the display ratio of the video frame to adapt to different resolutions and display requirements by calling the scale method of the CanvasAPI;

[0123] Audio Stream:

[0124] The audio stream is processed through WebRTC's RTCPeerConnection, using the Opus codec, and performing noise suppression and echo cancellation.

[0125] Combination of WebRTC and Canvas API

[0126] Dynamically crop and process video streams: Use the drawImage method of the Canvas API to draw the video stream received from WebRTC onto the Canvas. Using the clip function of the Canvas, you can partially crop the video stream, specify a rectangular area, and draw this area of ​​the video stream into the Canvas, so that you can only process the part of interest. Scaling and adjustment: The Canvas API also supports scaling and adjustment operations. The scale method of the Canvas can be used to scale the drawn content. In addition, you can use the size parameter of drawImage to adjust the display size of the video stream.

[0127] Processing of dynamic video frames

[0128] Draw the video stream obtained by WebRTC into Canvas, and use the drawImage method of CanvasAPI to capture each frame of video. The video frame can be cropped, scaled, and other image processing operations can be performed as required.

[0129] The system supports the application of customized HTML templates to render video content. These templates can include a variety of scene layouts, animation effects, and dynamic combinations of graphic elements to ensure that the video stream can adapt to a variety of live broadcast scenarios.

[0130] Definition and loading of custom templates

[0131] A multifunctional audio and video processing flexible interaction mode integrating CanvasAPI and WebRTC is proposed. The system can support custom HTML templates and animation effects. Users can define custom templates through HTML / CSS / JavaScript, which include layout information, static and dynamic elements (such as text, graphics, video embedding), and interaction logic (such as click-triggered animation). The template is loaded by the front-end rendering engine and combined with the audio and video streams received by WebRTC to form the final display output. The dynamic elements of the template are associated with the video stream or user input events to achieve real-time animation effects.

[0132] Implementation of animation effects

[0133] Animation effects are defined through JavaScript or WebAnimationsAPI and interact with Canvas elements. Animation can be triggered based on time, user input (such as click, hover), or changes in video stream content. For example, at a specific time point or after a specific user operation, an element in the template (such as graphics, text) begins to move, scale, fade in / out, etc. Through the requestAnimationFrame method of the CanvasAPI, ensure that the animation rendering is synchronized with the browser refresh rate to achieve smooth animation effects. Animation can involve multiple Canvas drawing operations, such as dynamically adjusting the image position and size in the Canvas, or switching between different frames.

[0134] Screen synchronization mechanism

[0135] In order to ensure the consistency of animation and video stream processing logic in a multi-user environment, the system transmits real-time data through the signaling channel or data channel of WebRTC. For example, when a user triggers an animation or changes the content in a template (such as switching the background or adjusting the position of a graphic), the operation will be transmitted to other users through the signaling channel. The client data synchronization includes but is not limited to the following:

[0136] Animation parameters: include the animation start time, duration, effect type (such as translation, rotation, scaling), and the identifier of the target element.

[0137] Template change information: such as background switching, new content loading, or template layout adjustment.

[0138] User interaction events: such as click, input or mouse hover events. The relevant information of these events is sent to all users through the signaling channel (combined with the WebRTC signaling server and ParseLiveQuery), ensuring that the animation and screen processing logic on all clients are executed synchronously.

[0139] Message notification and instant synchronization Signaling server: In WebRTC, the signaling server is responsible for exchanging SDP (Session Description Protocol) and ICE (Interactive Connectivity Establishment) candidates to establish and maintain peer-to-peer connections. In addition, the signaling server can also be used to transmit control messages, such as starting / stopping live broadcasts, turning on / off cameras, etc. Control message content: Control messages may include video cropping parameters, zoom factors, screen update frequency, etc. Robot agents and remote guests need to adjust their respective Canvas processing logic based on these control messages to ensure consistent display effects 8. Fusion of animation and video streams

[0140] In the specific implementation, the system uses the layer overlay function of the CanvasAPI to merge the animation effect with the video stream. For example, by defining a Canvas layer for rendering the video stream and another Canvas layer for rendering the animation effect. Then the two layers are overlaid together to form the final output. In this way, the rendering logic of the video stream and the animation can be controlled separately to ensure their synchronization and independence.

[0141] The processing of virtual background is also achieved through this layer overlay. The system first draws the virtual background in the bottom Canvas, then overlays the processed user video stream in the upper Canvas, and presents the final output in the browser.

[0142] Dynamic loading and updating of templates and animations:

[0143] Users can dynamically switch templates and animations during live broadcast. The system will notify other clients to load new templates or update the content of existing templates through the signaling channel. The loaded new templates and animations will be synchronized with the current video stream to ensure a smooth transition during switching.

[0144] During the template switching process, the system prioritizes the termination or transition of existing animations, and then gradually loads and initializes new animation elements to avoid screen freezes or inconsistencies.

[0145] The process of synchronously acquiring document information through the Parse-SDK-JS mechanism includes:

[0146] The client uses the HTTP protocol upgrade mechanism to establish a WebSocket connection with the Parse LiveQuery Server. The handshake protocol confirms the connection success through the HTTP 101Switching Protocols status code. Once the connection is completed, the communication between the client and the server can be full-duplex data transmission over the WebSocket connection;

[0147] This full-duplex connection allows the server to push event messages to the client at any time without the client having to poll frequently.

[0148] After establishing a WebSocket connection, the client sends a subscription message in JSON format through WebSocket. The message structure usually includes query conditions, subscription ID, name of related data model, etc. After receiving the subscription request, the server will parse the query conditions and create a subscription record for the query internally;

[0149] Parse Server data change detection (Change Detection):

[0150] When the client or server operates on the data in the database (for example, inserting, updating, or deleting data), Parse Server generates data change events through the Publisher module.

[0151] These events are passed to LiveQuery Server through MQ's PUBLISH / SUBSCRIBE (Pub / Sub) mechanism, and MQ acts as an efficient message middleware in this process.

[0152] The benefit of using MQ Pub / Sub is its efficient message broadcasting capability, which ensures that multiple LiveQuery servers can receive change events at the same time.

[0153] LiveQuery Server handles data changes

[0154] The Subscriber module of LiveQuery Server receives data change messages from the Redis channel. At this time, these change messages will be deserialized and checked whether they match the currently active subscription conditions.

[0155] The server maintains a mapping table based on the subscription ID (Subscription Table), which records the query conditions of each client and the corresponding WebSocket connection

[0156] Generate event notifications

[0157] Event Message: When the data matching the query condition changes in the database, Parse LiveQuery Server will send an event message to the client through WebSocket to notify the client that the data has changed.

[0158] If a data change matches the client's query conditions, LiveQuery Server will generate an event message and send it to the corresponding client via WebSocket.

[0159] Event messages are usually encapsulated in JSON format, including the changed data, operation type (such as CREATE, UPDATE, DELETE), etc.

[0160] This push mechanism is based on the full-duplex communication capability of WebSocket, achieving low-latency real-time data updates

[0161] Client handles event messages

[0162] After receiving the event message, the client updates the user interface (UI) or synchronizes local data according to the message content. For example, adding a message, updating the message status, or deleting a record

[0163] After receiving an event, the client may add new "unread" messages to the message list, thus ensuring that the user interface is synchronized with the background data.

[0164] Connection Management

[0165] To keep the WebSocket connection active, the client and server usually send heartbeat packets (ping / pong frames) to detect the connection status to avoid connection interruptions due to network fluctuations or firewall policies.

[0166] The frequency of sending heartbeat packets can be adjusted through configuration to ensure that the connection remains stable even when there is no data transmission for a long time.

[0167] Scalability and high availability

[0168] Parse LiveQuery Server supports multi-instance deployment and broadcasts and distributes change events through Redis. Multiple LiveQuery servers can handle subscription requests from different clients at the same time, achieving high concurrency and high availability.

[0169] In a multi-instance environment, a load balancer (such as NGINX) is usually used to distribute client connection requests to balance the load of each LiveQuery Server instance.

[0170] To keep the WebSocket connection active, the client and server usually send heartbeat packets (ping / pong frames) to detect the connection status to avoid connection interruptions caused by network fluctuations or firewall policies;

[0171] The sending frequency of the heartbeat packet can be adjusted through configuration to ensure that the connection remains stable when there is no data transmission for a long time.

[0172] A live broadcast method of a live broadcast system based on a Robot proxy mechanism, the method comprising the following steps:

[0173] Step S1, obtaining local video stream, remote video stream, local screen configuration information, and control operation information through a communication protocol;

[0174] Step S2, leaving the screen configuration and live broadcast control to local operation;

[0175] Step S3: After obtaining all the information, convert the rendering and streaming tasks to FunctionCompute operations.

[0176] The above is a detailed description of an embodiment of the present invention, but the content is only a preferred embodiment of the present invention and cannot be considered to limit the scope of implementation of the present invention. All equivalent changes and improvements made within the scope of the present invention should still fall within the scope of the patent coverage of the present invention.< / canvas> < / canvas> < / canvas> < / canvas> < / canvas>

Claims

1. A live broadcast system based on Robot agent mechanism, characterized in that: The system includes: a Puppeteer module, a rendering and streaming module; The Puppeteer module is used to control HeadlessChrome to simulate the behavior of all remote guests participating in the live broadcast joining the microphone room through the Web, in order to obtain the same picture as the main control end, and is also used to serve as a bridge for direct communication between the web page and the nodeserver; The rendering and streaming module is used to process the video content obtained by the Puppeteer module, stream it to multiple live servers, and monitor the streaming status in real time; The Puppeteer module and the rendering and streaming module run on the same FunctionCompute service.

2. A live broadcast system based on Robot proxy mechanism according to claim 1, characterized in that: The specific working process of the Puppeteer module includes: Simulation and verification of remote guest behavior: The Puppeteer module controls HeadlessChrome to simulate the behavior of all remote guests joining the live broadcast and microphone room; During the simulation, Puppeteer triggers the same WebRTC connection process as the actual user, including ICE connection establishment, SDP exchange, and the publishing and receiving of audio and video streams; Communication bridge between web page and NodeServer: Based on Puppeteer's evaluate method, the Node.js code on the server interacts with the JavaScript code in the page environment through Puppeteer to achieve instant data transmission and operation; Efficient data operation and page loading optimization: When specific metadata needs to be injected into the WebRTC stream, the Node.js server can write the data directly into the JavaScript environment of the page through Puppeteer; By preloading critical resources, delaying the loading of non-critical resources, and dynamically adjusting the loading order during page initialization; FunctionCompute interaction with the Web client: The server is allowed to send key control commands and data to the FunctionCompute service through WebSocket or other real-time communication protocols while controlling page loading and operation. After receiving these commands, FunctionCompute can immediately adjust the push stream.

3. A live broadcast system based on Robot proxy mechanism according to claim 2, characterized in that: The specific working process of the rendering and streaming module includes: Page content capture and video stream generation: On the server side, the same screen content as the control side is obtained synchronously through the input device, and the captured content is directly processed in the form of a video stream to generate the original data stream for streaming; Video stream encoding and processing: Select the appropriate codec for video encoding according to actual needs. During the encoding process, apply different video processing filters to optimize the output effect; Push streams to multiple live servers: After encoding is completed, the video stream is pushed to multiple specified live servers, and multiple output streams are created at the same time. Each stream corresponds to the RTMP address of a live server. Through parameter configuration, multiple streams can be pushed in the same operation; Real-time streaming status monitoring and fault recovery: Real-time tracking of streaming status, including bandwidth usage, frame loss rate, and latency parameters. If a network interruption or server failure occurs during streaming, the system can automatically retry the connection or switch to a backup server. Audio and video synchronization processing: During the streaming process, the built-in synchronization mechanism ensures that the video frames and audio frames are accurately matched according to the timestamps, thus avoiding the misalignment of the picture and sound. If synchronization deviation is detected, the system automatically adjusts the delay of the audio or video, recalibrating the timing relationship between the two to maintain high-quality streaming output.

4. A live broadcast system based on Robot agent mechanism according to claim 3, characterized in that: The working process of synchronously acquiring the same screen content as the control end includes: The audio and video streams of all users in the connected room are synchronously obtained through the WebRTC mechanism, and the document information is synchronously obtained through the Parse-SDK-JS mechanism.

5. A live broadcast system based on Robot proxy mechanism according to claim 4, characterized in that: The specific working process of synchronously acquiring the audio and video streams of all users in the connected microphone room through the WebRTC mechanism includes: WebRTC audio and video stream processing: The system receives the audio and video streams from the remote user through the RTCPeerConnection interface of WebRTC, and the audio and video streams are bound to the HTMLVideoElement in the web page for further processing; Video Stream: Whenever HTMLVideoElement updates the video frame, it uses the drawImage method of the CanvasAPI to draw the current frame to <canvas>Elements, <canvas> The element is used to carry and process video frame data;< / canvas> < / canvas> Video frame processing further includes scaling operations, which adjust the display ratio of the video frame to adapt to different resolutions and display requirements by calling the scale method of the CanvasAPI; Audio Stream: The audio stream is handled via WebRTC’s RTCPeerConnection, utilizing the Opus codec with noise suppression and echo cancellation.

6. A live broadcast system based on Robot proxy mechanism according to claim 4, characterized in that: The process of synchronously acquiring document information through the Parse-SDK-JS mechanism includes: The client uses the HTTP protocol upgrade mechanism to establish a WebSocket connection with the Parse LiveQuery Server. Once the connection is established, the communication between the client and the server can be full-duplex data transmission over the WebSocket connection; After establishing a WebSocket connection, the client sends a subscription message in JSON format through WebSocket. After receiving the subscription request, the server will parse the query conditions and create a subscription record for the query internally.

7. A live broadcast system based on Robot proxy mechanism according to claim 6, characterized in that: To keep the WebSocket connection active, the client and server usually send heartbeat packets (ping / pong frames) to detect the connection status to avoid connection interruptions caused by network fluctuations or firewall policies; The sending frequency of the heartbeat packet can be adjusted through configuration to ensure that the connection remains stable when there is no data transmission for a long time.

8. A live broadcast method using the live broadcast system based on the Robot proxy mechanism according to any one of claims 1 to 7, characterized in that: The method comprises the following steps: Step S1, obtaining local video stream, remote video stream, local screen configuration information, and control operation information through a communication protocol; Step S2, leaving the screen configuration and live broadcast control to local operation; Step S3: After obtaining all the information, convert the rendering and streaming tasks to FunctionCompute operations.

Images