Front-end-driven multi-channel video stream dynamic filtering and rendering method and system

Abstract

The invention discloses a front-end-driven multi-channel video stream dynamic filtering and rendering method and system, belongs to the technical field of computer communication, and aims to solve the technical problem of how to solve member state synchronization abnormity and picture residue caused by network fluctuation and detection delay. Comprising the following steps: establishing a persistent bidirectional connection between a client and a converged communication message service background through a WebSocket protocol; many-to-many real-time communication is realized based on a WebRTC and MCU scheme; in combination with a WebSocket automatic reconnection mechanism and a WebRTC packet loss recovery mechanism, the reliability and stability of connection between the client and the converged communication message service background in a high-concurrency environment are enhanced; a Vue.js framework is used for front-end development, and a componentized and responsive user interface is achieved; front-end calculation is carried out, and signaling-service double-track data difference detection is triggered when members change; and the screen fusion service is requested again, and screen fusion video stream rendering is carried out.

Description

Technical Field

[0001] This invention relates to the field of computer communication technology, specifically to a front-end driven method and system for dynamic filtering and rendering of multi-channel video streams. Background Technology

[0002] As a key component of smart city construction, the intelligent emergency command and dispatch system aims to manage the entire process before, during, and after an emergency. In the face of sudden incidents, it can quickly initiate emergency response, improving the city's emergency handling capabilities. Video conferencing is a core function of this system. For example, in the face of emergencies such as wildfires and flash floods, the command and dispatch platform can quickly conduct video conferences based on live video feeds to deploy relevant work. Video conferencing scenarios place higher demands on the data processing efficiency, rendering speed, and stability of multiple video streams.

[0003] Existing multi-channel video communication technologies, including WebSocket-based signaling exchange and WebRTC-based audio and video transmission, have been widely used in various real-time communication applications, but the following challenges remain: member status mainly relies on backend notifications, and the frontend lacks autonomous detection capabilities, resulting in delays in member status synchronization when the network fluctuates; delayed detection of member exits leads to rendering residues, resulting in "ghost images".

[0004] The technical problem that needs to be solved is how to resolve the issues of abnormal member status synchronization and image retention caused by network fluctuations and detection delays, and achieve high-performance, low-latency multi-person video rendering. Summary of the Invention

[0005] The technical objective of this invention is to address the above-mentioned shortcomings by providing a front-end driven dynamic filtering and rendering method and system for multi-channel video streams. This aims to solve the problems of abnormal member state synchronization and image retention caused by network fluctuations and detection delays, thereby achieving high-performance, low-latency multi-person video image rendering.

[0006] In a first aspect, the present invention provides a front-end driven dynamic filtering and rendering method for multiple video streams, comprising the following steps:

[0007] Connection establishment: A persistent bidirectional connection is established between the client and the converged communication messaging service backend via the WebSocket protocol;

[0008] Real-time communication implementation: Implementing many-to-many real-time communication based on WebRTC and MCU solutions;

[0009] Enhanced Communication: By combining the WebSocket automatic reconnection mechanism and the WebRTC packet loss recovery mechanism, the reliability and stability of the connection between the client and the converged communication messaging service backend are enhanced in high-concurrency environments;

[0010] Front-end development: Utilizing the Vue.js framework for front-end development to achieve component-based and responsive user interfaces;

[0011] Data discrepancy detection: The calculation is front-end-based, and signaling-business dual-track data discrepancy detection is triggered when members change;

[0012] Screen blending process: Re-request the screen blending service and render the blended video stream.

[0013] As a preferred option, front-end development specifically includes:

[0014] Develop reusable Vue components, including meeting pop-ups, which provide meeting room-related functions, such as receiving meeting invitations, displaying multiple screens, and muting and kicking out members.

[0015] Vuex is used for state management to maintain information such as meeting room host, member online / offline status, and incremental change sets when members change.

[0016] As a preferred option, the signaling-service dual-track data difference detection specifically includes:

[0017] L100: Obtain the current meeting room information by listening to the OnMeeting event broadcast by WebSocket;

[0018] When L100 members join a meeting, their information is added to the incremental change collection, including the meeting room name, RTSP streaming media address, and member's phone number.

[0019] L300, the meeting host initiates a screen merge request;

[0020] L400, when a meeting member leaves / is kicked out, execute the member's exit business logic;

[0021] L500: Determine if the current WebSocket heartbeat connection is normal;

[0022] L600 If the heartbeat connection is normal, the member will exit normally through the signaling-service dual-track data difference detection, and the incremental change set and member status will be updated synchronously.

[0023] L700: If the number of members is greater than 2, repeat step L300;

[0024] L800: If the number of members is less than 2, the meeting will end immediately, the incremental change set will be cleared, and the player will be destroyed.

[0025] L900 If the heartbeat connection is abnormal, it cannot be determined that the abnormal exit of the member is due to network fluctuations through the signaling-service dual-track data difference detection. There is no need to delete the member in the incremental change set and change the member status to "offline".

[0026] After LA00 and the network connection is normal, repeat step L300.

[0027] As a preferred option, screen convergence service requests include:

[0028] Get the array of RTSP streaming media addresses for the current conference room;

[0029] To resolve the issue of the window still occupying space and displaying a black screen after a member leaves the array, perform NOT NULL checks.

[0030] Set the pane index for each member;

[0031] Set the number of split screens;

[0032] The video is rendered based on the merged video stream address from the push notification, thus minimizing DOM manipulation.

[0033] In a second aspect, the present invention provides a front-end driven multi-channel video stream dynamic filtering and rendering system, comprising a connection establishment module, a real-time communication construction module, a communication enhancement module, a front-end development module, a data difference detection module, and a screen fusion processing module.

[0034] The connection establishment module is used to perform the following: establish a persistent bidirectional connection between the client and the converged communication messaging service backend via the WebSocket protocol;

[0035] The real-time communication building module is used to perform the following: implement many-to-many real-time communication based on WebRTC and MCU solutions;

[0036] The communication enhancement module is used to perform the following: combining the WebSocket automatic reconnection mechanism and the WebRTC packet loss recovery mechanism to enhance the reliability and stability of the connection between the client and the converged communication message service backend in a high-concurrency environment;

[0037] The front-end development module is used to perform the following: front-end development using the Vue.js framework to implement component-based and responsive user interfaces;

[0038] The data discrepancy detection module is used to perform the following: front-end calculations, and triggering signaling-business dual-track data discrepancy detection when members change;

[0039] The screen blending processing module is used to perform the following: re-request the screen blending service and render the screen blending video stream.

[0040] Preferably, the front-end development module is used to perform the following operations:

[0041] Develop reusable Vue components, including meeting pop-ups, which provide meeting room-related functions, such as receiving meeting invitations, displaying multiple screens, and muting and kicking out members.

[0042] Vuex is used for state management to maintain information such as meeting room host, member online / offline status, and incremental change sets when members change.

[0043] As a preferred option, the signaling-service dual-track data difference detection specifically includes:

[0044] L100: Obtain the current meeting room information by listening to the OnMeeting event broadcast by WebSocket;

[0045] When L100 members join a meeting, their information is added to the incremental change collection, including the meeting room name, RTSP streaming media address, and member's phone number.

[0046] L300, the meeting host initiates a screen merge request;

[0047] L400, when a meeting member leaves / is kicked out, execute the member's exit business logic;

[0048] L500: Determine if the current WebSocket heartbeat connection is normal;

[0049] L600 If the heartbeat connection is normal, the member will exit normally through the signaling-service dual-track data difference detection, and the incremental change set and member status will be updated synchronously.

[0050] L700: If the number of members is greater than 2, repeat step L300;

[0051] L800: If the number of members is less than 2, the meeting will end immediately, the incremental change set will be cleared, and the player will be destroyed.

[0052] L900 If the heartbeat connection is abnormal, it cannot be determined that the abnormal exit of the member is due to network fluctuations through the signaling-service dual-track data difference detection. There is no need to delete the member in the incremental change set and change the member status to "offline".

[0053] After LA00 and the network connection is normal, repeat step L300.

[0054] As a preferred option, screen convergence service requests include:

[0055] Get the array of RTSP streaming media addresses for the current conference room;

[0056] To resolve the issue of the window still occupying space and displaying a black screen after a member leaves the array, perform NOT NULL checks.

[0057] Set the pane index for each member;

[0058] Set the number of split screens;

[0059] The video image is rendered based on the merged video stream address from the push notification, minimizing DOM manipulation. The front-end-driven multi-channel video stream dynamic filtering and rendering method and system of this invention have the following advantages:

[0060] 1. Low state synchronization latency: By front-end computing, a signaling-service dual-track data difference detection algorithm is introduced, combined with a heartbeat mechanism, to solve the synchronization latency problem caused by network fluctuations;

[0061] 2. High resource recycling rate: Unlike relying entirely on backend notifications, this system is based on front-end computing. After the meeting, a destruction command is displayed to ensure zero resource residue. At the same time, the screen update delay when members change is controlled within 70ms, effectively solving the "ghost screen" problem.

[0062] 3. High system reliability: The system's fault tolerance mechanism ensures that stable communication services can still be maintained even in the event of network fluctuations. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] The invention will be further described below with reference to the accompanying drawings.

[0065] Figure 1 This is a flowchart of a front-end driven dynamic filtering and rendering method for multiple video streams, as described in Example 1. Detailed Implementation

[0066] The present invention will be further described below with reference to the accompanying drawings and specific embodiments, so that those skilled in the art can better understand and implement the present invention. However, the embodiments are not intended to limit the present invention. In the absence of conflict, the embodiments of the present invention and the technical features in the embodiments can be combined with each other.

[0067] This invention provides a front-end driven multi-channel video stream dynamic filtering and rendering method and system to solve the technical problem of how to solve the problems of abnormal member state synchronization and image retention caused by network fluctuations and detection delays, and to achieve high-performance, low-latency multi-person video image rendering.

[0068] Example 1:

[0069] This invention provides a front-end driven dynamic filtering and rendering method for multiple video streams, comprising six steps: connection establishment, real-time communication construction, communication enhancement, front-end development, data difference detection, and screen blending processing.

[0070] Step S100 Connection Establishment: A persistent bidirectional connection is established between the client and the converged communication message service backend via the WebSocket protocol.

[0071] Step S200 Real-time Communication Construction: Implement many-to-many real-time communication based on WebRTC and MCU solutions.

[0072] Step S300 Communication Enhancement: Combining the WebSocket automatic reconnection mechanism and the WebRTC packet loss recovery mechanism, the reliability and stability of the connection between the client and the converged communication message service backend are enhanced in high-concurrency environments.

[0073] Step S400 Front-end Development: Utilize the Vue.js framework for front-end development to achieve a component-based and responsive user interface.

[0074] As a specific implementation of front-end development, this step includes:

[0075] (1) Develop reusable Vue components, including meeting pop-ups, which provide meeting room related functions, including receiving meeting invitations, displaying multiple screens, and muting and kicking out members;

[0076] (2) Use Vuex for state management to maintain information on the meeting room host, members’ online / offline status, and incremental change sets when members change.

[0077] Step S500 Data Difference Detection: The calculation is front-end-based, and signaling-business dual-track data difference detection is triggered when members change.

[0078] Specifically, the signaling-service dual-track data difference detection includes:

[0079] L100: Obtain the current meeting room information by listening to the OnMeeting event broadcast by WebSocket;

[0080] When L100 members join a meeting, their information is added to the incremental change collection, including the meeting room name, RTSP streaming media address, and member's phone number.

[0081] L300, the meeting host initiates a screen merge request;

[0082] L400, when a meeting member leaves / is kicked out, execute the member's exit business logic;

[0083] L500: Determine if the current WebSocket heartbeat connection is normal;

[0084] L600 If the heartbeat connection is normal, the member will exit normally through the signaling-service dual-track data difference detection, and the incremental change set and member status will be updated synchronously.

[0085] L700: If the number of members is greater than 2, repeat step L300;

[0086] L800: If the number of members is less than 2, the meeting will end immediately, the incremental change set will be cleared, and the player will be destroyed.

[0087] L900 If the heartbeat connection is abnormal, it cannot be determined that the abnormal exit of the member is due to network fluctuations through the signaling-service dual-track data difference detection. There is no need to delete the member in the incremental change set and change the member status to "offline".

[0088] After LA00 and the network connection is normal, repeat step L300.

[0089] Step S600: Screen blending process: Re-request the screen blending service and render the screen blending video stream.

[0090] In this embodiment, the screen merging service request includes:

[0091] (1) Obtain the array of RTSP streaming media addresses for the current conference room;

[0092] (2) Perform NOT empty processing on the array to solve the problem that the window still occupies space and displays a black screen after a member exits;

[0093] (3) Set the pane index for each member;

[0094] (4) Set the number of split screens;

[0095] (5) Render the video frame according to the video stream address of the message push to minimize DOM operations.

[0096] The method in this embodiment achieves high-performance, low-latency dynamic rendering of multiple video streams through front-end filtering, and has certain social value and economic benefits.

[0097] Example 2:

[0098] The present invention provides a front-end driven multi-channel video stream dynamic filtering and rendering system, comprising a connection establishment module, a real-time communication construction module, a communication enhancement module, a front-end development module, a data difference detection module, and a screen fusion processing module.

[0099] The connection establishment module is used to perform the following: establish a persistent bidirectional connection between the client and the converged communication message service backend via the WebSocket protocol.

[0100] The real-time communication building module is used to perform the following: implement many-to-many real-time communication based on WebRTC and MCU solutions.

[0101] The communication enhancement module is used to perform the following: combining the WebSocket automatic reconnection mechanism and the WebRTC packet loss recovery mechanism to enhance the reliability and stability of the connection between the client and the converged communication message service backend in a high-concurrency environment.

[0102] The front-end development module is used to perform the following: front-end development using the Vue.js framework to implement component-based and responsive user interfaces.

[0103] As a specific implementation of the front-end development module, this module is used to perform the following operations:

[0104] (1) Develop reusable Vue components, including meeting pop-ups, which provide meeting room related functions, including receiving meeting invitations, displaying multiple screens, and muting and kicking out members;

[0105] (2) Use Vuex for state management to maintain information on the meeting room host, members’ online / offline status, and incremental change sets when members change.

[0106] The data discrepancy detection module is used to perform the following: front-end calculations and triggering signaling-business dual-track data discrepancy detection when members change.

[0107] Specifically, the signaling-service dual-track data difference detection includes:

[0108] L100: Obtain the current meeting room information by listening to the OnMeeting event broadcast by WebSocket;

[0109] When L100 members join a meeting, their information is added to the incremental change collection, including the meeting room name, RTSP streaming media address, and member's phone number.

[0110] L300, the meeting host initiates a screen merge request;

[0111] L400, when a meeting member leaves / is kicked out, execute the member's exit business logic;

[0112] L500: Determine if the current WebSocket heartbeat connection is normal;

[0113] L600 If the heartbeat connection is normal, the member will exit normally through the signaling-service dual-track data difference detection, and the incremental change set and member status will be updated synchronously.

[0114] L700: If the number of members is greater than 2, repeat step L300;

[0115] L800: If the number of members is less than 2, the meeting will end immediately, the incremental change set will be cleared, and the player will be destroyed.

[0116] L900 If the heartbeat connection is abnormal, it cannot be determined that the abnormal exit of the member is due to network fluctuations through the signaling-service dual-track data difference detection. There is no need to delete the member in the incremental change set and change the member status to "offline".

[0117] After LA00 and the network connection is normal, repeat step L300.

[0118] The screen blending processing module is used to perform the following: re-request the screen blending service and render the screen blending video stream.

[0119] In this embodiment, the screen merging service request includes:

[0120] (1) Obtain the array of RTSP streaming media addresses for the current conference room;

[0121] (2) Perform NOT empty processing on the array to solve the problem that the window still occupies space and displays a black screen after a member exits;

[0122] (3) Set the pane index for each member;

[0123] (4) Set the number of split screens;

[0124] (5) Render the video frame according to the video stream address of the message push to minimize DOM operations.

[0125] The system in this embodiment can execute the method disclosed in Embodiment 1 to achieve dynamic filtering and rendering of multiple video streams.

[0126] The foregoing has provided a detailed description of the front-end driven multi-channel video stream dynamic filtering and rendering method and system provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A front-end driven dynamic filtering and rendering method for multiple video streams, characterized in that, Includes the following steps: Connection establishment: A persistent bidirectional connection is established between the client and the converged communication messaging service backend via the WebSocket protocol; Real-time communication implementation: Implementing many-to-many real-time communication based on WebRTC and MCU solutions; Enhanced Communication: By combining the WebSocket automatic reconnection mechanism and the WebRTC packet loss recovery mechanism, the reliability and stability of the connection between the client and the converged communication messaging service backend are enhanced in high-concurrency environments; Front-end development: Utilizing the Vue.js framework for front-end development to achieve component-based and responsive user interfaces; Data discrepancy detection: The calculation is front-end-based, and signaling-business dual-track data discrepancy detection is triggered when members change; Screen blending process: Re-request the screen blending service and render the blended video stream.

2. The front-end driven multi-channel video stream dynamic filtering and rendering method according to claim 1, characterized in that, Front-end development specifically includes: Develop reusable Vue components, including meeting pop-ups, which provide meeting room-related functions, such as receiving meeting invitations, displaying multiple screens, and muting and kicking out members. Vuex is used for state management to maintain information such as meeting room host, member online / offline status, and incremental change sets when members change.

3. The front-end driven multi-channel video stream dynamic filtering and rendering method according to claim 1, characterized in that, Signaling-service dual-track data difference detection specifically includes: L100: Obtain the current meeting room information by listening to the OnMeeting event broadcast by WebSocket; When L100 members join a meeting, their information is added to the incremental change collection, including the meeting room name, RTSP streaming media address, and member's phone number. L300, the meeting host initiates a screen merge request; L400, when a meeting member leaves / is kicked out, execute the member's exit business logic; L500: Determine if the current WebSocket heartbeat connection is normal; L600 If the heartbeat connection is normal, the member will exit normally through the signaling-service dual-track data difference detection, and the incremental change set and member status will be updated synchronously. L700: If the number of members is greater than 2, repeat step L300; L800: If the number of members is less than 2, the meeting will end immediately, the incremental change set will be cleared, and the player will be destroyed. L900 If the heartbeat connection is abnormal, it cannot be determined that the abnormal exit of the member is due to network fluctuations through the signaling-service dual-track data difference detection. There is no need to delete the member in the incremental change set and change the member status to "offline". After LA00 and the network connection is normal, repeat step L300.

4. The front-end driven multi-channel video stream dynamic filtering and rendering method according to claim 1, characterized in that, Screen merge service requests include: Get the array of RTSP streaming media addresses for the current conference room; To resolve the issue of the window still occupying space and displaying a black screen after a member leaves the array, perform NOT NULL checks. Set the pane index for each member; Set the number of split screens; The video is rendered based on the merged video stream address from the push notification, thus minimizing DOM manipulation.

5. A front-end driven multi-channel video stream dynamic filtering and rendering system, characterized in that, It includes a connection establishment module, a real-time communication construction module, a communication enhancement module, a front-end development module, a data difference detection module, and a screen fusion processing module; The connection establishment module is used to perform the following: establish a persistent bidirectional connection between the client and the converged communication messaging service backend via the WebSocket protocol; The real-time communication building module is used to perform the following: implement many-to-many real-time communication based on WebRTC and MCU solutions; The communication enhancement module is used to perform the following: combining the WebSocket automatic reconnection mechanism and the WebRTC packet loss recovery mechanism to enhance the reliability and stability of the connection between the client and the converged communication message service backend in a high-concurrency environment; The front-end development module is used to perform the following: front-end development using the Vue.js framework to implement component-based and responsive user interfaces; The data discrepancy detection module is used to perform the following: front-end calculations, and triggering signaling-business dual-track data discrepancy detection when members change; The screen blending processing module is used to perform the following: re-request the screen blending service and render the screen blending video stream.

6. The front-end driven multi-channel video stream dynamic filtering and rendering system according to claim 5, characterized in that, The front-end development module is used to perform the following operations: Develop reusable Vue components, including meeting pop-ups, which provide meeting room-related functions, such as receiving meeting invitations, displaying multiple screens, and muting and kicking out members. Vuex is used for state management to maintain information such as meeting room host, member online / offline status, and incremental change sets when members change.

7. The front-end driven multi-channel video stream dynamic filtering and rendering system according to claim 5, characterized in that, Signaling-service dual-track data difference detection specifically includes: L100: Obtain the current meeting room information by listening to the OnMeeting event broadcast by WebSocket; When L100 members join a meeting, their information is added to the incremental change collection, including the meeting room name, RTSP streaming media address, and member's phone number. L300, the meeting host initiates a screen merge request; L400, when a meeting member leaves / is kicked out, execute the member's exit business logic; L500: Determine if the current WebSocket heartbeat connection is normal; L600 If the heartbeat connection is normal, the member will exit normally through the signaling-service dual-track data difference detection, and the incremental change set and member status will be updated synchronously. L700: If the number of members is greater than 2, repeat step L300; L800: If the number of members is less than 2, the meeting will end immediately, the incremental change set will be cleared, and the player will be destroyed. L900 If the heartbeat connection is abnormal, it cannot be determined that the abnormal exit of the member is due to network fluctuations through the signaling-service dual-track data difference detection. There is no need to delete the member in the incremental change set and change the member status to "offline". After LA00 and the network connection is normal, repeat step L300.

8. The front-end driven multi-channel video stream dynamic filtering and rendering system according to claim 5, characterized in that, Screen merge service requests include: Get the array of RTSP streaming media addresses for the current conference room; To resolve the issue of the window still occupying space and displaying a black screen after a member leaves the array, perform NOT NULL checks. Set the pane index for each member; Set the number of split screens; The video is rendered based on the merged video stream address from the push notification, thus minimizing DOM manipulation.

Images