A method and system for processing video information

Abstract

The application discloses a video information processing method, relates to video transmission, and comprises the following steps: acquiring a video stream of a video conference participant and auxiliary information from a video conference system; dividing the auxiliary information into control type information and content type information; establishing a first transmission channel for the control type information independently of the video stream through a WebSocket or a QUIC protocol; establishing a second transmission channel for the content type information in association with the video stream through an SEI mechanism of a video encoder or an RTCP channel; respectively transmitting the control type information and the content type information according to the first transmission channel and the second transmission channel, and adding a sequence number and a timestamp to the transmitted information; receiving the auxiliary information at a receiving end, and performing timestamp alignment and synchronization processing on the auxiliary information and the video stream according to the sequence number and the timestamp to generate a synchronization data structure. In view of the low reliability of auxiliary information transmission under the condition that a network is unstable, the application improves the reliability of auxiliary information transmission under the condition that the network is unstable.

Description

Technical Field

[0001] This application relates to the field of video transmission, and in particular to a video information processing method and system. Background Technology

[0002] With the widespread adoption of remote work, distance education, and global collaboration, video conferencing systems have become an essential component of modern communication infrastructure. In recent years, in particular, global events have spurred a significant demand for high-quality, highly reliable video conferencing systems. Against this backdrop, video conferencing systems not only need to transmit high-definition video and clear audio, but also need to support the real-time transmission of meeting control commands and auxiliary information such as shared whiteboards to achieve a rich collaborative experience.

[0003] Video conferencing systems typically operate on packet-switched networks, but the modern internet environment is complex and volatile. Especially in mobile network environments, home broadband networks, and cross-border communications, issues such as network bandwidth fluctuations, latency variations, and increased packet loss rates are common. Video coding technologies, through mechanisms such as adaptive bitrate control, forward error correction, and packet loss retransmission, have been able to effectively cope with network fluctuations. However, a systematic solution is still lacking for addressing the reliability issues of auxiliary information transmission.

[0004] On the one hand, conference control commands (such as audio / video on / off, screen sharing control, and conference permission changes) are characterized by their small quantity, high real-time requirements, and criticality. In traditional video conferencing systems, control commands are typically transmitted in the following ways: Signaling channel transmission: Control commands are transmitted through signaling channels using protocols such as SIP or H.323. However, these protocols are poorly adapted to network fluctuations and may lead to control failures in environments with high packet loss rates; WebSocket long connections: Based on an upgrade of the HTTP protocol, this enables full-duplex communication, but it is inefficient when reconnecting after network interruption and has compatibility issues in certain network environments; Periodic polling mechanisms: These use HTTP to periodically request status changes, resulting in poor real-time performance and control delays when network latency is high. None of these methods adequately consider the dynamic changes in network status, leading to the potential loss or delay of conference control commands in unstable network conditions, severely impacting the conference flow and user experience. Furthermore, the retransmission mechanism of control commands lacks an effective correlation with the video stream status, easily leading to inconsistencies between the control status and the video content.

[0005] On the other hand, shared whiteboard data, as content-based information, is characterized by large data volume, incremental changes, and close correlation with video content. Existing technologies primarily employ the following methods for transmitting whiteboard data: independent data channel transmission: transmitting whiteboard data via an independent TCP or WebSocket channel, but lacking a synchronization mechanism with the video stream, potentially leading to desynchronization between whiteboard content and video explanations during network fluctuations; screenshot transmission: encoding and transmitting whiteboard content as part of the video, wasting bandwidth and exhibiting poor interactivity, failing to support real-time multi-party collaboration; periodic synchronization of complete state: periodically transmitting the complete whiteboard state, consuming significant bandwidth and prone to incomplete content due to packet loss. Existing technologies have failed to effectively solve the problem of precise synchronization between whiteboard data and video streams, nor have they dynamically selected the optimal transmission strategy based on network conditions. Especially under fluctuating network conditions, updates to whiteboard content may be delayed or lost, resulting in inconsistent whiteboard states among meeting participants and severely impacting collaboration efficiency.

[0006] In existing technologies, video streams and auxiliary information typically use different transmission channels and processing mechanisms, which results in video streams and auxiliary information using different timestamp systems, making precise alignment difficult. Especially in video conferencing systems deployed across regions, due to large differences in network latency, traditional synchronization mechanisms cannot guarantee that all participants see the same meeting status, leading to interactive chaos and reduced meeting efficiency. Summary of the Invention

[0007] To address the low reliability of auxiliary information transmission under unstable network conditions, this application provides a video information processing method and system that improves the reliability of auxiliary information transmission under unstable network conditions based on dual-channel adaptive transmission and synchronous processing.

[0008] This application provides a video information processing method, comprising: S1, acquiring video streams from video conference participants and obtaining auxiliary information from the video conferencing system, the auxiliary information including conference control instructions and shared whiteboard data; S2, dividing the auxiliary information into control-type information and content-type information; S3, establishing a first transmission channel independent of the video stream for the control-type information via WebSocket or QUIC protocol; S4, establishing a second transmission channel associated with the video stream for the content-type information via the SEI mechanism of the video encoder or RTCP channel; S5, transmitting the control-type information and content-type information respectively according to the first and second transmission channels, and adding a sequence number and timestamp to the transmitted information; S6, receiving the auxiliary information at the receiving end, and performing timestamp alignment and synchronization processing between the auxiliary information and the video stream according to the sequence number and timestamp to generate a synchronization data structure, the synchronization data structure being used by the conference client to display video images and shared whiteboard content.

[0009] Specifically, traditional video conferencing systems typically process all auxiliary information (such as meeting control commands and whiteboard data) through the same transmission mechanism, ignoring the different transmission requirements of different types of information. Control information (such as mute and raise hand commands) has a small data volume but extremely high requirements for real-time performance and reliability, while content information (such as whiteboard data) has a large data volume but is closely related to the video content. Unified processing leads to unreasonable resource allocation and cannot meet the transmission needs of different types of information. This application, through step S2, clearly divides auxiliary information into "control information" and "content information," and establishes transmission channels with different characteristics for these two types of information in steps S3 and S4. This differentiated processing breaks the traditional video conferencing system's pattern of treating all auxiliary information as homogeneous data.

[0010] On the other hand, in traditional systems, content information (such as whiteboard data) is often transmitted through independent data channels, lacking an inherent temporal correlation with the video stream. This leads to a desynchronization problem between content and video under network fluctuations. For example, the presenter might point to a spot on the whiteboard before the content has finished loading, or the whiteboard content might lag behind the presenter's audio narration, severely impacting information delivery. This application innovatively utilizes the SEI mechanism or RTCP channel of the video encoder in step S4 to establish a transmission channel associated with the video stream for content information. This design enables content information to maintain an inherent synchronization relationship with the video.

[0011] Furthermore, traditional video conferencing systems typically employ simple time window matching or fixed delay compensation methods at the receiving end to handle synchronization issues, which cannot cope with latency jitter and transmission sequence changes in complex network environments. This results in users frequently seeing asynchronous video and auxiliary information when network conditions fluctuate, such as pointer positions not matching the actual narration, or control command execution order being disordered. This application implements intelligent synchronization and reassembly processing at the receiving end in step S6, precisely aligning auxiliary information from different channels with the video stream using sequence numbers and timestamps to generate a unified synchronization data structure.

[0012] Furthermore, control information includes conference control commands, and content information includes shared whiteboard data; the first transmission channel is used for real-time transmission of control information; the second transmission channel is used to support incremental transmission of large amounts of data and maintain synchronous transmission with video; the synchronous data structure includes video frame data, conference control status, and whiteboard content.

[0013] Furthermore, S3, establishing a first transmission channel independent of the video stream for control information via WebSocket or QUIC protocol, including: S31, acquiring indicators including RTT latency, network jitter and packet loss rate; and calculating a network quality score based on the indicators; S32, when the network quality score is greater than a threshold, selecting the WebSocket protocol to ensure compatibility, otherwise selecting the QUIC protocol to improve transmission efficiency.

[0014] In particular, traditional video conferencing systems generally use static protocol configurations, which cannot adapt to changes in the network environment, especially in mobile scenarios where network quality fluctuates greatly, such as switching from an enterprise network to public Wi-Fi and then to a 4G network; fixed protocol strategies often lead to low efficiency or even interruption of control information transmission. This application combines multi-dimensional quantitative assessment of network quality (S31) with dynamic switching of transmission protocols (S32) to establish an adaptive transmission decision.

[0015] S33, When the WebSocket protocol is selected, establish a TCP-based WebSocket connection and set the keep-alive frame interval time parameter to ensure continuous connection availability; S34, When the QUIC protocol is selected, establish a UDP-based QUIC connection and configure the 0-RTT parameter to accelerate connection recovery.

[0016] WebSocket, as a TCP-based long-connection protocol, may traverse multiple layers of NAT, firewalls, or proxy servers in the network. These intermediate devices typically perform timeout cleanup for inactive connections (typically 30-90 seconds). Keep-alive frames are lightweight control frames that do not carry application data. Regularly sending these frames indicates to intermediate network devices that the connection remains active, preventing accidental closure. In video conferencing scenarios, setting a reasonable keep-alive frame interval is crucial. Too long an interval may lead to connection termination by intermediate devices, while too short an interval will increase unnecessary network overhead. In this application, this parameter is set to 50%-70% of the minimum timeout of the intermediate device (e.g., 25 seconds) to achieve a balance between connection stability and network efficiency.

[0017] QUIC is a UDP-based transport layer protocol developed by Google and is now the foundation of HTTP / 3. Zero Round Trip Time (0-RTT) is a key feature of the QUIC protocol. The 0-RTT parameter is a configuration feature in the QUIC protocol used to accelerate connection recovery, allowing clients to send application data immediately upon resuming a previous connection without a full handshake process. Traditional TLS connection recovery requires a round trip (1-RTT) before sending application data. QUIC's 0-RTT feature, by caching the session ticket from the previous connection and server configuration parameters, allows the client to include application data in the first packet without waiting for a server response, thus saving a full network round trip time. This parameter configuration includes enabling the 0-RTT feature, setting the session ticket validity period (usually several hours), configuring the acceptable 0-RTT packet size limit, and replay attack protection strategies. In video conferencing scenarios, the session ticket validity period is typically set to the maximum expected duration of the meeting (e.g., 4-8 hours) to cover possible disconnections and reconnections throughout the meeting. After network fluctuations cause connection interruptions, the 0-RTT parameter can significantly shorten the reconnection time, reducing it from 300-500 milliseconds in traditional protocols to approximately 50 milliseconds, which is crucial for the timely transmission of conference control commands. Especially in mobile network environments where frequent network switching occurs, this parameter ensures that the control channel can be quickly restored, maintaining the continuity of conference control and real-time responsiveness.

[0018] S35, Based on the exponential backoff algorithm, set up a reconnection mechanism; S36, According to steps S33 to S35, establish a first transmission channel and verify the stability of the first transmission channel through a heartbeat mechanism: send heartbeat probe packets at fixed intervals and measure the response time; update the indicators in step S31 according to the response time; when the response time is greater than the threshold, trigger the reconnection mechanism in step S35.

[0019] In particular, traditional heartbeat mechanisms are typically only used to detect connection interruptions, lacking the ability to predict network quality trends. Problems are often only detected after the connection is completely lost. In this application, the heartbeat mechanism (S36) cleverly forms a closed loop with network quality assessment. It not only monitors connection status but also feeds the monitoring results back to the network scoring system in S31, using the heartbeat response time as a real-time input for network quality assessment. This solves the problems of delayed network quality prediction and passive fault detection.

[0020] Further, in step S31, a network quality score is calculated, including obtaining the Round Trip Time (RTT) by sending probe packets. The RTT refers to the total round-trip time required for a data packet to travel from the sender to the receiver, be received and processed, and then return to the sender, typically measured in milliseconds (ms). In this application, the system sends a probe packet with a specific structure to the server and records a precise timestamp during transmission. The server immediately returns a response upon receiving the packet. Upon receiving the response, the client calculates the time difference between transmission and reception, thus obtaining one RTT sample. The system typically sends a probe packet every 100-500 milliseconds, continuously collecting multiple RTT samples (typically 10-20) to form an RTT dataset for subsequent analysis.

[0021] Calculate the standard deviation of multiple consecutive RTT delay samples to obtain the network jitter value; count the number of probe packets sent and received to calculate the packet loss rate; and obtain the network quality score by weighted calculation based on the RTT delay time, network jitter value, and packet loss rate.

[0022] Furthermore, in step S35, based on the exponential backoff algorithm, a reconnection mechanism is set up, including: when the connection established in step S33 or step S34 is detected to be interrupted, a reconnection process is executed; the initial reconnection interval is set to T1, and the reconnection interval is doubled after each reconnection failure until the maximum reconnection interval T0 is reached; after reaching the preset maximum number of reconnections, a connection failure notification is sent; during each reconnection, the protocol of the current network environment is reselected according to the results of steps S31 and S32; during the reconnection, newly generated control information in the video conference is sorted by priority and cached; when the reconnection is successful, the conference status is synchronized and the current conference status on the server is obtained; the locally cached control information is compared with the conference status on the server, redundant or conflicting control information is eliminated, and valid cached control information is obtained; the valid cached control information is sent through the first transmission channel in priority order to ensure the continuity of conference control.

[0023] The Exponential Backoff Algorithm is a retry strategy in network communication. In the reconnection mechanism of a video conferencing system, it exponentially extends the waiting time by dynamically increasing the time interval between consecutive failed reconnection attempts, thus avoiding waste of network resources and improving the eventual reconnection success rate. In this application, the initial reconnection interval T1 is typically set to 100-300 milliseconds, balancing the need for rapid response and avoiding immediate reconnection. For low-latency information such as control commands, a shorter initial interval ensures that the system can quickly recover from minor network fluctuations. The maximum reconnection interval T0 is generally set to 20-30 seconds to prevent the reconnection interval from growing indefinitely and causing prolonged system unavailability, while also allowing sufficient time for network recovery. The maximum number of reconnections is set based on the tolerance of the video conferencing scenario, typically 10-15 times, so that the total reconnection time covers most temporary network interruption situations (approximately 5-10 minutes). If this limit is exceeded, the user is notified to handle the issue manually.

[0024] In particular, traditional systems often experience inconsistent control states after network interruption recovery, such as inconsistent microphone status displayed on different clients or chaotic whiteboard operation permissions. This is because the network layer reconnection mechanism and the application layer state recovery mechanism are disconnected. This application deeply integrates the network layer connection recovery mechanism (S35) with the application layer state synchronization mechanism. It intelligently controls the reconnection rhythm through an exponential backoff algorithm, while maintaining control information caching and priority sorting. After successful reconnection, it implements server state synchronization and conflict resolution strategies, constructing a cross-layer fault self-healing system. This solves the problem of inconsistent meeting states caused by network fluctuations.

[0025] On the other hand, this application uses an exponential backoff algorithm to set up network reconnection, avoiding the "reconnection storm" that may be caused by traditional fixed-interval reconnection, reducing the burden on the network and server. Especially in large-scale conference systems, it can effectively prevent server overload caused by multiple users reconnecting simultaneously. Combining network quality assessment and protocol selection using S31 and S32, the optimal protocol is dynamically selected each time a reconnection occurs, enabling the system to adapt to changes in the network environment and improving the reconnection success rate. Working in conjunction with control information caching, priority sorting, and conflict resolution mechanisms, it ensures that even after a long network outage, critical control commands can be reliably transmitted in the correct order, maintaining the continuity and consistency of conference control.

[0026] Further, in step S4, a second transmission channel associated with the video stream is established for the content information through the SEI mechanism or RTCP channel of the video encoder, including: S41, obtaining characteristic parameters including data volume, real-time requirements, and relevance to the video based on the content information; S42, simultaneously establishing the SEI mechanism and RTCP channel as candidate paths based on the characteristic parameters and the current network status, and calculating the transmission adaptability scores of the two candidate paths respectively; S43, processing the content information into segments, and allocating the optimal transmission channel to each content segment based on the transmission adaptability score; S44, setting the incremental transmission strategy for the content information and the time synchronization mechanism with the video stream based on the optimal transmission channel; S45, establishing the second transmission channel according to steps S41 to S44, and associating the second transmission channel with the video stream encoder to perform synchronous transmission of the content information and the video stream.

[0027] Specifically, traditional video conferencing systems generally use a single channel to transmit whiteboard and other content information, which cannot cope with various problems in complex network environments: when using an independent data channel, content information is prone to desynchronization with the video under network congestion; when embedded in the video stream, it increases the video encoding pressure and affects video quality. This application utilizes two complementary transmission paths—the SEI (Supplemental Enhancement Information) mechanism and the RTCP (RTP Control Protocol) channel—to establish a transmission system with redundant backup capabilities. The SEI mechanism, as part of the video encoding standard, can directly embed auxiliary data into the video stream; while RTCP, as an independent control channel, offers flexibility and reliability. Simultaneously establishing and dynamically evaluating these two transmission paths provides the system with unprecedented transmission flexibility.

[0028] On the other hand, traditional content delivery systems treat whiteboard data as a whole, neglecting the differentiated needs of different content elements. For example, pointer positions require high real-time performance but have a small data volume, while image content has a large data volume but lower real-time requirements. A uniform transmission strategy leads to unreasonable resource allocation and fails to meet the transmission needs of different elements. This improvement solves the problem of differentiated needs in content information transmission through feature parameter analysis and quantification. This application establishes a feature parameter evaluation system based on three dimensions: "data volume, real-time requirements, and relevance to video," innovatively quantifying the transmission needs of content information. This multi-dimensional feature analysis breaks away from the traditional "one-size-fits-all" transmission strategy, providing differentiated transmission solutions for different types of content elements.

[0029] Preferably, in step S44, based on the optimal transmission channel, an incremental transmission strategy for content-related information and a time synchronization mechanism with the video stream are set, including: establishing a version control system for content-related information, detecting changes in the shared whiteboard data obtained in step S1, and generating an incremental update package containing version numbers and difference data; setting a transmission priority for the incremental update package based on the optimal transmission channel allocated in step S43 and the real-time requirements of the content, and executing the transmission; implementing a keyframe synchronization mechanism between content-related information and the video stream, triggering the transmission of the complete whiteboard state when a video I-frame is generated, ensuring the consistency of the content state when the video scene is switched; establishing a time mapping relationship between content segments and video frames, assigning corresponding video timestamps to content segments, and implementing rendering synchronization at the receiving end based on the timestamp information, dynamically adjusting rendering parameters when a synchronization error exceeds a threshold.

[0030] In particular, whiteboard content in video conferencing typically changes incrementally. Each drawing or annotation by a user adds to or modifies existing content. Furthermore, in a multi-person collaborative environment, each participant needs to see each other's actions and content changes in near real-time. Moreover, in video conferencing scenarios, whiteboard content usually needs to be synchronized with the explanatory video.

[0031] Therefore, this application employs an incremental transmission strategy, transmitting only the changed content rather than the complete whiteboard state, significantly reducing data volume while maintaining visual quality. This is particularly important for bandwidth-constrained mobile networks and unstable home networks. Furthermore, by combining the I-frame characteristics of video encoding technology with whiteboard state synchronization, the transmission of the complete whiteboard state is triggered when keyframes are generated in the video, cleverly solving the multimedia synchronization problem. This mechanism ensures that even under network fluctuations, the whiteboard content and video explanation maintain semantic consistency. Establishing a temporal mapping relationship between content fragments and video frames ensures that the whiteboard content seen by the user remains synchronized with the video content; in presentation scenarios, this synchronization is crucial for user experience.

[0032] Further, in step S42, the transmission adaptability scores of the two candidate paths are calculated respectively, including: obtaining network performance indicators of the SEI mechanism and the RTCP channel, wherein the network performance indicators include current latency, packet loss rate and available bandwidth; setting the weights of content class information feature parameters; and calculating the transmission adaptability scores of the SEI mechanism and the RTCP channel respectively based on the weights and network performance indicators.

[0033] Further, in step S5, control information and content information are transmitted separately, and sequence numbers and timestamps are added to the transmitted information, including: S51, prioritizing the control information and transmitting it according to priority on the first transmission channel; S52, transmitting the content information in segments according to the allocated optimal transmission channel and executing the transmission according to the incremental transmission strategy in step S44; S53, assigning globally unique sequence numbers and timestamps synchronized with the video time base to the control information and content information segments; S54, monitoring the status of the first transmission channel through the heartbeat mechanism in step S36, and triggering the reconnection mechanism in step S35 when an abnormality is detected; simultaneously monitoring the status of the second transmission channel, and re-triggering the transmission adaptability scoring in step S42 when an abnormality is detected.

[0034] Specifically, traditional video conferencing systems typically employ a unified transmission strategy to handle different types of information. This can lead to high-priority control commands being blocked by low-priority content transmission under bandwidth constraints, or a large amount of content data crowding out control information transmission resources. This application deeply decouples control and content information transmission while maintaining logical association through a unified sequence number and timestamp system. S51 establishes a priority transmission queue for control commands, while S52 implements segmented transmission channel allocation for content information. This breaks the limitations of mutual interference between different types of information transmission in traditional systems, constructing an adaptive, reliable, and efficient transmission architecture.

[0035] On the other hand, in traditional systems, different information streams typically use independent sequence numbering systems and time bases, making it difficult for the receiving end to accurately reconstruct the temporal relationships between various information streams. Especially when network latency fluctuates significantly, control commands and content updates may have incorrect timing, impacting user experience. For example, the narrator's pointer movement may precede or lag the audio narration display, causing comprehension difficulties. This application introduces a dual identification mechanism of globally unique sequence numbers and video synchronization timestamps for all auxiliary information. This design establishes a unified time reference framework, enabling auxiliary information from different sources and channels to be precisely aligned with the video stream at the receiving end. In particular, the timestamp design directly references the video time base, fundamentally solving the problem of inconsistent synchronization bases for multi-source data.

[0036] Further, in step S6, a synchronization data structure is generated, including: S61, receiving transmitted control information and content information, sorting them according to sequence numbers, and detecting packet loss: for control information, when a discontinuous sequence number is detected, a request is made to the sender to retransmit lost control commands; for content information, lost data is recovered according to the incremental transmission strategy in step S44. S62, establishing a unified time base and aligning the video stream timestamp with the auxiliary information timestamp added in step S53: calculating the time offset between the video stream and control information to compensate for network transmission delay differences; calibrating the content information timestamp according to the time synchronization mechanism in step S44; S63, constructing an association structure between video frames and auxiliary information based on the aligned timestamps: associating video frames, control information, and content information within the time window; dynamically adjusting the time window size when a synchronization error exceeds a threshold; S64, generating a synchronization data structure containing video frame data, conference control status, and whiteboard content based on the association structure for use by the conference client.

[0037] Compared to existing technologies, the advantages of this application are:

[0038] Video conferencing suffers from low reliability in auxiliary information transmission when network conditions are unstable. Existing technologies generally use a single channel to process all types of auxiliary information, but this has the drawback of not being able to optimize transmission strategies for information with different characteristics. This application solves the differentiated transmission needs of information with different characteristics by classifying auxiliary information and using a dual-channel transmission mechanism, thereby improving the overall transmission efficiency and reliability. Attached Figure Description

[0039] This application will be further described by way of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting; in these embodiments, the same reference numerals denote the same structures, wherein:

[0040] Figure 1 This is an exemplary flowchart of a video information processing method according to some embodiments of this application;

[0041] Figure 2 This is an exemplary flowchart illustrating the classification of auxiliary information according to some embodiments of this application;

[0042] Figure 3 This is an exemplary flowchart illustrating auxiliary information transmission according to some embodiments of this application;

[0043] Figure 4 This is an exemplary flowchart illustrating the establishment of a first transmission channel according to some embodiments of this application;

[0044] Figure 5This is an exemplary flowchart of video transmission according to some embodiments of this application. Detailed Implementation

[0045] The methods and systems provided in the embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0046] like Figure 1 As shown, the video conferencing auxiliary information transmission method provided in this embodiment of the invention mainly includes the following steps: acquiring video streams and auxiliary information, classifying and processing them, establishing a dual-channel transmission system, serializing and transmitting the data, and synchronously reassembling and displaying it. This method achieves efficient synchronous transmission of auxiliary information and video by distinguishing between control-type and content-type information and establishing differentiated transmission channels.

[0047] Specifically, the system acquires video streams from video conference participants and obtains auxiliary information from the video conferencing system, including conference control commands and shared whiteboard data. This auxiliary information is categorized into control-related information and content-related information. A first transmission channel, independent of the video stream, is established for the control-related information via WebSocket or QUIC protocol. A second transmission channel, associated with the video stream, is established for the content-related information via the SEI mechanism of the video encoder or RTCP channel. Control-related information and content-related information are transmitted according to the first and second transmission channels, with sequence numbers and timestamps added to the transmitted information. At the receiving end, the auxiliary information is received, and the auxiliary information is time-stamp aligned and synchronized with the video stream based on the sequence number and timestamp to generate a synchronization data structure. This synchronization data structure is used by the conference client to display video feeds and share whiteboard content.

[0048] like Figure 2 As shown, auxiliary information is divided into two main categories: control information and content information. Control information mainly includes meeting control commands, such as speaking requests, screen sharing controls, and meeting status changes; content information mainly includes shared whiteboard data, such as whiteboard pages, drawing elements, text annotations, and pointer positions.

[0049] like Figure 3 As shown, the acquisition and classification of video streams and auxiliary information specifically includes: calling the system camera API to acquire the raw video streams of video conference participants, and simultaneously acquiring auxiliary information data from the conference system service layer. Preprocessing the acquired video streams, including resolution adaptation, frame rate adjustment, and image enhancement, generates standard YUV format video frame data. Parsing the auxiliary information data packets acquired from the conference system, and extracting the type field from the auxiliary information header.

[0050] Based on the type field in the auxiliary information header, auxiliary information is classified into control information and content information: auxiliary information with type values ​​of 0x01-0x1F is classified as control information; auxiliary information with type values ​​of 0x20-0x3F is classified as content information.

[0051] After classification, the control information and content information are transformed and standardized respectively: the control information is converted into JSON format for subsequent transmission via WebSocket or QUIC; the content information is converted into binary format for subsequent transmission via SEI or RTCP.

[0052] like Figure 4 As shown, it includes four main stages: network quality assessment, protocol selection, connection establishment, and connection maintenance. Network quality assessment specifically includes:

[0053] (a) Initialize the network probing module and configure the probe packet size (64 bytes), sending interval (100ms), and number of samples (20).

[0054] (b) Create a UDP socket, send a probe packet to the specified port of the conference server and start a timer.

[0055] (c) Receive the probe packet response returned by the server and record the difference between the sending time and the receiving time as an RTT sample.

[0056] (d) Collect 20 consecutive RTT samples and calculate the average RTT value as the RTT delay time (in ms).

[0057] (e) Apply the standard deviation calculation formula to the collected RTT sample set to obtain the network jitter value: jitter value = √(∑(RTTi - RTT average value)² / n).

[0058] (f) Calculate the packet loss rate by counting the number of probe packets sent (Ns) and received (Nr): Packet loss rate = (Ns-Nr) / Ns×100%.

[0059] (g) Based on the measurement data, the network quality score is obtained by applying the weighted calculation formula: Network quality score = 100 - (0.4 × RTT / 200 + 0.3 × jitter value / 50 + 0.3 × packet loss rate). The score range is 0-100, and the higher the score, the better the network quality.

[0060] The protocol selection includes:

[0061] (a) Set the network quality score threshold to 65 points as the basis for protocol selection.

[0062] (b) When the network quality score is greater than 65, the WebSocket protocol is selected as the underlying protocol for the first transmission channel;

[0063] (c) When the network quality score is less than or equal to 65, the QUIC protocol is selected as the underlying protocol of the first transmission channel.

[0064] (d) Record the protocol selection result and pass it to the connection establishment module.

[0065] Establishing a WebSocket connection includes:

[0066] (a) Create a WebSocket client instance and configure the connection URL as "wss: / / meeting.server.com / control".

[0067] (b) Set the TLS handshake timeout to 3 seconds and the connection timeout to 5 seconds.

[0068] (c) Configure the WebSocket header information, adding the meeting ID, user ID, and authentication token.

[0069] (d) Execute the WebSocket handshake process to establish a secure WebSocket connection based on TCP.

[0070] (e) After a successful connection, set the keep-alive frame interval to 25 seconds to ensure that the connection remains available.

[0071] (f) Configure message compression options and enable the permessage-deflate extension to improve transmission efficiency.

[0072] (g) Send a connection confirmation message to the server, including the client version and supported message types.

[0073] Establishing a QUIC connection includes:

[0074] (a) Initialize the QUIC client and load the TLS certificate and key.

[0075] (b) Configure QUIC parameters, including maximum number of streams, initial window size, and maximum packet size.

[0076] (c) Enable 0-RTT functionality, save and load the recovery token from the previous session.

[0077] (d) Establish a QUIC connection with the target address "quic: / / meeting.server.com:8443".

[0078] (e) After a successful connection, a bidirectional stream is created for control message transmission.

[0079] (f) Configure the congestion control algorithm as BBR to improve transmission performance in weak network environments.

[0080] (g) Register client information with the server and complete connection initialization.

[0081] The reconnection mechanism includes:

[0082] (a) Initialize the reconnection parameters, set the initial reconnection interval T1 to 200ms, the maximum reconnection interval T0 to 30 seconds, and the maximum number of reconnections to 15.

[0083] (b) When a connection interruption is detected, initiate the reconnection process: record the current number of reconnections and set the current reconnection interval to T1; re-execute S31 and S32 based on the current network environment to determine the protocol to be used for reconnection; execute the connection establishment process of step S33 or S34 according to the selected protocol; if the connection fails, double the reconnection interval (T=min(T×2, T0)) and wait for a specified time before trying to connect again; when the maximum number of reconnections is reached, send a connection failure notification to the user interface.

[0084] (c) During reconnection, create a control information cache queue and sort and cache newly generated control information according to the following priorities: Priority 1: Changes in core meeting status (such as meeting end, emergency evacuation); Priority 2: Controls related to speaking permissions (such as requesting to speak, granting speaking rights); Priority 3: Media control instructions (such as audio and video on / off, sharing control); Priority 4: Regular UI interactions (such as raising hand, liking, sending messages).

[0085] (d) After successful reconnection, perform state synchronization and caching: request the current meeting state from the server and obtain the latest participant list, speaking status, sharing status, etc.; compare the locally cached control information with the server meeting state and delete redundant or conflicting control information; send the valid cached control information to the server through the newly established connection according to priority; send a reconnection success notification to the client UI layer and restore normal meeting control functions.

[0086] The heartbeat mechanism includes:

[0087] (a) Initialize the heartbeat parameters, set the heartbeat interval to 5 seconds, the response timeout threshold to 3 seconds, and the consecutive failure threshold to 3 times.

[0088] (b) Start the timer to send heartbeat detection packets at set intervals;

[0089] (c) Record the heartbeat packet sending time and wait for the server response.

[0090] (d) Receive the heartbeat response returned by the server, calculate the round-trip time and update the RTT sample set.

[0091] (e) Update the network quality metrics in S31 based on the latest RTT sample.

[0092] (f) When the heartbeat response time exceeds the threshold or no response is received for several consecutive times, the reconnection mechanism of S35 is triggered.

[0093] (g) Perform protocol evaluations periodically and consider switching transport protocols when network conditions continue to change.

[0094] Establishing a second transmission channel includes:

[0095] First, obtain the feature parameters of the content type information:

[0096] (a) Analyze the data characteristics of content information and extract the following parameters: Data volume: Calculate the byte size of the content data, which is divided into small (<5KB), medium (5KB-50KB) and large (>50KB); Real-time requirements: Determine the real-time requirements based on the content type, which is divided into high (e.g., pointer position, <100ms), medium (e.g., handwriting, <300ms) and low (e.g., background image, <1000ms); Video relevance: Evaluate the degree of relevance between the content and the current video, which is divided into high (e.g., pointer in the explanation), medium (e.g., the graphic being drawn) and low (e.g., static background).

[0097] (b) Quantize the parameters and convert them into scores of 0-100 to construct feature vectors.

[0098] (c) Record the characteristic parameters and pass them to the transmission path evaluation module.

[0099] Then, the transport adaptability score is calculated:

[0100] (a) Initialize the SEI and RTCP channels and establish test connections respectively.

[0101] (b) Obtain network performance metrics for two channels: send probe packets to measure the current latency (ms); count packet loss during continuous transmission and calculate the packet loss rate (%); use the bandwidth probing algorithm to estimate the available bandwidth (Kbps).

[0102] (c) Set the weight matrix of content feature parameters: Data volume weight: SEI channel 0.4, RTCP channel 0.3; Real-time weight: SEI channel 0.4, RTCP channel 0.5; Video relevance weight: SEI channel 0.2, RTCP channel 0.2.

[0103] (d) Calculate the transmission adaptability scores for the two channels respectively: SEI score = data volume score × 0.4 + real-time performance score × 0.4 + video correlation score × 0.2 - delay penalty - packet loss penalty; RTCP score = data volume score × 0.3 + real-time performance score × 0.5 + video correlation score × 0.2 - delay penalty - packet loss penalty; where delay penalty and packet loss penalty are calculated proportionally according to the measurement indicators.

[0104] Next, content fragmentation and channel allocation:

[0105] (a) Content information is segmented according to content type and size: large whiteboard background image: segmented into 2KB pieces; medium-sized drawing element set: segmented into 1KB pieces; small control data (such as pointer position): treated as a single piece.

[0106] (b) Assign a unique identifier and fragment number to each fragment.

[0107] (c) Based on the transmission adaptability score calculated in step 3-2, select the optimal transmission channel for each fragment: when the SEI score is higher than the RTCP score, select the SEI channel; when the RTCP score is higher than the SEI score, select the RTCP channel; when the score difference is less than the threshold (5 points), consider load balancing and assign the fragment to the channel with the lighter current load.

[0108] (d) Generate a fragmented transmission plan and record the fragment ID, target channel, and transmission priority.

[0109] Configure incremental transmission and time synchronization mechanisms:

[0110] Establish a content version control system: assign a globally incrementing version number to the whiteboard content status; calculate the difference data with the previous version each time the content changes; package the difference data together with the current version number to form an incremental update package.

[0111] Set up a transmission priority mechanism: Assign transmission priorities based on the real-time requirements of content fragments; high real-time content such as pointer positions has a priority of 1; medium real-time content such as handwriting trajectories has a priority of 2; and low real-time content such as background images has a priority of 3.

[0112] Implement a synchronization mechanism with video keyframes: listen for video encoder I-frame generation events; when I-frame generation is detected, trigger the transmission of the complete whiteboard state; associate the current complete whiteboard state with the I-frame and package it into a key state packet; send the key state packet through the selected transmission channel.

[0113] Establish time mapping relationship: Obtain the timestamp (PTS) of the current video frame from the video encoder; add timestamps corresponding to the video PTS to the content segments; write timestamp information in the segment header for synchronization at the receiving end; set the synchronization error threshold to 50ms, and dynamically adjust the rendering timing when the threshold is exceeded.

[0114] Among them, SEI channel transmission:

[0115] (a) Initialize the SEI packaging module, configure the company UUID as "ABCDEF12" and the business UUID as "34567890".

[0116] (b) Set the maximum payload size of the SEI packet to 1400 bytes to avoid transmission fragmentation.

[0117] (c) Perform Huffman coding compression on the content fragment data to reduce the amount of data transmitted.

[0118] (d) Generate SEI data packets, the specific implementation is as follows: Figure 5 The process is as follows: Write NAL header information, type 0x06; write SEI type 0x05, indicating custom user data; write data length according to 0xFF segmentation logic; write company UUID and business UUID identifiers; write compressed payload data; write end marker (0x00 0x80).

[0119] (e) Associate SEI packets with video frame data: Determine the current video frame type. For IDR frames, first write SPS and PPS; insert the SEI packet into the video NAL array, before the video frame NAL; combine to form a complete video ES data stream.

[0120] (f) Submit the generated ES data to the video encoder for processing.

[0121] Among them, RTCP channel transmission:

[0122] (a) Initialize the RTCP extension module and configure the custom RTCP packet type to 205.

[0123] (b) Create a custom RTCP format containing the following fields: Packet type (8 bits): fixed value 205; Subtype (8 bits): 0x01 indicates blank data; Length (16 bits): total length of the RTCP packet; Synchronization source identifier (32 bits): SSRC matching the RTP stream; Timestamp (32 bits): timestamp corresponding to the video PTS; Fragment ID (16 bits): unique identifier of the content fragment; Fragment sequence number (16 bits): position of the current fragment in the sequence; Version number (32 bits): version identifier of the content update; Payload length (16 bits): length of the data payload; Data payload: compressed content data.

[0124] (c) Send RTCP packets through the control channel of the RTP session, sharing the same transport session as the video RTP stream.

[0125] (d) Set the RTCP sending interval to 20ms to avoid network congestion.

[0126] Auxiliary information transmission and serialization:

[0127] Control information is transmitted according to priority:

[0128] (a) Initialize the control information queue and set four priority levels: Priority 1: Core meeting status control (such as ending the meeting, forced exit); Priority 2: Media stream control (such as mute, stop video); Priority 3: Sharing control (such as start / stop screen sharing); Priority 4: Regular interaction (such as raising hand, facial expression feedback).

[0129] (b) Receive control commands from the UI layer and assign priorities based on command type.

[0130] (c) Insert the control command into the queue of the corresponding priority.

[0131] (d) Start the transmission scheduler and process the queues in order of priority: when the high priority queue is not empty, high priority control instructions are processed first; in each transmission round, at least one low priority instruction is processed to avoid starvation; the transmission frequency is dynamically adjusted according to the network conditions, and high priority instructions are guaranteed to be transmitted in the event of congestion.

[0132] Content-type information is transmitted in segments:

[0133] (a) Receive the fragmentation transmission plan from step S43.

[0134] (b) Based on the planned channel allocation, route the fragments to the SEI channel or the RTCP channel.

[0135] (c) For SEI channel fragmentation: call the SEI packing method in step S43 to generate SEI data packets; associate the SEI data packets with the next available video frame; and send them together through the video encoding process.

[0136] (d) For RTCP channel fragmentation: Call the RTCP packet generation method in step S43 to create an RTCP control packet; send it through the control channel of the RTP session; monitor the sending status and record the sending time and acknowledgment status.

[0137] (e) Implement the incremental transmission strategy in step S44: track content changes and transmit only the changed parts; send a complete status periodically (e.g., every 30 seconds) to ensure synchronization; and send a complete status immediately when a new client is detected.

[0138] The serial number and timestamp allocation are as follows:

[0139] (a) Initialize the global serial number generator with an initial value of 1 and a step size of 1.

[0140] (b) Assign an incrementing globally unique sequence number to each control class information and content class information segment.

[0141] (c) The serial number encoding format is a 64-bit integer, with the high 32 bits being the information type encoding and the low 32 bits being the sequence number.

[0142] (d) Obtain the PTS (Presentation Time Stamp) of the current video frame from the video encoder as the base timestamp.

[0143] (e) Add timestamps to control information: record the system time when the control command is generated; calculate the time offset from the current video PTS; add the offset to the video PTS as the associated timestamp of the control command; write a 32-bit timestamp field to the header of the control message.

[0144] (f) Add timestamps to content information: Calculate the corresponding video PTS based on the generation time of the content segment; write the PTS value to the timestamp field in the segment header; for content that needs precise synchronization (such as pointer position), add additional fine-tuning offsets.

[0145] (g) Before sending the information, confirm the correctness of the sequence number and timestamp.

[0146] Among them, channel monitoring and anomaly handling:

[0147] (a) Set up heartbeat monitoring for the first transmission channel: According to the heartbeat mechanism in step S36, periodically check the channel status; record the RTT delay value and update the network quality score; set the channel abnormality judgment criteria: three consecutive heartbeat timeouts or RTT sudden increase of more than 200%; when a channel abnormality is detected, trigger the reconnection mechanism in S35.

[0148] (b) Set up performance monitoring for the second transmission channel: periodically (every 5 seconds) send probe data to measure the current performance of the SEI and RTCP channels; calculate the actual transmission success rate and latency data; re-trigger the transmission adaptability score when the performance of any channel drops by more than 30%; dynamically adjust the channel allocation of the fragments based on the updated score results.

[0149] (c) Implement anomaly recovery strategy: For the failure of the first channel, use the alternative protocol for recovery first; for the failure of the second channel, redirect the critical content to the available channel first; in extreme cases (both channels fail), start the degraded service mode and transmit only the most critical control information.

[0150] Receiver processing steps:

[0151] Auxiliary information reception and sorting:

[0152] (a) Initialize the receive buffer for control information and content information.

[0153] (b) Receive control information from the first transmission channel: parse WebSocket or QUIC messages and extract control instruction content; verify message integrity and signature validity; extract global sequence number from message header; insert messages into the ordered control information buffer according to sequence number.

[0154] (c) Receive content class information of the second transmission channel: For the SEI channel, extract the SEI packet from the video frame: parse the video ES stream and identify the SEI NAL type (0x06); verify whether the SEI type is 0x05 (user data); verify whether the company UUID and business UUID match; extract the SEI payload data.

[0155] For the RTCP channel, parse the RTCP control packet: identify the custom RTCP packet type (205); verify the subtype and synchronization source identifier; extract the content data from the RTCP packet.

[0156] Extract the sequence number and fragment information from the content fragments; insert the fragments into the content information buffer according to their sequence numbers.

[0157] (d) Perform packet loss detection and processing: check the sequence number continuity in the control information buffer; when a non-contiguous sequence number is detected (interval > 1), mark the packet as lost; send a retransmission request to the sender, including the range of lost sequence numbers; set the retransmission timeout to 500ms, and re-request after the timeout; attempt retransmission a maximum of 3 times, and if it still fails, perform state synchronization recovery.

[0158] (e) Handling packet loss of content information: Identify lost content fragments based on version number and fragment sequence number; for loss of incremental updates, request the most recent complete state; reconstruct the state using existing incremental updates; if reconstruction is not possible, wait for the next complete state update.

[0159] Among them, the timestamp alignment is as follows:

[0160] (a) Establish a unified time reference system: use the PTS of the video frame output by the video decoder as the main time reference; maintain a sliding window of size 10 to record the timestamp of the most recent video frame; calculate the average time interval between video frames for interpolation calculation.

[0161] (b) Calculate the time offset of the control information: measure the network transmission delay of the control information from sending to receiving; calculate the network delay difference between the video stream and the control stream; calculate the time compensation value based on the delay difference: time compensation value = network delay of control information - network delay of video stream, subtract the compensation value from the timestamp of the control information to achieve time alignment with the video stream.

[0162] (c) Timestamp calibration of content information: Extract the associated timestamps of content segments according to the time synchronization mechanism; for content transmitted through the SEI channel, directly use the PTS of the associated video frame; for content transmitted through the RTCP channel, apply delay compensation similar to that of control information; use a linear interpolation algorithm to handle the transition between timestamps: interpolated timestamp = previous timestamp + (current time - previous update time) / (next update time - previous update time) × (next timestamp - previous timestamp).

[0163] Among them, constructing the association structure:

[0164] (a) Define time window parameters: default window size is ±100ms; minimum window size is ±50ms; maximum window size is ±300ms; synchronization error threshold is 50ms.

[0165] (b) Time association processing after alignment: Create a time window based on the video frame timestamp; search for control information and content information within the window; establish a ternary association structure of <video frame, control set, content set>; sort the control information corresponding to the same video frame at multiple time points by sequence number; retain only the latest version of the same content for different updated versions.

[0166] (c) Dynamic time window adjustment: Monitor synchronization error and calculate the actual time deviation statistics; when the synchronization error exceeds the threshold for 3 consecutive times, expand the time window; when the synchronization error is less than 50% of the threshold for 10 consecutive times, shrink the time window; the window size adjustment step is 25ms to ensure a smooth transition.

[0167] (d) Abnormal situation handling: When a video frame is missing, use interpolation between the preceding and following frames to estimate the time point; when critical control information exceeds the window range, expand the single window; when network jitter is severe, temporarily increase the window to accommodate latency fluctuations.

[0168] Constructing a synchronous data structure:

[0169] (a) Define a unified synchronous data structure format:

[0170] struct SyncFrame {

[0171] uint64_t frameId; / / Unique identifier for the frame

[0172] uint64_t timestamp; / / Video timestamp

[0173] VideoFrameData videoData; / / Video frame data

[0174] vector <controlmessage>controls; / / List of control information

[0175] ContentState contentState; / / Whiteboard content state

[0176] uint32_t syncQuality; / / Synchronize quality metrics};

[0177] (b) Based on the association structure, populate the synchronization data structure: copy the current video frame data into the structure; add the control information within the time window to the control list; construct the complete content state at the current time point according to the content update sequence; calculate the synchronization quality index to reflect the current synchronization accuracy.

[0178] (c) Perform data optimization and conflict handling: merge consecutive control instructions of the same type and retain the latest state; handle the dependencies and conflicts between control instructions; ensure that the operation sequence of whiteboard content is logically consistent; remove operation instructions that have been overwritten or canceled.

[0179] (d) Generate the final synchronization data structure and provide it to the conference client UI layer through the callback interface.

[0180] (e) Implement a buffer queue to smoothly process synchronous data streams: maintain a synchronous frame buffer queue of size 5; provide a stable data source for UI rendering and reduce stuttering; maintain the continuous display of video and auxiliary information in the event of network fluctuations.

[0181] This completes the end-to-end video conferencing auxiliary information transmission process from the sending end to the receiving end, achieving reliable transmission and accurate synchronization of control and content information, and effectively solving the problem of reliable auxiliary information transmission in video conferencing under unstable network conditions.

[0182] The foregoing illustrative description of the invention and its embodiments is not restrictive and can be implemented in other specific forms without departing from the spirit or essential characteristics of the invention. The accompanying drawings are only one embodiment of the invention, and the actual structure is not limited thereto. No reference numerals in the claims should limit the scope of the claims. Therefore, if a person skilled in the art, inspired by this description, designs a similar structure and embodiment without departing from the spirit of the invention, such design should fall within the scope of protection of this application. Furthermore, the word "comprising" does not exclude other elements or steps, and the word "a" preceding an element does not exclude the inclusion of "a plurality" of that element. Multiple elements stated in the product claims may also be implemented by a single element through software or hardware. The terms "first," "second," etc., are used to indicate names and do not indicate any specific order.< / controlmessage>

Claims

1. A video information processing method, characterized in that, include: S1, acquire the video streams of the video conference participants and obtain auxiliary information from the video conferencing system, the auxiliary information including conference control instructions and shared whiteboard data; S2 divides auxiliary information into control information and content information; S3 establishes a first transmission channel for control information that is independent of the video stream via WebSocket or QUIC protocol; S4, establishes a second transmission channel associated with the video stream for content-type information through the SEI mechanism or RTCP channel of the video encoder; S5, according to the first transmission channel and the second transmission channel, transmit control information and content information respectively, and add a sequence number and timestamp to the transmitted information; S6, the receiving end receives auxiliary information and performs time stamp alignment and synchronization processing between the auxiliary information and the video stream according to the sequence number and timestamp to generate a synchronization data structure. The synchronization data structure is used by the conference client to display video images and share whiteboard content. Step S3 includes: S31, Obtain metrics including RTT latency, network jitter, and packet loss rate; and calculate a network quality score based on the metrics. S32. When the network quality score is greater than the threshold, select the WebSocket protocol to ensure compatibility; otherwise, select the QUIC protocol to improve transmission efficiency. S33, When the WebSocket protocol is selected, a TCP-based WebSocket connection is established, and the keep-alive frame interval time parameter is set to ensure that the connection is continuously available; S34, When the QUIC protocol is selected, establish a UDP-based QUIC connection and configure the 0-RTT parameter to accelerate connection recovery; S35, based on the exponential backoff algorithm, sets up a reconnection mechanism; S36, According to steps S33 to S35, establish the first transmission channel and verify the stability of the first transmission channel through a heartbeat mechanism: Send heartbeat detection packets at fixed intervals and measure the response time; Update the metrics in step S31 based on the response time; When the response time exceeds the threshold, the reconnection mechanism in step S35 is triggered. Step S4 includes: S41, Based on the content type information, obtain feature parameters including data volume, real-time requirements, and relevance to the video; S42, based on the characteristic parameters and the current network status, simultaneously establish the SEI mechanism and RTCP channel as candidate paths, and calculate the transmission adaptability scores of the two candidate paths respectively. S43, the content information is segmented and the optimal transmission channel is allocated to each content segment based on the transmission adaptability score; S44, Based on the optimal transmission channel, set the incremental transmission strategy for content-type information, as well as the time synchronization mechanism with the video stream; S45, according to steps S41 to S44, establish a second transmission channel and associate the second transmission channel with the video stream encoder to perform synchronous transmission of content information and video stream.

2. The video information processing method according to claim 1, characterized in that: Control information includes meeting control instructions, and content information includes shared whiteboard data; The first transmission channel is used for the real-time transmission of control information; The second transmission channel is used to support incremental transmission of large amounts of data and maintain synchronous transmission with video. The synchronized data structure includes video frame data, conference control status, and whiteboard content.

3. The video information processing method according to claim 1, characterized in that: S31, Calculate the network quality score, including: The RTT delay time is obtained by sending probe packets; Calculate the standard deviation of multiple consecutive RTT delay samples to obtain the network jitter value; Count the number of probe packets sent and received, and calculate the packet loss rate; A network quality score is obtained by weighting the RTT latency, network jitter, and packet loss rate.

4. The video information processing method according to claim 1, characterized in that: S35, based on the exponential backoff algorithm, sets up a reconnection mechanism, including: When the connection established in step S33 or step S34 is detected to be interrupted, the reconnection process is executed. Set the initial reconnection interval to T1, and double the reconnection interval after each reconnection failure until the maximum reconnection interval T0 is reached. After reaching the preset maximum number of reconnections, a connection failure notification is sent. Each time a connection is reconnected, the protocol for the current network environment is reselected based on the results of steps S31 and S32. During the reconnection period, newly generated control information in the video conference will be sorted by priority and cached. Once the reconnection is successful, synchronize the meeting status and obtain the current meeting status from the server. Compare the local cached control information with the server-side meeting status, eliminate redundant or conflicting control information, and obtain effective cached control information; Effective cache control information is sent through the first transmission channel in priority order to ensure the continuity of conference control.

5. The video information processing method according to claim 1, characterized in that: S42, and calculate the transmission adaptability scores for the two candidate paths respectively, including: Obtain network performance metrics for the SEI mechanism and RTCP channel, including current latency, packet loss rate, and available bandwidth; Set the weights of content-type information feature parameters; Based on weights and network performance metrics, the transmission adaptability scores for the SEI mechanism and the RTCP channel are calculated respectively.

6. The video information processing method according to any one of claims 2 to 5, characterized in that: S5, according to the first transmission channel and the second transmission channel, transmit control information and content information respectively, including: S51, prioritize the control information and transmit it according to priority based on the first transmission channel; S52, according to the allocated optimal transmission channel, the content information is transmitted in segments, and the transmission is performed according to the incremental transmission strategy in step S44. S53 assigns globally unique sequence numbers to control information and content information segments, as well as timestamps synchronized with the video time base; S54, monitor the status of the first transmission channel through the heartbeat mechanism in step S36. When an abnormality is detected in the channel, trigger the reconnection mechanism in step S35. At the same time, monitor the status of the second transmission channel. When an abnormality is detected, re-trigger the transmission adaptability score in step S42.

7. The video information processing method according to claim 6, characterized in that: S6 generates synchronized data structures, including: S61, Receive transmitted control and content information, sort them according to sequence number, and perform packet loss detection: For control information, when a discontinuous sequence number is detected, a request is made to the sender to retransmit the lost control command. For content-related information, the lost data is recovered according to the incremental transmission strategy in step S44; S62, Establish a unified time reference and align the video stream timestamp with the auxiliary information timestamp added in step S53: Calculate the time offset between the video stream and control information to compensate for differences in network transmission latency; According to the time synchronization mechanism in step S44, calibrate the timestamp of the content information; S63, construct the association structure between video frames and auxiliary information based on the aligned timestamps: Associate video frames, control information, and content information within the time window; When a synchronization error exceeds a threshold, the time window size is dynamically adjusted. S64 generates a synchronized data structure containing video frame data, meeting control status, and whiteboard content based on the association structure, for use by the meeting client.

Images