Video information processing method and system

Abstract

The invention discloses a video information processing method, and relates to video transmission, and the method comprises the steps: obtaining a video stream of a video conference participant, and obtaining auxiliary information from a video conference system; dividing the auxiliary information into control type information and content type information; establishing a first transmission channel independent of a video stream for the control type information through a WebSocket or QUIC protocol; establishing a second transmission channel associated with the video stream for the content information through an SEI mechanism or an RTCP channel of the video encoder; respectively transmitting the control type information and the content type information according to the first transmission channel and the second transmission channel, and adding a serial number and a timestamp to the transmitted information; and receiving the auxiliary information at a receiving end, and performing timestamp alignment and synchronization processing on the auxiliary information and the video stream according to the serial number and the timestamp to generate a synchronous data structure. Aiming at low transmission reliability of the auxiliary information under the condition that the network condition is unstable, the application improves the transmission reliability of the auxiliary information under the condition that the network is unstable.

Description

Technical Field

[0001] The present application relates to the field of video transmission, and in particular to a video information processing method and system. Background Art

[0002] With the prevalence of remote work, distance education, and global collaboration, video conferencing systems have become a vital component of modern communications infrastructure. In recent years, global events have particularly driven a surge in demand for high-quality, highly reliable video conferencing systems. In this context, video conferencing systems must not only deliver high-definition video and crystal-clear audio but also support real-time transmission of auxiliary information, such as meeting control commands and shared whiteboards, to deliver a rich collaborative experience.

[0003] Video conferencing systems typically operate over packet-switched networks, yet the modern internet environment is complex and volatile. This is particularly true in scenarios such as mobile networks, home broadband networks, and international communications, where bandwidth fluctuations, latency variations, and increased packet loss are common. Video coding technologies, through mechanisms such as adaptive bitrate control, forward error correction, and packet retransmission, have proven effective in addressing network fluctuations. However, a systematic solution to the reliability of auxiliary information transmission remains lacking.

[0004] On the one hand, conference control commands (such as turning audio and video on and off, controlling screen sharing, and changing conference permissions) are small in volume, highly real-time, and critical. In traditional video conferencing systems, control commands are typically transmitted through the following methods: signaling channels: These are transmitted over signaling channels using protocols such as SIP or H.323. However, these protocols are less adaptable to network fluctuations and can cause control failures in environments with high packet loss rates; WebSocket persistent connections: Based on an upgrade of the HTTP protocol, they enable full-duplex communication, but are inefficient during network disconnections and reconnections, and have compatibility issues in certain network environments; and periodic polling mechanisms: These mechanisms use HTTP to periodically request status changes, but this lacks real-time performance and can introduce control delays in situations with high network latency. These methods fail to fully account for dynamic network conditions, resulting in the loss or delay of conference control commands in unstable network conditions, severely impacting the conference process and user experience. Furthermore, the retransmission mechanism for control commands lacks effective correlation with the video stream status, easily leading to inconsistencies between the control status and the video content.

[0005] On the other hand, shared whiteboard data, as content-based information, has the characteristics of large data volume, incremental changes, and close connection with video content. In the existing technology, there are mainly the following ways to transmit whiteboard data: independent data channel transmission: whiteboard data is transmitted through an independent TCP or WebSocket channel, but there is a lack of synchronization mechanism with the video stream, which may cause the whiteboard content to be out of sync with the video explanation when the network fluctuates; screenshot transmission: the whiteboard content is encoded and transmitted as part of the video, which wastes bandwidth and has poor interactivity, and cannot support real-time collaboration among multiple parties; complete state periodic synchronization: the complete state of the whiteboard is transmitted regularly, which occupies a large amount of bandwidth and can easily lead to incomplete whiteboard content in the event of packet loss. The existing technologies have failed to effectively solve the problem of precise synchronization between whiteboard data and video streams, nor have they been able to dynamically select the optimal transmission strategy based on network conditions. Especially when network conditions fluctuate, the update of whiteboard content may lag or be lost, resulting in inconsistent whiteboard status between meeting participants, seriously affecting collaboration efficiency.

[0006] In existing technologies, video streams and auxiliary information typically use different transmission channels and processing mechanisms, which results in different timestamp systems for the video streams and auxiliary information, making precise alignment difficult. Especially in video conferencing systems deployed across regions, due to large differences in network latency, traditional synchronization mechanisms cannot ensure that all participants see a consistent conference status, resulting in chaotic interactions and reduced conference efficiency. Summary of the Invention

[0007] In view of the low reliability of auxiliary information transmission under unstable network conditions, the present application provides a video information processing method and system, which improves the reliability of auxiliary information transmission under unstable network conditions based on dual-channel adaptive transmission and synchronous processing.

[0008] The present application provides a video information processing method, including: S1, obtaining a video stream of a video conference participant and obtaining auxiliary information from a video conference system, wherein the auxiliary information includes conference control instructions and shared whiteboard data; S2, dividing the auxiliary information into control information and content information; S3, establishing a first transmission channel independent of the video stream for the control information through the WebSocket or QUIC protocol; S4, establishing a second transmission channel associated with the video stream for the content information through the SEI mechanism or RTCP channel of the video encoder; S5, transmitting the control information and the content information respectively according to the first transmission channel and the second transmission channel, and adding a serial number and a timestamp to the transmitted information; S6, receiving the auxiliary information at the receiving end, and performing timestamp alignment and synchronization processing on the auxiliary information and the video stream according to the serial number and the timestamp to generate a synchronization data structure, wherein the synchronization data structure is used for the conference client to display the video screen and the shared whiteboard content.

[0009] In particular, on the one hand, traditional video conferencing systems usually process all auxiliary information (such as conference control instructions and whiteboard data) through the same transmission mechanism, ignoring the differences in transmission requirements for different types of information. Control information (such as mute and hand raising) has a small amount of data but has extremely high requirements for real-time and reliability, while content information (such as whiteboard data) has a large amount of data but is closely related to the video content. Unified processing leads to unreasonable resource allocation and cannot meet the transmission requirements of different types of information. In this application, the auxiliary information is clearly divided into "control information" and "content information" through step S2, and transmission channels with different characteristics are established for these two types of information in steps S3 and S4. This differentiated processing breaks the pattern of treating all auxiliary information as homogeneous data in traditional video conferencing systems.

[0010] On the other hand, in traditional systems, content information (such as whiteboard data) is often transmitted through independent data channels and lacks an inherent temporal relationship with the video stream. This can easily lead to content-video desynchronization issues under fluctuating network conditions. For example, a lecturer may point to a certain part of the whiteboard while the whiteboard content has not yet loaded, or the whiteboard content may be updated later than the lecturer's voice commentary, severely impacting information delivery. In this application, in step S4, the video encoder's SEI mechanism or RTCP channel is innovatively utilized to establish a transmission channel for content information associated with the video stream. This design enables content information to maintain an inherent synchronization relationship with the video.

[0011] In addition, traditional video conferencing systems usually use simple time window matching or fixed delay compensation methods to deal with synchronization problems at the receiving end, which cannot cope with delay jitter and changes in transmission order in complex network environments. This results in users often seeing unsynchronized video and auxiliary information when network conditions fluctuate, such as the pointer position not matching the actual commentary, or the control command execution order being disordered. In this application, in step S6, intelligent synchronization and reorganization processing is implemented at the receiving end, and auxiliary information from different channels is accurately aligned with the video stream through sequence numbers and timestamps to generate a unified synchronization data structure.

[0012] Furthermore, control information includes conference control instructions, and content information includes shared whiteboard data; the first transmission channel is used for real-time transmission of control information; the second transmission channel is used to support incremental transmission of large amounts of data and maintain synchronous transmission with the video; the synchronous data structure includes video frame data, conference control status and whiteboard content.

[0013] Furthermore, S3 establishes a first transmission channel independent of the video stream for control information through the WebSocket or QUIC protocol, including: S31, obtaining indicators including RTT delay, network jitter and packet loss rate; and calculating the network quality score based on the indicators; S32, when the network quality score is greater than the threshold, selecting the WebSocket protocol to ensure compatibility, otherwise selecting the QUIC protocol to improve transmission efficiency.

[0014] In particular, traditional video conferencing systems generally use static protocol configurations, which are unable to adapt to changes in the network environment, especially in mobile scenarios where network quality fluctuates significantly, such as switching from an enterprise network to public Wi-Fi and then to a 4G network. Fixed protocol strategies often lead to inefficient control information transmission or even interruption. This application combines a multi-dimensional quantitative assessment of network quality (S31) with dynamic switching of transmission protocols (S32) to establish adaptive transmission decision-making.

[0015] S33: When the WebSocket protocol is selected, a WebSocket connection based on TCP is established, and the keep-alive frame interval parameter is set to ensure continuous connection availability. S34: When the QUIC protocol is selected, a QUIC connection based on UDP is established, and the 0-RTT parameter is configured to speed up connection recovery.

[0016] WebSocket, as a TCP-based long-connection protocol, may pass through multiple layers of NAT, firewalls, or proxy servers in the network. These intermediate devices typically time out inactive connections (typical timeouts are 30 to 90 seconds). Keep-alive frames are lightweight control frames that do not carry application data. Periodically sending these frames indicates to intermediate network devices that the connection is still active, preventing the connection from being accidentally closed. In video conferencing scenarios, properly setting the keep-alive frame interval is crucial. An interval that is too long may cause the connection to be disconnected by the intermediate device, while an interval that is too short will increase unnecessary network overhead. In this application, this parameter is set to 50%-70% of the minimum timeout of the intermediate device (for example, 25 seconds) to achieve a balance between connection stability and network efficiency.

[0017] QUIC is a UDP-based transport layer protocol developed by Google and now forms the foundation of HTTP / 3. 0-RTT (Zero Round Trip Time) is a key feature of the QUIC protocol. The 0-RTT parameter is a QUIC feature that accelerates connection recovery, enabling clients to immediately send application data upon resuming a previous connection without completing a full handshake. Traditional TLS connection resumption requires a single round-trip (1-RTT) before sending application data. The QUIC protocol's 0-RTT feature caches session tickets (SessionTickets) from previous connections and server configuration parameters, allowing clients to include application data in the first packet without waiting for a server response, thus saving a full network round-trip. Configuration parameters include enabling 0-RTT, setting the session ticket validity period (typically several hours), configuring the maximum acceptable 0-RTT packet size, and configuring replay attack protection policies. In video conferencing scenarios, the session ticket validity period is typically set to the maximum expected duration of the meeting (e.g., 4-8 hours) to cover potential disconnections and reconnections throughout the meeting. When network fluctuations cause a connection interruption, the 0-RTT parameter significantly reduces reconnection time from 300-500 milliseconds in traditional protocols to approximately 50 milliseconds, which is crucial for the timely delivery of conference control commands. Especially in mobile networks with frequent network switching, this parameter ensures that the control channel can be quickly restored, maintaining conference control continuity and real-time responsiveness.

[0018] S35, set up a reconnection mechanism based on the exponential backoff algorithm; S36, establish a first transmission channel according to steps S33 to S35, and verify the stability of the first transmission channel through the heartbeat mechanism: send heartbeat detection packets at fixed intervals and measure the response time; update the indicator in step S31 according to the response time; when the response time is greater than the threshold, trigger the reconnection mechanism in step S35.

[0019] In particular, traditional heartbeat mechanisms are typically used only to detect connection interruptions, lacking the ability to predict network quality trends, and often only detect problems after the connection is completely disconnected. The heartbeat mechanism (S36) in this application cleverly forms a closed loop with network quality assessment. It not only monitors connection status but also feeds monitoring results back to the network scoring system in S31. It uses heartbeat response time as a real-time input for network quality assessment, resolving the issues of delayed network quality prediction and passive fault detection.

[0020] Furthermore, S31 calculates the network quality score, including: obtaining the RTT delay time by sending a probe packet; wherein the RTT delay time (Round Trip Time) refers to the total time required for a data packet to be sent from the sender, received and processed by the receiver, and then returned to the sender, and the unit is usually milliseconds (ms). In this application, the system sends a probe packet (Probe Packet) with a specific structure to the server and records the precise timestamp when sending; the server returns a response immediately after receiving it; after the client receives the response, it calculates the time difference from sending to receiving, that is, obtains an RTT sample. The system usually sends a probe packet every 100-500 milliseconds, continuously collects multiple RTT samples (typically 10-20), and forms an RTT data set for subsequent analysis.

[0021] Calculate the standard deviation of multiple consecutive RTT delay samples to obtain the network jitter value; count the number of sent and received probe packets to calculate the packet loss rate; and obtain the network quality score through weighted calculation based on the RTT delay time, network jitter value and packet loss rate.

[0022] Furthermore, S35 sets a reconnection mechanism based on an exponential backoff algorithm, including: when it is detected that the connection established in step S33 or step S34 is interrupted, executing a reconnection process; setting an initial reconnection interval to T1, and doubling the reconnection interval time after each reconnection failure until the maximum reconnection interval T0 is reached; sending a connection failure notification after reaching the preset maximum number of reconnections; each time the connection is reconnected, reselecting the protocol of the current network environment according to the results of steps S31 and S32; during the reconnection period, sorting and caching the newly generated control information in the video conference by priority; when the reconnection is successful, synchronizing the conference status and obtaining the current conference status of the server; comparing the locally cached control information with the server-side conference status, eliminating redundant or conflicting control information, and obtaining valid cached control information; sending the valid cached control information through the first transmission channel in order of priority to ensure the continuity of conference control.

[0023] The Exponential Backoff Algorithm (EBA) is a retry strategy used in network communications. In the reconnection mechanism of a video conferencing system, it exponentially increases the waiting time by dynamically increasing the interval between consecutive failed reconnection attempts, avoiding wasted network resources and improving the final reconnection success rate. In this application, the initial reconnection interval, T1, is typically set to 100-300 milliseconds, balancing the requirements for fast response and avoiding immediate reconnections. For low-latency messages such as control commands, a shorter initial interval ensures the system can quickly recover from minor network fluctuations. The maximum reconnection interval, T0, is typically set to 20-30 seconds to prevent the reconnection interval from increasing indefinitely, causing prolonged system unavailability, while also allowing sufficient time for network recovery. The maximum number of retries is set based on the tolerance of the video conferencing scenario, typically 10-15 times, to ensure that the total reconnection time covers most temporary network outages (approximately 5-10 minutes). If this limit is exceeded, the user is notified to manually resolve the issue.

[0024] In particular, in traditional systems, inconsistent control status often occurs after network interruption and recovery, such as inconsistent display of a participant's microphone status on different clients, or confusion in whiteboard operation permissions. This is because the network layer reconnection and application layer state recovery mechanisms are separated from each other. This application deeply integrates the network layer connection recovery mechanism (S35) with the application layer state synchronization mechanism, intelligently controls the reconnection rhythm through an exponential backoff algorithm, maintains control information cache and priority sorting, and implements server state synchronization and conflict elimination strategies after successful reconnection, thus building a cross-layer fault self-healing system. This solves the problem of inconsistent conference status caused by network fluctuations.

[0025] On the other hand, this application sets up network reconnection based on the exponential backoff algorithm, avoiding the "reconnection storm" that may be caused by traditional fixed-interval reconnection, reducing the burden on the network and server, especially in large-scale conference systems, which can effectively prevent server overload caused by simultaneous reconnection of multiple users. Combined with the network quality evaluation and protocol selection of S31 and S32, the optimal protocol is dynamically selected each time reconnection occurs, so that the system can adapt to changes in the network environment and improve the reconnection success rate. Working in conjunction with the control information cache, priority sorting and conflict elimination mechanism, it ensures that even after a long period of network interruption, key control instructions can be reliably transmitted in the correct order to maintain the continuity and consistency of conference control.

[0026] Furthermore, S4 establishes a second transmission channel associated with the video stream for the content information through the SEI mechanism or RTCP channel of the video encoder, including: S41, obtaining characteristic parameters including data volume, real-time requirements, and relevance to the video based on the content information; S42, establishing the SEI mechanism and RTCP channel as candidate paths based on the characteristic parameters and the current network status, and calculating the transmission adaptability scores of the two candidate paths respectively; S43, fragmenting the content information and allocating the optimal transmission channel to each content fragment based on the transmission adaptability score; S44, setting an incremental transmission strategy for the content information and a time synchronization mechanism with the video stream based on the optimal transmission channel; S45, establishing a second transmission channel according to steps S41 to S44, and associating the second transmission channel with the video stream encoder for synchronous transmission of the content information and the video stream.

[0027] In particular, on the one hand, traditional video conferencing systems generally use a single channel to transmit content information such as whiteboards, which cannot cope with various problems in complex network environments: when using independent data channels, it is easy to cause content information to be out of sync with the video in the case of network congestion; when embedded in the video stream, it will increase the pressure on video encoding and affect the video quality. This application uses the SEI (Supplemental Enhancement Information) mechanism and the RTCP (RTP Control Protocol) channel, two transmission paths with complementary technical characteristics, to establish a transmission system with redundant backup characteristics. As part of the video coding standard, the SEI mechanism can embed auxiliary data directly into the video stream; and RTCP, as an independent control channel, is flexible and reliable. Establishing and dynamically evaluating these two transmission paths at the same time provides the system with unprecedented transmission flexibility.

[0028] On the other hand, traditional content transmission systems process whiteboard data as a whole, ignoring the differentiated needs of different content elements. For example, the pointer position requires high real-time performance but a small amount of data, while the image content requires a large amount of data but has lower real-time requirements. The unified transmission strategy leads to unreasonable resource allocation and cannot take into account the transmission needs of different elements. This improvement solves the problem of differentiated needs in content information transmission through feature parameter analysis and quantification. This application establishes a feature parameter evaluation system based on the three dimensions of "data volume, real-time requirements, and relevance to video", and innovatively quantifies the transmission needs of content information. This multi-dimensional feature analysis breaks the traditional "one-size-fits-all" transmission strategy and provides differentiated transmission solutions for different types of content elements.

[0029] Preferably, S44 sets an incremental transmission strategy for content information and a time synchronization mechanism with the video stream according to the optimal transmission channel, including: establishing a version control system for content information, detecting changes in the shared whiteboard data obtained in step S1, and generating an incremental update package containing a version number and difference data; setting a transmission priority for the incremental update package based on the optimal transmission channel and content real-time requirements allocated in step S43, and executing the transmission; implementing a key frame synchronization mechanism between content information and video streams, triggering a complete whiteboard state transmission when a video I frame is generated, and ensuring consistency of content state when the video scene switches; establishing a time mapping relationship between content fragments and video frames, assigning corresponding video timestamps to content fragments, and implementing rendering synchronization at the receiving end based on the timestamp information, and dynamically adjusting the rendering parameters when it is detected that the synchronization error exceeds the threshold.

[0030] In particular, whiteboard content in video conferencing usually changes incrementally. Each drawing and annotation by the user is an addition or modification to the existing content. In addition, in a multi-person collaborative environment, each participant needs to see each other's operations and content changes in near real time. Moreover, in video conferencing scenarios, whiteboard content usually needs to be synchronized with the lecture video.

[0031] Therefore, this application, through an incremental transmission strategy, only transmits the changed content instead of the complete whiteboard state, significantly reducing the amount of data while maintaining the visual effect. This is especially important for mobile networks with limited bandwidth and unstable home networks. In addition, the I-frame feature in video encoding technology is combined with whiteboard state synchronization to trigger the transmission of the complete whiteboard state when the video key frame is generated, cleverly solving the multimedia synchronization problem. This mechanism ensures that the whiteboard content and video explanation can maintain semantic consistency even in the case of network fluctuations. A time mapping relationship between content fragments and video frames is established to ensure that the whiteboard content seen by the user is synchronized with the video content; in the explanation and demonstration scenario, this synchronization is crucial to the user experience.

[0032] Furthermore, S42 calculates the transmission adaptability scores of the two candidate paths respectively, including: obtaining network performance indicators of the SEI mechanism and the RTCP channel, the network performance indicators including current delay, packet loss rate and available bandwidth; setting weights of content-type information feature parameters; and calculating the transmission adaptability scores of the SEI mechanism and the RTCP channel respectively according to the weights and network performance indicators.

[0033] Furthermore, S5 transmits control information and content information separately, and adds serial numbers and timestamps to the transmitted information, including: S51, prioritizing the control information and transmitting it according to the priority according to the first transmission channel; S52, transmitting the content information in fragments according to the allocated optimal transmission channel, and executing the transmission according to the incremental transmission strategy of step S44; S53, assigning globally unique serial numbers and timestamps synchronized with the video time base to the control information and content information fragments; S54, monitoring the status of the first transmission channel through the heartbeat mechanism of step S36, and triggering the reconnection mechanism of step S35 when a channel abnormality is detected; and simultaneously monitoring the status of the second transmission channel, and re-triggering the transmission adaptability score of step S42 when an abnormality is detected.

[0034] In particular, on the one hand, traditional video conferencing systems usually use a unified transmission strategy to process different types of information, resulting in high-priority control instructions being blocked by low-priority content transmission when bandwidth is limited, or a large amount of content data crowding out the transmission resources of control information. This application deeply decouples the transmission of control and content information, but at the same time maintains a logical connection through a unified serial number and timestamp system. S51 establishes a priority transmission queue for control instructions, while S52 implements fragment-level transmission channel allocation for content information. It breaks the limitation of mutual interference of various types of information transmission in traditional systems and builds an adaptive, reliable and efficient transmission architecture.

[0035] On the other hand, in traditional systems, different information streams usually use independent serial number systems and time bases, which makes it difficult for the receiving end to accurately reconstruct the timing relationship between the various information. Especially when the network delay fluctuates greatly, the control commands and content updates may have incorrect timing, affecting the user experience. For example, the pointer movement of the narrator may be earlier or later than his voice commentary display, making it difficult to understand. This application introduces a dual identification mechanism of globally unique serial numbers and video synchronization timestamps for all auxiliary information. This design establishes a unified time reference framework, so that auxiliary information from different sources and different channels can be accurately aligned with the video stream at the receiving end. In particular, the timestamp design directly references the video time base, which fundamentally solves the problem of inconsistent synchronization bases for multi-source data.

[0036] Furthermore, S6 generates a synchronization data structure, including: S61, receiving the transmitted control information and content information, sorting and detecting packet loss according to the sequence number: for the control information, when the sequence number is detected to be discontinuous, requesting the sender to retransmit the lost control instructions; for the content information, recovering the lost data according to the incremental transmission strategy of step S44. S62, establishing a unified time base, aligning the video stream timestamp with the auxiliary information timestamp added in step S53: calculating the time offset between the video stream and the control information, compensating for the network transmission delay difference; calibrating the timestamp of the content information according to the time synchronization mechanism of step S44; S63, constructing an association structure between the video frame and the auxiliary information based on the aligned timestamps: associating the video frame, control information and content information within the time window; dynamically adjusting the time window size when the synchronization error is detected to exceed the threshold; S64, generating a synchronization data structure containing video frame data, conference control status and whiteboard content based on the association structure for use by the conference client.

[0037] Compared with the existing technology, the advantages of this application are:

[0038] When the network conditions are unstable for video conferencing, there is a phenomenon of low reliability in the transmission of auxiliary information. The existing technology generally adopts a single channel to uniformly process all types of auxiliary information, but there is a defect that the transmission strategy cannot be optimized for information with different characteristics. This application solves the differentiated transmission needs of information with different characteristics through auxiliary information classification and dual-channel transmission mechanism, thereby improving the overall transmission efficiency and reliability. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The present application will be further described in the form of exemplary embodiments, which will be described in detail with reference to the accompanying drawings. These embodiments are not limiting, and in these embodiments, the same numbers represent the same structures, wherein:

[0040] Figure 1 is an exemplary flow chart of a video information processing method according to some embodiments of the present application;

[0041] Figure 2 is an exemplary flow chart of auxiliary information classification according to some embodiments of the present application;

[0042] Figure 3 is an exemplary flow chart of auxiliary information transmission according to some embodiments of the present application;

[0043] Figure 4 is an exemplary flowchart of establishing a first transmission channel according to some embodiments of the present application;

[0044] Figure 5This is an exemplary flowchart of video transmission according to some embodiments of the present application. DETAILED DESCRIPTION

[0045] The method and system provided in the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0046] like Figure 1 As shown, the video conferencing auxiliary information transmission method provided by an embodiment of the present invention mainly includes the steps of obtaining video streams and auxiliary information, classifying and processing them, establishing a dual-channel transmission system, serializing and transmitting them, and synchronously reassembling and presenting them. This method establishes differentiated transmission channels by distinguishing and processing control-related and content-related information, thus achieving efficient synchronous transmission of auxiliary information and video.

[0047] Specifically, the video stream of the video conference participants is obtained, and auxiliary information is obtained from the video conference system, wherein the auxiliary information includes conference control instructions and shared whiteboard data; the auxiliary information is divided into control information and content information; a first transmission channel independent of the video stream is established for the control information through the WebSocket or QUIC protocol; a second transmission channel associated with the video stream is established for the content information through the SEI mechanism or RTCP channel of the video encoder; according to the first transmission channel and the second transmission channel, the control information and the content information are transmitted respectively, and a serial number and a timestamp are added to the transmitted information; the auxiliary information is received at the receiving end, and the auxiliary information is timestamp-aligned and synchronized with the video stream according to the serial number and timestamp to generate a synchronization data structure, and the synchronization data structure is used for the conference client to display the video screen and the shared whiteboard content.

[0048] like Figure 2 As shown in Figure 1, auxiliary information is divided into two categories: control information and content information. Control information mainly includes conference control commands, such as speaking requests, screen sharing controls, and conference status changes. Content information mainly includes shared whiteboard data, such as whiteboard pages, drawing elements, text annotations, and pointer positions.

[0049] like Figure 3 As shown in the figure, the acquisition and classification of video streams and auxiliary information specifically involves calling the system camera API to obtain the original video stream of the video conference participants and simultaneously obtaining auxiliary information data from the conference system service layer. The obtained video stream is preprocessed, including resolution adaptation, frame rate adjustment, and image enhancement, to generate video frame data in standard YUV format. The auxiliary information data packet obtained from the conference system is parsed and the type field in the auxiliary information header is extracted.

[0050] According to the type field in the auxiliary information header, the auxiliary information is classified into control information and content information: auxiliary information with a type value of 0x01-0x1F is classified as control information; auxiliary information with a type value of 0x20-0x3F is classified as content information.

[0051] The classified control information and content information are converted into data structures and standardized respectively: the control information is converted into JSON format to facilitate subsequent transmission via WebSocket or QUIC; the content information is converted into binary format to facilitate subsequent transmission via SEI or RTCP.

[0052] like Figure 4 As shown in the figure, it includes four main steps: network quality assessment, protocol selection, connection establishment and connection maintenance. Network quality assessment specifically includes:

[0053] (a) Initialize the network detection module and configure the detection packet size (64 bytes), sending interval (100ms), and number of samples (20).

[0054] (b) Create a UDP socket, send a probe packet to the specified port of the conference server and start a timer.

[0055] (c) Receive the probe packet response returned by the server and record the difference between the sending time and the receiving time as an RTT sample.

[0056] (d) Collect 20 consecutive RTT samples and calculate the average RTT value as the RTT delay time (unit: ms).

[0057] (e) Apply the standard deviation calculation formula to the collected RTT sample set to obtain the network jitter value: Jitter value = √(∑(RTTi - RTT average)² / n).

[0058] (f) Count the number of probe packets sent, Ns, and received, Nr, and calculate the packet loss rate: Packet loss rate = (Ns - Nr) / Ns × 100%.

[0059] (g) Based on the measured data, a weighted calculation formula is applied to derive the network quality score: Network Quality Score = 100 - (0.4 × RTT / 200 + 0.3 × Jitter / 50 + 0.3 × Packet Loss Rate). The score ranges from 0 to 100, with a higher score indicating better network quality.

[0060] Among them, the protocol selection includes:

[0061] (a) Set the network quality score threshold to 65 points as the basis for judging protocol selection.

[0062] (b) When the network quality score is greater than 65, the WebSocket protocol is selected as the underlying protocol of the first transmission channel;

[0063] (c) When the network quality score is less than or equal to 65 points, the QUIC protocol is selected as the underlying protocol of the first transmission channel.

[0064] (d) Record the protocol selection result and pass it to the connection establishment module.

[0065] Establishing a WebSocket connection includes:

[0066] (a) Create a WebSocket client instance and configure the connection URL to "wss: / / meeting.server.com / control".

[0067] (b) Set the TLS handshake timeout to 3 seconds and the connection timeout to 5 seconds.

[0068] (c) Configure the WebSocket header information and add the conference ID, user ID, and authentication token.

[0069] (d) Execute the WebSocket handshake process to establish a secure WebSocket connection based on TCP.

[0070] (e) After the connection is successful, set the keep-alive frame interval to 25 seconds to ensure that the connection remains available.

[0071] (f) Configure message compression options and enable the permessage-deflate extension to improve transmission efficiency.

[0072] (g) Send a connection confirmation message to the server, including the client version and supported message types.

[0073] Among them, establishing a QUIC connection includes:

[0074] (a) Initialize the QUIC client and load the TLS certificate and key.

[0075] (b) Configure QUIC parameters, including the maximum number of streams, initial window size, and maximum packet size.

[0076] (c) Enable 0-RTT functionality and save and load resume tokens from previous sessions.

[0077] (d) Establish a QUIC connection with the destination address "quic: / / meeting.server.com:8443".

[0078] (e) After the connection is successful, a bidirectional stream is created for control message transmission.

[0079] (f) Configure the congestion control algorithm to BBR to improve transmission performance in weak network environments.

[0080] (g) Register client information with the server to complete connection initialization.

[0081] Among them, the reconnection mechanism includes:

[0082] (a) Initialize the reconnection parameters, set the initial reconnection interval T1 to 200ms, the maximum reconnection interval T0 to 30 seconds, and the maximum number of reconnections to 15 times.

[0083] (b) When a connection interruption is detected, the reconnection process is initiated: the current number of reconnections is recorded and the current reconnection interval is set to T1; S31 and S32 are re-executed based on the current network environment to determine the protocol to be used for reconnection; the connection establishment process of step S33 or S34 is executed according to the selected protocol; if the connection fails, the reconnection interval is doubled (T = min (T × 2, T0)), and the connection is attempted again after waiting for a specified time; when the maximum number of reconnections is reached, a connection failure notification is sent to the user interface.

[0084] (c) During the reconnection period, a control information cache queue is created to sort and cache newly generated control information according to the following priorities: Priority 1: changes in the core status of the meeting (such as meeting end, emergency evacuation); Priority 2: control related to speaking rights (such as applying for speaking, granting speaking rights); Priority 3: media control instructions (such as audio and video switching, sharing control); Priority 4: regular UI interactions (such as raising hands, liking, and sending messages).

[0085] (d) After successful reconnection, status synchronization and cache processing are performed: the current conference status is requested from the server to obtain the latest participant list, speaking status, sharing status, etc.; the locally cached control information is compared with the server conference status, and redundant or conflicting control information is deleted; the valid cached control information is sent to the server through the newly established connection according to priority; a reconnection success notification is sent to the client UI layer to restore normal conference control functions.

[0086] Among them, the heartbeat mechanism includes:

[0087] (a) Initialize the heartbeat parameters, set the heartbeat interval to 5 seconds, the response timeout threshold to 3 seconds, and the continuous failure threshold to 3 times.

[0088] (b) Start the timer and send heartbeat detection packets at the set interval;

[0089] (c) Record the time when the heartbeat packet is sent and wait for the server to respond.

[0090] (d) Receive the heartbeat response returned by the server, calculate the round-trip time and update the RTT sample set.

[0091] (e) Update the network quality indicator in S31 based on the latest RTT sample.

[0092] (f) When the heartbeat response time exceeds the threshold or no response is received for multiple consecutive times, the reconnection mechanism of S35 is triggered.

[0093] (g) Perform protocol evaluations periodically and consider switching transport protocols when network conditions continue to change.

[0094] Establishing a second transmission channel, including:

[0095] First, obtain the content information characteristic parameters:

[0096] (a) Analyze the data characteristics of content information and extract the following parameters: Data volume: Calculate the byte size of the content data and categorize it into small (<5KB), medium (5KB-50KB), and large (>50KB); Real-time requirements: Determine the real-time requirements based on the content type and categorize them into high (e.g., pointer position, <100ms), medium (e.g., handwriting, <300ms), and low (e.g., background image, <1000ms); Video relevance: Evaluate the degree of relevance of the content to the current video and categorize it into high (e.g., a pointer during a lecture), medium (e.g., a graphic being drawn), and low (e.g., a static background).

[0097] (b) Quantify the parameters, convert them into scores between 0 and 100, and construct feature vectors.

[0098] (c) Record the characteristic parameters and pass them to the transmission path evaluation module.

[0099] Then, calculate the transfer suitability score:

[0100] (a) Initialize SEI and RTCP channels and establish test connections respectively.

[0101] (b) Obtain network performance metrics for the two channels: send probe packets and measure the current latency (ms); count packet loss during continuous transmission and calculate the packet loss rate (%); use a bandwidth detection algorithm to estimate the available bandwidth (Kbps).

[0102] (c) Set the weight matrix of content feature parameters: data volume weight: SEI channel 0.4, RTCP channel 0.3; real-time weight: SEI channel 0.4, RTCP channel 0.5; video relevance weight: SEI channel 0.2, RTCP channel 0.2.

[0103] (d) Calculate the transmission adaptability scores of the two channels separately: SEI score = data volume score × 0.4 + real-time score × 0.4 + video relevance score × 0.2 - delay penalty - packet loss penalty; RTCP score = data volume score × 0.3 + real-time score × 0.5 + video relevance score × 0.2 - delay penalty - packet loss penalty; among which, the delay penalty and packet loss penalty are calculated proportionally based on the measurement indicators.

[0104] Next, content segmentation and channel allocation:

[0105] (a) Content information is fragmented according to content type and size: large whiteboard background images are fragmented into 2KB pieces; medium-sized drawing element collections are fragmented into 1KB pieces; small control data (such as pointer position) are treated as a single fragment.

[0106] (b) Assign a unique identifier and a shard sequence number to each shard.

[0107] (c) Based on the transmission adaptability score calculated in step 3-2, the optimal transmission channel is selected for each fragment: when the SEI score is higher than the RTCP score, the SEI channel is selected; when the RTCP score is higher than the SEI score, the RTCP channel is selected; when the score difference is less than the threshold (5 points), considering load balancing, the fragment is assigned to the channel with the lighter current load.

[0108] (d) Generate a shard transmission plan, recording the shard ID, target channel, and transmission priority.

[0109] Set up incremental transmission and time synchronization mechanisms:

[0110] Establish a content version control system: assign a global incremental version number to the whiteboard content status; each time the content changes, calculate the difference data with the previous version; package the difference data together with the current version number to form an incremental update package.

[0111] Set up a transmission priority mechanism: Assign transmission priorities based on the real-time requirements of content segments; high-real-time content such as pointer position has a priority of 1; medium-real-time content such as handwriting tracks has a priority of 2; low-real-time content such as background images has a priority of 3.

[0112] Implement a synchronization mechanism with the video key frame: monitor the video encoder I-frame generation event; when I-frame generation is detected, trigger the complete whiteboard state transmission; associate the current complete whiteboard state with the I-frame and package it into a key state package; and send the key state package through the selected transmission channel.

[0113] Establish a time mapping relationship: obtain the timestamp (PTS) of the current video frame from the video encoder; add a timestamp corresponding to the video PTS to the content fragment; write the timestamp information in the fragment header for synchronization on the receiving end; set the synchronization error threshold to 50ms, and dynamically adjust the rendering timing when it is detected that the threshold is exceeded.

[0114] Among them, SEI channel transmission:

[0115] (a) Initialize the SEI packaging module and configure the company UUID to "ABCDEF12" and the business UUID to "34567890".

[0116] (b) Set the maximum payload size of the SEI packet to 1400 bytes to avoid transmission fragmentation.

[0117] (c) Perform Huffman coding compression on content fragment data to reduce the amount of transmitted data.

[0118] (d) Generate SEI data packet, the specific implementation is as follows Figure 5 The process shown is as follows: write NAL header information, type 0x06; write SEI type 0x05, indicating custom user data; write data length according to 0xFF segmentation logic; write company UUID and business UUID identifiers; write compressed payload data; and write end marker (0x00 0x80).

[0119] (e) Associate the SEI packet with the video frame data: Determine the current video frame type. For IDR frames, write the SPS and PPS first. Insert the SEI packet into the video NAL array before the video frame NAL. Combine to form a complete video ES data stream.

[0120] (f) Submit the generated ES data to the video encoder for processing.

[0121] Among them, RTCP channel transmission:

[0122] (a) Initialize the RTCP extension module and configure the custom RTCP packet type to 205.

[0123] (b) Create a custom RTCP format containing the following fields: Packet Type (8 bits): fixed value 205; Subtype (8 bits): 0x01 indicates whiteboard data; Length (16 bits): total length of the RTCP packet; Synchronization Source Identifier (32 bits): SSRC that matches the RTP stream; Timestamp (32 bits): timestamp corresponding to the video PTS; Fragment ID (16 bits): unique identifier of the content fragment; Fragment Sequence Number (16 bits): position of the current fragment in the sequence; Version Number (32 bits): version identifier of the content update; Payload Length (16 bits): length of the data payload; Data payload: compressed content data.

[0124] (c) Send the RTCP packet through the control channel of the RTP session, sharing the same transmission session with the video RTP stream.

[0125] (d) Set the RTCP sending interval to 20ms to avoid network congestion.

[0126] Auxiliary information transmission and serialization:

[0127] Among them, control information is transmitted according to priority:

[0128] (a) Initialize the control information queue and set four levels of priority: Priority 1: Conference core status control (such as ending the meeting and forced exit); Priority 2: Media stream control (such as mute and stop video); Priority 3: Sharing control (such as start / stop screen sharing); Priority 4: General interaction (such as raising hands and expression feedback).

[0129] (b) Receive control instructions from the UI layer and assign priorities based on instruction types.

[0130] (c) Insert the control instruction into the queue of corresponding priority.

[0131] (d) Start the transmission scheduler and process the queues in order of priority: when the high-priority queue is not empty, high-priority control instructions are processed first; in each transmission round, at least one low-priority instruction is processed to avoid starvation; the sending frequency is dynamically adjusted according to the network conditions, and high-priority instructions are given priority in congestion.

[0132] Among them, content information is transmitted in fragments:

[0133] (a) Receive the slice transmission plan from step S43.

[0134] (b) Segments are routed to SEI channels or RTCP channels according to the planned channel allocation.

[0135] (c) For the slices of the SEI channel: call the SEI packaging method of step S43 to generate an SEI data packet; associate the SEI data packet with the next available video frame; and send it together through the video encoding process.

[0136] (d) For the fragmentation of the RTCP channel: call the RTCP packet generation method of step S43 to create an RTCP control packet; send it through the control channel of the RTP session; monitor the sending status, and record the sending time and confirmation status.

[0137] (e) Implement the incremental transmission strategy in step S44: track content changes and transmit only the changed parts; send the full status periodically (e.g., every 30 seconds) to ensure synchronization; and send the full status immediately when a new client is detected.

[0138] Among them, the sequence number and timestamp are allocated:

[0139] (a) Initialize the global sequence number generator with a starting value of 1 and a step size of 1.

[0140] (b) Assign an increasing globally unique sequence number to each control information and content information fragment.

[0141] (c) The serial number encoding format is a 64-bit integer, with the upper 32 bits encoding the information type and the lower 32 bits encoding the sequence number.

[0142] (d) Obtain the PTS (Presentation Time Stamp) of the current video frame from the video encoder as the reference timestamp.

[0143] (e) Add timestamps to control information: record the system time when the control instruction is generated; calculate the time offset from the current video PTS; add the offset to the video PTS as the associated timestamp of the control instruction; write a 32-bit timestamp field in the control message header.

[0144] (f) Add timestamps to content information: Calculate the corresponding video PTS based on the generation time of the content segment; write the PTS value into the timestamp field in the segment header; and add additional fine-tuning offsets for content that requires precise synchronization (such as pointer position).

[0145] (g) Final confirmation of the correctness of the sequence number and timestamp before the information is sent.

[0146] Among them, channel monitoring and exception handling:

[0147] (a) Set up heartbeat monitoring for the first transmission channel: According to the heartbeat mechanism in step S36, regularly check the channel status; record the RTT delay value and update the network quality score; set the channel anomaly judgment standard: three consecutive heartbeat timeouts or a sudden increase in RTT exceeding 200%; when a channel anomaly is detected, trigger the reconnection mechanism in S35.

[0148] (b) Set up performance monitoring for the second transmission channel: send probe data regularly (every 5 seconds) to measure the current performance of the SEI and RTCP channels; calculate the actual transmission success rate and delay data; when the performance of any channel drops by more than 30%, re-trigger the transmission adaptability scoring; and dynamically adjust the channel allocation of the shards based on the updated scoring results.

[0149] (c) Implement an abnormal recovery strategy: For the first channel failure, prioritize using an alternative protocol for recovery; for the second channel failure, prioritize redirecting critical content to an available channel; in extreme cases (failure of both channels), initiate a degraded service mode and only transmit the most core control information.

[0150] Processing steps at the receiving end:

[0151] Auxiliary information reception and sorting:

[0152] (a) Initialize the receiving buffer for control information and content information.

[0153] (b) Receive control information from the first transmission channel: parse the WebSocket or QUIC message and extract the control instruction content; verify the message integrity and signature validity; extract the global sequence number from the message header; and insert the message into the control information ordered buffer according to the sequence number.

[0154] (c) Receive content class information for the second transmission channel: For the SEI channel, extract the SEI packet from the video frame: parse the video ES stream and identify the SEI NAL type (0x06); verify that the SEI type is 0x05 (user data); verify that the company UUID and business UUID match; and extract the SEI payload data.

[0155] For RTCP channels, parse RTCP control packets: identify the custom RTCP packet type (205); verify the subtype and synchronization source identifier; and extract the content data in the RTCP packet.

[0156] Extract the sequence number and segment information from the content segment; insert the segment into the content information buffer according to the sequence number.

[0157] (d) Perform packet loss detection and processing: Check the sequence number continuity of the control information buffer; mark the packet as lost when a discontinuous sequence number is detected (interval > 1); send a retransmission request to the sender, including the range of lost sequence numbers; set the retransmission timeout to 500ms and re-request after the timeout; attempt to retransmit up to three times; if the retransmission fails, perform state synchronization recovery.

[0158] (e) Handling content packet loss: Identify lost content fragments based on version numbers and fragment sequence numbers; request the most recent complete state for lost incremental updates; rebuild the state using the existing incremental updates; if reconstruction is not possible, wait for the next complete state update.

[0159] Among them, the alignment timestamp:

[0160] (a) Establish a unified time base system: use the PTS of the video frame output by the video decoder as the main time base; maintain a sliding window of size 10 to record the timestamp of the most recent video frame; calculate the average time interval between video frames for interpolation calculation.

[0161] (b) Calculate the time offset of the control information: Measure the network transmission delay of the control information from sending to receiving; calculate the network delay difference between the video stream and the control stream; and calculate the time offset value based on the delay difference: time offset value = control information network delay - video stream network delay. Subtract the compensation value from the control information timestamp to achieve time alignment with the video stream.

[0162] (c) Calibrating the timestamps of content information: Extract the associated timestamps of the content segments based on the time synchronization mechanism; for content transmitted via the SEI channel, directly use the PTS of the associated video frame; for content transmitted via the RTCP channel, apply delay compensation similar to that for control information; use a linear interpolation algorithm to handle the transition between timestamps: interpolated timestamp = previous timestamp + (current time - previous update time) / (next update time - previous update time) × (next timestamp - previous timestamp).

[0163] Among them, build the association structure:

[0164] (a) Define time window parameters: the default window size is ±100ms; the minimum window size is ±50ms; the maximum window size is ±300ms; and the synchronization error threshold is 50ms.

[0165] (b) Temporal association processing after alignment: Create a time window based on the video frame timestamp; search for control information and content information within the window; establish a ternary association structure of <video frame, control set, content set>; sort the control information corresponding to the same video frame at multiple time points by serial number; and retain only the latest version of different updated versions of the same content.

[0166] (c) Dynamic time window adjustment: Monitor synchronization errors and calculate actual time deviation statistics. When the synchronization error exceeds the threshold for three consecutive times, the time window is expanded. When the synchronization error is less than 50% of the threshold for 10 consecutive times, the time window is reduced. The window size is adjusted in steps of 25ms to ensure a smooth transition.

[0167] (d) Abnormal situation handling: When a video frame is missing, the time point is estimated by interpolating the previous and next frames; when key control information exceeds the window range, the single window is expanded; when the network jitter is severe, the window is temporarily increased to accommodate delay fluctuations.

[0168] Constructing a synchronized data structure:

[0169] (a) Define a unified synchronization data structure format:

[0170] struct SyncFrame {

[0171] uint64_t frameId; / / frame unique identifier

[0172] uint64_t timestamp; / / video timestamp

[0173] VideoFrameData videoData; / / video frame data

[0174] vector <controlmessage>controls; / / control information list

[0175] ContentState contentState; / / Whiteboard content state

[0176] uint32_t syncQuality; / / synchronization quality indicator};

[0177] (b) Based on the association structure, fill the synchronization data structure: copy the current video frame data into the structure; add the control information within the time window to the control list; construct the complete content state at the current time point based on the content update sequence; calculate the synchronization quality index to reflect the current synchronization accuracy.

[0178] (c) Perform data optimization and conflict resolution: merge consecutive control instructions of the same type to retain the latest status; handle dependencies and conflicts between control instructions; ensure that the operation sequence of the whiteboard content is logically consistent; and remove operation instructions that have been overwritten or canceled.

[0179] (d) Generate the final synchronization data structure and provide it to the conference client UI layer through the callback interface.

[0180] (e) Implement a buffer queue to smooth the synchronous data flow: maintain a synchronous frame buffer queue of size 5; provide a stable data source for UI rendering to reduce lag; and maintain the continuous display of video and auxiliary information in the event of network fluctuations.

[0181] At this point, the full-link video conferencing auxiliary information transmission process from the sending end to the receiving end has been completed, achieving reliable transmission and precise synchronization of control information and content information, and effectively solving the problem of auxiliary information transmission reliability in video conferencing under unstable network conditions.

[0182] The above schematically describes the invention of the present application and its implementation methods. This description is not restrictive. Without departing from the spirit or basic features of the present application, the present application can be implemented in other specific forms. What is shown in the drawings is only one of the implementation methods of the invention of the present application. The actual structure is not limited to this. Any figure mark in the claims should not limit the claims involved. Therefore, if a person of ordinary skill in the art is inspired by it and designs a structural method and embodiment similar to the technical solution without creativity without departing from the purpose of the invention, they should all fall within the scope of protection of the present application. In addition, the word "including" does not exclude other elements or steps, and the word "one" before an element does not exclude the inclusion of "multiple" elements. The multiple elements stated in the product claim can also be implemented by one element through software or hardware. Words such as first and second are used to indicate names and do not indicate any specific order.< / controlmessage>

Claims

1. A video information processing method, characterized in that: include: S1, obtaining video streams of video conference participants and obtaining auxiliary information from the video conference system, wherein the auxiliary information includes conference control instructions and shared whiteboard data; S2, divides the auxiliary information into control information and content information; S3, establishes a first transmission channel for control information independent of the video stream through the WebSocket or QUIC protocol; S4, establishing a second transmission channel associated with the video stream for the content information through the SEI mechanism or RTCP channel of the video encoder; S5, transmitting the control information and the content information respectively according to the first transmission channel and the second transmission channel, and adding a sequence number and a timestamp to the transmitted information; S6, receiving the auxiliary information at the receiving end, and aligning and synchronizing the auxiliary information with the video stream according to the sequence number and timestamp to generate a synchronization data structure, which is used by the conference client to display the video screen and share the whiteboard content.

2. The video information processing method according to claim 1, wherein: Control information includes conference control instructions, and content information includes shared whiteboard data; The first transmission channel is used for real-time transmission of control information; The second transmission channel is used to support the incremental transmission of large amounts of data and maintain synchronous transmission with the video; The synchronization data structure includes video frame data, conference control status and whiteboard content.

3. The video information processing method according to claim 2, wherein: S3, using WebSocket or QUIC protocol to establish a primary transmission channel for control information independent of the video stream, including: S31, obtaining indicators including RTT delay, network jitter, and packet loss rate; and calculating a network quality score based on the indicators; S32, when the network quality score is greater than the threshold, the WebSocket protocol is selected to ensure compatibility, otherwise the QUIC protocol is selected to improve transmission efficiency; S33, when the WebSocket protocol is selected, establish a TCP-based WebSocket connection and set a keep-alive frame interval parameter to ensure continuous availability of the connection; S34, when the QUIC protocol is selected, establish a QUIC connection based on UDP and configure 0-RTT parameters to speed up connection recovery; S35, set up a reconnection mechanism based on the exponential backoff algorithm; S36: According to steps S33 to S35, a first transmission channel is established, and the stability of the first transmission channel is verified through a heartbeat mechanism: Send heartbeat probe packets at fixed intervals and measure the response time; Update the indicator in step S31 according to the response time; When the response time is greater than the threshold, the reconnection mechanism in step S35 is triggered.

4. The video information processing method according to claim 3, wherein: S31, calculates the network quality score, including: Get the RTT delay time by sending a probe packet; Calculate the standard deviation of multiple consecutive RTT delay samples to obtain the network jitter value; Count the number of sent and received probe packets and calculate the packet loss rate; The network quality score is obtained through weighted calculation based on RTT delay time, network jitter value and packet loss rate.

5. The video information processing method according to claim 3, wherein: S35, based on the exponential backoff algorithm, sets the reconnection mechanism, including: When it is detected that the connection established in step S33 or step S34 is interrupted, a reconnection process is executed; Set the initial reconnection interval to T1, and double the reconnection interval after each reconnection failure until the maximum reconnection interval T0 is reached; After reaching the preset maximum number of reconnections, a connection failure notification is sent; Each time the connection is reconnected, the protocol of the current network environment is reselected according to the results of steps S31 and S32; During the reconnection period, newly generated control information in the video conference is prioritized and cached; When the reconnection is successful, synchronize the conference status and obtain the current conference status on the server; Compare the locally cached control information with the server-side conference status, eliminate redundant or conflicting control information, and obtain valid cached control information; The effective cache control information is sent through the first transmission channel in order of priority to ensure the continuity of conference control.

6. The video information processing method according to claim 3, wherein: S4: Establishing a second transmission channel associated with the video stream for the content information through the SEI mechanism or RTCP channel of the video encoder, including: S41, acquiring characteristic parameters including data volume, real-time requirements, and relevance to the video based on the content information; S42, based on the characteristic parameters and the current network status, establish the SEI mechanism and the RTCP channel as candidate paths, and calculate the transmission adaptability scores of the two candidate paths respectively; S43, fragmenting the content information and assigning an optimal transmission channel to each content fragment based on the transmission adaptability score; S44, setting an incremental transmission strategy for content information and a time synchronization mechanism with the video stream based on the optimal transmission channel; S45 , according to steps S41 to S44 , establish a second transmission channel, and associate the second transmission channel with the video stream encoder to perform synchronous transmission of the content information and the video stream.

7. The video information processing method according to claim 6, wherein: S42 calculates the transmission adaptability scores of the two candidate paths respectively, including: Obtain network performance indicators of the SEI mechanism and RTCP channel, wherein the network performance indicators include current delay, packet loss rate, and available bandwidth; Set the weight of content information feature parameters; According to the weights and network performance indicators, the transmission adaptability scores of the SEI mechanism and RTCP channel are calculated respectively.

8. The video information processing method according to any one of claims 2 to 7, characterized in that: S5, transmitting control information and content information according to the first transmission channel and the second transmission channel respectively, including: S51, sorting the control information by priority and transmitting it according to the priority through the first transmission channel; S52, according to the allocated optimal transmission channel, the content information is transmitted in segments, and the transmission is performed according to the incremental transmission strategy of step S44; S53, assigning globally unique serial numbers and timestamps synchronized with the video time reference to the control information and content information fragments; S54, monitor the status of the first transmission channel through the heartbeat mechanism of step S36, and when a channel abnormality is detected, trigger the reconnection mechanism of step S35; at the same time, monitor the status of the second transmission channel, and when an abnormality is detected, re-trigger the transmission adaptability scoring of step S42.

9. The video information processing method according to claim 8, wherein: S6, generating a synchronization data structure, including: S61: Receive the transmitted control information and content information, sort them according to the sequence number, and detect packet loss. For control information, when a discontinuous sequence number is detected, the sender is requested to retransmit the lost control instruction; For content information, the lost data is restored according to the incremental transmission strategy of step S44; S62, establish a unified time base, align the video stream timestamp with the auxiliary information timestamp added in step S53: Calculate the time offset between the video stream and control information to compensate for network transmission delay differences; According to the time synchronization mechanism of step S44, the timestamp of the content information is calibrated; S63: Construct an association structure between the video frame and the auxiliary information based on the aligned timestamps: Associating video frames, control information, and content information within a time window; When the synchronization error is detected to exceed the threshold, the time window size is dynamically adjusted; S64: Generate a synchronization data structure including video frame data, conference control status, and whiteboard content according to the association structure for use by the conference client.

Images