A method for low-latency transmission of audio and video interaction supporting multi-terminal synchronization

Abstract

This invention discloses a low-latency audio and video interactive transmission method supporting multi-terminal synchronization, belonging to the field of audio and video interactive transmission technology. It includes: enabling data communication between the client and server to achieve audio and video interactive data transmission according to the WebSocket communication protocol; adaptively selecting the optimal streaming media transmission channel based on the ICE algorithm; transmitting audio and video interactive data from the client to the server according to the optimal streaming media transmission channel; monitoring the transmission status of the audio and video interactive data in real time; and dynamically adjusting and optimizing the audio and video interactive data transmission based on the monitoring results. This invention solves the problems of existing methods that cannot effectively achieve low-latency transmission and cannot achieve high-quality real-time audio and video interaction across multiple terminals. This invention can ensure synchronous low-latency transmission of audio and video interactive data, providing users with a smooth and natural interactive experience, and enabling high-quality real-time audio and video interaction across multiple terminals.

Description

Technical Field

[0001] This invention relates to the field of audio and video interactive transmission technology, specifically to a low-latency audio and video interactive transmission method that supports multi-terminal synchronization. Background Technology

[0002] Audio and video interactive transmission refers to the bidirectional or multidirectional transmission of audio and video data between two or more users through real-time audio and video technology; it is widely used in online education, telemedicine, interactive entertainment, video conferencing and other fields, and can significantly improve user experience and interaction efficiency.

[0003] Existing audio and video interactive data cannot effectively achieve low-latency transmission during transmission, and cannot provide users with a smooth and natural interactive experience, resulting in poor audio and video interactive data transmission quality and the inability to achieve high-quality real-time audio and video interaction across multiple terminals. Summary of the Invention

[0004] The purpose of this invention is to provide a low-latency transmission method for audio and video interaction that supports multi-terminal synchronization. This method ensures synchronous and low-latency transmission of audio and video interaction data, provides users with a smooth and natural interactive experience, improves the data transmission effect of audio and video interaction, and enables high-quality real-time audio and video interaction across multiple terminals, thus solving the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A low-latency transmission method for audio and video interaction supporting multi-terminal synchronization includes:

[0007] The WebSocket communication protocol enables data communication between the client and the server to achieve interactive audio and video data transmission. The optimal streaming media transmission channel is adaptively selected based on the ICE algorithm.

[0008] Based on the optimal streaming media transmission channel, the client transmits audio and video interactive data to the server and monitors the transmission status of the audio and video interactive data in real time. Based on the monitoring results, the client dynamically adjusts and optimizes the transmission of audio and video interactive data to ensure synchronous and low-latency transmission of audio and video interactive data.

[0009] Preferably, the client and server communicate using the WebSocket communication protocol to achieve audio and video interactive data transmission, and the following operations are performed:

[0010] The client sends a command to the server requesting the establishment of an audio and video interactive data transmission connection. After receiving the command from the client, the server performs security verification on the client's audio and video interactive data transmission port.

[0011] When the client's audio and video interactive data transmission port security verification is successful, the server transmits an instruction to the client agreeing to establish an audio and video interactive data transmission connection. After receiving the instruction from the server agreeing to establish an audio and video interactive data transmission connection, the client establishes an audio and video interactive data transmission connection between the client and the server based on the audio and video interactive data transmission port for audio and video interactive data transmission.

[0012] Preferably, the server performs security verification on the client's audio and video interactive data transmission port and performs the following operations:

[0013] Extract the audio and video interactive data transmission port on the server side;

[0014] Compare the audio and video interactive data transmission ports of the client and the server to analyze the inclusion relationship between the audio and video interactive data transmission ports of the client and the server.

[0015] Assess whether the client is qualified to transmit audio and video data interactively with the server, and verify the security of the client's audio and video data transmission port.

[0016] When the server's audio and video interactive data transmission port includes the client's audio and video interactive data transmission port, the client is qualified to conduct audio and video interactive data transmission with the server, and the client's audio and video interactive data transmission port security verification is successful.

[0017] When the server's audio and video interactive data transmission port does not include the client's audio and video interactive data transmission port, the client is not qualified to conduct audio and video interactive data transmission with the server, and the client's audio and video interactive data transmission port security verification fails.

[0018] Preferably, based on the ICE algorithm, the optimal streaming media transmission channel is adaptively selected, and the following operations are performed:

[0019] Collect local IP addresses, STUN-generated public network mapping addresses, and TURN relay addresses to form a candidate address list;

[0020] The client exchanges SDP messages with the signaling server, which contain a list of candidate addresses. Based on the SDP offer / answer mode, the client and server complete the exchange of candidate addresses. At this time, both the client and the server have each other's list of candidate addresses.

[0021] Based on local and remote candidate addresses, generate all possible candidate address pairs and sort them by priority according to address type and configuration parameters;

[0022] For each candidate address pair, a STUN binding request is executed. The connection is confirmed through a four-way handshake. A STUN request is sent from the local address to the remote address and a response is awaited. After receiving the request, the peer actively sends a STUN request to the local address to speed up the connectivity check.

[0023] When a candidate address pair successfully completes the check in both directions, the address pair is considered valid, and the optimal streaming media transmission channel is selected from all valid candidate address pairs according to priority.

[0024] When a candidate address pair fails the connectivity check, a TURN server is used for relay transmission.

[0025] Preferably, the candidate address pairs are prioritized according to address type and configuration parameters, with the priority from high to low as follows: local IP address > public network mapping address generated by STUN > TURN relay address.

[0026] Preferably, based on the optimal streaming media transmission channel, the client transmits audio and video interactive data to the server and performs the following operations:

[0027] Based on the optimal streaming media transmission channel, the client transmits RTP data packets for the transmission of audio and video interactive data. Combined with packet loss feedback mechanism, forward error correction and jitter buffer, the real-time performance and reliability of audio and video interactive data during transmission are ensured.

[0028] The RTP protocol is used to transmit data packets. Each data packet carries a sequence number and a timestamp. The server buffers the data packets using a jitter buffer and outputs them in sequence.

[0029] When the server detects packet loss, it requests the client to retransmit the lost data packets through the packet loss feedback mechanism. At the same time, it uses forward error correction to decode redundant data packets to recover the lost data packets, and dynamically adjusts the buffer size based on the jitter buffer to adapt to network jitter and balance latency and packet loss rate.

[0030] Preferably, the buffer size is dynamically adjusted based on the jitter buffer, and the following operations are performed:

[0031] Real-time monitoring of network jitter and packet loss rate;

[0032] The network jitter and network packet loss rate are normalized to obtain the normalized network jitter and network packet loss rates.

[0033] The normalized network jitter value is compared with the preset jitter threshold.

[0034] When the normalized network jitter value exceeds the preset jitter threshold, the real-time monitored FEC decoding success rate is retrieved.

[0035] The FEC decoding success rate is normalized to obtain the normalized FEC decoding success rate.

[0036] The buffer setting adjustment coefficient is obtained using the normalized network jitter value, network packet loss rate, and FEC decoding success rate, and the buffer size is dynamically adjusted using the buffer setting adjustment coefficient.

[0037] Preferably, the buffer setting adjustment coefficient is obtained using the normalized network jitter value, network packet loss rate, and FEC decoding success rate, and the buffer size is dynamically adjusted using the buffer setting adjustment coefficient, performing the following operations:

[0038] Retrieve the normalized network jitter value, network packet loss rate, and FEC decoding success rate;

[0039] The network packet loss rate and FEC decoding success rate after normalization are used;

[0040] A reference value for the packet loss rate is obtained using the normalized network packet loss rate and the FEC decoding success rate.

[0041] Retrieve the preset target packet loss rate;

[0042] The target packet loss rate is compared with the reference packet loss rate to obtain the packet loss rate ratio; wherein, the packet loss rate ratio = reference packet loss rate / target packet loss rate;

[0043] The buffer setting adjustment coefficient is obtained by combining the packet loss rate reference value and the packet loss rate ratio with the normalized network jitter value;

[0044] The buffer size is dynamically adjusted using the buffer setting adjustment coefficient.

[0045] Preferably, when the server detects packet loss, it requests the client to retransmit the lost data packets through a packet loss feedback mechanism and performs the following operations:

[0046] The server sends a packet loss feedback message to the client via Real-Time Transmission Control Protocol (RTP), informing the client of the sequence number of the data packet that needs to be retransmitted. The client retrieves the corresponding data packet from its cache based on the sequence number in the packet loss feedback message and retransmits it to the server. Upon receiving the retransmitted data packet, the server inserts it into the correct order to restore data integrity.

[0047] Preferably, the transmission of audio and video interactive data is monitored in real time, and the data transmission is dynamically adjusted and optimized based on the monitoring results, by performing the following operations:

[0048] After receiving the data packet, the server stores it in a buffer according to the timestamp or sequence number, and the data packets in the buffer are arranged in the order of timestamp to ensure the continuity of playback;

[0049] The jitter of audio and video interactive data transmission is monitored, and the standard deviation of the time interval between consecutive data packets is calculated as the jitter value. The larger the jitter value, the more severe the network fluctuation.

[0050] The jitter value is compared and analyzed with a preset threshold. When the jitter value exceeds the threshold, the buffer is gradually expanded to store more data packets to smooth the jitter. When the jitter value is below the threshold, the buffer is gradually reduced to reduce end-to-end latency.

[0051] The system monitors packet loss during the transmission of audio and video interactive data, detects the number of lost data packets by sequence number, and dynamically adjusts the buffer size based on the packet loss rate. When the packet loss rate is high, the buffer capacity is increased to provide more time for packet loss feedback mechanism requests and forward error correction decoding.

[0052] Monitor the transmission latency of audio and video interactive data, record the time difference between sending and receiving data packets, and minimize the buffer size to reduce latency while ensuring audio and video quality.

[0053] Compared with the prior art, the beneficial effects of the present invention are:

[0054] This invention enables data communication between the client and server using the WebSocket communication protocol to achieve interactive audio and video data transmission. Based on the ICE algorithm, it adaptively selects the optimal streaming media transmission channel. According to the optimal streaming media transmission channel, the client transmits interactive audio and video data to the server. The transmission status of interactive audio and video data is monitored in real time, and the data transmission is dynamically adjusted and optimized based on the monitoring results to ensure synchronous and low-latency transmission of interactive audio and video data. This provides users with a smooth and natural interactive experience, improves the effect of interactive audio and video data transmission, and enables high-quality real-time audio and video interaction across multiple terminals. Attached Figure Description

[0055] Figure 1 This is a flowchart of the low-latency audio and video interactive transmission method supporting multi-terminal synchronization according to the present invention. Detailed Implementation

[0056] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0057] To address the issues of poor data transmission quality and the inability to achieve high-quality real-time audio and video interaction across multiple devices due to ineffective low-latency transmission of existing audio and video interactive data, which prevents a smooth and natural interactive experience for users, please refer to [link to relevant documentation]. Figure 1 This embodiment provides the following technical solution:

[0058] A low-latency transmission method for audio and video interaction supporting multi-terminal synchronization includes:

[0059] The WebSocket communication protocol enables data communication between the client and the server to achieve interactive audio and video data transmission.

[0060] In this embodiment, the client and server communicate using the WebSocket communication protocol to achieve audio and video interactive data transmission, and the following operations are performed:

[0061] The client sends a command to the server requesting the establishment of an audio and video interactive data transmission connection. After receiving the command from the client, the server performs security verification on the client's audio and video interactive data transmission port.

[0062] When the client's audio and video interactive data transmission port security verification is successful, the server transmits an instruction to the client agreeing to establish an audio and video interactive data transmission connection. After receiving the instruction from the server agreeing to establish an audio and video interactive data transmission connection, the client establishes an audio and video interactive data transmission connection between the client and the server based on the audio and video interactive data transmission port for audio and video interactive data transmission.

[0063] In this embodiment, the server performs security verification on the client's audio and video interactive data transmission port by performing the following operations:

[0064] Extract the audio and video interactive data transmission port on the server side;

[0065] Compare the audio and video interactive data transmission ports of the client and the server to analyze the inclusion relationship between the audio and video interactive data transmission ports of the client and the server.

[0066] Assess whether the client is qualified to transmit audio and video data interactively with the server, and verify the security of the client's audio and video data transmission port.

[0067] When the server's audio and video interactive data transmission port includes the client's audio and video interactive data transmission port, the client is qualified to conduct audio and video interactive data transmission with the server, and the client's audio and video interactive data transmission port security verification is successful.

[0068] When the server's audio and video interactive data transmission port does not include the client's audio and video interactive data transmission port, the client is not qualified to conduct audio and video interactive data transmission with the server, and the client's audio and video interactive data transmission port security verification fails.

[0069] Among them, the ICE algorithm adaptively selects the optimal streaming media transmission channel. The ICE algorithm prioritizes the optimal path and transmits through a relay server when a direct connection is not possible, effectively reducing latency.

[0070] It should be noted that the ICE (Interactive Connectivity Establishment) algorithm is a key technology for NAT traversal and firewall crossing, and is widely used in real-time audio and video communication. By combining signaling servers, STUN servers, and TURN servers, the ICE algorithm can adaptively select the optimal streaming media transmission channel, thereby ensuring low latency and high reliability of audio and video interaction between multiple terminals.

[0071] In this embodiment, the optimal streaming media transmission channel is adaptively selected based on the ICE algorithm, and the following operations are performed:

[0072] Collect local IP addresses, STUN-generated public network mapping addresses, and TURN relay addresses to form a candidate address list;

[0073] The client exchanges SDP messages with the signaling server, which contain a list of candidate addresses. Based on the SDP offer / answer mode, the client and server complete the exchange of candidate addresses. At this time, both the client and the server have each other's list of candidate addresses.

[0074] Based on local and remote candidate addresses, generate all possible candidate address pairs and sort them by priority according to address type and configuration parameters;

[0075] For each candidate address pair, a STUN binding request is executed. The connection is confirmed through a four-way handshake. A STUN request is sent from the local address to the remote address and a response is awaited. After receiving the request, the peer actively sends a STUN request to the local address to speed up the connectivity check.

[0076] When a candidate address pair successfully completes the check in both directions, the address pair is considered valid, and the optimal streaming media transmission channel is selected from all valid candidate address pairs according to priority.

[0077] When a candidate address pair fails the connectivity check, a TURN server is used for relay transmission.

[0078] In this embodiment, candidate address pairs are prioritized according to address type and configuration parameters, with the priority from high to low as follows: local IP address > public network mapping address generated by STUN > TURN relay address.

[0079] It should be noted that the signaling server is responsible for the signaling exchange between the client and the server, including the transmission of SDP messages and ICE candidate addresses. It establishes a real-time connection through the WebSocket communication protocol to ensure reliable signaling transmission. The STUN server provides binding discovery functionality to help clients obtain their public IP address after NAT translation. The TURN server acts as a relay server, responsible for forwarding data between the client and the server when a direct connection cannot be established. It provides relay candidate addresses to ensure the feasibility of communication.

[0080] Based on the optimal streaming media transmission channel, the client transmits audio and video interactive data to the server and monitors the transmission status of the audio and video interactive data in real time. Based on the monitoring results, the client dynamically adjusts and optimizes the transmission of audio and video interactive data to ensure synchronous and low-latency transmission of audio and video interactive data.

[0081] In this embodiment, based on the optimal streaming media transmission channel, the client transmits audio and video interactive data to the server and performs the following operations:

[0082] Based on the optimal streaming media transmission channel, the client transmits RTP data packets for the transmission of audio and video interactive data. Combined with packet loss feedback mechanism, forward error correction and jitter buffer, the real-time performance and reliability of audio and video interactive data during transmission are ensured.

[0083] The RTP protocol is used to transmit data packets. Each data packet carries a sequence number and a timestamp. The server buffers the data packets using a jitter buffer and outputs them in sequence.

[0084] When the server detects packet loss, it requests the client to retransmit the lost data packets through the packet loss feedback mechanism. At the same time, it uses forward error correction to decode redundant data packets to recover the lost data packets, and dynamically adjusts the buffer size based on the jitter buffer to adapt to network jitter and balance latency and packet loss rate.

[0085] Specifically, based on the jitter buffer, dynamically adjust the buffer size and perform the following operations:

[0086] Real-time monitoring of network jitter and packet loss rate;

[0087] The network jitter and network packet loss rate are normalized to obtain the normalized network jitter and network packet loss rates.

[0088] The normalized network jitter value is compared with the preset jitter threshold.

[0089] When the normalized network jitter value exceeds the preset jitter threshold, the real-time monitored FEC decoding success rate is retrieved.

[0090] The FEC decoding success rate is normalized to obtain the normalized FEC decoding success rate.

[0091] The buffer setting adjustment coefficient is obtained using the normalized network jitter value, network packet loss rate, and FEC decoding success rate, and the buffer size is dynamically adjusted using the buffer setting adjustment coefficient.

[0092] In this embodiment, the buffer settings are optimized in real time based on a comprehensive assessment of network jitter, packet loss rate, and FEC decoding success rate. This ensures that the buffer size is highly adapted to the current network conditions, effectively mitigating data flow instability caused by network jitter and reducing packet loss or latency accumulation due to network fluctuations. Simultaneously, adjustments are made in conjunction with the FEC decoding success rate. This ensures data transmission reliability while avoiding unnecessary latency caused by an excessively large buffer, and also prevents a buffer that is too small to withstand network jitter. Ultimately, this improves the continuity and timeliness of data reception, optimizing overall data transmission performance and user experience.

[0093] Specifically, the buffer setting adjustment coefficient is obtained using the normalized network jitter value, network packet loss rate, and FEC decoding success rate. The buffer size is then dynamically adjusted using this adjustment coefficient, and the following operations are performed:

[0094] Retrieve the normalized network jitter value, network packet loss rate, and FEC decoding success rate;

[0095] The network packet loss rate and FEC decoding success rate after normalization are used;

[0096] A reference value for the packet loss rate is obtained using the normalized network packet loss rate and the FEC decoding success rate.

[0097] The packet loss rate reference value is obtained by the following formula:

[0098] ;

[0099] Where R represents the reference value for packet loss rate; P L P represents the network packet loss rate after normalization. f This represents the normalized FEC decoding success rate; specifically, at the level of packet loss risk assessment, this formula represents the normalized network packet loss rate (P...). L ) and FEC decoding success rate (P f The FEC decoding success rate directly reflects the error correction mechanism's ability to compensate for packet loss when P is calculated using a reference value (R). f A higher value for R means a lower effective packet loss pressure that the buffer actually needs to handle; therefore, the value of R will increase with P. L Increases and rises (the more severe the packet loss, the higher the risk), with P f Increase the error correction capability and decrease the risk (the stronger the error correction capability, the lower the risk), thereby objectively quantifying the actual effective packet loss risk currently faced by the network.

[0100] Retrieve the preset target packet loss rate;

[0101] The target packet loss rate is compared with the reference packet loss rate to obtain the packet loss rate ratio; wherein, the packet loss rate ratio = reference packet loss rate / target packet loss rate. Simultaneously, at the packet loss deviation quantification level, the actual packet loss risk is compared with the expected acceptable risk by calculating the ratio of the reference packet loss rate (R) to the preset target packet loss rate. This ratio directly reflects the degree of deviation between the current packet loss state and the target state: when the ratio is greater than 1, it indicates that the actual packet loss risk exceeds expectations, and the buffer's anti-packet loss capability needs to be enhanced; when the ratio is less than 1, it indicates that the packet loss risk is within a controllable range, and the buffer can be reduced to lower latency.

[0102] The buffer setting adjustment coefficient is obtained by combining the packet loss rate reference value and the packet loss rate ratio with the normalized network jitter value;

[0103] The buffer setting adjustment coefficient is obtained by the following formula:

[0104] ;

[0105] Where H represents the buffer setting adjustment coefficient; y and x represent the weight values ​​corresponding to the network jitter value and the packet loss rate ratio, respectively; J represents the normalized network jitter value; Jmax This represents the normalized network jitter value corresponding to the maximum network jitter value in historical records; P s This represents the network latency rate at the current moment. Specifically, at the adjustment coefficient calculation level, the formula for setting the adjustment coefficient (H) in the buffer zone achieves comprehensive decision-making through multi-parameter weighted fusion. Among them, the normalized network jitter value (y weight) and the packet loss rate ratio (x weight) respectively reflect the impact weights of jitter and packet loss risks, which can be dynamically allocated according to the sensitivity of the service to jitter or packet loss; the historical maximum jitter (J...) max The baseline value is used to avoid over-adjustment caused by a single abnormal fluctuation, ensuring decision stability; the current network latency rate (Ps) is used to constrain the magnitude of the buffer increase, that is, when the latency is already high, Ps... s It will limit the growth of H to prevent the buffer from becoming too large and exacerbating latency, thus achieving a balance between jitter resistance and low latency.

[0106] The buffer size is dynamically adjusted using the buffer setting adjustment coefficient.

[0107] The dynamically adjusted buffer size is obtained using the following formula:

[0108] ;

[0109] Where S represents the dynamically adjusted buffer size; S0 represents the buffer size before dynamic adjustment.

[0110] In this embodiment, firstly, a reference value for the packet loss rate is obtained based on the network packet loss rate and the FEC decoding success rate. This integrates the interaction between packet loss itself and the FEC error correction capability, more objectively reflecting the effective packet loss pressure faced by data transmission. Next, the ratio of the reference value to the target packet loss rate is used to clarify the deviation between the current network packet loss situation and the expected target. Then, combined with normalized network jitter values, historical jitter extremes, and network latency, an adjustment coefficient is set using a weighted buffer to achieve fine-grained and dynamic adjustment of the jitter buffer size. This technical solution can specifically adjust the buffer size to buffer data and resist network fluctuations when network jitter is severe, packet loss is high, and FEC compensation is insufficient. Furthermore, it can avoid latency degradation caused by excessive buffer enlargement when network conditions improve. Thus, while ensuring the continuity and reliability of data transmission, it effectively controls transmission latency, achieving an optimal balance between resistance to network fluctuations and real-time transmission, significantly improving overall data transmission performance and user experience stability.

[0111] In this embodiment, when the server detects packet loss, it requests the client to retransmit the lost data packets through a packet loss feedback mechanism and performs the following operations:

[0112] The server sends a packet loss feedback message to the client via Real-Time Transmission Control Protocol (RTP), informing the client of the sequence number of the data packet that needs to be retransmitted. The client retrieves the corresponding data packet from its cache based on the sequence number in the packet loss feedback message and retransmits it to the server. Upon receiving the retransmitted data packet, the server inserts it into the correct order to restore data integrity.

[0113] In this embodiment, the transmission of audio and video interactive data is monitored in real time, and the data transmission is dynamically adjusted and optimized based on the monitoring results, by performing the following operations:

[0114] After receiving the data packet, the server stores it in a buffer according to the timestamp or sequence number, and the data packets in the buffer are arranged in the order of timestamp to ensure the continuity of playback;

[0115] The jitter of audio and video interactive data transmission is monitored, and the standard deviation of the time interval between consecutive data packets is calculated as the jitter value. The larger the jitter value, the more severe the network fluctuation.

[0116] The jitter value is compared and analyzed with a preset threshold. When the jitter value exceeds the threshold, the buffer is gradually expanded to store more data packets to smooth the jitter. When the jitter value is below the threshold, the buffer is gradually reduced to reduce end-to-end latency.

[0117] The system monitors packet loss during the transmission of audio and video interactive data, detects the number of lost data packets by sequence number, and dynamically adjusts the buffer size based on the packet loss rate. When the packet loss rate is high, the buffer capacity is increased to provide more time for packet loss feedback mechanism requests and forward error correction decoding.

[0118] Monitor the transmission latency of audio and video interactive data, record the time difference between sending and receiving data packets, and minimize the buffer size to reduce latency while ensuring audio and video quality.

[0119] In summary, this method enables data communication between the client and server via the WebSocket communication protocol to achieve interactive audio and video data transmission. Based on the ICE algorithm, it adaptively selects the optimal streaming media transmission channel. Using this optimal channel, the client transmits the interactive audio and video data to the server. The transmission status of the interactive audio and video data is monitored in real time, and the data transmission is dynamically adjusted and optimized based on the monitoring results. This ensures synchronous, low-latency transmission of the interactive audio and video data, providing users with a smooth and natural interactive experience. It improves the quality of interactive audio and video data transmission and enables high-quality real-time audio and video interaction across multiple terminals. Its applications include the following scenarios:

[0120] 1) Online education: Low-latency transmission ensures real-time interaction between teachers and students, which can improve teaching effectiveness, especially in scenarios that require audio and video synchronization, such as language learning and music teaching.

[0121] 2) Video conferencing: In enterprise meetings and remote collaboration, low-latency transmission and multi-terminal synchronization can significantly improve meeting efficiency and reduce time wastage caused by delays or asynchrony.

[0122] 3) Telemedicine: In remote diagnosis and surgical guidance, low latency and high synchronization are crucial to ensuring the accuracy and real-time nature of medical procedures.

[0123] 4) Virtual Reality / Augmented Reality: In VR / AR scenarios, low latency and high resolution transmission can provide a more immersive experience and meet the needs of real-time interaction.

[0124] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0125] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for low-latency audio and video interactive transmission supporting multi-terminal synchronization, characterized in that, include: The WebSocket communication protocol enables data communication between the client and the server to achieve interactive audio and video data transmission. The optimal streaming media transmission channel is adaptively selected based on the ICE algorithm. Based on the optimal streaming media transmission channel, the client transmits audio and video interactive data to the server and monitors the transmission status of the audio and video interactive data in real time. Based on the monitoring results, the client dynamically adjusts and optimizes the transmission of audio and video interactive data.

2. The method for low-latency audio and video interactive transmission supporting multi-terminal synchronization according to claim 1, characterized in that, To enable data communication between the client and server and achieve interactive audio and video data transmission using the WebSocket communication protocol, the following operations are performed: The client sends a command to the server requesting the establishment of an audio and video interactive data transmission connection. After receiving the command from the client, the server performs security verification on the client's audio and video interactive data transmission port. When the client's audio and video interactive data transmission port security verification is successful, the server transmits an instruction to the client agreeing to establish an audio and video interactive data transmission connection. After receiving the instruction from the server agreeing to establish an audio and video interactive data transmission connection, the client establishes an audio and video interactive data transmission connection between the client and the server based on the audio and video interactive data transmission port for audio and video interactive data transmission.

3. The method for low-latency audio and video interactive transmission supporting multi-terminal synchronization according to claim 2, characterized in that, The server performs security verification on the client's audio and video interactive data transmission port and performs the following operations: Extract the audio and video interactive data transmission port on the server side; Compare the audio and video interactive data transmission ports of the client and the server to analyze the inclusion relationship between the audio and video interactive data transmission ports of the client and the server. Assess whether the client is qualified to transmit audio and video data interactively with the server, and verify the security of the client's audio and video data transmission port. When the server's audio and video interactive data transmission port includes the client's audio and video interactive data transmission port, the client is qualified to conduct audio and video interactive data transmission with the server, and the client's audio and video interactive data transmission port security verification is successful. When the server's audio and video interactive data transmission port does not include the client's audio and video interactive data transmission port, the client is not qualified to conduct audio and video interactive data transmission with the server, and the client's audio and video interactive data transmission port security verification fails.

4. The method for low-latency audio and video interactive transmission supporting multi-terminal synchronization according to claim 3, characterized in that, Based on the ICE algorithm, the optimal streaming media transmission channel is adaptively selected, and the following operations are performed: Collect local IP addresses, STUN-generated public network mapping addresses, and TURN relay addresses to form a candidate address list; The client exchanges SDP messages with the signaling server, which contain a list of candidate addresses. Based on the SDP offer / answer mode, the client and server complete the exchange of candidate addresses. At this time, both the client and the server have each other's list of candidate addresses. Based on local and remote candidate addresses, generate all possible candidate address pairs and sort them by priority according to address type and configuration parameters; For each candidate address pair, a STUN binding request is executed. The connection is confirmed through a four-way handshake. A STUN request is sent from the local address to the remote address and a response is awaited. After receiving the request, the peer actively sends a STUN request to the local address to speed up the connectivity check. When a candidate address pair successfully completes the check in both directions, the address pair is considered valid, and the optimal streaming media transmission channel is selected from all valid candidate address pairs according to priority. When a candidate address pair fails the connectivity check, a TURN server is used for relay transmission.

5. The method for low-latency audio and video interactive transmission supporting multi-terminal synchronization according to claim 4, characterized in that, Candidate address pairs are prioritized according to address type and configuration parameters, with priority from high to low as follows: local IP address > STUN-generated public network mapping address > TURN relay address.

6. The method for low-latency audio and video interactive transmission supporting multi-terminal synchronization according to claim 5, characterized in that, Based on the optimal streaming media transmission channel, the client transmits audio and video interactive data to the server and performs the following operations: Based on the optimal streaming media transmission channel, the client transmits RTP data packets for the transmission of audio and video interactive data, and combines a packet loss feedback mechanism, forward error correction and jitter buffer. The RTP protocol is used to transmit data packets. Each data packet carries a sequence number and a timestamp. The server buffers the data packets using a jitter buffer and outputs them in sequence. When the server detects packet loss, it requests the client to retransmit the lost data packets through the packet loss feedback mechanism. At the same time, it uses forward error correction to decode redundant data packets to recover the lost data packets, and dynamically adjusts the buffer size based on the jitter buffer to adapt to network jitter and balance latency and packet loss rate.

7. The method for low-latency audio and video interactive transmission supporting multi-terminal synchronization according to claim 6, characterized in that, Based on the jitter buffer, dynamically adjust the buffer size and perform the following operations: Real-time monitoring of network jitter and packet loss rate; The network jitter and network packet loss rate are normalized to obtain the normalized network jitter and network packet loss rates. The normalized network jitter value is compared with the preset jitter threshold. When the normalized network jitter value exceeds the preset jitter threshold, the real-time monitored FEC decoding success rate is retrieved. The FEC decoding success rate is normalized to obtain the normalized FEC decoding success rate. The buffer setting adjustment coefficient is obtained using the normalized network jitter value, network packet loss rate, and FEC decoding success rate, and the buffer size is dynamically adjusted using the buffer setting adjustment coefficient.

8. A method for low-latency audio and video interactive transmission supporting multi-terminal synchronization according to claim 7, characterized in that, Using the normalized network jitter value, network packet loss rate, and FEC decoding success rate, a buffer setting adjustment coefficient is obtained. The buffer size is then dynamically adjusted using this coefficient, and the following operations are performed: Retrieve the normalized network jitter value, network packet loss rate, and FEC decoding success rate; The network packet loss rate and FEC decoding success rate after normalization are used; A reference value for the packet loss rate is obtained using the normalized network packet loss rate and the FEC decoding success rate. Retrieve the preset target packet loss rate; The target packet loss rate is compared with the reference packet loss rate to obtain the packet loss rate ratio; wherein, the packet loss rate ratio = reference packet loss rate / target packet loss rate; The buffer setting adjustment coefficient is obtained by combining the packet loss rate reference value and the packet loss rate ratio with the normalized network jitter value; The buffer size is dynamically adjusted using the buffer setting adjustment coefficient.

9. A method for low-latency audio and video interactive transmission supporting multi-terminal synchronization according to claim 6, characterized in that, When the server detects packet loss, it requests the client to retransmit the lost data packets through the packet loss feedback mechanism and performs the following operations: The server sends a packet loss feedback message to the client via Real-Time Transmission Control Protocol (RTP), informing the client of the sequence number of the data packet that needs to be retransmitted. The client retrieves the corresponding data packet from its cache based on the sequence number in the packet loss feedback message and retransmits it to the server. Upon receiving the retransmitted data packet, the server inserts it into the correct order to restore data integrity.

10. A method for low-latency audio and video interactive transmission supporting multi-terminal synchronization according to claim 9, characterized in that, The transmission of audio and video interactive data is monitored in real time, and the data transmission is dynamically adjusted and optimized based on the monitoring results. The following operations are performed: After receiving the data packet, the server stores it in a buffer according to the timestamp or sequence number, and the data packets are arranged in the buffer in the order of timestamps; The jitter of audio and video interactive data transmission is monitored, and the standard deviation of the time interval between consecutive data packets is calculated as the jitter value. The larger the jitter value, the more severe the network fluctuation. The jitter value is compared and analyzed with a preset threshold. When the jitter value exceeds the threshold, the buffer is expanded to store more data packets to smooth the jitter. When the jitter value is below the threshold, the buffer is reduced to reduce end-to-end latency. The system monitors packet loss during the transmission of audio and video interactive data, detects the number of lost data packets by sequence number, and dynamically adjusts the buffer size based on the packet loss rate. When the packet loss rate is high, the buffer capacity is increased to provide more time for packet loss feedback mechanism requests and forward error correction decoding. Monitor the transmission delay of audio and video interactive data and record the time difference between sending and receiving data packets.

Images