A method for transmitting audio and video over weak networks with full link state awareness

Abstract

The present invention proposes a weak network audio and video transmission method with full link status perception, which belongs to the field of audio and video communication technology. It includes: the client and the server conduct media negotiation to confirm the communication capability, the sending client sends the audio and video media stream and the uplink network status report, and the server forwards it to the receiving client; the receiving client sends the downlink network status report to the server, and the server selects the worst downlink network status report and forwards it to the sending client; when a weak network situation occurs, the sending client determines the weak network transmission strategy, and sends the media stream to the server after corresponding processing; after the server receives the media stream data, if there is a redundant packet, it decodes it, and then encodes the media stream data and forwards it or directly forwards it; the receiving client determines the media cache length and receives the media stream data. The present invention can obtain the state parameters of the current network more accurately, greatly improving the audio and video interaction experience in a weak network environment.

Description

Technical Field

[0001] The present invention relates to the technical field of audio and video communications, and in particular to a weak network audio and video transmission method with full link state awareness. Background Art

[0002] As the application fields and scenarios of audio and video continue to expand, customers have higher and higher requirements for the quality of audio and video communications. However, in some environments where the communication signal is weak and there are serious obstacles, audio and video freezes, video mosaics, and screen distortion often occur. Traditional audio and video communication methods are difficult to ensure effective data transmission in such a weak network environment with high packet loss and high latency. Therefore, a weak network transmission method is urgently needed to ensure that audio and video data can be effectively transmitted in weak network environments such as network packet loss, jitter, and disorder. At the same time, the existing network status perception method is at the sending or receiving end of the network, and can only obtain the uplink or downlink network status separately, and cannot accurately perceive the status of the overall system network. Summary of the invention

[0003] In view of this, the present invention proposes a weak network audio and video transmission method with full link state awareness. The method can still ensure the quality of audio and video communication in weak network environments such as network packet loss, jitter and disorder, thereby improving the user's audio and video interaction experience.

[0004] In order to achieve the above object, the present invention adopts the following technical solution:

[0005] A method for transmitting audio and video in a weak network with full link state awareness is applied to an audio and video system based on an SFU architecture, wherein the system includes a server and multiple clients; and includes the following steps:

[0006] (1) After the server is started, each client conducts media negotiation with the server to confirm the transmission strategy of the media data, that is, whether the client supports the retransmission strategy or the redundancy strategy; after confirming the transmission strategy, the sending client starts to send the audio and video media stream, and at the same time sends the uplink network status to the server in the form of an uplink network status report. The server forwards the received media stream and uplink network status report to the receiving client;

[0007] (2) The receiving client sends the downlink network status to the server in the form of a downlink network status report. After receiving the downlink network status reports from all the receiving clients, the server selects the worst downlink network status report by comparison and records it, and forwards the worst downlink network status report to the sending client in a timely manner;

[0008] (3) When a weak network situation occurs, that is, when there is packet loss, jitter, or disorder in the network, the sending client calculates the packet loss rate and round-trip delay based on the current uplink network status and the latest downlink network status report received, and then determines the weak network transmission strategy, so as to realize the full-link network status perception of the sending end, and sends the media stream to the server after corresponding processing;

[0009] (4) After receiving the media stream data from the sending client, the server first determines whether there are redundant packets. If there are redundant packets, it decodes them. Then, according to the transmission strategy set by the receiving client, it encodes the media stream data and forwards it directly or forwards it directly.

[0010] (5) After receiving the latest uplink network status report from the sending client forwarded by the server, the receiving client calculates the jitter value and packet loss rate based on the current downlink network status it perceives, thereby determining the media cache length, realizing full link status perception at the receiving end, and receiving media stream data;

[0011] (6) Repeat steps (2) to (5) until the media communication ends.

[0012] Furthermore, the specific method of step (3) is:

[0013] When 0 < log 丢包率 When (0.1)×Round-trip delay ≦ 250, the retransmission strategy and redundancy strategy are activated;

[0014] When 250 < log 丢包率 When (0.1)×RTT ≤ 500, the number of retransmitted packets in the retransmission strategy and the number of redundant packets in the redundancy strategy are continuously increased;

[0015] When log 丢包率 When (0.1)×RTT> 500, the retransmission strategy and redundancy strategy are adjusted to the maximum, and the predictive retransmission strategy is enabled.

[0016] Furthermore, in step (4), according to the transmission strategy set by the receiving client, the media stream data is encoded and then sent or directly forwarded, specifically in the following manner:

[0017] If the receiving client supports the retransmission strategy or the redundancy strategy, the server encodes the media stream data and then forwards it; if the receiving client does not support the retransmission strategy and the redundancy strategy, the server directly forwards the media stream data.

[0018] Furthermore, in step (5), the specific method of determining the media cache length is:

[0019] Buffer_Size = Base_Delay + α×Jitter + β×Packet_Loss

[0020] Where Buffer_Size is the buffer length, α and β are weight parameters used to balance the impact of jitter and packet loss, Base_Delay is the initial setting value of the buffer length, Jitter is the jitter value, which is calculated by the weighted average method of the timestamp of the data packet at the sender and the actual arrival time at the receiver, and Packet_Loss is the packet loss rate.

[0021] The beneficial effects of the above solution of the present invention are:

[0022] (1) The present invention senses and obtains the network status of the entire link through the media messages of the uplink and downlink, thereby determining the weak network transmission method and adaptively selecting weak network strategies such as the sender media retransmission mechanism, redundancy strategy, and congestion control, so as to better balance the contradiction between anti-packet loss and bandwidth utilization.

[0023] (2) The present invention can more accurately estimate the media cache length at the receiving end, greatly reduce the video freeze rate, and improve the anti-packet loss capability of the audio and video system.

[0024] (3) The present invention applies weak network technology to the media server, and can determine the weak network method to be adopted according to the type of client, expanding the application scenarios of the SFU (Selective Forwarding Unit) architecture and improving the flexibility of the system's weak network application. The present invention has good performance in multiple combinations of different delays and different packet loss rates, greatly improving the user experience.

[0025] (4) Combined with the overall link perception and weak network strategy control capabilities of the system, in a network environment with an RTT of 60 ms and a packet loss rate of 70%, the present invention can ensure that the system video freeze rate is less than 10% and the audio MOS value is not less than 3 (based on the ITU-TP.863 POLQA standard), greatly ensuring the quality of the system's audio and video communication in a weak network environment. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 It is a structural diagram of an audio and video weak network transmission system based on the SFU architecture in an embodiment of the present invention.

[0027] Figure 2 This is a schematic diagram of a media sending end obtaining a network status report in an embodiment of the present invention.

[0028] Figure 3 This is a schematic diagram of a media receiving terminal acquiring a network status report in an embodiment of the present invention.

[0029] Figure 4 The present invention is a flowchart of a method for transmitting audio and video in a weak network with full link status awareness in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention is described in detail below with reference to the accompanying drawings.

[0031] A method for transmitting audio and video in weak networks with full link state awareness is applied to Figure 1 In the audio and video communication system based on the SFU architecture shown in the figure. The system is composed of a server and various types of clients. The server adopts the SFU architecture, and the weak network transmission method of the present invention is applied to the server and all clients. The server forwards media data and media messages, and the client is divided into a media sending end and a media receiving end, which obtains local network environment information and sends and receives data packets.

[0032] like Figure 4 As shown, the method comprises the following steps:

[0033] (1) After the server is started, each client conducts media negotiation with the server to confirm the transmission strategy of the media data, that is, whether the client supports the retransmission strategy or the redundancy strategy; after confirming the transmission strategy, the sending client starts to send the audio and video media stream, and at the same time sends the uplink network status to the server in the form of an uplink network status report. The server forwards the received media stream and uplink network status report to the receiving client;

[0034] (2) The receiving client sends the downlink network status to the server in the form of a downlink network status report. After receiving the downlink network status reports from all the receiving clients, the server records them and selects the worst downlink network status report by comparison, and forwards the worst downlink network status report to the sending client;

[0035] (3) When a weak network situation occurs, that is, network packet loss, jitter or disorder occurs, the sending client calculates the packet loss rate and round-trip delay based on the current uplink network status and the latest downlink network status report received, thereby determining the weak network transmission strategy, realizing the full-link network status perception of the sending end, and sending the media stream to the server after corresponding processing; the specific method is as follows:

[0036] When 0 < log 丢包率 When (0.1)×Round-trip delay ≦ 250, the retransmission strategy and redundancy strategy are activated;

[0037] When 250 < log 丢包率 When (0.1)×RTT ≤ 500, the number of retransmitted packets in the retransmission strategy and the number of redundant packets in the redundancy strategy are continuously increased;

[0038] When log 丢包率 When (0.1)×Round-trip delay > 500, the retransmission strategy and redundancy strategy are adjusted to the maximum, and the predictive retransmission strategy is enabled;

[0039] (4) After receiving the media stream data from the sending client, the server first determines whether there are redundant packets, and decodes them if there are redundant packets, and then encodes and forwards the media stream data or directly forwards it according to the transmission strategy set by the receiving client. Specifically, if the receiving client supports the retransmission strategy or the redundancy strategy, the server encodes and forwards the media stream data; if the receiving client does not support the retransmission strategy and the redundancy strategy, the server directly forwards the media stream data.

[0040] (5) After receiving the latest uplink network status report from the sending client forwarded by the server, the receiving client calculates the jitter value and packet loss rate based on the current downlink network status it perceives, thereby determining the media cache length, realizing the full link status perception of the receiving end, and receiving the media stream data; the specific method is as follows:

[0041] Buffer_Size = Base_Delay + α×Jitter + β×Packet_Loss

[0042] Where Buffer_Size is the buffer length, α and β are weight parameters used to balance the impact of jitter and packet loss, Base_Delay is the initial setting value of the buffer length, Jitter is the jitter value, which is calculated by the weighted average method of the timestamp of the data packet at the sender and the actual arrival time at the receiver, and Packet_Loss is the packet loss rate;

[0043] (6) Repeat steps (2) to (5) until the media communication ends.

[0044] Here is a more specific example:

[0045] A method for transmitting audio and video over a weak network with full link state awareness comprises the following steps:

[0046] (1) After the server is started normally, each client enters the system, and each client negotiates with the server to confirm the transmission strategy of media data. After confirming the strategy, audio and video communication is established between each client and the server. Figure 1 shown.

[0047] (2) The media receiver sends the downlink network status report to the server. After receiving the reports from all the receivers, the server selects the worst downlink network status report by comparison and records it, and then forwards the worst downlink network status report to the media sender. Figure 2 shown.

[0048] (3) The sender sends the media stream and the uplink network status report to the server at the same time. The server forwards the received media stream and network status report to the media receiver. Figure 3 shown.

[0049] (4) When weak network conditions such as packet loss, jitter, and disorder occur, the sender will determine the appropriate weak network transmission strategy such as redundancy and retransmission based on its own transmission strategy, the current uplink network status, and the latest downlink network status report received, and process the media stream according to the transmission plan and send it to the server.

[0050] (5) Under the premise of step (4), considering that the receiving end does not support data redundancy, the server receives the media data sent by the sender, decodes the redundant packets, and then encodes the media data according to the strategy set by the receiving end before sending or directly forwarding it. Figure 4 shown.

[0051] (6) The media receiver will determine the appropriate media buffer length to receive media information based on the current downlink network status and the latest uplink network status.

[0052] (7) Continue to repeat steps (2) to (6) until the media communication ends.

[0053] In summary, the present invention comprehensively applies redundant error correction, packet loss retransmission, congestion control and dynamic caching technologies, and can forward the worst network status of all downlinks to the client of the uplink in real time through the server, and also forward the client sending status of the uplink to the client of the downlink in real time, so that each client can perceive the full-link network status in real time, and adjust the media redundancy strategy, retransmission strategy and cache length in real time according to the network status. Compared with point-to-point communication and the network status perception method of separate uplink and downlink, the present invention has better scalability, and can more accurately obtain the status parameters of the current network, which greatly improves the audio and video interaction experience of system users in weak network environments such as packet loss, jitter and disorder.

[0054] It should be noted that any modifications, equivalent substitutions, improvements, etc. made outside the spirit and principles of the present invention are included in the protection scope of the present invention.

Claims

1. A method for transmitting audio and video in a weak network with full link state awareness, characterized in that: In an audio and video system applied to an SFU architecture, the system includes a server and multiple clients; and includes the following steps: (1) After the server is started, each client conducts media negotiation with the server to confirm the transmission strategy of the media data, that is, whether the client supports the retransmission strategy or the redundancy strategy; after confirming the transmission strategy, the sending client starts to send the audio and video media stream, and at the same time sends the uplink network status to the server in the form of an uplink network status report. The server forwards the received media stream and uplink network status report to the receiving client; (2) The receiving client sends the downlink network status to the server in the form of a downlink network status report. After receiving the downlink network status reports from all the receiving clients, the server selects the worst downlink network status report by comparison and records it, and forwards the worst downlink network status report to the sending client; (3) When a weak network situation occurs, that is, when there is packet loss, jitter, or disorder in the network, the sending client calculates the packet loss rate and round-trip delay based on the current uplink network status and the latest downlink network status report received, thereby determining the weak network transmission strategy, realizing the full-link network status perception of the sending end, and sending the media stream to the server after corresponding processing; (4) After receiving the media stream data from the sending client, the server first determines whether there are redundant packets. If there are redundant packets, it decodes them. Then, according to the transmission strategy set by the receiving client, it encodes the media stream data and forwards it directly or forwards it directly. (5) After receiving the latest uplink network status report from the sending client forwarded by the server, the receiving client calculates the jitter value and packet loss rate based on the current downlink network status it perceives, thereby determining the media cache length, realizing full link status perception at the receiving end, and receiving media stream data; (6) Repeat steps (2) to (5) until the media communication ends.

2. According to claim 1, a method for transmitting audio and video in a weak network with full link state awareness is characterized in that: The specific method of step (3) is: When 0 < log 丢包率 When (0.1)×Round-trip delay ≦ 250, the retransmission strategy and redundancy strategy are activated; When 250 < log 丢包率 When (0.1)×RTT ≤ 500, the number of retransmitted packets in the retransmission strategy and the number of redundant packets in the redundancy strategy are continuously increased; When log 丢包率 When (0.1)×RTT> 500, the retransmission strategy and redundancy strategy are adjusted to the maximum, and the predictive retransmission strategy is enabled.

3. According to claim 1, a method for transmitting audio and video in a weak network with full link state awareness is characterized in that: In step (4), according to the media data transmission strategy set by the receiving client, the media stream data is encoded and sent or directly forwarded, specifically in the following manner: If the receiving client supports the retransmission strategy or the redundancy strategy, the server encodes the media stream data and then forwards it; if the receiving client does not support the retransmission strategy and the redundancy strategy, the server directly forwards the media stream data.

4. According to claim 1, a method for transmitting audio and video in a weak network with full link state awareness, characterized in that: In step (5), the specific method of determining the media cache length is: Buffer_Size = Base_Delay + α×Jitter + β×Packet_Loss Where Buffer_Size is the buffer length, α and β are weight parameters used to balance the impact of jitter and packet loss, Base_Delay is the initial setting value of the buffer length, Jitter is the jitter value, which is calculated by the weighted average method of the timestamp of the data packet at the sender and the actual arrival time at the receiver, and Packet_Loss is the packet loss rate.

Images