WebRTC video service quality improvement method and device

Abstract

The invention relates to the technical field of real-time audio and video transmission, and provides a WebRTC video service quality improvement method and device, and the method comprises the steps: responding to a session description proposal sent by a client, and sending a session description response containing candidate information to the client, the candidate information comprises transport layer protocol type parameters and internet protocol address parameters; a session traversing request sent by the client is received, a session traversing reply is sent to the client, and interactive connection establishment with the client is completed; sending media data to the client based on interactive connection establishment; receiving statistical information for the media data fed back by the client, wherein the statistical information comprises time delay information and packet loss information; and determining a packet loss change trend of the media data according to the statistical information, and adjusting a sending code rate of the media data based on the packet loss change trend. According to the technical scheme provided by one or more embodiments of the invention, the service quality of the WebRTC video service can be effectively improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of real-time audio and video transmission, and in particular to a method and device for improving the quality of WebRTC video services. Background Art

[0002] With the rapid development of 5G networks and audio and video technologies, video communication has penetrated into every aspect of people's lives. WebRTC (Web Real-Time Communications) is a real-time audio and video open source project that can use browsers as carriers and has the advantages of cross-platform, low latency, high real-time, high security, and scalability. WebRTC can provide technical support for remote conferencing, online education, telemedicine, live video, real-time monitoring and other fields.

[0003] At the same time, as an open source project, WebRTC has been widely used and borrowed by the industry for its powerful core capabilities such as encoding and decoding, network transmission, media negotiation, and peer-to-peer, bringing the development of real-time audio and video to a new level.

[0004] However, the unpredictable network conditions and the increasingly high video resolution requirements have made communication delays, image quality degradation and packet loss a major problem affecting the quality of WebRTC video. Summary of the invention

[0005] In view of this, one or more embodiments of the present disclosure provide a method and device for improving the quality of WebRTC video services, which can effectively improve the service quality of WebRTC video services.

[0006] In a first aspect, the present disclosure provides a method for improving the quality of WebRTC video services, the method being applied to a WebRTC server, the method comprising: in response to a session description proposal sent by a WebRTC client, sending a session description response containing candidate information to the WebRTC client, the candidate information comprising a transport layer protocol type parameter and an Internet Protocol address parameter, the transport layer protocol type parameter comprising a transmission control protocol and a user datagram protocol; receiving a session crossing request sent by the WebRTC client, and sending a session crossing reply to the WebRTC client, and completing interactive connection establishment with the WebRTC client; wherein the session crossing request is determined by the WebRTC client according to the candidate information; based on the interactive connection establishment, sending media data to the WebRTC client; receiving statistical information for the media data fed back by the WebRTC client, the statistical information comprising delay information and packet loss information; determining a packet loss change trend of the media data according to the statistical information, and adjusting the sending bit rate of the media data based on the packet loss change trend.

[0007] In a second aspect, the present disclosure provides a method for improving the quality of WebRTC video services, which is applied to a WebRTC client, and includes: sending a session description proposal to a WebRTC server, and receiving a session description response containing candidate information returned by the WebRTC server, wherein the candidate information includes a transport layer protocol type parameter and an Internet Protocol address parameter, and the transport layer protocol type parameter includes a transmission control protocol and a user datagram protocol; determining a session crossing request based on the candidate information, and sending the session crossing request to the WebRTC server; receiving a session crossing reply returned by the WebRTC server, and completing interactive connection establishment with the WebRTC server; based on the interactive connection establishment, receiving media data sent by the WebRTC server, and determining statistical information for the media data, wherein the statistical information includes delay information and packet loss information; sending the statistical information to the WebRTC server, so that the WebRTC server determines a packet loss change trend of the media data based on the statistical information, and adjusts the sending bit rate of the media data based on the packet loss change trend.

[0008] In a third aspect, the present disclosure provides a device for improving the quality of WebRTC video services, the device being applied to a WebRTC server, the device comprising: a session description response unit, configured to respond to a session description proposal sent by a WebRTC client and send a session description response containing candidate information to the WebRTC client, the candidate information comprising a transport layer protocol type parameter and an Internet Protocol address parameter, the transport layer protocol type parameter comprising a transmission control protocol and a user datagram protocol; a session traversal response unit, configured to receive a session traversal request sent by the WebRTC client, and send a session traversal reply to the WebRTC client, to complete interactive connection establishment with the WebRTC client; wherein the session traversal request is determined by the WebRTC client according to the candidate information; a media data sending unit, configured to send media data to the WebRTC client based on the interactive connection establishment; a statistical information receiving unit, configured to receive statistical information for the media data fed back by the WebRTC client, the statistical information comprising delay information and packet loss information; a bit rate adjustment unit, configured to determine a packet loss change trend of the media data according to the statistical information, and adjust the sending bit rate of the media data based on the packet loss change trend.

[0009] In a fourth aspect, the present disclosure provides a device for improving the quality of WebRTC video services, the device being applied to a WebRTC client, the device comprising: a session description request unit, configured to send a session description proposal to a WebRTC server, and receive a session description response including candidate information returned by the WebRTC server, the candidate information comprising a transport layer protocol type parameter and an Internet Protocol address parameter, the transport layer protocol type parameter comprising a transmission control protocol and a user datagram protocol; a session traversal request unit, configured to determine a session traversal request according to the candidate information, and send the session traversal request to the WebRTC server; an interactive connection A connection establishment unit is used to receive a session crossing reply returned by the WebRTC server and complete the interactive connection establishment with the WebRTC server; a media data receiving unit is used to receive the media data sent by the WebRTC server based on the interactive connection establishment, and determine statistical information for the media data, wherein the statistical information includes delay information and packet loss information; a statistical information sending unit is used to send the statistical information to the WebRTC server, so that the WebRTC server determines the packet loss change trend of the media data according to the statistical information, and adjusts the sending bit rate of the media data based on the packet loss change trend.

[0010] In a fifth aspect, the present disclosure provides an electronic device, comprising a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, the quality improvement method of the WebRTC video service of the first aspect or the second aspect is implemented.

[0011] In a sixth aspect, the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store a computer program. When the computer program is executed by a processor, the quality improvement method of the WebRTC video service of the first aspect or the second aspect is implemented.

[0012] The technical solution provided by one or more embodiments of the present disclosure is that during the session description interaction process, the candidate information sent by the WebRTC server to the WebRTC client includes not only the Internet Protocol address parameters, but also the transport layer protocol type parameters, so that the two can flexibly choose the transmission control protocol or the user datagram protocol according to the actual network environment to complete the transmission of media data. On this basis, a session crossing interaction is set up to open up the interactive connection establishment between the WebRTC server and the WebRTC client, ensuring that the media data can be transmitted accurately and stably. According to the statistical information on the media data fed back by the WebRTC client, the WebRTC server can determine the packet loss change trend of the media data. Based on the packet loss change trend, the WebRTC server can adjust the sending bit rate of the media data to optimize the video service quality.

[0013] The technical solution provided by one or more embodiments of the present disclosure proposes a WebRTC service model based on the transmission control protocol, designs a feasible technical solution for implementing WebRTC services based on the transmission control protocol, and sorts out the data interaction process of the WebRTC service model after adding the transmission control protocol. In this way, the real-time audio and video transmission requirements of the WebRTC server in the case where the user datagram protocol network is not applicable can be met, and the comprehensiveness of the WebRTC video service in different network conditions can be expanded, thereby improving the service quality of the WebRTC video service.

[0014] The technical solution provided by one or more embodiments of the present disclosure improves the bitrate estimation algorithm of the WebRTC server by adding the analysis of the trend change state of the packet loss rate. In this way, the transmission bitrate of the video stream can respond quickly to the situation of severe network fluctuations, and can be adaptively adjusted according to the fluctuation of the current real-time network available bandwidth and the congestion level of the system, thereby improving the quality of WebRTC video service.

[0015] According to the technical solution provided by one or more embodiments of the present disclosure, the improved bit rate prediction algorithm can be directly performed on the server side, without additional resource consumption and delay compared to adding a machine learning model separately. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The features and advantages of the embodiments of the present disclosure may be more clearly understood by referring to the accompanying drawings, which are schematic and should not be construed as limiting the present disclosure in any way. In the accompanying drawings:

[0017] Figure 1 A schematic diagram of the calculation process of a network congestion control algorithm in the related art is shown;

[0018] Figure 2 A schematic diagram of a system model of a method for improving the quality of WebRTC video services in one embodiment of the present disclosure is shown;

[0019] Figure 3 A schematic diagram showing the steps of a method for improving the quality of a WebRTC video service in one embodiment of the present disclosure is shown;

[0020] Figure 4 A schematic diagram of a process of data interaction between a WebRTC server and a browser in some practical applications of the present disclosure is shown;

[0021] Figure 5 A schematic diagram of the composition structure of candidate information in one embodiment of the present disclosure is shown;

[0022] Figure 6 A schematic diagram of the calculation logic of a network congestion control algorithm in the related art is shown;

[0023] Figure 7 A schematic diagram of the calculation logic of a network congestion control algorithm in one embodiment of the present disclosure is shown;

[0024] Figure 8 A schematic diagram showing the steps of another method for improving the quality of WebRTC video services in one embodiment of the present disclosure is shown;

[0025] Fig. 9 A schematic diagram of functional units of a device for improving the quality of WebRTC video services in one embodiment of the present disclosure is shown;

[0026] Fig.10 A schematic diagram of functional units of another device for improving the quality of WebRTC video services in one embodiment of the present disclosure is shown;

[0027] Fig.11 A schematic structural diagram of an electronic device in one embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0029] In the related technology, the problems that affect the quality of WebRTC video may include communication delay, image quality degradation, packet loss and lag. For example, the contradiction between bitrate and bandwidth: in order to obtain good playback effects, a larger bitrate is usually selected, but under limited bandwidth, when the network is congested, it will lead to the loss of data packets, resulting in frame loss and screen distortion. Another example is the contradiction between the transmission protocol and the image quality: for high real-time performance, the User Datagram Protocol (UDP) is usually selected as the transmission protocol, but UDP is prone to packet loss and disorder, making it difficult to ensure the quality of the picture.

[0030] In related technologies, WebRTC users try to improve the video service quality of WebRTC in less-than-ideal network environments through QoS (Quality of Service) measures such as the video engine's image enhancement algorithm, network congestion control algorithm (GCC), receiving-end interframe interpolation algorithm, jitter buffer control algorithm, NACK packet loss retransmission technology, and FEC forward error correction algorithm.

[0031] In related technologies, the GCC algorithm can estimate the bandwidth and calculate the bit rate based on the delay gradient and packet loss rate. TFB-GCC (Transport-wide Feedback based Congestion Control Algorithm) is a new version of the GCC algorithm commonly used by WebRTC. Compared with the old version of REMB-GCC, it can transfer the logic of bandwidth calculation to the sender, so that the receiver only periodically feedbacks the packet receiving status to the sender through the Real-time Transport Control Protocol (RTCP).

[0032] See also Figure 1 In the related art, the GCC algorithm can achieve a balance between network delay and image quality through congestion control and bandwidth adjustment strategies based on delay and packet loss feedback. The overall operation process of the GCC algorithm is as follows Figure 1As shown, the receiver can perform arrival time and packet loss statistics based on the received Real-time Transport Protocol (RTP) message, and send an RTCP message to the sender. The remote bit rate estimator of the sender can obtain the current network queuing delay and packet loss through the RTCP message. The remote bit rate estimator can dynamically adjust the encoding bit rate of the sender based on the delay and packet loss information, and change the bit rate of the smooth sender (pacer). According to the adjusted bit rate, the pacer can change its own network sending speed, and use this sending speed to regularly trigger the packet sending time of the sender (sender). The above GCC algorithm operation process can be repeated, so that in a weak network environment, the sender can reduce the sending bit rate, reduce the picture clarity, and improve the continuity of the video picture; when the network quality improves, the sender can increase the sending bit rate to improve the picture effect.

[0033] However, in the related art, the WebRTC video service combined with the GCC algorithm may still not be able to guarantee stable service when facing changeable and complex network conditions.

[0034] The present disclosure finds that WebRTC video services in related technologies generally have the following difficulties:

[0035] First of all, WebRTC data transmission is generally based on point-to-point. In the face of complex Internet environments such as cross-regional and low-bandwidth environments, it is difficult to ensure transmission quality. Most existing WebRTC servers transmit RTP messages based on UDP. However, under security management policies, UDP ports are not open in some scenarios, and UDP cannot be used to transmit media data. In other actual application scenarios, some access devices or specific network systems have limitations. UDP networks are available but packet loss is serious, while the Transmission Control Protocol (TCP) can be used normally. In this case, traditional UDP-based WebRTC servers may have poor transmission quality or even be unable to provide video services.

[0036] Secondly, although the GCC algorithm is powerful, it still has some defects. For example, the GCC algorithm is a typical algorithm based on fixed rules. In some networks, it may behave too conservatively, resulting in low bandwidth utilization. The network conditions in actual applications are complex and changeable. For example, in a complex network topology with multiple link jumps, the routing state of each hop has a lot of randomness, which affects the entire network link. At the same time, depending on the scale of the accessed video stream, the concurrent viewing volume, etc., the network is also prone to rapid fluctuations. In this case, the GCC algorithm, which calculates the delay between two adjacent packets based on fixed rules and adjusts the sending bit rate accordingly, is difficult to adjust the sending bit rate in time. The GCC algorithm is likely to make delayed or wrong judgments due to extremely short network fluctuations, affecting the quality of WebRTC video service.

[0037] In addition, to solve the contradiction between network bandwidth and bitrate, machine learning methods can be introduced to add a separate model service. The network parameters transmitted by the WebRTC server can be analyzed by the machine learning model service to obtain the bitrate result and return it to the WebRTC server. However, this round trip also depends on network access, which can easily increase network burden and latency.

[0038] In view of this, a method for improving the quality of WebRTC video services provided in an embodiment of the present disclosure solves the problem of WebRTC service failure when UDP network is not applicable by implementing the design idea of ​​WebRTC based on TCP; and improves the shortcomings of GCC algorithm by using adaptive control algorithm. By simultaneously addressing these two aspects, the comprehensiveness, stability and reliability of WebRTC video services can be improved.

[0039] See also Figure 2 The quality improvement method of WebRTC video service proposed in this disclosure can be applied in Figure 2 In the system model shown. The system model mainly consists of three modules: audio and video acquisition equipment, WebRTC server, and WebRTC client. Audio and video acquisition equipment such as cameras, computers, and mobile phones can push the collected original audio and video to the media server. As the central media processor, the WebRTC server can distribute the media stream data processed by encoding and decoding, image enhancement, congestion control and other strategies to WebRTC clients such as web, ios applications, and android applications for decoding and playback. Among them, the process of audio and video acquisition equipment pushing streams to the WebRTC server and the WebRTC client live streaming and pulling streams from the WebRTC server can negotiate the use of TCP or UDP data transmission methods according to the network environment.

[0040] See also Figure 3A method for improving the quality of a WebRTC video service provided by an embodiment of the present disclosure can be applied to a WebRTC server and may include the following steps.

[0041] S11: In response to a session description proposal sent by a WebRTC client, a session description response including candidate information is sent to the WebRTC client, wherein the candidate information includes a transport layer protocol type parameter and an Internet Protocol address parameter, and the transport layer protocol type parameter includes a transmission control protocol and a user datagram protocol.

[0042] In this embodiment, the WebRTC server and the WebRTC client can first perform the interaction process of the Session Description Protocol (SDP). The communicating parties can exchange SDP information through the offer and answer model. The initiator (such as the WebRTC client) can create an offer and use it as a local description, and then send it to the other party (such as the WebRTC server) through the signaling server. After receiving the offer, the other party can save it as a remote description, and create an answer based on this remote description, and then send it back to the initiator through the signaling server. The offer mainly includes the media type (such as audio, video) that the proposer wants to use, the supported codecs, and the transmission parameters of the media stream, such as IP address and port. The offer generally includes the media capabilities of the initiator, such as supported codecs, resolution, frame rate, etc. After receiving the offer, the receiver will compare its own media capabilities with the media capability information in the offer to determine the media capabilities supported by both parties. The main function of the offer is to provide a session parameter proposal to the other party (answerer) so that both parties can reach an agreement on the media parameters supported by both parties. The SDP interaction process is dynamic, allowing both parties to negotiate a set of session parameters acceptable to both parties based on their respective media capabilities and network conditions.

[0043] For some practical application examples, see Figure 4, the WebRTC client is a browser (Web). The browser can send a session description proposal sdp-offer to the WebRTC server through a Hypertext Transfer Protocol (HTTP) request. In order to select the network transmission protocol, the browser can add a TransType field parameter (the parameter value can be tcp or udp) in the Uniform Resource Locator (URL) or in the HTTP request body, so that the WebRTC server can quickly determine the network transmission protocol to be used for the WebRTC video service.

[0044] After receiving the session description proposal from the browser, the WebRTC server can select the audio and video codec format according to its own media capabilities; it can confirm the network transmission method according to the TransType field parameter; it can also form candidate information and encapsulate the candidate information in the session description response sdp-answer and reply to the browser. The WebRTC server can package the candidate information and media capability information in the same session description response sdp-answer and send them together.

[0045] See also Figure 5 In a practical application example, the candidate information is constructed as follows Figure 5 shown.

[0046] Specifically, foundation is an identifier used to distinguish different candidate information. This identifier can be the same for candidates of the same type and sharing the same foundation from the same STUN (Session Traversal Utilities for NAT) server. The componentID field can indicate whether the candidate is used to transmit RTP packets or RTCP packets. If the componentID field parameter is 1, it can indicate that the candidate is used to transmit RTP packets. If the componentID field parameter is 2, it can indicate that the candidate is used to transmit RTCP packets. The TransType field is used to confirm the candidate's network transmission method (TCP or UDP). The priority field indicates the priority of the candidate. The Address field includes the IP (Internet Protocol Address) field and the Port field, which can indicate the candidate's IP address and port number. The IP address can be in IPv4 format, IPv6 format, or domain name format (if it is in domain name format, the receiving end needs to perform DNS resolution). The port number is the port used for network communication. The Candidate Type field indicates the type of the candidate. Common types include host candidate (host), server reflexive candidate (ServerReflexive, srflx), peer reflexive candidate (Peer Reflexive, prflx), and relay candidate (relay).

[0047] The above fields can collectively describe a candidate information for ICE (Interactive Connectivity Establishment) to help determine the best network connection path for WebRTC services so as to establish a point-to-point connection for WebRTC services behind NAT (Network Address Translation) and network firewalls.

[0048] It should be noted that if the browser does not transmit the parameter value of the TransType field parameter when sending the session description proposal, the WebRTC server can fill in the corresponding part of the candidate information as udp by default.

[0049] S12: receiving a session traversal request sent by the WebRTC client, and sending a session traversal reply to the WebRTC client, thereby completing the establishment of an interactive connection with the WebRTC client; wherein the session traversal request is determined by the WebRTC client according to the candidate information.

[0050] In this embodiment, based on multiple ICE candidate information, the WebRTC client can select target candidate information and open ICE with the WebRTC server through a session traversal request (such as a STUN request) and a session traversal reply (such as a STUN reply). The target candidate information can be determined by the priority and order of the candidate information.

[0051] In this embodiment, the interactive connection is established when the network connection is made, which usually involves the interaction of the STUN protocol. The STUN protocol is used to help the client discover its public network address after NAT (Network Address Translation) and determine the restrictions of NAT on direct connection, thereby achieving NAT traversal. In the ICE framework, the STUN protocol is not only used to obtain the public network mapping address, but also to perform connectivity checks and specify transmission paths. Therefore, ICE connections do need to be completed through STUN interactions in most cases.

[0052] In some implementations, the interactive connection establishment can be completed in ICE-Lite mode. Only one end (usually the end initiating the call, that is, the WebRTC client) needs to perform the complete ICE process, including collecting candidate information and performing connectivity tests. The other end (WebRTC server) only needs to respond to the other party's STUN request, because it does not need to actively collect other candidates except the host candidate.

[0053] ICE-Lite mode can reduce network bandwidth consumption and latency by simplifying STUN protocol interactions. Compared to FULL-ICE mode, ICE-Lite mode reduces candidate collection and connectivity testing, which can speed up the connection establishment process.

[0054] For a practical example, see Figure 4, the WebRTC client is a browser. After the browser receives the session description answer (sdp-answer) provided by the WebRTC server, which contains media capability information and ICE candidate information, it can open ICE with the WebRTC server. First, the browser can sort the priority of the candidate information in the session description answer. The candidate with the highest priority is usually the most likely to provide the path with the best connection quality. Then, the browser can send a STUN request to the IP address of the target candidate information from high to low according to the sorted candidate list to detect and verify the network connectivity between the browser and the WebRTC server. When the WebRTC server receives the STUN request from the browser, it will parse the request. The WebRTC server will verify the binding request in the STUN request to ensure that it is legal and meets the requirements of the ICE protocol. Once the request is verified, the WebRTC server can send a STUN reply to the browser. This STUN reply can contain the public IP address and port of the browser seen by the WebRTC server, which is necessary for NAT traversal. This application example uses the ICE-Lite method, which only needs to be opened on one end, which can improve the robustness and speed of the connection.

[0055] S13: Based on the interactive connection establishment, media data is sent to the WebRTC client.

[0056] In this implementation, after ICE is established, the WebRTC server can send media data to the WebRTC client, and provide video services to the user on the WebRTC client.

[0057] In some implementations, before sending media data to the WebRTC client based on establishing the interactive connection, the method further includes: exchanging keys with the WebRTC client based on a data packet transport layer security protocol to generate a session key, wherein the session key is used to encrypt and decrypt the media data.

[0058] Specifically, Datagram Transport Layer Security (DTLS) can provide end-to-end data confidentiality and integrity protection. In WebRTC video services, DTLS is a key part of ensuring communication security. In WebRTC, DTLS can be used for two main parts: negotiating and managing SRTP (Secure Real-time Transport Protocol) keys, and providing encrypted data channels.

[0059] For a practical example, see Figure 4 According to the setting attributes in the SDP negotiation, the identity of the WebRTC server or WebRTC client in the DTLS interaction can be determined. For example, the setting attribute of the browser (WebRTC client) in the SDP negotiation is active, and it can actively initiate DTLS negotiation to the server (WebRTC server) as a client (actively sending clienthello). Subsequently, the WebRTC server and the WebRTC client can exchange keys to determine the session key. The session key can be used to encrypt subsequent RTP data. Based on DTLS interaction, link certificate verification can also be performed between the WebRTC server and the WebRTC client to further ensure data security.

[0060] It should be noted that although DTLS is designed for UDP, in the present disclosure, even if data is transmitted between the WebRTC server and the WebRTC client via TCP mode, DTLS and SRTP data encryption can be performed.

[0061] In some embodiments, sending the media data to the WebRTC client includes: sending the media data to the WebRTC client in response to a real-time transport control protocol request sent by the WebRTC client; or actively sending the encrypted media data to the WebRTC client after performing a data packet transport layer security protocol interaction process with the WebRTC client.

[0062] In an actual application example, after completing ICE, DTLS interaction and other links, the browser can send the first RTCP data packet to the WebRTC server, requesting the WebRTC server to transmit media data. After receiving the RTCP data packet, the WebRTC server can respond and start sending the media data to the browser.

[0063] In an actual application example, after completing ICE, DTLS interaction and other links, the WebRTC server can actively send encrypted media data to the browser immediately or after a preset time interval (such as a fixed number of seconds or minutes).

[0064] S14: Receive statistical information on the media data fed back by the WebRTC client, where the statistical information includes delay information and packet loss information.

[0065] In this embodiment, the WebRTC client can count the reception of media data, including the arrival time of RTP packets reflecting the delay information, and the packet loss amount and packet loss rate statistics reflecting the packet loss information. According to the preset feedback interval, the WebRTC client can regularly feedback these statistical information to the WebRTC server.

[0066] In a practical application example, the browser can trigger an operation at a fixed time to send an RTCP packet to the WebRTC server to feedback the current network delay and packet loss.

[0067] S15: Determine a packet loss variation trend of the media data according to the statistical information, and adjust a sending bit rate of the media data based on the packet loss variation trend.

[0068] In this embodiment, the WebRTC server can dynamically adjust the transmission bit rate of the media data through a bit rate adjustment algorithm. In the bit rate adjustment algorithm, adding analysis of the trend change state of the packet loss rate can improve the accuracy and timeliness of the bit rate adjustment.

[0069] Specifically, see Figure 6 The code rate control logic of the GCC algorithm in the related technology is mainly divided into code rate calculation based on delay and bandwidth estimation based on packet loss rate. Among them, based on the estimation of packet loss rate, the current network status can be judged according to the packet loss rate at this moment and the set threshold. When the packet loss rate is less than 2%, it can be considered that the network condition is very good and the sending code rate can be appropriately increased; when the packet loss rate is between 2% and 10%, the original code rate can be maintained; when the packet loss rate is greater than 10%, it can be considered that the network is congested and the code rate can be reduced according to the specific size of the packet loss rate. Based on the code rate estimation of delay, the delay gradient of the packet can be calculated according to the RTCP message sent by the receiving end, and the change trend of the delay gradient can be accumulated through linear filtering calculation. Then, based on the relationship between this trend and the size of the adaptive threshold, overload detection can be performed to judge the current bandwidth status (for example, overuse overload state, normal normal state, underuse underuse state). The rate controller can drive the speed control according to the signal output by the overload detection to estimate the final code rate. Finally, the GCC algorithm can take the smaller value between the predicted value based on packet loss and the predicted value based on the delay gradient as the final transmission code rate.

[0070] However, the present disclosure finds that when the network jitters violently, the GCC algorithm of the related art is too conservative. Selecting the minimum value as the final transmission bit rate can easily lead to low bandwidth utilization, resulting in low image clarity. The main reason is that when the network fluctuates, the estimated bit rate of the GCC algorithm will decrease exponentially. When the network improves, the estimated bit rate of the GCC algorithm rises very slowly, and cannot be adjusted in time. It still sends packets at a lower bit rate, resulting in the image clarity of the receiving end being continuously low. For example: before a certain moment, the packet loss rate was about 3%, and the data transmission bit rate A(i-1) of the WebRTC server was 2MB / s. At the current moment, the network suddenly jitters violently, and the packet loss rate suddenly increases by 50%. The data transmission bit rate A(i) of the WebRTC server is changed to 1.5MB / s. At the next moment, the packet loss rate returns to about 3%. Since the bandwidth based on delay is higher than that based on packet loss, the data transmission bit rate A(i+1) of the WebRTC server will still be estimated based on the packet loss method. At this point, although the network situation has greatly improved, the WebRTC server still continues to send media data at a bandwidth of 1.5MB / s. The data transmission bit rate of the WebRTC server will not recover until the packet loss rate drops below 2%, and the recovery speed is relatively slow.

[0071] Based on the above problems, the present disclosure believes that the bit rate estimation value based on the packet loss rate should not only be calculated based on the size of the packet loss rate and the last bit rate estimation value, but also the judgment of the packet loss rate change trend should be added.

[0072] In some embodiments, adjusting the sending bit rate of the media data based on the packet loss change trend may include: calculating the packet loss rate difference between the packet loss rates of adjacent time nodes, the packet loss rate difference representing the packet loss change trend of the target time period; when the packet loss rate difference is greater than a preset threshold, determining an adjustment coefficient that matches the packet loss rate difference; and correcting the sending bit rate based on the packet loss rate difference and the adjustment coefficient.

[0073] For a practical example, see Figure 7 , based on the packet loss change trend, adjust the sending bit rate of media data. The process can be as follows:

[0074] When the current packet loss rate is between 2% and 10%, in order to avoid the delay of the bit rate remaining unchanged for a long time, the judgment of the packet loss rate change trend can be increased. When the packet loss rate difference diff of adjacent time nodes is greater than the start threshold α, the sending bit rate can be adjusted. The specific formula for adjusting the sending bit rate can be:

[0075] A(i)=A(i-1)(1+β i ·diff(P i-1 -Pi ))

[0076] Where A(i) represents the target transmission bit rate generated at time i, and A(i-1) represents the transmission bit rate at the previous time. i-1 is the packet loss rate at the previous moment, P i is the packet loss rate at this moment, β is the adjustment coefficient, and diff is the difference in packet loss rate between time i and time i-1, which represents the packet loss change trend during the period formed by these two moments. If diff is greater than 0, it means that the packet loss rate is reduced and the A(i) code rate can be increased; if diff is less than 0, it means that the packet loss rate is increased and the A(i) code rate can be reduced; if diff is equal to 0, the A(i) code rate can remain unchanged.

[0077] In an actual application example, the startup threshold α and the adjustment coefficient β can be selected as fixed values ​​according to the actual application scenario, or can be adaptively changed according to the packet loss rate. For example, the adjustment coefficient β can be set according to the following rules:

[0078]

[0079] When the absolute value of the packet loss rate change difference is greater than 10%, it may represent a huge difference change, and the adjustment coefficient β may be 1.05; when the absolute value of the difference is between 5% and 10%, the adjustment coefficient β may be 1; when the absolute value of the difference is less than 5%, it may represent a small difference change, and the adjustment coefficient β may be 0.95.

[0080] In some implementations, if the original image received by the WebRTC server is distorted and incomplete, image enhancement technology can be used to repair the image, and a NACK (Negative-Acknowledgment) mechanism can be used to periodically trigger data packet retransmission.

[0081] It is worth noting that in the method for improving the quality of WebRTC video services provided in this embodiment, since the candidate information of the session description response in step S11 includes the transport layer protocol type parameter, the SDP interaction between the WebRTC server and the WebRTC client can select the network transmission mode of UDP or TCP according to the specific network environment and requirements. In the subsequent possible ICE, DTLS, and media data transmission links, the WebRTC server and the WebRTC client will exchange data according to the network transmission mode (TCP / UDP) determined in the SDP exchange link.

[0082] See also Figure 8 A method for improving the quality of a WebRTC video service provided in one embodiment of the present disclosure can be applied to a WebRTC client and may include the following steps.

[0083] S21: sending a session description proposal to a WebRTC server, and receiving a session description response including candidate information returned by the WebRTC server, wherein the candidate information includes a transport layer protocol type parameter and an Internet Protocol address parameter, and the transport layer protocol type parameter includes a transmission control protocol and a user datagram protocol;

[0084] S22: Determine a session traversal request according to the candidate information, and send the session traversal request to the WebRTC server;

[0085] S23: receiving a session traversal response returned by the WebRTC server, and completing the establishment of an interactive connection with the WebRTC server;

[0086] S24: Based on the interactive connection establishment, receiving media data sent by the WebRTC server, and determining statistical information for the media data, the statistical information including delay information and packet loss information;

[0087] S25: Sending the statistical information to the WebRTC server, so that the WebRTC server determines the packet loss change trend of the media data according to the statistical information, and adjusts the sending bit rate of the media data based on the packet loss change trend.

[0088] The method for improving the quality of WebRTC video services provided in steps S21 to S25 of this embodiment is the counterpart method of the technical solution defined in steps S11 to S15 above. For relevant explanations, please refer to the corresponding contents of steps S11 to S15, which will not be repeated here.

[0089] In some implementations, before receiving the media data sent by the WebRTC server based on the establishment of the interactive connection in step S24, the method may further include: exchanging keys with the WebRTC server based on the packet transport layer security protocol to generate a session key, wherein the session key is used to encrypt and decrypt the media data.

[0090] For relevant explanations, please refer to the corresponding content of step S13, which will not be repeated here.

[0091] In some implementations, receiving the media data sent by the WebRTC server in step S24 may include: sending a real-time transport control protocol request to the WebRTC server, and receiving the media data sent by the WebRTC server in response to the real-time transport control protocol request; or, after performing a data packet transport layer security protocol interaction process with the WebRTC server, receiving the encrypted media data actively sent by the WebRTC client.

[0092] For relevant explanations, please refer to the corresponding content of step S13, which will not be repeated here.

[0093] In some embodiments, adjusting the sending bit rate of the media data based on the packet loss change trend in step S25 may include: calculating the packet loss rate difference between the packet loss rates of adjacent time nodes, the packet loss rate difference representing the packet loss change trend of the target time period; when the packet loss rate difference is greater than a preset threshold, determining an adjustment coefficient that matches the packet loss rate difference; and correcting the sending bit rate based on the packet loss rate difference and the adjustment coefficient.

[0094] For relevant explanations, please refer to the corresponding content of step S15, which will not be repeated here.

[0095] The technical solution provided by one or more embodiments of the present disclosure is that during the session description interaction process, the candidate information sent by the WebRTC server to the WebRTC client includes not only the Internet Protocol address parameters, but also the transport layer protocol type parameters, so that the two can flexibly choose the transmission control protocol or the user datagram protocol according to the actual network environment to complete the transmission of media data. On this basis, a session crossing interaction is set up to open up the interactive connection establishment between the WebRTC server and the WebRTC client, ensuring that the media data can be transmitted accurately and stably. According to the statistical information on the media data fed back by the WebRTC client, the WebRTC server can determine the packet loss change trend of the media data. Based on the packet loss change trend, the WebRTC server can adjust the sending bit rate of the media data to optimize the video service quality.

[0096] The technical solution provided by one or more embodiments of the present disclosure proposes a WebRTC service model based on TCP, designs a feasible technical solution for implementing WebRTC services based on TCP, and sorts out the data interaction process of the WebRTC service model after adding TCP. In this way, the real-time audio and video transmission requirements of the WebRTC server in the case where the UDP network is not applicable can be met, and the comprehensiveness of the WebRTC video service in different network conditions can be expanded, thereby improving the service quality of the WebRTC video service.

[0097] TCP can provide reliable transmission in IP environment and has a natural congestion control strategy. The slow start algorithm at the beginning of TCP communication is a congestion avoidance mechanism that can play a role in smooth transition, slow down the rate growth, and avoid the uncontrollability caused by the rapid rate growth. TCP's fast retransmission and fast recovery strategies can improve network throughput. TCP's congestion control mechanism has undergone a long period of development and precipitation and is now relatively complete. The combination of TCP and WebRTC can meet customer needs and have excellent performance in specific scenarios (for example, UDP is unavailable, UDP network is available but packet loss is severe).

[0098] The technical solution provided by one or more embodiments of the present disclosure improves the bitrate estimation GCC algorithm of the WebRTC server by adding the analysis of the trend change state of the packet loss rate. In some embodiments, the present disclosure details the process steps, main formulas and formula parameter definitions of the optimized GCC algorithm. In this way, the transmission bitrate of the video stream can respond quickly to the situation of severe network fluctuations, and can be adaptively adjusted according to the fluctuation of the current real-time network available bandwidth and the congestion level of the system, thereby improving the quality of WebRTC video service.

[0099] According to the technical solution provided by one or more embodiments of the present disclosure, the GCC improvement method can be directly run on the server side, without additional resource consumption and delay compared to adding a machine learning model separately.

[0100] See also Fig. 9 The present disclosure also provides a device for improving the quality of WebRTC video services, the device being applied to a WebRTC server, the device comprising:

[0101] The session description response unit 901 is configured to send a session description response including candidate information to the WebRTC client in response to the session description proposal sent by the WebRTC client, wherein the candidate information includes a transport layer protocol type parameter and an Internet protocol address parameter, and the transport layer protocol type parameter includes a transmission control protocol and a user datagram protocol;

[0102] The session traversal response unit 902 is configured to receive a session traversal request sent by the WebRTC client, and send a session traversal reply to the WebRTC client to complete the establishment of an interactive connection with the WebRTC client; wherein the session traversal request is determined by the WebRTC client according to the candidate information;

[0103] A media data sending unit 903, configured to send media data to the WebRTC client based on the interactive connection establishment;

[0104] A statistical information receiving unit 904 is used to receive statistical information for the media data fed back by the WebRTC client, where the statistical information includes delay information and packet loss information;

[0105] The bit rate adjustment unit 905 is used to determine the packet loss change trend of the media data according to the statistical information, and adjust the sending bit rate of the media data based on the packet loss change trend.

[0106] In one embodiment, the device further includes a data encryption unit 906, which is used to perform a key exchange with the WebRTC client based on the data packet transport layer security protocol to generate a session key, and the session key is used to encrypt and decrypt the media data.

[0107] In one embodiment, the media data sending unit 903 is specifically used to send the media data to the WebRTC client in response to a real-time transport control protocol request sent by the WebRTC client; or, after performing a data packet transport layer security protocol interaction process with the WebRTC client, actively send the encrypted media data to the WebRTC client.

[0108] In one embodiment, the bit rate adjustment unit 905 includes a bit rate calculation subunit. The bit rate calculation subunit is specifically used to calculate the packet loss rate difference between the packet loss rates of adjacent time nodes, and the packet loss rate difference represents the packet loss change trend of the target period; when the packet loss rate difference is greater than a preset threshold, determine an adjustment coefficient matching the packet loss rate difference; based on the packet loss rate difference and the adjustment coefficient, correct the sending bit rate.

[0109] See also Fig.10 The present disclosure also provides a device for improving the quality of WebRTC video services, the device being applied to a WebRTC client, the device comprising:

[0110] The session description request unit 1001 is used to send a session description proposal to the WebRTC server, and receive a session description response including candidate information returned by the WebRTC server, wherein the candidate information includes a transport layer protocol type parameter and an Internet Protocol address parameter, and the transport layer protocol type parameter includes a transmission control protocol and a user datagram protocol;

[0111] The session traversal request unit 1002 is used to determine a session traversal request according to the candidate information, and send the session traversal request to the WebRTC server;

[0112] An interactive connection establishing unit 1003 is used to receive a session traversal reply returned by the WebRTC server and complete the interactive connection establishment with the WebRTC server;

[0113] A media data receiving unit 1004 is used to receive media data sent by the WebRTC server based on the interactive connection establishment, and determine statistical information for the media data, wherein the statistical information includes delay information and packet loss information;

[0114] The statistical information sending unit 1005 is used to send the statistical information to the WebRTC server, so that the WebRTC server determines the packet loss change trend of the media data according to the statistical information, and adjusts the sending bit rate of the media data based on the packet loss change trend.

[0115] In one embodiment, the device further includes a data decryption unit 1006, which is used to perform a key exchange with the WebRTC server based on the data packet transport layer security protocol to generate a session key, and the session key is used to encrypt and decrypt the media data.

[0116] In one embodiment, the media data receiving unit 1004 is specifically used to send a real-time transport control protocol request to the WebRTC server, and receive the media data sent by the WebRTC server in response to the real-time transport control protocol request; or, after performing a data packet transport layer security protocol interaction process with the WebRTC server, receive the encrypted media data actively sent by the WebRTC client.

[0117] In one embodiment, the WebRTC server determines the packet loss change trend of the media data according to the statistical information, and adjusts the sending bit rate of the media data based on the packet loss change trend, including: the WebRTC server calculates the packet loss rate difference between the packet loss rates of adjacent time nodes, and the packet loss rate difference represents the packet loss change trend of the target time period; when the packet loss rate difference is greater than a preset threshold, the WebRTC server determines an adjustment coefficient that matches the packet loss rate difference; the WebRTC server corrects the sending bit rate based on the packet loss rate difference and the adjustment coefficient.

[0118] Above Fig. 9 or Fig.10Each unit described in the corresponding embodiment may be implemented by a computer chip or a product having a certain function. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0119] For the convenience of description, the above devices are described in terms of functions and are described separately in various units. Of course, when implementing the present application, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0120] See also Fig.11 The present disclosure also provides an electronic device, which includes a memory and a processor, wherein the memory is used to store a computer program, and when the computer program is executed by the processor, a method for improving the quality of the WebRTC video service of the above-mentioned WebRTC server or WebRTC client is implemented.

[0121] The present disclosure also provides a computer-readable storage medium, which is used to store a computer program. When the computer program is executed by a processor, the quality improvement method of the WebRTC video service of the above-mentioned WebRTC server or WebRTC client is implemented.

[0122] The processor may be a central processing unit (CPU). The processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0123] As a non-transitory computer-readable storage medium, the memory can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as program instructions / modules corresponding to the method in the embodiment of the present disclosure. The processor executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory, that is, implementing the method in the above method embodiment.

[0124] The memory may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created by the processor, etc. In addition, the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0125] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0126] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the device, equipment, and storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0127] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

[0128] Although the embodiments of the present disclosure have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present disclosure, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A method for improving the quality of WebRTC video service, characterized in that: The method is applied to a WebRTC server, and the method comprises: In response to the session description proposal sent by the WebRTC client, sending a session description response including candidate information to the WebRTC client, the candidate information including a transport layer protocol type parameter and an Internet Protocol address parameter, the transport layer protocol type parameter including a transmission control protocol and a user datagram protocol; Receiving a session traversal request sent by the WebRTC client, and sending a session traversal reply to the WebRTC client, to complete the establishment of an interactive connection with the WebRTC client; wherein the session traversal request is determined by the WebRTC client according to the candidate information; Based on the interactive connection establishment, sending media data to the WebRTC client; Receiving statistical information about the media data fed back by the WebRTC client, the statistical information including delay information and packet loss information; According to the statistical information, a packet loss variation trend of the media data is determined, and based on the packet loss variation trend, a sending bit rate of the media data is adjusted.

2. The method according to claim 1, characterized in that Before sending media data to the WebRTC client based on the interactive connection establishment, the method further includes: Based on the Data Packet Transport Layer Security protocol, a key exchange is performed with the WebRTC client to generate a session key, where the session key is used to encrypt and decrypt the media data.

3. The method according to claim 1, characterized in that The sending of media data to the WebRTC client includes: In response to a real-time transport control protocol request sent by the WebRTC client, sending the media data to the WebRTC client; Alternatively, after performing a data packet transport layer security protocol interaction process with the WebRTC client, the encrypted media data is actively sent to the WebRTC client.

4. The method according to any one of claims 1 to 3, characterized in that: The adjusting the sending bit rate of the media data based on the packet loss change trend includes: Calculating a packet loss rate difference between packet loss rates at adjacent time nodes, wherein the packet loss rate difference represents a packet loss change trend in a target period; When the packet loss rate difference is greater than a preset threshold, determining an adjustment coefficient matching the packet loss rate difference; The sending bit rate is corrected based on the packet loss rate difference and the adjustment coefficient.

5. A method for improving the quality of WebRTC video service, characterized in that: The method is applied to a WebRTC client, and the method comprises: Sending a session description proposal to a WebRTC server, and receiving a session description response including candidate information returned by the WebRTC server, wherein the candidate information includes a transport layer protocol type parameter and an Internet Protocol address parameter, and the transport layer protocol type parameter includes a transmission control protocol and a user datagram protocol; Determine a session traversal request according to the candidate information, and send the session traversal request to the WebRTC server; Receive a session traversal reply returned by the WebRTC server, and complete interactive connection establishment with the WebRTC server; Based on the interactive connection establishment, receiving media data sent by the WebRTC server, and determining statistical information for the media data, the statistical information including delay information and packet loss information; The statistical information is sent to the WebRTC server, so that the WebRTC server determines a packet loss variation trend of the media data according to the statistical information, and adjusts a sending bit rate of the media data based on the packet loss variation trend.

6. The method according to claim 5, characterized in that Before the receiving the media data sent by the WebRTC server based on the establishment of the interactive connection, the method further includes: Based on the data packet transport layer security protocol, a key exchange is performed with the WebRTC server to generate a session key, where the session key is used to encrypt and decrypt the media data.

7. The method according to claim 5, characterized in that The receiving the media data sent by the WebRTC server includes: Sending a real-time transport control protocol request to the WebRTC server, and receiving the media data sent by the WebRTC server in response to the real-time transport control protocol request; Alternatively, after performing a data packet transport layer security protocol interaction process with the WebRTC server, the encrypted media data actively sent by the WebRTC client is received.

8. The method according to any one of claims 5 to 7, characterized in that: The adjusting the sending bit rate of the media data based on the packet loss change trend includes: Calculating a packet loss rate difference between packet loss rates at adjacent time nodes, wherein the packet loss rate difference represents a packet loss change trend in a target period; When the packet loss rate difference is greater than a preset threshold, determining an adjustment coefficient matching the packet loss rate difference; The sending bit rate is corrected based on the packet loss rate difference and the adjustment coefficient.

9. A device for improving the quality of WebRTC video service, characterized in that: The device is applied to a WebRTC server, and the device includes: a session description response unit, configured to send a session description response including candidate information to the WebRTC client in response to a session description proposal sent by the WebRTC client, wherein the candidate information includes a transport layer protocol type parameter and an Internet protocol address parameter, and the transport layer protocol type parameter includes a transmission control protocol and a user datagram protocol; A session traversal response unit, configured to receive a session traversal request sent by the WebRTC client, and send a session traversal reply to the WebRTC client, so as to complete the establishment of an interactive connection with the WebRTC client; wherein the session traversal request is determined by the WebRTC client according to the candidate information; A media data sending unit, configured to send media data to the WebRTC client based on the interactive connection establishment; A statistical information receiving unit, configured to receive statistical information for the media data fed back by the WebRTC client, wherein the statistical information includes delay information and packet loss information; The bit rate adjustment unit is used to determine the packet loss change trend of the media data according to the statistical information, and adjust the sending bit rate of the media data based on the packet loss change trend.

10. A device for improving the quality of WebRTC video services, characterized in that: The device is applied to a WebRTC client, and the device includes: a session description request unit, configured to send a session description proposal to a WebRTC server, and receive a session description response including candidate information returned by the WebRTC server, wherein the candidate information includes a transport layer protocol type parameter and an Internet Protocol address parameter, and the transport layer protocol type parameter includes a transmission control protocol and a user datagram protocol; A session traversal request unit, configured to determine a session traversal request according to the candidate information, and send the session traversal request to the WebRTC server; An interactive connection establishing unit, configured to receive a session traversal reply returned by the WebRTC server and complete interactive connection establishment with the WebRTC server; A media data receiving unit, configured to receive media data sent by the WebRTC server based on the interactive connection establishment, and determine statistical information for the media data, wherein the statistical information includes delay information and packet loss information; The statistical information sending unit is used to send the statistical information to the WebRTC server, so that the WebRTC server determines the packet loss change trend of the media data according to the statistical information, and adjusts the sending bit rate of the media data based on the packet loss change trend.

Images