Video frame loss and frame supplement transmission method, system and application based on WebRTC weak network environment

Abstract

The present invention discloses a video frame loss and frame supplement transmission method, system and application based on a WebRTC weak network environment. The transmission method comprises: S01, a transmitting end sends a video frame to a receiving end through a network controller; S02, the network controller counts the video frames and RCTP statistical information received by it, and updates the current network state parameters; S03, judging the network state information according to the current network state parameters and outputting it to the receiving end; S04, the receiving end receives the video frame and network state information sent by the network controller, makes a judgment according to the received video frame and network state information, and generates request change information and sends it to the transmitting end when the result meets the expectation; S05, the transmitting end receives the request change information and generates response information and sends it to the receiving end when it meets the expectation, the receiving end updates the receiving decoding parameters according to the response information and initializes the decoder, and the transmitting end updates the encoding parameters according to the change information and initializes the encoder; the implementation of this scheme is reliable, data interaction is flexible and the response is fast.

Description

Technical Field

[0001] The present invention relates to the technical fields of data communication and audio and video transmission, and in particular to a video frame loss and frame supplement transmission method, system and application based on a WebRTC weak network environment. Background Art

[0002] As a popular transmission technology nowadays, video data transmission is not only widely used in the field of surveillance, but also widely used in video calls in social occasions. However, the transmission of video data needs to rely on the support of the network environment. According to the applicant's survey and market feedback, it is common for the network environment to fluctuate or switch during video data transmission. Sometimes the network switching or network conditions are limited, which can easily cause a sharp drop in network bandwidth, resulting in the inability to transmit audio and video data normally. This situation is also a major pain point in the application of video data transmission and a technical difficulty that needs to be overcome urgently. At present, most network communication scenarios are wired networks. When there are weak or extremely weak network scenarios, most users turn a blind eye, have no way to complain, or try again. Even if a few manufacturers have made similar optimizations, the optimization effect is not ideal. They also use commonly used weak network countermeasure-related technologies, and the video fluency cannot be met, resulting in a very poor user experience. It should also be noted that the equipment performance of most service providers is uneven, and the hardware conditions cannot meet the extremely weak network scenario; at the same time, no real-time video interpolation mechanism has been introduced. Especially in the field of VoIP equipment, it is less used; therefore, how to ensure the stability and reliability of video transmission in a weak network environment is a very realistic topic. Summary of the invention

[0003] In view of this, the purpose of the present invention is to propose a video frame loss and frame supplement transmission method, system and application based on a WebRTC weak network environment with reliable implementation, flexible data interaction and fast response.

[0004] In order to achieve the above technical objectives, the technical solution adopted by the present invention is:

[0005] A video frame loss and frame supplement transmission method based on a WebRTC weak network environment, comprising:

[0006] S01, the sending end sends the video frame to the receiving end through the network controller;

[0007] S02, the network controller counts the video frames and RTCP statistics it receives, and then updates the current network status parameters;

[0008] S03, judging network status information according to current network status parameters, and then outputting the network status information to the receiving end;

[0009] S04, the receiving end receives the video frame and network status information sent by the network controller, and then makes a judgment based on the received video frame and network status information. When the judgment result meets the preset conditions, a request change information is generated and sent to the sending end;

[0010] S05. The sending end receives the request change information and generates a response message and sends it to the receiving end when the preset conditions are met. The receiving end updates the decoding parameters and initializes the decoder according to the response information. In addition, after generating the response information, the sending end updates the encoding parameters according to the request change information and initializes the encoder.

[0011] As a possible implementation mode, further, in the present solution S01, the sending end packages the video frames into video data packets and sends them to the receiving end through the network controller; on this basis, in the present solution S02, the network controller receives the video data packets sent by the sending end, and counts the video frames in the received video data packets and the video frames in the sent video data packets, and then updates the current network status parameters.

[0012] As a preferred implementation option, preferably, in this solution S04, the receiving end receives the video data packet and network status information sent by the network controller, then decodes the video frame in the video data packet and obtains the frame rate obtained by decoding the video data packet, and then makes a judgment based on the frame rate and network status information corresponding to the video data packet. When the judgment result meets the preset conditions, a request change information is generated and sent to the sending end through the network controller; wherein the network status information includes the video data packet sent by the sending end received by the network controller and the packet loss rate of sending the video data packet to the receiving end; in addition, the network jitter parameters can also be monitored and recorded, so that it can also be used as one of the reference factors for judging the network quality.

[0013] As a preferred implementation option, preferably, in the present solution S01, the sending end packages multiple video frames encoded to form a preset frame rate into a video data packet and sends it to the receiving end through a network controller; wherein the data in the video data packet includes video frames or video frames and audio frame data.

[0014] As a better implementation option, preferably, in this scheme S03, the judgment result includes a weak network environment and a non-weak network environment; wherein, when the frame rate is less than the preset threshold value, the packet loss rate is greater than the preset threshold value, and the delay is greater than the preset threshold value, the judgment result is a weak network environment, otherwise it is a non-weak network environment.

[0015] As a preferred implementation option, preferably, in this solution S04, when the judgment result is a weak network environment, the receiving end sends a TSTR message to the sending end to request the sending end to reduce the number of video frames of the data packets it sends, and the TSTR message is set to request change information.

[0016] As a preferred implementation option, preferably, in this solution S05, the sending end receives the TSTR message and generates a TSTN message and sends it to the receiving end when it meets the processing conditions. The receiving end updates the decoding processing parameters according to the TSTN message. In addition, after generating the TSTN message, the sending end updates the encoding processing parameters of the sent data packet according to the TSTR message; the TSTN message is set as the response information.

[0017] As a preferred implementation option, preferably, in the present solution S04, the receiving end performs decoding through a decoder, and when updating the decoding processing parameters, the decoder is initialized.

[0018] As a preferred implementation option, preferably, in the present solution S05, the transmitting end performs encoding through an encoder, and when updating the encoding processing parameters, the encoder is initialized.

[0019] As a preferred implementation option, preferably, in this scheme S04, when the judgment result is a weak network environment and the frame rate corresponding to the video data packet received by the receiving end is lower than the encoding frame rate of the sending end, the receiving end performs interpolation processing on the received video data packet so that the frame rate corresponding to the video data packet after the interpolation processing is equal to the encoding frame rate of the sending end or the difference is lower than a preset value.

[0020] As a preferred implementation option, preferably, in the present solution S01, the sending end sends the packaged video data packets to the receiving end in sequence via the network controller in a queue manner.

[0021] As a preferred implementation option, preferably, in this scheme S05, when the judgment result is a weak network environment and the sending end updates the encoding processing parameters of the sent data packet according to the TSTR message, the video data packet that has been packaged and not sent to the receiving end by the network controller is also subjected to frame loss processing, so that the frame rate corresponding to the video data packet after the frame loss processing matches the updated encoding frame rate.

[0022] Based on the above scheme, the present invention also provides a video frame loss and frame supplement transmission system based on a WebRTC weak network environment, which includes:

[0023] A collector configured to collect images or image and audio data;

[0024] An encoder configured to encode the image or the image and audio data collected by the collector into a video frame or a video frame and an audio frame;

[0025] A transmitter configured to receive the video frame or the video frame and the audio frame encoded by the encoder and encapsulate them into a video data packet, and then send it out;

[0026] a frame drop processor configured to perform video frame drop on a video data packet sent by the transmitter;

[0027] a network controller configured to receive video data packets sent by the transmitter and forward them;

[0028] A receiver configured to receive video data packets forwarded by a network controller;

[0029] a decoder configured to decode a video data packet received by the receiver;

[0030] The display is configured to play the video frames decoded by the decoder.

[0031] Among them, the collector, encoder, transmitter and frame loss processor constitute the transmitting end; the receiver, decoder, frame insertion processor and display constitute the receiving end.

[0032] Based on the above scheme, the present invention also provides a monitoring video or call video data transmission method, including the above-mentioned video frame loss and frame compensation transmission method based on the WebRTC weak network environment; in addition, the method of this scheme can also be optimized for VOIP devices in extremely weak network environments. It can effectively improve video playback when the network conditions between devices are very poor, thereby improving the VOIP video call experience.

[0033] Based on the above scheme, the present invention also provides a computer-readable storage medium, in which at least one instruction, at least one program, code set or instruction set is stored. The at least one instruction, at least one program, code set or instruction set is loaded and executed by a processor to implement the above-mentioned video frame loss and frame compensation transmission method based on a WebRTC weak network environment.

[0034] By adopting the above-mentioned technical scheme, the present invention has the following beneficial effects compared with the prior art: the scheme cleverly monitors the packet loss rate of the audio and video data packets when they are transmitted by the network controller and the frame rate after decoding by the receiving end, and uses the preset threshold value to judge the forward network transmission situation and quality, so as to judge whether the audio and video transmission environment is in a weak network or a non-weak network environment, so that when processing in a weak network environment, timely feedback can be given through the receiving end to realize the encoding control of the sending end, and by reducing the encoding frame rate and performing video data frame loss on the already packaged audio and video data packets, data simplification is realized on the upstream sending end, so that the audio and video data packets can realize stable and good data transmission in a weak network environment, and The receiving end at the downstream end optimizes the frame rate of the streamlined audio and video data packets through frame insertion processing, so that the audio and video quality after the final decoding and playback can present high-quality and smooth playback effects. This solution is not only reliable in implementation, but also flexible in mechanism coordination and high in response efficiency. It can provide a high-quality optimized transmission mechanism for network fluctuations during current video transmission. Not only as follows, this solution optimizes the problems existing in WebRTC, but also can use JitterBuffer partial framework adjustment, increase the judgment mechanism for removing the sequence number of retransmitted packets, give priority to sending video packets in weak network environments, and introduce TSTR (Temporal-Spatial The Trade-off Request (Trade-off Request) and TSTN (Temporal-Spatial Trade-off Notification) mechanisms are used to negotiate the frame rate information in the current network so that the frame rate of the current call can be reduced when the frame rate is insufficient (less than 10FPS); after the network conditions are improved (the statistical frame rate is greater than 15FPS), the frame rate is gradually increased and the previous configuration is restored, thereby effectively improving the video quality in a weak network environment; in addition, this solution determines the network status based on parameters such as packet loss, bit rate, and jitter. Under extremely weak network conditions (packet loss exceeds 50%, average jitter and delay exceed 250ms, and the weak network optimization effect is not ideal, resulting in an average frame rate of less than 10FPS), the encoding end reduces the frame rate, and the decoding end performs a combined strategy of intelligent frame insertion to ensure the improvement of video fluency, thereby improving the user's subjective fluency experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0036] Figure 1 It is a schematic diagram of a brief implementation process of the scheme of the present invention;

[0037] Figure 2 It is a simplified implementation connection diagram of the system of the present invention. DETAILED DESCRIPTION

[0038] The present invention will be further described in detail below in conjunction with the accompanying drawings and examples. It is particularly noted that the following examples are only used to illustrate the present invention, but are not intended to limit the scope of the present invention. Similarly, the following examples are only partial embodiments of the present invention rather than all embodiments, and all other embodiments obtained by those of ordinary skill in the art without creative work are within the scope of protection of the present invention.

[0039] like Figure 1 As shown, this solution is a video frame loss and frame supplement transmission method based on a WebRTC weak network environment, which includes:

[0040] S01, the sending end sends the video frame to the receiving end through the network controller;

[0041] S02, the network controller counts the video frames and RTCP statistics it receives, and then updates the current network status parameters;

[0042] S03, judging network status information according to current network status parameters, and then outputting the network status information to the receiving end;

[0043] S04, the receiving end receives the video frame and network status information sent by the network controller, and then makes a judgment based on the received video frame and network status information. When the judgment result meets the preset conditions, a request change information is generated and sent to the sending end;

[0044] S05. The sending end receives the request change information and generates a response message and sends it to the receiving end when the preset conditions are met. The receiving end updates the decoding parameters according to the response information and initializes the decoder. In addition, after generating the response information, the sending end updates the encoding parameters according to the request change information and initializes the encoder.

[0045] In order to facilitate the control of the frame rate during video playback, in this solution S01, the sending end packages multiple video frames into a video data packet and sends it to the receiving end through the network controller; on this basis, in this solution S02, the network controller receives the video data packet sent by the sending end, and counts the video frames in the received video data packet and the video frames in the sent video data packet, and then updates the current network status parameters.

[0046] In this way, the network controller and the receiving end can quickly determine the frame rate corresponding to the video data packet after receiving and decoding the video data packet. It also makes it easier for the network controller to count the packet loss rate, thereby achieving the data reception quality of the receiving end and the network status of the network controller.

[0047] In the present solution S01, the transmitting end packages the video frames encoded to form a preset frame rate into video data packets and then sends them to the receiving end through the network controller; wherein the data in the video data packets include video frames or video frames and audio frames. At the same time, in the present solution S01, the transmitting end sends the packaged video data packets to the receiving end in sequence through the network controller in a queue manner.

[0048] In addition, in the present scheme S04, the receiving end receives the video data packet and network status information sent by the network controller, then decodes the video frame in the video data packet and obtains the frame rate obtained by decoding the video data packet, and then makes a judgment based on the frame rate and network status information corresponding to the video data packet. When the judgment result meets the preset conditions, a request change information is generated and sent to the sending end through the network controller; wherein the network status information includes the video data packet sent by the sending end received by the network controller and the packet loss rate of sending the video data packet to the receiving end.

[0049] In order to facilitate the identification and judgment of network communication conditions, in this solution S03, the judgment results include weak network environment and non-weak network environment; among them, when the frame rate is less than the preset threshold value (for example, the frame rate is less than 10FPS) and the packet loss rate is greater than the preset threshold value (for example, the packet loss rate is greater than 50%), the judgment result is a weak network environment, otherwise it is a non-weak network environment; comprehensive factors such as jitter or delay causing decoding failure and inability to display can also be introduced, so that when the final display frame rate is less than 10FPS, it is judged as a weak network environment.

[0050] In a weak network environment, the limitations of network conditions make it impossible for the receiving end or network control end to carry more video data packets. If conventional weak network countermeasure technology is used, on the one hand, it will increase the network load, not only failing to improve the smoothness of the receiving end after decoding and playback, but more likely to cause network deterioration; on the other hand, conventional weak network countermeasure technology is more about improving network transmission related performance, and is powerless for scenarios where the network environment is limited and cannot meet the standards.

[0051] In order to enable the receiving end to adjust the transmission rate of the video data packet in time according to the current network conditions after receiving the video data packet to adapt to different network transmission qualities, in this solution S04, when the judgment result is a weak network environment, the receiving end sends a TSTR message to the sending end to request the sending end to reduce the number of video frames of the data packet it sends, and the TSTR message is set as request change information; and on this basis, in this solution S05, the sending end receives the TSTR message and generates a TSTN message and sends it to the receiving end when it meets the processing conditions, and the receiving end updates the decoding processing parameters according to the TSTN message. In addition, after generating the TSTN message, the sending end updates the encoding processing parameters of the sent data packet according to the TSTR message; the TSTN message is set as response information.

[0052] In order to avoid the decoder and encoder being unable to encode or decode normally after the processing parameters are updated, in this scheme S04, the receiving end decodes through the decoder, and the decoder is initialized when the decoding processing parameters are updated; at the same time, in this scheme S05, the sending end encodes through the encoder, and the encoder is initialized when the encoding processing parameters are updated.

[0053] In order to improve the smoothness of playback and avoid a significant decrease in the performance of audio and video data playback due to a weak network environment, in this solution S04, when the judgment result is a weak network environment and the frame rate corresponding to the video data packet received by the receiving end is less than the encoding frame rate of the sending end, the receiving end performs frame insertion processing on the received video data packet, so that the frame rate corresponding to the video data packet after the frame insertion processing is equal to the encoding frame rate of the sending end or the difference is less than a preset value; for example: the receiving end determines whether to start the frame insertion processor (for example, when the frame rate is less than 10FPS) based on the network status information fed back by the network control module. If it is less than 10FPS, a request TSTR to reduce the frame rate is sent to the sending end through the RTCP command, and the sending end is waited for to generate TSTN and send it to the receiving end. After receiving the TSTN message, the receiving end performs a decoder initialization action to complete the update of relevant data. At the same time, if the frame rate still does not reach 10FPS, the frame insertion processor is started to optimize the insertion of some video frames to maximize the proximity or equality to the currently received encoding frame rate, thereby improving the video smoothness. In this link, the interpolation technology can improve the fluency at the codec end and minimize the dependence on network conditions. Among them, when the receiving end performs interpolation processing through the interpolation processor, if the video frame in the video data packet is a scene switch, the key frame is reused; if it is not a scene switch, the linear motion vector estimation scheme is adopted to estimate the video frame to be inserted for the complete video frame, update PTS and other data, realize the construction of the video frame, and send it to the decoder to autonomously improve the fluency of the video call. Among them, the key frame can be pre-marked and set during encoding, for example, the first frame of each video data packet is set as the key frame, and the rest are non-key frames.

[0054] In order to improve the transmission efficiency of video data packets and avoid the situation where the network controller still occupies a large amount of resources when the network is in a weak network environment, in this solution S05, when the judgment result is a weak network environment and the sending end updates the encoding processing parameters of the sent data packet according to the TSTR message, the video data packet that has completed the packetization and has not been sent to the receiving end by the network controller is also subjected to frame loss processing, so that the frame rate corresponding to the video data packet after the frame loss processing matches the updated encoding frame rate, wherein the non-key frames in the data packet are subjected to frame loss processing, and after processing, a retransmission packet and a corresponding retransmission packet sequence are formed.

[0055] Similarly, in S04 of this solution, when the judgment result is a non-weak network environment, if the frame rate processing parameter decoded by the receiving end is less than the preset value and the frame rate counted by the current network controller is greater than the preset value (for example, 15FPS), that is, when it is judged that the network quality can meet the normal video data frame rate transmission, the receiving end sends a TSTR message to the sending end to request the sending end to increase the number of video frames of the data packet it sends, and the TSTR message is set as a request change information; and on this basis, in S05 of this solution, the sending end receives the TSTR message and generates a TSTN message and sends it to the receiving end when it meets the processing conditions, and the receiving end updates the decoding processing parameters according to the TSTN message. In addition, after generating the TSTN message, the sending end updates the encoding processing parameters of the sent data packet according to the TSTR message; the TSTN message is set as a response information, thereby enabling the receiving end and the sending end to adjust the frame rate of the video data packet and update the processing parameters of the encoder and decoder in a timely and efficient manner when the network environment fluctuates, thereby improving the humanization and adaptability of the scheme mechanism.

[0056] Based on the above scheme, the present invention also provides a monitoring video or call video data transmission method, including the above-mentioned video frame loss and frame supplementation transmission method based on the WebRTC weak network environment.

[0057] like Figure 2 As shown, based on the above scheme, the present invention also provides a video frame loss and frame supplement transmission system based on a WebRTC weak network environment, which includes:

[0058] A collector configured to collect images or image and audio data;

[0059] An encoder configured to encode the image or image and audio data collected by the collector into video frames or image frames and audio frames;

[0060] A transmitter configured to receive the video frame or the video frame and the audio frame encoded by the encoder and encapsulate them into a video data packet, and then send it out;

[0061] a frame drop processor configured to perform video frame drop on a video data packet sent by the transmitter;

[0062] a network controller configured to receive video data packets sent by the transmitter and forward them;

[0063] A receiver configured to receive video data packets forwarded by a network controller;

[0064] a decoder configured to decode a video data packet received by the receiver;

[0065] The display is configured to play the video frames decoded by the decoder.

[0066] Among them, the collector, encoder, transmitter and frame loss processor constitute the transmitting end; the receiver, decoder, frame insertion processor and display constitute the receiving end.

[0067] In one of the application implementation examples, the coordination relationship of the various components of the system of this solution is briefly described as follows:

[0068] R1 (display): This component plays the received video sequence and has a display function only.

[0069] S1 (collector): This component plays and collects camera data. The collected data is raw data, which is sent to the encoder for encoding and transmission.

[0070] R2 (decoder): This component is used to accept the video sequence decoding operation of the receiver, send the video sequence frame by frame to the bottom decoder to complete the video decoding, and deliver it to the display for display. Among them, the decoder needs to be reinitialized when the frame rate changes to prevent decoding failure.

[0071] S2 (encoder): This component is used to receive the image frames from the collector and perform encoding operations, send the image sequence frame by frame to the underlying encoder to complete the video encoding, and deliver it to the transmitter to complete the packet transmission. Among them, the encoder needs to be reinitialized when the frame rate changes to prevent the encoded data from being unable to be decoded at the receiving end.

[0072] R3 (receiver) This component is used to complete the reception of device video and audio frames, and to determine whether it is necessary to send a retransmission packet sequence to the sender after receiving the retransmission packet queue. This part will determine whether to start the interpolation processor (frame rate is less than 10FPS) based on the network status information fed back by the network control module. If it is less than 10FPS, it will send a request TSTR to reduce the frame rate to the sender through the RTCP command. After waiting for the TSTN message, it will initialize the decoder and complete the update of related data. At the same time, if the frame rate still does not reach 10FPS, the interpolation processor will be started to optimize the insertion of some video frames to maximize the closeness or equalness to the currently received encoding frame rate, thereby improving the video fluency.

[0073] S3 (sender): This component is used to complete the sending of device video and audio frames. This part will confirm whether to start the frame rate update operation based on the received TSTR request. If the conditions are met, it will send a TSTN message to the other party and reinitialize the encoder to complete the encoding frame rate update; determine whether to start the frame loss processor to send non-critical video frame data in the queue to reduce the current network's video frame payload.

[0074] R4 (frame insertion processor): This component is used to intelligently insert frames into data before decoding. In a weak network environment, when the received video frame rate does not meet the decoded video frame rate, intelligent frame insertion is performed to make the video sequence after insertion as close to or equal to the video frame rate of the encoding end as possible, so as to improve the current video fluency.

[0075] S4 (frame loss processor): This component is used to discard some video frames after receiving network control feedback in order to alleviate network congestion, achieve the negotiated frame rate, and ensure the effective transmission of key video frames. At the same time, it counts the video frame rate of the content sent over a period of time and feeds it back to the encoder to complete the encoding frame rate refresh.

[0076] N (Network Controller): This device receives and sends audio and video data packets, and performs data statistics based on the received data to update the current network status in order to determine whether the current network environment is extremely weak. If so, the frame insertion processor and frame loss processor are started at the same time to complete the pre-processing of decoding and encoding.

[0077] The system and transmission method of this solution can be implemented in computer languages ​​such as C++ or C, and then encapsulated into a so library, and the corresponding API can be directly called in specific application frameworks such as Android, iOS, and Linux, and then installed in a specific device for use.

[0078] In addition, the method of this solution can also be optimized for applications when VOIP devices are in extremely weak network environments. It can effectively improve video playback when the network conditions between devices are very poor, thereby improving the VOIP video call experience.

[0079] In the video call scenario, this solution is based on the WebRTC video call framework, introduces video frame insertion technology, and improves related processes to enhance the video call experience. Compared with traditional technologies, this solution has at least the following advantages:

[0080] (1) The original WebRTC framework does not contain any interpolation-related content. Under extremely weak network conditions, due to bandwidth limitations, increased packet loss, and increased jitter and delay, the video may freeze or stop playing.

[0081] (2) Even if WebRTC adds technologies related to weak network resistance, it still has limited improvement in extremely weak network conditions. The main reasons are as follows: when network bandwidth is prioritized and congestion is severe, packet loss and retransmission will aggravate network congestion; although there is a forward error correction function called FEC, it cannot guarantee effective resistance for the entire video call; although lowering the resolution can alleviate the problem to a certain extent, there will still be freezes and inability to play.

[0082] Therefore, the intelligent frame insertion technology is introduced. When facing a very poor network during a VoIP call, the sending and receiving ends first reduce the frame rate to give priority to the transmission of key frames; and insert some video frames through estimated calculation to improve the fluency of the call.

[0083] In terms of optimizing hardware requirements, this solution has been verified and tested on a phone device with a CPU main frequency of 1.8 GHz and a memory of 2 GB. It has adjusted the intelligent frame insertion for very poor network environments and achieved playback of less than 10 FPS based on the H264 encoding format. It can be seen that under the mechanism of this solution, the requirements for hardware performance during video calls can be optimized and a better working state can be maintained under weak network conditions.

[0084] Although some parts of the industry currently use the WebRTC framework for video transmission, it has not proposed a response mechanism for weak or extremely weak network environments. When the network becomes very poor, all products are stuck or directly display a black screen, and basically no improvement is made to the video fluency. In addition, although there are already solutions in the Internet industry that use optical flow to insert frames and improve fluency, and there are also solutions based on AI algorithm improvement, these require strong computing power and high performance to complete; and after investigation, the applicant found that the solutions are all based on the premise that the video fluency is relatively stable, and the solutions to increase the frame rate are not suitable for real-time call scenarios in weak or extremely weak networks.

[0085] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0086] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0087] The above descriptions are only some embodiments of the present invention, and are not intended to limit the protection scope of the present invention. Any equivalent device or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A video frame loss and frame supplement transmission method based on a WebRTC weak network environment, characterized in that: include: The sending end sends the video frame to the receiving end through the network controller; The network controller counts the video frames and RTCP statistics it receives, and then updates the current network status parameters; Determine the network status information according to the current network status parameters, and then output the network status information to the receiving end; The receiving end receives the video frame and network status information sent by the network controller, and then makes a judgment based on the received video frame and network status information. When the judgment result meets the preset conditions, it generates a request change information and sends it to the sending end; The sending end receives the request change information and generates a response information and sends it to the receiving end when the preset conditions are met. The receiving end updates the receiving decoding parameters and initializes the decoder according to the response information. In addition, after generating the response information, the sending end updates the encoding parameters according to the request change information and initializes the encoder; The transmitting end sends the packaged video data packets to the receiving end in sequence through the network controller in a queue manner; The judgment result includes a weak network environment and a non-weak network environment; wherein, when the frame rate is less than a preset threshold value, the packet loss rate is greater than a preset threshold value, and the delay is greater than a preset threshold value, the judgment result is a weak network environment, otherwise it is a non-weak network environment; When the judgment result is a weak network environment, the receiving end sends a TSTR message to the sending end to request the sending end to reduce the video frame rate of the data packet sent by it, and the TSTR message is set as request change information; The sending end receives the TSTR message and generates a TSTN message and sends it to the receiving end when it meets the processing conditions. The receiving end updates the decoding processing parameters according to the TSTN message. In addition, after generating the TSTN message, the sending end updates the encoding processing parameters of the sent data packet according to the TSTR message. When the judgment result is a weak network environment and the sending end updates the encoding processing parameters of the sent data packet according to the TSTR message, frame loss processing is also performed on the video data packet that has been packaged and not sent to the receiving end by the network controller, so that the frame rate corresponding to the video data packet after frame loss processing matches the updated encoding frame rate.

2. The video frame loss and frame supplement transmission method based on the WebRTC weak network environment as claimed in claim 1 is characterized in that: The sending end packages the video frames into video data packets and sends them to the receiving end through the network controller; The network controller receives the video data packets sent by the transmitter, counts the video frames and RTCP information in the received video data packets, and then updates the current network status parameters.

3. The video frame loss and frame supplement transmission method based on the WebRTC weak network environment as claimed in claim 2 is characterized in that: The receiving end receives the video data packet and network status information sent by the network controller, then decodes the video frame in the video data packet and obtains the frame rate obtained by decoding the video data packet, and then makes a judgment based on the frame rate and network status information corresponding to the video data packet. When the judgment result meets the preset conditions, it generates a request change information and sends it to the sending end through the network controller; The network status information includes the video data packets sent by the network controller and received by the sender and the statistical information obtained in the RTCP.

4. The video frame loss and frame supplement transmission method based on a WebRTC weak network environment as claimed in claim 3 is characterized in that: The transmitting end packages the video frames encoded to form a preset frame rate into video data packets and then sends them to the receiving end through the network controller; The data in the video data packet includes video frames or video frames and audio frames.

5. The video frame loss and frame supplement transmission method based on the WebRTC weak network environment as claimed in claim 4 is characterized in that: The receiving end performs decoding by a decoder, and when updating the decoding processing parameters, the decoder is initialized; The transmitting end performs encoding through an encoder, and when updating encoding processing parameters, the encoder is initialized.

6. The video frame loss and frame supplement transmission method based on a WebRTC weak network environment as claimed in claim 5, characterized in that: When the judgment result is a weak network environment and the frame rate corresponding to the video data packet received by the receiving end is lower than the encoding frame rate of the sending end, the receiving end performs frame interpolation on the received video data packet so that the frame rate corresponding to the video data packet after frame interpolation is equal to the encoding frame rate of the sending end or the difference is less than a preset value.

7. A video frame loss and frame supplement transmission system based on a WebRTC weak network environment, which is applied with the video frame loss and frame supplement transmission method based on a WebRTC weak network environment according to any one of claims 1 to 6, characterized in that: It includes: A collector configured to collect images or image and audio data; An encoder configured to encode the image or image and audio data collected by the collector into video frames or image frames and audio frames; A transmitter configured to receive the video frame or the video frame and the audio frame encoded by the encoder and encapsulate them into a video or video and audio data packet, and then send them out; a frame drop processor configured to perform video frame drop on a video data packet sent by the transmitter; a network controller configured to receive video data packets sent by the transmitter and forward them; A receiver configured to receive video data packets forwarded by a network controller; a decoder configured to decode a video data packet received by the receiver; The display is configured to play the video frames decoded by the decoder.

8. A method for transmitting monitoring video or call video data, characterized in that: It includes the video frame loss and frame supplement transmission method based on a WebRTC weak network environment as described in one of claims 1 to 6.

Images