QoS (Quality of Service) comprehensive processing method of video conference system

Through the QoS comprehensive processing method of the video conferencing system, using technologies such as streaming media receiver, FEC decoder and jitter buffer module, the QoS indicators are both taken into account in different network environments, and the stable and high-quality communication of video conferencing is achieved, and users are enhanced. Experience and reduce application costs.

CN118945147AActive Publication Date: 2024-11-12FUJIAN STAR NET WISDOM TECH CO LTD +1

Patent Information

Application Number
CN202410941971.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-15
Publication Date
2024-11-12
Estimated Expiration
2044-07-15

AI Technical Summary

Technical Problem

The existing video conferencing system cannot take into account multiple QoS indicators under different network environments, resulting in unstable communication quality and impact on user experience.

Method used

A comprehensive QoS processing method of video conferencing system is adopted to receive audio and video streams through streaming media receivers, and data classification and distribution are performed. Lost data is restored by using FEC decoder and jitter buffer module, and the jitter buffer length and redundancy rate are dynamically adjusted. , combined with RTCP feedback and RTP private protocol, dynamically adjust the video encoding rate and redundant configuration to ensure the quality of video conferencing.

Benefits of technology

Balance various service quality indicators in different network environments, improve video conferencing quality and user experience, low application cost, and easy integration of new QoS indicators or optimization strategies.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118945147A_ABST
    Figure CN118945147A_ABST
Patent Text Reader

Abstract

The invention provides a QoS comprehensive processing method of a video conference system, which is based on universal RTCP feedback and a part of RTP private protocol, obtains network data in real time through a transmission statistics module and a broadband detection module, adjusts the FEC adaptive redundancy rate in real time according to the current network change condition, improves the FEC recovery rate, and improves the quality of the video conference system. And meanwhile, the jitter buffer length is dynamically adjusted according to changing network conditions, so that the frames can be smoothly output in a weak network environment. The method is low in application cost and high in network adaptability, the video conference quality is improved, and the user experience is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the technical field of video communication, and in particular to a QoS comprehensive processing method for a video conference system. Background Art

[0002] The main function of the video conferencing system is to centrally process the audio and video streams of different conference terminals (mixing, screen mixing, transcoding, etc.) and distribute them to other participating terminals. However, video conferencing faces many challenges under different network conditions, such as network delay, bandwidth limitation, packet loss rate, and jitter. These problems may cause blurry images, sound delays, and freezes in video conferencing, seriously affecting the user experience.

[0003] In order to provide stable and high-quality video conferencing services, multiple QoS indicators need to be considered comprehensively to ensure an excellent user experience under various network conditions. The current technologies used in video conferencing system QoS (quality of service) cover many aspects, mainly including: network bandwidth management, traffic adjustment and priority, network congestion control, packet loss recovery and error correction, etc. The main shortcomings of some existing video conferencing optimization strategies are:

[0004] First, lack of adaptability. Some QoS technologies may only be effective in certain network environments or situations, but may not work optimally in other environments. Lack of adaptability may result in the inability to achieve stable communication quality under different network conditions.

[0005] Second, the overall effect is not good. Usually, only optimizing a certain QoS indicator may have a negative impact on other indicators. For example, focusing only on reducing latency may lead to excessive bandwidth usage, thus affecting video quality.

[0006] In view of the shortcomings of the existing relatively single and fixed QoS solutions, the present invention proposes a comprehensive QoS processing strategy to effectively improve the user experience of video conferencing. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a QoS comprehensive processing method for a video conferencing system to overcome the problem that various QoS indicators cannot be taken into account in different network environments, provide stable and high-quality video communications, and improve the user experience of video conferencing.

[0008] The problem of the present invention is achieved as follows: A QoS comprehensive processing method for a video conferencing system comprises the following steps:

[0009] Step S1: Receive the uplink audio and video stream through the streaming media receiver, classify and distribute the received data, and count the bandwidth and number of received packets of different types of messages and aggregate them to the transmission statistics module;

[0010] Step S2: The FEC decoder compares the redundant information with the cached original message and redundant message, restores the lost original message, and sends it to the jitter buffer module;

[0011] Step S3: sort the received original messages through the jitter buffer module, determine whether packet loss occurs through the RTP sequence number, and feed back the packet loss sequence number to the transmission statistics module, while dynamically adjusting the jitter buffer length and sending the buffered RTP messages in frames to the error concealment module;

[0012] Step S4: Determine whether to generate padding data according to the continuity of the RTP sequence number through the error concealment module. If the RTP sequence number is discontinuous, the audio sample values ​​between adjacent frames are interpolated to create new audio frames to fill the lost frames. If the RTP sequence number is continuous, no padding data is required. After that, the continuous media data is sent to the audio and video decoder for decoding, and the decoded data is processed according to different conference scenarios and then sent to the audio and video encoder.

[0013] Step S5: When the RTCP processing module parses the key frame request FIR / PLI, it directly notifies the audio and video encoder to generate a key frame through the transmission statistics module;

[0014] Step S6: The transmission statistics module transmits the currently counted downlink bandwidth, downlink delay recv_delta data and the current downlink packet loss rate to the bandwidth detection module. The bandwidth detection module evaluates the current downlink bandwidth based on the statistical information, and refers to the downlink packet loss situation to modify the video coding rate and FEC redundancy configuration information in real time.

[0015] Step S7, the audio and video encoder encodes the received data according to the modified video coding rate and passes it to the FEC encoder. The FEC encoder dynamically adjusts the redundancy rate according to the data received by the bandwidth detection module, adds redundant packet group information to the original message data, inserts a specified number of redundant packets in each redundant packet group according to the current redundancy rate, and sends them to the streaming media sender smoothly through the sending rate control module.

[0016] Furthermore, the FEC redundancy configuration information includes a ratio of redundant messages to original messages in the FEC packet group.

[0017] Furthermore, the step S1 of "classifying and distributing the received data" specifically includes: dividing the received messages into RTP messages and RTCP messages, wherein the RTP messages are divided into redundant messages, original messages and detection filling messages, wherein the redundant messages are directly passed to the FEC decoder, a copy of the original messages is given to the FEC decoder and placed in the jitter buffer module, and the RTCP messages are directly sent to the RTCP message processing module for parsing.

[0018] Furthermore, the step S5 also includes an RTCP processing module that obtains downlink jitter, delay rtt, and congestion control delay information recv_delta by parsing RR, SR, and TWCC messages, and aggregates them to the transmission statistics module, and when parsing the packet loss retransmission request NACK, notifies the media stream sender to retransmit the corresponding RTP message, wherein the NACK request interval control adopts dynamic interval control, and the calculation formula of the dynamic interval interval is: interval = 0.6*current interval + 0.4*rtt, wherein the initial value of interval is 30ms.

[0019] Furthermore, the specific process of restoring the lost original message in step S2 is: extending the private RTP protocol, carrying the packet group sequence number, the number of original packets in the packet group and the position information of the current message in the packet group in the RTP header extension of each original message;

[0020] The number of original packets k and the number of redundant packets r in the preset packet group, when the number of original packets + redundant packets received by the receiving end in the packet group reaches k, the FEC decoder recovers all messages in the packet group in combination with the NACK strategy.

[0021] Furthermore, the specific implementation method of dynamically adjusting the jitter buffer length in step S3 is:

[0022] The jitter buffer module sorts the received data packets according to their timestamp information and verifies the sequence number to confirm the packet loss;

[0023] Put the sorted data packets into the jitter buffer queue jitterbuffer and index them by the packet sequence number seq;

[0024] The jitter buffer length is dynamically adjusted according to the changing network conditions. The specific adjustment method is as follows:

[0025] The minimum buffer length is preset to 140ms and the maximum buffer length is 1s;

[0026] For audio, if one or more packet reading fails within a 250-packet reading cycle, the current jitter buffer length is increased by one; if 250 consecutive packet readings are successful, the current jitter buffer length is reduced by one;

[0027] For video, in a weak network environment, the jitter buffer length is increased, and after the network is restored, the jitter buffer length is reduced; the judgment condition of the weak network environment is: when a frame reading timeout or frame loss occurs, it is determined to be a weak network environment.

[0028] Furthermore, for video, the specific process of adjusting the jitter buffer length is as follows:

[0029] For weak network environments, set the frame reading cycle as the maximum buffer length, and adjust the jitter buffer length once in each frame reading cycle; if the frame loss is caused by incomplete frames, the jitter buffer length is increased by 1 in a regular manner, that is, when an incomplete frame occurs in each frame reading cycle, the jitter buffer length is increased by 1. If the number of complete frames in the current buffer queue accounts for 2 / 3, the jitter buffer length stops increasing; if there is no frame loss in the cycle, the buffer length is reduced in a regular manner, that is, the buffer length is reduced by 1 in each frame reading cycle. If 1 / 4 of the frames in the cycle are read out of time, the jitter buffer length stops decreasing;

[0030] In the early stage of joining the conference, the jitter buffer length is increased by 1 without being controlled by the frame reading cycle. However, if the number of frames exceeds 1 / 3 of the current buffer queue, the jitter buffer length is adjusted once per frame reading cycle.

[0031] For a large packet loss scenario with a sudden packet loss rate greater than 3%, the large packet loss scenario control strategy is as follows:

[0032] If the packet loss rate is lower than 10%, or the current jitter buffer queue length exceeds 2 / 3 of the maximum buffer length, the jitter buffer length increase is reduced to 1;

[0033] If the packet loss rate is in the range of [10%, 15%], increase the increase and set the jitter buffer length increase to packet loss rate - 10;

[0034] If the packet loss rate is greater than 15%, the jitter buffer length is increased rapidly, and the increase is set to 10;

[0035] The packet loss rate refers to the packet loss rate inside the current buffer queue.

[0036] Furthermore, the processing in step S4 includes mixing audio and / or mixing screen data after audio and video decoding.

[0037] Furthermore, the specific method of dynamically adjusting the redundancy rate in step S7 is as follows:

[0038] The FEC encoding end receives real-time bandwidth data inferred from the bandwidth detection module and dynamically adjusts the redundancy rate. Specifically, the maximum redundancy rate is set to (detection bandwidth-coding bit rate) / coding bit rate. When the downlink packet loss rate is less than 3%, the redundancy rate is adjusted to the maximum redundancy rate / 4; when the downlink packet loss rate is between 3% and 10%, the redundancy rate is adjusted to the maximum redundancy rate / 2; when the downlink packet loss rate is above 10%, the redundancy rate is adjusted to the maximum redundancy rate.

[0039] Furthermore, step S7 also includes the sending rate control module feeding back the existing downlink packet sending situation to the transmission statistics module.

[0040] Furthermore, the sending rate control module uses a sliding window algorithm to control the smooth sending of data packets, reduce the risk of packet loss caused by insufficient bandwidth, and control the sending priority of data packets to: retransmission packet>audio packet>video packet.

[0041] The advantages of the present invention are: based on the general RTCP feedback and part of the RTP private protocol, the dynamic jitter buffer and adaptive redundancy processing of the present invention are realized, so that the video conference can adapt to different network environments, balance various service quality indicators, improve the quality of the video conference, and enhance the user's comprehensive experience. At the same time, the application cost of the method of the present invention is low, and new QoS indicators or new optimization strategies can be relatively easily integrated into the method.

[0042] The accompanying drawings illustrate the advantages of low application cost.

[0043] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0044] Figure 1 The present invention is a flowchart of the execution of a QoS comprehensive processing method for a video conferencing system.

[0045] Figure 2 The present invention is an overall block diagram of a QoS comprehensive processing method for a video conferencing system.

[0046] Figure 3 It is a schematic diagram of the FEC-RTP header extension structure of the present invention. DETAILED DESCRIPTION

[0047] In order to make the present invention more clearly understood, a preferred embodiment is now described in detail with reference to the accompanying drawings.

[0048] like Figure 1 As shown, a QoS comprehensive processing method for a video conferencing system of the present invention comprises the following steps:

[0049] Step S1, receiving the uplink audio and video stream through the streaming media receiver, classifying and distributing the received data, specifically, dividing the received messages into RTP messages and RTCP messages, wherein the RTP messages are divided into redundant messages, original messages and detection filling messages (the content of the messages is fixed, there is no actual media data, and the receiver can directly discard it after receiving it), and directly passing the redundant messages to the FEC decoder, and copying a copy of the original message to the FEC decoder, and putting it into the jitter buffer module at the same time, and directly sending the RTCP message to the RTCP message processing module for parsing. The above-mentioned streaming media receiver also counts the bandwidth and number of received packets of different types of messages and aggregates them to the transmission statistics module;

[0050] Step S2: The FEC decoder compares the redundant information with the cached original message and redundant message, restores the lost original message, and sends it to the jitter buffer module;

[0051] Step S3: sort the received original messages through the jitter buffer module, determine whether packet loss occurs through the RTP sequence number, and feed back the packet loss sequence number to the transmission statistics module, while dynamically adjusting the jitter buffer length and sending the buffered RTP messages in frames to the error concealment module;

[0052] Step S4, the error concealment module determines whether it is necessary to generate filling data according to the continuity of the RTP sequence number (the message passed to the error concealment module at this stage is sorted by the jitter buffer module, and the continuity is judged by comparing whether the current sequence number is continuous with the last incoming message sequence number). If the RTP sequence number is discontinuous, the audio sample values ​​between adjacent frames are interpolated to create new audio frames to fill the lost frames. If the RTP sequence number is continuous, no filling data is required; thereafter, the continuous media data is sent to the audio and video decoder for decoding, and the decoded data is processed according to different conference scenarios and then sent to the audio and video encoder; the above processing includes mixing and screen mixing of the decoded audio and video data.

[0053] Step S5: When the RTCP processing module parses the key frame request FIR / PLI, it directly notifies the audio and video encoder to generate a key frame through the transmission statistics module;

[0054] Step S6: The transmission statistics module transmits the currently counted downlink bandwidth, downlink delay recv_delta data and the current downlink packet loss rate to the bandwidth detection module. The bandwidth detection module evaluates the current downlink bandwidth based on the statistical information, and refers to the downlink packet loss situation to modify the video coding rate and FEC redundancy configuration information in real time.

[0055] Step S7, the audio and video encoder encodes the received data according to the modified video coding rate and passes it to the FEC encoder. The FEC encoder dynamically adjusts the redundancy rate according to the data received by the bandwidth detection module, adds redundant packet group information to the original message data, inserts a specified number of redundant packets in each redundant packet group according to the current redundancy rate, and sends them to the streaming media transmitter smoothly by the sending rate control module. The sending rate control module uses a sliding window algorithm to control the smooth sending of data packets, reduces the risk of packet loss caused by insufficient bandwidth, and controls the sending priority of data packets to be: retransmission packet>audio packet>video packet. In addition, the sending rate control module feeds back the existing downlink packet sending situation to the transmission statistics module.

[0056] Preferably, the FEC redundancy configuration information includes the ratio of redundant messages to original messages in the FEC packet group.

[0057] Preferably, step S5 also includes an RTCP processing module that parses RR, SR, and TWCC messages to obtain downlink jitter, delay rtt, and congestion control delay information recv_delta, which are aggregated to the transmission statistics module, and when parsing the packet loss retransmission request NACK, the media stream transmitter is notified to retransmit the corresponding RTP message, wherein the NACK request interval control adopts dynamic interval control, and the calculation formula of the dynamic interval interval is: interval = 0.6*current interval + 0.4*rtt, wherein the initial value of interval is 30ms.

[0058] The above-mentioned dynamic interval control of NACK request is implemented through the transmission statistics module. As the statistical data aggregation module of the entire system, the transmission statistics module is used to set the NACK request interval control in addition to saving the transmission data statistical information. The traditional NACK request interval is a fixed interval, that is, NACK is requested according to a fixed time period. In order to avoid the serious increase in network bandwidth caused by frequent responses to retransmission packets, this period is usually set to a relatively large value, such as 100ms. Then, when the retransmission packet is lost again during the transmission process, the video frame needs to wait for a long time to be complete, and the overall delay will increase. In order to reduce the delay caused by packet loss and retransmission, we optimize the interval control to a dynamic interval. When packet loss is detected, the sequence packet initiates the first NACK request after 30ms (to avoid redundant retransmission processing in a small jitter scenario), and then repeats the request according to the dynamic interval interval until the lost message is received or the number of requests reaches the upper limit of 10 times. The dynamic interval interval is related to the statistical delay rtt. From the receiving end requesting NACK to the sending end responding to the retransmission packet and then to the receiving end receiving the retransmission packet, this cycle length can be determined to be close to the statistical rtt value. When the retransmission request interval is lower than this period, there will be redundant NACK requests, and the extra retransmission packets will cause bandwidth waste. Therefore, we set the above dynamic interval calculation method to: interval = 0.6*current interval + 0.4*rtt (the initial interval is 30ms).

[0059] Preferably, the specific process of restoring the lost original message in step S2 is: extending the private RTP protocol, carrying the packet group sequence number, the number of original packets in the packet group and the position information of the current message in the packet group in the RTP header extension of each original message;

[0060] An example of an extended header structure is Figure 3As shown in the figure, SBN is the sequence number of the packet group, which is used to identify that the redundant packet and the original packet belong to the same packet group, that is, to identify which packet group composed of the original packet + the redundant packet the packet belongs to. ESI is the position of the current packet in the packet group, in the order of the original packet first and the redundant packet later. For example, a packet group composed of 10 original packets and 5 redundant packets, the position sequence number in the group starts from 0, then the original packet position is 0-9, and the redundant packet position is 10-14. In the picture, k is the number of original packets in the current packet group, and pad is the padding bit, which is used to fill 4 bytes. The condition for the packet group to be fully recovered: the total number of packets received in the current packet group is greater than or equal to the original number k of packets in the packet.

[0061] Packet group content: k original messages (i.e. original packets), plus r redundant messages (i.e. redundant packets);

[0062] Redundancy rate calculation: number of redundant messages r / number of original messages k;

[0063] The number of original packets k and the number of redundant packets r in the preset packet group (i.e., redundant packet group) are set. When the number of original packets + redundant packets received by the receiving end in the packet group reaches k, the FEC decoder recovers all messages in the packet group in combination with the NACK strategy. The decoding end adjusts the interval between two consecutive nack requests for the same message according to the statistical transmission interval. Here, the number of redundant messages r does not need to be transmitted through messages. The receiving end only needs to receive the number of messages that reaches the number of original messages to meet the recovery condition.

[0064] Preferably, the specific method of dynamically adjusting the redundancy rate in step S7 is as follows:

[0065] The FEC encoding end receives real-time bandwidth data inferred from the bandwidth detection module and dynamically adjusts the redundancy rate. Specifically, the maximum redundancy rate is set to (detection bandwidth-coding bit rate) / coding bit rate. When the downlink packet loss rate is less than 3%, the redundancy rate is adjusted to the maximum redundancy rate / 4; when the downlink packet loss rate is between 3% and 10%, the redundancy rate is adjusted to the maximum redundancy rate / 2; when the downlink packet loss rate is above 10%, the redundancy rate is adjusted to the maximum redundancy rate.

[0066] The FEC (forward error correction unit) of the present invention is implemented through a private RTP protocol extension. By carrying the redundant packet group sequence number, the number of original packets in the packet group, and the number information of the message in the packet group in the RTP header extension of each original message, the receiving end can parse the redundant information required for FEC decoding when receiving any message in the packet group, so as to achieve the purpose of flexibly changing the redundancy rate during the meeting without the need for secondary negotiation. The FEC recovery judgment rule in this scheme is: the number of original packets in the redundant packet group k, the number of redundant packets r, when the receiving end receives the number of original packets + redundant packets in the packet group reaches k, all messages in the packet group can be restored. Among them, the FEC decoding end combines the NACK strategy of the present application, and both the original packet and the retransmitted packet need to be placed in the FEC decoder, which speeds up the time for the packet group to reach the recovery standard, reduces the packet group recovery delay, and makes it easier for the packet group to reach the recovery standard, thereby improving the FEC recovery rate. The real-time bandwidth data inferred from the bandwidth detection module is received at the FEC encoding end, and the redundancy rate is dynamically adjusted in combination with the current packet loss rate to adapt to changes in the network environment.

[0067] Preferably, the specific implementation method of dynamically adjusting the jitter buffer length in step S3 is:

[0068] The jitter buffer module sorts the received data packets according to their timestamp information and verifies the sequence number to confirm the packet loss;

[0069] Put the sorted data packets into the jitter buffer queue jitterbuffer and index them by the packet sequence number seq;

[0070] The jitter buffer length is dynamically adjusted according to the changing network conditions. A larger jitter buffer can handle greater delay jitter and more severe disorder, but it will introduce greater processing delay. This solution uses a dynamic buffer length method to adapt to changing network conditions. The specific adjustment method is as follows:

[0071] The minimum buffer length is preset to 140ms (7 audio frames, 4 video frames), and the maximum buffer length is 1s (50 audio frames, 30 video frames);

[0072] For audio, if one or more packet reading fails within a 250-packet reading cycle, the current jitter buffer length frame_len is increased by one; if 250 consecutive packet readings are successful, the current jitter buffer length frame_len is reduced by one;

[0073] For video, in a weak network environment, in order to reduce jamming, the number of cached frames is increased to ensure that the frames can be output smoothly, and after the network is restored, the number of cached frames is reduced to reduce latency; wherein, the judgment condition for a weak network environment is: when a frame read timeout occurs (i.e., the current frame read interval is larger than the frame interval calculated by the negotiated frame rate, indicating that a frame read timeout exists) or a frame is lost, it is determined to be a weak network environment; specifically, for a weak network environment, the method for adjusting the number of cached frames is as follows: set the frame read cycle to the maximum number of cached frames (i.e., the maximum buffer length), adjust the number of cached frames once in each frame read cycle (i.e., "increase|decrease" once in the frame read cycle), and during adjustment, if it is caused by an incomplete frame. If a frame is lost, the jitter buffer length will increase by 1 as usual, that is, when an incomplete frame appears in each frame reading cycle, the jitter buffer length will increase by 1. If it is in the early stage of joining the meeting, the growth will be accelerated (faster growth is required in the early stage of joining the meeting, ignoring the configured frame reading cycle), that is, the jitter buffer length will increase by 1 every time a frame is lost. If the number of frames exceeds 1 / 3 of the current buffer length, the growth will be slowed down, and the jitter buffer length will be adjusted once per frame reading cycle; if the number of complete frames in the current jitter buffer reaches 2 / 3, the growth will stop. If no frame is lost in the cycle, the normal reduction will be performed, that is, the buffer length will decrease by 1 in each frame reading cycle. If 1 / 4 of the frames in the cycle are read out of time, the reduction will stop (extremely low probability).

[0074] For a large packet loss scenario with a sudden packet loss rate greater than 3%, the large packet loss scenario control strategy is as follows:

[0075] Note: The following packet loss rate refers to the internal packet loss rate of JB.

[0076] When the packet loss rate is lower than 10%, or frame_len is close to the upper limit of JB length (i.e. the current jitter conversion length has exceeded 2 / 3 of the maximum buffer length), the increase is reduced to 1. Usually 4 video frames and 7 audio frames are cached, and the cache time is close to 140ms. Therefore, the maximum length of video jitter buffer is set to 30 frames and audio jitter buffer is set to 50 frames, that is, the maximum buffer length is 1s. When the video buffer length exceeds 2 / 3, i.e. 20 frames, the growth will slow down (the increase is 1 per cycle).

[0077] The packet loss rate is in the range of [10,15]. The increase is increased by packet loss rate lost_rate - 10.

[0078] The packet loss rate is greater than 15% and increases rapidly by 10.

[0079] Preferably, the implementation of the bandwidth detection module of the present invention is based on the open source Send-side BWE algorithm, and the downlink bandwidth detection value is evaluated based on the statistical congestion control delay information and the current downlink bit rate. The present invention mainly realizes the control of FEC redundancy rate and encoding bit rate based on the detection value. The detected bandwidth is mainly used for redundant packet bit rate + original packet bit rate. The original packet bit rate is effective by modifying the encoding bit rate of the encoder. The bit rate of the redundant packet needs to modify the redundancy rate, that is, the ratio of the redundant packet to the original packet is effective. The larger the redundant packet bit rate, the better the anti-packet loss effect; the larger the encoding bit rate, the clearer the picture quality. Therefore, the present invention provides users with two configuration methods for the ratio of the two: picture quality priority or fluency priority; a. Picture quality priority: the ratio of redundant packet bit rate to encoding bit rate is 3:10; b. Fluency priority: the ratio of redundant packet bit rate to encoding bit rate is 1:1; the configuration provided is based on a single participating terminal as a unit to achieve a flexible bit rate control effect.

[0080] The following is combined with Figure 2 The comprehensive QoS strategy data flow interaction of the video conferencing system of the present invention is further described:

[0081] like Figure 2 As shown, the modules involved in the present invention include a media stream receiver, a media stream transmitter, a transmission statistics module, a broadband detection module, a jitter buffer module, an RTCP message processing module, an error concealment module, an FEC decoder, an FEC encoder, an audio and video decoder, an audio and video encoder, a conference audio / image processing module and a transmission rate control module. The interaction process between the modules is as follows:

[0082] 1. The uplink audio and video streams of the participating terminals are uniformly received through the media stream receiver. The media stream receiver can classify the received message information, mainly RTP messages and RTCP messages. Among them, the RTP message is divided into redundant messages, original messages, and detection filling messages (the content of the message is fixed, there is no actual media data, and the receiver can directly discard it after receiving it). The redundant message is directly passed to the FEC decoder, and a copy of the original message needs to be given to the FEC decoder and placed in the jitter buffer module (corresponding to Figure 2 The media stream receiver also counts the bandwidth and number of packets received for different types of messages, and sends the statistics to the transmission statistics module (corresponding to Figure 2 The RTCP message is directly sent to the RTCP message processing module for parsing (corresponding to Figure 2 Arrow No. 4).

[0083] 2. The FEC decoder compares the redundant information with the cached original message and redundant message, restores the lost original message, and sends it to the jitter buffer module (corresponding to Figure 2 Arrow No. 5).

[0084] 3. Sort the received original messages through the jitter buffer module, determine whether packet loss occurs through the RTP sequence number, and feedback the packet loss sequence number to the transmission module (corresponding to Figure 2 The buffered RTP message is sent to the error concealment module in frames (corresponding to Figure 2 Arrow No. 7).

[0085] 4. The error concealment module determines whether to generate filler data based on the continuity of the RTP sequence number. If necessary, the audio sample values ​​between adjacent frames are interpolated to create new audio frames to fill the lost frames. Finally, the error concealment module sends the continuous media data to the audio and video decoder for processing (corresponding to Figure 2 The error concealment unit is the final processing before decoding the audio and video data. By comparing the data difference before and after the lost packet, a virtual data packet is constructed and sent to the decoder to reduce the subjective experience impact caused by packet loss.

[0086] 5. According to different conference scenarios, the conference audio / image processing module mixes the audio and video decoded data, mixes the screen, etc., and encodes it into data to be sent (corresponding to Figure 2 Arrow No. 9).

[0087] 6. The RTCP processing module obtains the downlink jitter, delay rtt, and congestion control delay information recv_delta by parsing the RR, SR, and TWCC messages, and all of them are aggregated to the transmission statistics module (corresponding to Figure 2 When parsing the packet loss retransmission request NACK, notify the media stream sender to retransmit the corresponding RTP message (corresponding to Figure 2 When the key frame request FIR / PLI is parsed, the encoder is notified to generate a key frame directly through the parameter statistics module (corresponding to Figure 2 Arrow No. 17).

[0088] 7. The transmission statistics module passes the current statistics of downlink bandwidth, downlink delay recv_delta data and current downlink packet loss rate to the bandwidth detection module (corresponding to Figure 2 Arrow number 11).

[0089] 8. The bandwidth detection module evaluates the current downlink bandwidth based on the statistical information, and refers to the downlink packet loss situation to modify the video encoding bit rate in real time (corresponding to Figure 2 Arrow 12) and FEC redundancy configuration information (corresponding to Figure 2 Arrow No. 14).

[0090] 9. The encoded data needs to be passed to the FEC encoder first (corresponding to Figure 2The redundancy packet group information is added to the original message data for the receiving end to recover the lost data. A specified number of redundancy packets are inserted into each redundancy packet group according to the current redundancy rate, and the sending rate control module sends them smoothly (corresponding to Figure 2 Arrow No. 15).

[0091] 10. The sending rate control module feeds back the current downlink packet sending status to the transmission statistics module (corresponding to Figure 2 The media packets are sent uniformly through the media stream sender (corresponding to Figure 2 Arrow No. 18).

[0092] The QoS comprehensive strategy of the video conferencing system of the present invention is based on the general RTCP feedback, and part of the RTP private protocol to implement the dynamic jitter buffer and adaptive redundancy processing of the present invention. It can balance various service quality indicators under different network environments, ensure the communication quality of participating terminals under different network conditions, and improve the comprehensive user experience. At the same time, the application cost of the method of the present invention is low. When new QoS indicators or new optimization strategies appear in the future, they can be relatively easily integrated into the method to ensure that the system maintains a leading position in the process of technological evolution. Since the optimization strategies of different processing nodes can be replaced separately, such as replacing the FEC algorithm separately, it is only necessary to standardize the data transmission process, input the original message + redundant message, and output the restored original message, so as to realize the arbitrary replacement of the FEC algorithm without changing other modules of the original system.

[0093] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A QoS comprehensive processing method for a video conferencing system, Features: The steps include: Step S1: Receive the uplink audio and video stream through the streaming media receiver, classify and distribute the received data, and count the bandwidth and number of received packets of different types of messages and aggregate them to the transmission statistics module; Step S2: The FEC decoder compares the redundant information with the cached original message and redundant message, restores the lost original message, and sends it to the jitter buffer module; Step S3: sort the received original messages through the jitter buffer module, determine whether packet loss occurs through the RTP sequence number, and feed back the packet loss sequence number to the transmission statistics module, while dynamically adjusting the jitter buffer length and sending the buffered RTP messages in frames to the error concealment module; Step S4: Determine whether to generate padding data according to the continuity of the RTP sequence number through the error concealment module. If the RTP sequence number is discontinuous, the audio sample values ​​between adjacent frames are interpolated to create new audio frames to fill the lost frames. If the RTP sequence number is continuous, no padding data is required. After that, the continuous media data is sent to the audio and video decoder for decoding, and the decoded data is processed according to different conference scenarios and then sent to the audio and video encoder. Step S5: When the RTCP processing module parses the key frame request, it directly notifies the audio and video encoder to generate the key frame through the transmission statistics module; Step S6: The transmission statistics module transmits the currently counted downlink bandwidth, downlink delay data and the current downlink packet loss rate to the bandwidth detection module. The bandwidth detection module evaluates the current downlink bandwidth based on the statistical information, and refers to the downlink packet loss situation to modify the video coding rate and FEC redundancy configuration information in real time. Step S7, the audio and video encoder encodes the received data according to the modified video coding rate and passes it to the FEC encoder. The FEC encoder dynamically adjusts the redundancy rate according to the data received by the bandwidth detection module, adds redundant packet group information to the original message data, inserts a specified number of redundant packets in each redundant packet group according to the current redundancy rate, and sends them to the streaming media sender smoothly through the sending rate control module.

2. A QoS comprehensive processing method for a video conferencing system as claimed in claim 1, Features: In the step S1, "classifying and distributing the received data" specifically includes: classifying the received messages into RTP messages and RTCP messages, wherein the RTP messages are divided into redundant messages, original messages and detection filling messages, wherein the redundant messages are directly passed to the FEC decoder, a copy of the original messages is given to the FEC decoder and simultaneously put into the jitter buffer module, and the RTCP messages are directly sent to the RTCP message processing module for parsing.

3. A QoS comprehensive processing method for a video conferencing system as claimed in claim 1, Features: The step S5 also includes the RTCP processing module obtaining downlink jitter, delay rtt, and congestion control delay information recv_delta by parsing RR, SR, and TWCC messages, and aggregating them to the transmission statistics module, and notifying the media stream sender to retransmit the corresponding RTP message when parsing the packet loss retransmission request NACK, wherein the NACK request interval control adopts dynamic interval control, and the calculation formula of the dynamic interval interval is: interval = 0.6*current interval + 0.4*rtt, wherein the initial value of interval is 30ms.

4. A QoS comprehensive processing method for a video conferencing system as claimed in claim 1 or 3, Features: The specific process of restoring the lost original message in step S2 is: extending the private RTP protocol, carrying the packet group sequence number, the number of original packets in the packet group and the position information of the current message in the packet group in the RTP header extension of each original message; The number of original packets k and the number of redundant packets r in the preset packet group, when the number of original packets + redundant packets received by the receiving end in the packet group reaches k, the FEC decoder recovers all messages in the packet group in combination with the NACK strategy.

5. A QoS comprehensive processing method for a video conferencing system as claimed in claim 1, Features: The specific implementation method of dynamically adjusting the jitter buffer length in step S3 is: The jitter buffer module sorts the received data packets according to their timestamp information and verifies the sequence number to confirm the packet loss; Put the sorted data packets into the jitter buffer queue and index them by the message sequence number seq; The jitter buffer length is dynamically adjusted according to the changing network conditions. The specific adjustment method is as follows: The minimum buffer length is preset to 140ms and the maximum buffer length is 1s; For audio, if one or more packet reading fails within a 250-packet reading cycle, the current jitter buffer length is increased by one; if 250 consecutive packet readings are successful, the current jitter buffer length is reduced by one; For video, in a weak network environment, the jitter buffer length is increased, and after the network is restored, the jitter buffer length is reduced; the judgment condition of the weak network environment is: when a frame read timeout or frame loss occurs, it is determined to be a weak network environment; specifically, for a weak network environment, the frame read cycle is set as the maximum buffer length, and the jitter buffer length is adjusted once in each frame read cycle; if the frame loss is caused by incomplete frames, the regular increase by 1 is performed, that is, when an incomplete frame occurs in each frame read cycle, the jitter buffer length is increased by 1, and if the number of complete frames in the current buffer queue accounts for 2 / 3, the jitter buffer length stops increasing; if there is no frame loss in the cycle, the regular reduction is performed, that is, the buffer length is reduced by 1 in each frame read cycle, and if 1 / 4 of the frames in the cycle are read out of time, the jitter buffer length stops reducing; In the early stage of joining the conference, the jitter buffer length is increased by 1 without being controlled by the frame reading cycle. However, if the number of frames exceeds 1 / 3 of the current buffer queue, the jitter buffer length is adjusted once per frame reading cycle. For a large packet loss scenario with a sudden packet loss rate greater than 3%, the large packet loss scenario control strategy is as follows: If the packet loss rate is lower than 10%, or the current jitter buffer queue length exceeds 2 / 3 of the maximum buffer length, the jitter buffer length increase is reduced to 1; If the packet loss rate is in the range of [10%, 15%], increase the increase and set the jitter buffer length increase to packet loss rate - 10; If the packet loss rate is greater than 15%, the jitter buffer length is increased rapidly, and the increase is set to 10; The packet loss rate refers to the packet loss rate inside the current buffer queue.

6. A QoS comprehensive processing method for a video conferencing system as claimed in claim 1, Features: The processing in step S4 includes mixing audio and / or mixing screen data after audio and video decoding.

7. A QoS comprehensive processing method for a video conferencing system as claimed in claim 1, Features: The specific method of dynamically adjusting the redundancy rate in step S7 is as follows: The FEC encoding end receives real-time bandwidth data inferred from the bandwidth detection module and dynamically adjusts the redundancy rate. Specifically, the maximum redundancy rate is set to (detection bandwidth-coding bit rate) / coding bit rate. When the downlink packet loss rate is less than 3%, the redundancy rate is adjusted to the maximum redundancy rate / 4; when the downlink packet loss rate is between 3% and 10%, the redundancy rate is adjusted to the maximum redundancy rate / 2; when the downlink packet loss rate is above 10%, the redundancy rate is adjusted to the maximum redundancy rate.

8. A QoS comprehensive processing method for a video conferencing system as claimed in claim 1, Features: Step S7 also includes the sending rate control module feeding back the existing downlink packet sending situation to the transmission statistics module.

9. A QoS comprehensive processing method for a video conferencing system as claimed in claim 1, Features: The sending rate control module uses a sliding window algorithm to control the smooth sending of data packets, reduce the risk of packet loss caused by insufficient bandwidth, and control the sending priority of data packets to: retransmission packet>audio packet>video packet.

Citation Information

Patent Citations

  • Data transmission method, device and equipment, vehicle and storage medium

    CN115208864A

  • Lost packet retransmission system and method in video conference scene

    CN116405630A

  • Adaptive forward error correction

    WO2006060036A1

Cited By

  • Efficient collaboration method for video data

    CN119854435A

  • Efficient collaboration method for video data

    CN119854435B

  • Self-adaptive video parameter adjusting system and method for video conference

    CN120416424A

  • Bank remote monitoring video acquisition system and method thereof

    CN121174005A

  • Bank remote monitoring video acquisition system and method thereof

    CN121174005B