A method for QoS comprehensive processing of a video conference system

Abstract

This invention provides a comprehensive QoS processing method for video conferencing systems. Based on the common RTCP feedback and some proprietary RTP protocols, it acquires network data in real time through a transmission statistics module and a bandwidth detection module. It adjusts the FEC adaptive redundancy rate in real time according to current network changes, improving the FEC recovery rate. Simultaneously, it dynamically adjusts the jitter buffer length based on changing network conditions, ensuring smooth frame output even in weak network environments. This invention offers advantages such as low application cost, high network adaptability, improved video conferencing quality, and enhanced user experience.

Description

Technical Field

[0001] This invention relates to the field of video communication technology, and in particular to a comprehensive QoS processing method for a video conferencing system. Background Technology

[0002] The primary function of a video conferencing system is to centrally process (mix audio, screen, transcode, etc.) the audio and video streams from different conferencing terminals and distribute them to other participating terminals. However, video conferencing faces numerous challenges under varying network conditions, such as network latency, bandwidth limitations, packet loss, and jitter. These issues can lead to blurry images, audio delays, and stuttering during video conferences, severely impacting the user experience.

[0003] To provide stable, high-quality video conferencing services, multiple QoS metrics need to be considered comprehensively to ensure an excellent user experience under various network conditions. Current QoS (Quality of Service) technologies applied to video conferencing systems cover several aspects, primarily including: network bandwidth management, traffic adjustment and prioritization, network congestion control, packet loss recovery, and error correction. Existing video conferencing optimization strategies have several main drawbacks:

[0004] First, there is a lack of adaptability. Some QoS technologies may only be effective in specific network environments or situations, but may not perform optimally in other environments. This lack of adaptability can lead to inconsistent communication quality under different network conditions.

[0005] Second, the overall effect is poor. Optimizing only one QoS metric may negatively impact other metrics. For example, focusing solely on reducing latency may lead to excessive bandwidth consumption, thereby affecting video quality.

[0006] To address the shortcomings of existing, relatively simple and fixed QoS schemes, this invention proposes a comprehensive QoS processing strategy that effectively improves the user experience in video conferencing. Summary of the Invention

[0007] The technical problem to be solved by this invention is to provide a comprehensive QoS processing method for a video conferencing system, so as to overcome the problem that various QoS indicators cannot be taken into account in different network environments, provide stable and high-quality video communication, and improve the user experience of video conferencing.

[0008] The problem of this invention is solved as follows: a comprehensive QoS processing method for a video conferencing system, comprising the following steps:

[0009] Step S1: Receive the uplink audio and video streams through the streaming media receiver, classify and distribute the received data, and collect the bandwidth and packet count of different types of messages to the transmission statistics module.

[0010] Step S2: The FEC decoder compares the cached original packets and redundant packets with the redundant information, restores the lost original packets, and sends them to the jitter buffer module.

[0011] Step S3: Sort the received raw packets through the jitter buffer module, determine whether packet loss has occurred through the RTP sequence number, and feed back the packet loss sequence number to the transmission statistics module. At the same time, dynamically adjust the jitter buffer length and send the buffered RTP packets to the error hiding module in units of frames.

[0012] Step S4: The error hiding module determines whether padding data needs to be generated based on the continuity of the RTP sequence number. If the RTP sequence number is not continuous, the audio sample values ​​between adjacent frames are interpolated to create new audio frames to fill the missing frames. If the RTP sequence number is continuous, no padding data is needed. Then, the continuous media data is sent to the audio and video decoder for decoding, and the decoded data is processed according to different conference scenarios before being sent to the audio and video encoder.

[0013] Step S5: When the RTCP processing module parses the keyframe request FIR / PLI, it directly notifies the audio and video encoder to generate the keyframe through the transmission statistics module.

[0014] Step S6: The transmission statistics module transmits the current downlink bandwidth, downlink latency recv_delta data, and current downlink packet loss rate to the bandwidth detection module. The bandwidth detection module evaluates the current downlink bandwidth based on the statistical information and, with reference to the downlink packet loss situation, modifies the video coding rate and FEC redundancy configuration information in real time.

[0015] Step S7: The audio and video encoder encodes the received data according to the modified video encoding rate and then transmits it to the FEC encoder. The FEC encoder dynamically adjusts the redundancy rate according to the data received by the bandwidth detection module, adds redundant packet group information to the original message data, inserts a specified number of redundant packets into each redundant packet group according to the current redundancy rate, and then hands it over to the transmission rate control module for smooth transmission to the streaming media transmitter.

[0016] Furthermore, the FEC redundancy configuration information includes the ratio of redundant packets to original packets in the FEC packet group.

[0017] Furthermore, the step S1 of "classifying and distributing the received data" specifically includes: dividing the received packets into RTP packets and RTCP packets. The RTP packets are further divided into redundant packets, original packets, and probe padding packets. The redundant packets are directly passed to the FEC decoder. A copy of the original packets is sent to the FEC decoder and placed into the jitter buffer module. The RTCP packets are directly sent to the RTCP packet processing module for parsing.

[0018] Furthermore, step S5 also includes the RTCP processing module obtaining downlink jitter, RTT, and congestion control delay information recv_delta by parsing RR, SR, and TWCC packets, which is then aggregated to the transmission statistics module. When parsing the packet loss retransmission request NACK, the module notifies the media stream transmitter to retransmit the corresponding RTP packet. The NACK request interval control adopts dynamic interval control, and the dynamic interval interval is calculated as: interval = 0.6 * current interval + 0.4 * RTT, where the initial value of interval is 30ms.

[0019] Furthermore, the specific process of restoring the lost original message in step S2 is as follows: the private RTP protocol is extended, and the packet group sequence number, the number of original packets in the packet group and the current message's position information in the packet group are carried in the RTP header extension of each original message.

[0020] The FEC decoder, in conjunction with the NACK strategy, recovers all packets in a packet group when the number of original packets plus redundant packets in the packet group reaches k.

[0021] Furthermore, the specific implementation method for dynamically adjusting the jitter buffer length in step S3 is as follows:

[0022] The jitter buffer module sorts the received data packets according to their timestamp information and verifies the sequence number to confirm packet loss.

[0023] The sorted data packets are placed into the jitter buffer queue, indexed by the packet sequence number (seq).

[0024] The jitter buffer length is dynamically adjusted according to changing network conditions, and the specific adjustment method is as follows:

[0025] The minimum buffer length is preset to 140ms, and the maximum buffer length is 1s.

[0026] For audio, if one or more packet readings fail within 250 packet reading cycles, the current jitter buffer length is incremented by one; if 250 consecutive packet readings are successful, the current jitter buffer length is decremented by one.

[0027] For video, in a weak network environment, the jitter buffer length is increased, and after the network recovers, the jitter buffer length is decreased; the condition for judging a weak network environment is: when a frame read timeout or frame loss occurs, it is judged as a weak network environment.

[0028] Furthermore, for video, the specific process for adjusting the jitter buffer length is as follows:

[0029] For weak network environments, the frame reading period is set to the maximum buffer length, and the jitter buffer length is adjusted once within each frame reading period. If frame loss is caused by incomplete frames, the buffer length is increased by 1 in a normal manner. That is, when an incomplete frame occurs in each frame reading period, the jitter buffer length is increased by 1. If the proportion of complete frames in the current buffer queue reaches 2 / 3, the jitter buffer length stops increasing. If no frames are lost in the period, the buffer length is decreased by 1 in a normal manner. If 1 / 4 of the frames in the period time out, the jitter buffer length stops decreasing.

[0030] In the initial stage of joining, accelerated growth is implemented without being controlled by the frame reading cycle. That is, the jitter buffer length is increased by 1 every time a frame is lost. However, if the number of frames exceeds 1 / 3 of the current buffer queue, the jitter buffer length is adjusted once according to each frame reading cycle.

[0031] For sudden packet loss scenarios with a packet loss rate exceeding 3%, the following packet loss scenario control strategy will be implemented:

[0032] If the packet loss rate is less than 10%, or the current jitter buffer queue length has exceeded 2 / 3 of the maximum buffer length, then the jitter buffer length increment will be reduced to 1.

[0033] If the packet loss rate is in the range of [10%, 15%], then increase the increment and set the jitter buffer length increment to packet loss rate - 10.

[0034] If the packet loss rate is greater than 15%, then increase it rapidly by setting the jitter buffer length increment to 10.

[0035] The packet loss rate refers to the packet loss rate within the current buffer queue.

[0036] Furthermore, the processing in step S4 includes mixing and / or screen mixing of the decoded audio and video data.

[0037] Furthermore, the specific method for dynamically adjusting the redundancy rate in step S7 is as follows:

[0038] The FEC encoder receives real-time bandwidth data inferred from the bandwidth detection module and dynamically adjusts the redundancy rate. Specifically, the maximum redundancy rate is set to (detection bandwidth - encoding rate) / encoding rate. When the downlink packet loss rate is less than 3%, the redundancy rate is adjusted to the maximum redundancy rate / 4. When the downlink packet loss rate is between 3% and 10%, the redundancy rate is adjusted to the maximum redundancy rate / 2. When the downlink packet loss rate is above 10%, the redundancy rate is adjusted to the maximum redundancy rate.

[0039] Furthermore, step S7 also includes the transmission rate control module feeding back the existing downlink packet transmission status to the transmission statistics module.

[0040] Furthermore, the transmission rate control module uses a sliding window algorithm to control the smooth transmission of data packets, reducing the risk of packet loss due to insufficient bandwidth, and controls the transmission priority of data packets as follows: retransmission packets > audio packets > video packets.

[0041] The advantages of this invention are: based on the general RTCP feedback and some RTP private protocols, the dynamic jitter buffer and adaptive redundancy processing of this invention are implemented, thereby enabling video conferencing to adapt to different network environments, balance various service quality indicators, improve video conferencing quality, and enhance the overall user experience. At the same time, the method of this invention has low application cost, and new QoS indicators or new optimization strategies can be relatively easily integrated into this method.

[0042] The attached diagram illustrates that it has low application costs.

[0043] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0044] Figure 1 This is an execution flowchart of a QoS comprehensive processing method for a video conferencing system according to the present invention.

[0045] Figure 2 This is an overall block diagram of a QoS comprehensive processing method for a video conferencing system according to the present invention.

[0046] Figure 3 This is a schematic diagram of the FEC-RTP header extension structure of the present invention. Detailed Implementation

[0047] To make the present invention more apparent and understandable, a preferred embodiment is described in detail below with reference to the accompanying drawings.

[0048] like Figure 1 As shown, a QoS comprehensive processing method for a video conferencing system according to the present invention includes the following steps:

[0049] Step S1: Receive the uplink audio and video streams through the streaming media receiver. Classify and distribute the received data. Specifically, the received packets are divided into RTP packets and RTCP packets. The RTP packets are further divided into redundant packets, original packets, and probe padding packets (these packets have fixed content, contain no actual media data, and can be discarded by the receiver). The redundant packets are directly passed to the FEC decoder. A copy of the original packets is sent to the FEC decoder and simultaneously placed into the jitter buffer module. The RTCP packets are directly sent to the RTCP packet processing module for parsing. The streaming media receiver also collects statistics on the bandwidth and number of packets received for different types of packets and aggregates this data to the transmission statistics module.

[0050] Step S2: The FEC decoder compares the cached original packets and redundant packets with the redundant information, restores the lost original packets, and sends them to the jitter buffer module.

[0051] Step S3: Sort the received raw packets through the jitter buffer module, determine whether packet loss has occurred through the RTP sequence number, and feed back the packet loss sequence number to the transmission statistics module. At the same time, dynamically adjust the jitter buffer length and send the buffered RTP packets to the error hiding module in units of frames.

[0052] Step S4: The error concealment module determines whether padding data needs to be generated based on the continuity of the RTP sequence numbers (the messages passed to the error concealment module at this stage are sorted by the jitter buffer module; continuity is determined by comparing whether the current sequence number is continuous with the sequence number of the previous incoming message). If the RTP sequence numbers are not continuous, new audio frames are created to fill the missing frames by interpolating the audio sample values ​​between adjacent frames. If the RTP sequence numbers are continuous, no padding data is needed. Then, the continuous media data is sent to the audio / video decoder for decoding, and the decoded data is processed according to different conference scenarios before being sent to the audio / video encoder. The above processing includes mixing and screen mixing of the decoded audio / video data.

[0053] Step S5: When the RTCP processing module parses the keyframe request FIR / PLI, it directly notifies the audio and video encoder to generate the keyframe through the transmission statistics module.

[0054] Step S6: The transmission statistics module transmits the current downlink bandwidth, downlink latency recv_delta data, and current downlink packet loss rate to the bandwidth detection module. The bandwidth detection module evaluates the current downlink bandwidth based on the statistical information and, with reference to the downlink packet loss situation, modifies the video coding rate and FEC redundancy configuration information in real time.

[0055] Step S7: The audio / video encoder encodes the received data according to the modified video encoding rate and then transmits it to the FEC encoder. The FEC encoder dynamically adjusts the redundancy rate based on the data received by the bandwidth detection module, adds redundant packet group information to the original message data, inserts a specified number of redundant packets into each redundant packet group according to the current redundancy rate, and then hands it over to the transmission rate control module for smooth transmission to the streaming media transmitter. The transmission rate control module uses a sliding window algorithm to control the smooth transmission of data packets, reducing the risk of packet loss due to insufficient bandwidth, and controls the data packet transmission priority as: retransmission packets > audio packets > video packets. In addition, the transmission rate control module feeds back the current downlink packet transmission status to the transmission statistics module.

[0056] Preferably, the FEC redundancy configuration information includes the ratio of redundant packets to original packets in the FEC packet group.

[0057] Preferably, step S5 further includes the RTCP processing module obtaining downlink jitter, RTT, and congestion control delay information recv_delta by parsing RR, SR, and TWCC messages, converging them to the transmission statistics module, and notifying the media stream sender to retransmit the corresponding RTP message when parsing the packet loss retransmission request NACK. The NACK request interval control adopts dynamic interval control, and the dynamic interval interval is calculated as: interval = 0.6 * current interval + 0.4 * RTT, where the initial value of interval is 30ms.

[0058] The aforementioned dynamic interval control for NACK requests is implemented through the transmission statistics module. As the system's statistical data aggregation module, the transmission statistics module not only stores transmission data statistics but also sets the NACK request interval control. Traditionally, the NACK request interval is fixed, meaning NACK requests are made at fixed time periods. To avoid significant increases in network bandwidth caused by frequent retransmission packets, this period is usually set relatively large, such as 100ms. Therefore, if a retransmission packet is lost during transmission, the video frame will need to wait a long time to become complete, increasing the overall latency. To reduce the latency caused by packet loss and retransmission, we optimize the interval control to a dynamic interval. When packet loss is detected, the sequence of packets initiates the first NACK request after 30ms (to avoid redundant retransmission processing in scenarios with minor jitter). Then, requests are repeated according to the dynamic interval until the lost packet is received or the number of requests reaches the maximum of 10. The dynamic interval is related to the statistical RTT (Round-Trip Time). The length of this period, from the receiver requesting NACK to the sender responding with a retransmission packet and then the receiver receiving the retransmission packet, can be considered close to the statistical RTT value. When the retransmission request interval is shorter than this period, there will be redundant NACK requests, and the extra retransmission packets will cause bandwidth waste. Therefore, we set the above dynamic interval calculation method as follows: interval = 0.6 * current interval + 0.4 * rtt (initial interval is 30ms).

[0059] Preferably, the specific process of restoring the lost original message in step S2 is as follows: the private RTP protocol is extended, and the packet group sequence number, the number of original packets in the packet group and the current message's position information in the packet group are carried in the RTP header extension of each original message.

[0060] Example of an extended header structure: Figure 3As shown, SBN is the packet group sequence number, used to identify that the redundant packet and the original packet belong to the same packet group, that is, to identify which packet group consisting of the original packet and the redundant packet the message belongs to. ESI is the position of the current message in the packet group, following the order of original packets first, then redundant packets. For example, in a packet group consisting of 10 original packets and 5 redundant packets, the position number within the group starts from 0, then the positions of the original packets are 0-9, and the positions of the redundant packets are 10-14. In the image, k is the number of original packets in the current packet group, and pad is the padding bit, used to pad to make up 4 bytes. The condition for a packet group to be completely recovered is that the total number of messages received in the current packet group is greater than or equal to the original number of packets k in the packet group.

[0061] Packet group content: k original messages (i.e., original packets), plus r redundant messages (i.e., redundant packets);

[0062] Redundancy rate calculation: Number of redundant packets r / Number of original packets k;

[0063] The preset packet group (i.e., the redundant packet group) contains the original packet number k and the redundant packet number r. When the receiver receives the original packet plus the redundant packet number k in this packet group, the FEC decoder, combined with the NACK strategy, recovers all packets in the packet group. The decoder adjusts the interval between two consecutive NACK requests for the same packet according to the statistical transmission interval. Here, the redundant packet number r does not need to be transmitted through packets; the receiver only needs to receive packets that reach the original packet number to satisfy the recovery condition.

[0064] Preferably, the specific method for dynamically adjusting the redundancy rate in step S7 is as follows:

[0065] The FEC encoder receives real-time bandwidth data inferred from the bandwidth detection module and dynamically adjusts the redundancy rate. Specifically, the maximum redundancy rate is set to (detection bandwidth - encoding rate) / encoding rate. When the downlink packet loss rate is less than 3%, the redundancy rate is adjusted to the maximum redundancy rate / 4. When the downlink packet loss rate is between 3% and 10%, the redundancy rate is adjusted to the maximum redundancy rate / 2. When the downlink packet loss rate is above 10%, the redundancy rate is adjusted to the maximum redundancy rate.

[0066] The FEC (Forward Error Correction Unit) of this invention is implemented through a proprietary RTP protocol extension. By carrying redundant packet group sequence numbers, the number of original packets in the packet group, and the quantity of the packet in the packet group in the RTP header extension of each original message, the receiving end can parse the redundancy information required for FEC decoding when it receives any message in the packet group. This achieves the goal of flexibly changing the redundancy rate during the conference without the need for secondary negotiation. In this scheme, the FEC recovery determination rule is: the number of original packets in the redundant packet group is k, and the number of redundant packets is r. When the receiving end receives the original packets plus the number of redundant packets in the packet group and they reach k, all messages in the packet group can be recovered. Furthermore, the FEC decoding end, combined with the NACK strategy of this application, requires both original packets and retransmission packets to be included in the FEC decoder, accelerating the time for the packet group to reach the recovery standard, reducing packet group recovery latency, making it easier for the packet group to reach the recovery standard, and improving the FEC recovery rate. At the FEC encoding end, real-time bandwidth data inferred from the bandwidth detection module is received, and the redundancy rate is dynamically adjusted based on the current packet loss rate to adapt to changes in the network environment.

[0067] Preferably, the specific implementation method for dynamically adjusting the jitter buffer length in step S3 is as follows:

[0068] The jitter buffer module sorts the received data packets according to their timestamp information and verifies the sequence number to confirm packet loss.

[0069] The sorted data packets are placed into the jitter buffer queue, indexed by the packet sequence number (seq).

[0070] The jitter buffer length is dynamically adjusted according to changing network conditions. A larger jitter buffer can handle greater latency jitter and more severe out-of-order processing, but it will introduce greater processing latency. This solution uses a dynamic buffer length approach to adapt to changing network conditions. The specific adjustment method is as follows:

[0071] The minimum buffer length is preset to 140ms (7 frames for audio, 4 frames for video), and the maximum buffer length is 1s (50 frames for audio, 30 frames for video).

[0072] For audio, if one or more packet readings fail within 250 packet reading cycles, the current jitter buffer length frame_len is incremented by one; if 250 consecutive packet readings are successful, the current jitter buffer length frame_len is decremented by one.

[0073] For video, in weak network environments, to reduce stuttering, the number of buffered frames is increased to ensure smooth frame output. After network recovery, the number of buffered frames is decreased to reduce latency. The criteria for determining a weak network environment are: a frame read timeout (i.e., the current frame read interval is larger than the frame interval calculated by the negotiated frame rate, indicating a frame read timeout) or frame drops are identified as a weak network environment. Specifically, for weak network environments, the adjustment method for the number of buffered frames is as follows: a frame read cycle is set to the maximum number of buffered frames (i.e., the maximum buffer length), and the number of buffered frames is adjusted once within each frame read cycle (i.e., increasing / decreasing once within the frame read cycle). During adjustment, if the issue is caused by incomplete frames... If a frame is lost, the jitter buffer length is increased by 1 in a normal increment. This means that the jitter buffer length is increased by 1 whenever an incomplete frame occurs within each frame read cycle. In the early stages of membership, the jitter buffer length is increased by 1 (faster growth is required in the early stages, ignoring the configured frame read cycle). If the number of frames exceeds 1 / 3 of the current buffer length, the growth is slowed down, and the jitter buffer length is adjusted once per frame read cycle. If the proportion of complete frames in the current jitter buffer reaches 2 / 3, the growth stops. If no frames are lost within the cycle, the jitter buffer length is decreased by 1 in a normal reduction. If 1 / 4 of the frames in the cycle time out, the reduction stops (extremely low probability).

[0074] For sudden packet loss scenarios with a packet loss rate exceeding 3%, the following packet loss scenario control strategy will be implemented:

[0075] Note: The packet loss rate below refers to the internal packet loss rate of JB.

[0076] When the packet loss rate is below 10%, or when frame_len is close to the JB length limit (i.e., the current jitter buffer length has exceeded 2 / 3 of the maximum buffer length), the increment drops to 1. Typically, 4 video frames and 7 audio frames are buffered, with buffer times close to 140ms. Therefore, the maximum video jitter buffer length is set to 30 frames, and the audio jitter buffer to 50 frames, meaning a maximum buffer length of 1 second. Once the video buffer length exceeds 2 / 3 (20 frames), the increment slows down (increasing by 1 per cycle).

[0077] If the packet loss rate is in the range of [10, 15], increase the increment by the packet loss rate lost_rate - 10.

[0078] The packet loss rate is greater than 15%, and it increases rapidly, with an increase of 10.

[0079] Preferably, the bandwidth detection module of this invention is implemented based on the open-source Send-side BWE algorithm. It evaluates and obtains the downlink bandwidth detection value based on statistical congestion control delay information and the current downlink bitrate. This invention mainly implements FEC redundancy rate and encoding bitrate control based on the detection value. The detected bandwidth is primarily used for the redundant packet bitrate + original packet bitrate. The original packet bitrate takes effect by modifying the encoder's encoding bitrate, while the redundant packet bitrate requires modification of the redundancy rate, i.e., the ratio of redundant packets to original packets. A higher redundant packet bitrate results in better packet loss resistance; a higher encoding bitrate results in clearer image quality. Therefore, this invention provides two configuration options for the ratio: image quality priority or smoothness priority; a) Image quality priority: redundant packet bitrate to encoding bitrate ratio is 3:10; b) Smoothness priority: redundant packet bitrate to encoding bitrate ratio is 1:1. The provided configuration is based on a single participating terminal, achieving flexible bitrate control.

[0080] The following is in conjunction with the appendix Figure 2 The interaction of the integrated QoS policy data stream in the video conferencing system of the present invention will be further explained as follows:

[0081] like Figure 2 As shown, the modules involved in this invention include a media stream receiver, a media stream transmitter, a transmission statistics module, a bandwidth detection module, a jitter buffer module, an RTCP packet processing module, an error concealment module, an FEC decoder, an FEC encoder, an audio / video decoder, an audio / video encoder, a conference audio / image processing module, and a transmission rate control module. The interaction process between the modules is as follows:

[0082] 1. All participating terminals receive uplink audio and video streams through a media stream receiver. The media stream receiver categorizes received packets, primarily RTP and RTCP packets. RTP packets are further divided into redundant packets, original packets, and probe padding packets (these packets have fixed content, contain no actual media data, and can be discarded by the receiver). Redundant packets are directly passed to the FEC decoder, while the original packets are copied to the FEC decoder and simultaneously placed into the jitter buffer module (corresponding to...). Figure 2 (Arrow No. 1, 2). The media stream receiver also counts the bandwidth and number of packets received for different types of messages, and aggregates the statistical data to the transmission statistics module (corresponding to...). Figure 2 Arrow number 3). The RTCP message is then directly sent to the RTCP message processing module for parsing (corresponding to...). Figure 2 Arrow 4).

[0083] 2. The FEC decoder compares the redundant information with the cached original and redundant packets, reconstructs the lost original packets, and sends them to the jitter buffer module (corresponding to...). Figure 2 Arrow 5).

[0084] 3. The received raw packets are sorted using the jitter buffer module, and the RTP sequence number is used to determine whether packet loss has occurred. The packet loss sequence number is then sent back to the transmission module (corresponding to...). Figure 2 Arrow number 6). Buffered RTP packets are sent to the error concealment module (corresponding to) in frames. Figure 2 Arrow No. 7).

[0085] 4. The error concealment module determines whether padding data needs to be generated based on the continuity of the RTP sequence number. If so, it interpolates the audio sample values ​​between adjacent frames to create new audio frames to fill in the missing frames. Finally, the error concealment module sends the continuous media data to the audio / video decoder for processing (corresponding to...). Figure 2 (Arrow No. 8). The error concealment unit is the final processing step before decoding audio and video data. By comparing the data differences before and after packet loss, a virtual data packet is constructed and sent to the decoder, reducing the subjective impact of packet loss on user experience.

[0086] 5. Depending on the meeting scenario, the meeting audio / image processing module performs mixing, screen mixing, and other processing on the decoded audio and video data, and encodes it into data to be sent (corresponding to...). Figure 2 Arrow No. 9).

[0087] 6. The RTCP processing module parses RR, SR, and TWCC messages to obtain downlink jitter, RTT (rttime to time), and congestion control delay information recv_delta, all of which are then aggregated to the transmission statistics module (corresponding to...). Figure 2 Arrow 10). When parsing the NACK request for packet loss retransmission, the media stream sender is notified to retransmit the corresponding RTP packet (corresponding to...). Figure 2 When the keyframe request FIR / PLI is parsed (arrow number 19), the encoder is directly notified to generate the keyframe (corresponding to) through the parameter statistics module. Figure 2 Arrow No. 17).

[0088] 7. The transmission statistics module transmits the currently calculated downlink bandwidth, downlink latency recv_delta data, and current downlink packet loss rate to the bandwidth detection module (corresponding to...). Figure 2 Arrow 11).

[0089] 8. The bandwidth detection module assesses the current downlink bandwidth based on statistical information and, taking into account downlink packet loss, adjusts the video encoding bitrate in real time (corresponding to...). Figure 2 Arrow No. 12) and FEC redundancy configuration information (corresponding to Figure 2 Arrow 14).

[0090] 9. The encoded data needs to be transmitted to the FEC encoder first (corresponding to...). Figure 2Arrow 13) adds redundant packet group information to the original message data to help the receiver recover lost data. A specified number of redundant packets are inserted into each redundant packet group according to the current redundancy rate, and then smoothly transmitted by the transmission rate control module (corresponding to...). Figure 2 Arrow 15).

[0091] 10. The transmission rate control module feeds back the current downlink packet transmission status to the transmission statistics module (corresponding to...). Figure 2 (Arrow No. 16), and achieves millisecond-level flow control. Finally, media messages are uniformly sent through the media stream sender (corresponding to...). Figure 2 Arrow No. 18).

[0092] This invention's video conferencing system QoS comprehensive strategy is based on the general RTCP feedback and incorporates some proprietary RTP protocols to implement dynamic jitter buffering and adaptive redundancy processing. This ensures balanced service quality indicators across different network environments, guaranteeing communication quality for participating terminals and improving the overall user experience. Furthermore, the method has low application costs, and new QoS indicators or optimization strategies can be easily integrated into this method, ensuring the system maintains a leading position in technological evolution. Since the optimization strategies for different processing nodes can be replaced individually—for example, replacing the FEC algorithm—simply requires standardizing the data transmission process, inputting the original packet + redundant packet, and outputting the restored original packet. This allows for arbitrary replacement of the FEC algorithm without modifying other modules of the original system.

[0093] While specific embodiments of the present invention have been described above, those skilled in the art should understand that the specific embodiments described are merely illustrative and not intended to limit the scope of the present invention. Equivalent modifications and variations made by those skilled in the art in accordance with the spirit of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for QoS synthesis processing of a video conference system, characterized in that: Includes the following steps: Step S1: Receive the uplink audio and video streams through the streaming media receiver, classify and distribute the received data, and collect the bandwidth and packet count of different types of messages to the transmission statistics module. Step S2: The FEC decoder compares the cached original packets and redundant packets with the redundant information, restores the lost original packets, and sends them to the jitter buffer module. Step S3: Sort the received raw packets through the jitter buffer module, determine whether packet loss has occurred through the RTP sequence number, and feed back the packet loss sequence number to the transmission statistics module. At the same time, dynamically adjust the jitter buffer length and send the buffered RTP packets to the error hiding module in units of frames. Step S4: The error hiding module determines whether padding data needs to be generated based on the continuity of the RTP sequence number. If the RTP sequence number is not continuous, the audio sample values ​​between adjacent frames are interpolated to create new audio frames to fill the missing frames. If the RTP sequence number is continuous, no padding data is needed. Then, the continuous media data is sent to the audio and video decoder for decoding, and the decoded data is processed according to different conference scenarios before being sent to the audio and video encoder. Step S5: When the RTCP processing module parses a keyframe request, it directly notifies the audio / video encoder to generate a keyframe through the transmission statistics module. Step S6: The transmission statistics module transmits the current downlink bandwidth, downlink latency data and current downlink packet loss rate to the bandwidth detection module. The bandwidth detection module evaluates the current downlink bandwidth based on the statistical information and, with reference to the downlink packet loss situation, modifies the video encoding rate and FEC redundancy configuration information in real time. Step S7: The audio and video encoder encodes the received data according to the modified video encoding rate and then transmits it to the FEC encoder. The FEC encoder dynamically adjusts the redundancy rate according to the data received by the bandwidth detection module, adds redundant packet group information to the original message data, inserts a specified number of redundant packets into each redundant packet group according to the current redundancy rate, and then hands it over to the transmission rate control module for smooth transmission to the streaming media transmitter.

2. The method for QoS synthesis processing of a video conference system according to claim 1, characterized in that: The step S1, "classifying and distributing the received data," specifically includes: dividing the received packets into RTP packets and RTCP packets. The RTP packets are further divided into redundant packets, original packets, and probe padding packets. The redundant packets are directly passed to the FEC decoder. A copy of the original packets is sent to the FEC decoder and simultaneously placed into the jitter buffer module. The RTCP packets are directly sent to the RTCP packet processing module for parsing.

3. The method for QoS synthesis processing of a video conference system according to claim 1, wherein: Step S5 further includes the RTCP processing module obtaining downlink jitter, RTT, and congestion control delay information recv_delta by parsing RR, SR, and TWCC messages, which is then aggregated to the transmission statistics module. When parsing the packet loss retransmission request NACK, the module notifies the media stream transmitter to retransmit the corresponding RTP message. The NACK request interval control adopts dynamic interval control, and the dynamic interval interval is calculated as: interval = 0.6 * current interval + 0.4 * RTT, where the initial value of interval is 30ms.

4. A QoS comprehensive processing method for a video conferencing system as described in claim 1 or 3, characterized in that: The specific process of restoring the lost original message in step S2 is as follows: the private RTP protocol is extended, and the packet group sequence number, the number of original packets in the packet group and the current message's position information in the packet group are carried in the RTP header extension of each original message. The FEC decoder, in conjunction with the NACK strategy, recovers all packets in a packet group when the number of original packets plus redundant packets in the packet group reaches k.

5. The QoS comprehensive processing method for a video conferencing system as described in claim 1, characterized in that: The specific implementation method for dynamically adjusting the jitter buffer length in step S3 is as follows: The jitter buffer module sorts the received data packets according to their timestamp information and verifies the sequence number to confirm packet loss. The sorted data packets are placed in the jitter buffer queue and indexed by the packet sequence number (seq). The jitter buffer length is dynamically adjusted according to changing network conditions, and the specific adjustment method is as follows: The minimum buffer length is preset to 140ms, and the maximum buffer length is 1s. For audio, if one or more packet readings fail within 250 packet reading cycles, the current jitter buffer length is incremented by one; if 250 consecutive packet readings are successful, the current jitter buffer length is decremented by one. For video, in weak network environments, the jitter buffer length is increased, and after the network recovers, the jitter buffer length is decreased. The criteria for determining a weak network environment are: when a frame read timeout or frame drop occurs, it is determined to be a weak network environment. Specifically, for weak network environments, the frame read period is set to the maximum buffer length, and the jitter buffer length is adjusted once within each frame read period. If the frame drop is caused by incomplete frames, a normal increment of 1 is performed, that is, when an incomplete frame occurs within each frame read period, the jitter buffer length is increased by 1. If the proportion of complete frames in the current buffer queue reaches 2 / 3, the jitter buffer length stops increasing. If no frames are dropped within the period, a normal reduction is performed, that is, the buffer length decreases by 1 within each frame read period. If 1 / 4 of the frames within the period experience read timeouts, the jitter buffer length stops decreasing. In the initial stage of joining, accelerated growth is implemented without being controlled by the frame reading cycle. That is, the jitter buffer length is increased by 1 every time a frame is lost. However, if the number of frames exceeds 1 / 3 of the current buffer queue, the jitter buffer length is adjusted once according to each frame reading cycle. For sudden packet loss scenarios with a packet loss rate exceeding 3%, the following packet loss scenario control strategy will be implemented: If the packet loss rate is less than 10%, or the current jitter buffer queue length has exceeded 2 / 3 of the maximum buffer length, then the jitter buffer length increment will be reduced to 1. If the packet loss rate is in the range of [10%, 15%], then increase the increment and set the jitter buffer length increment to packet loss rate - 10. If the packet loss rate is greater than 15%, then increase it rapidly by setting the jitter buffer length increment to 10. The packet loss rate refers to the packet loss rate within the current buffer queue.

6. The QoS comprehensive processing method for a video conferencing system as described in claim 1, characterized in that: The processing in step S4 includes mixing and / or screen mixing of the decoded audio and video data.

7. The QoS comprehensive processing method for a video conferencing system as described in claim 1, characterized in that: The specific method for dynamically adjusting the redundancy rate in step S7 is as follows: The FEC encoder receives real-time bandwidth data inferred from the bandwidth detection module and dynamically adjusts the redundancy rate. Specifically, the maximum redundancy rate is set to (detection bandwidth - encoding rate) / encoding rate. When the downlink packet loss rate is less than 3%, the redundancy rate is adjusted to the maximum redundancy rate / 4. When the downlink packet loss rate is between 3% and 10%, the redundancy rate is adjusted to the maximum redundancy rate / 2. When the downlink packet loss rate is above 10%, the redundancy rate is adjusted to the maximum redundancy rate.

8. The QoS comprehensive processing method for a video conferencing system as described in claim 1, characterized in that: Step S7 also includes the transmission rate control module feeding back the current downlink packet transmission status to the transmission statistics module.

9. The QoS comprehensive processing method for a video conferencing system as described in claim 1, characterized in that: The transmission rate control module uses a sliding window algorithm to control the smooth transmission of data packets, reducing the risk of packet loss due to insufficient bandwidth, and controls the transmission priority of data packets as follows: retransmission packets > audio packets > video packets.

Images