Traffic regulation method and device of audio and video stream, electronic equipment and storage medium

Abstract

This application relates to a method, apparatus, electronic device, and storage medium for traffic control of audio and video streams. The method includes: acquiring an audio and video stream to be sent and parsing the audio and video stream to obtain the frame type and single-frame data size in the audio and video stream; determining the key frame data size based on the frame type and single-frame data size; acquiring the bandwidth threshold of the link used to send the audio and video stream, wherein the bandwidth threshold is used to characterize the upper limit of bandwidth that can be allocated to the link; determining whether the triggering conditions of the traffic control strategy are met based on the key frame data size and the bandwidth threshold; and if the triggering conditions of the traffic control strategy are met, splitting each key frame in the audio and video stream into multiple data sub-packets for transmission. This can achieve traffic control of the audio and video stream, avoiding the peak traffic of the audio and video stream from exceeding the bandwidth threshold of the link, which could lead to key frame data loss and problems such as video distortion, stuttering, and audio disconnection.

Description

Technical Field

[0001] This application relates to the field of audio and video transmission technology, and in particular to a method, apparatus, electronic device and storage medium for controlling the flow of audio and video streams. Background Technology

[0002] Before transmission, audio and video streams typically need to be encoded to compress the data volume. Currently, mainstream video coding methods such as High Efficiency Video Coding (HEVC, also known as H.265) and Advanced Video Coding (AVC, also known as H.264), when using inter-frame prediction mechanisms for encoding, often result in keyframes (i.e., I-frames) having a much larger data volume than ordinary frames (i.e., P-frames or B-frames), thus forming periodic frame-level traffic pulses. Furthermore, current mainstream audio coding methods such as OPUS (an open-source audio codec standardized by the Internet Engineering Task Force (IETF)) and Advanced Audio Coding (AAC) use variable bitrate modes, which further exacerbates the uncertainty of the traffic.

[0003] However, when the peak traffic of audio and video streams exceeds the link bandwidth limit, the router's queue dropping mechanism will prioritize the loss of key frames and other critical data. This can easily lead to problems such as video screen tearing, stuttering, and audio disconnection. Therefore, how to accurately control the traffic of audio and video streams has become an urgent technical problem to be solved. Summary of the Invention

[0004] This application provides a method, apparatus, electronic device, and storage medium for controlling the flow of audio and video streams, in order to solve the problems in related technologies where the flow of audio and video streams cannot be accurately controlled, resulting in video screen tearing, stuttering, and audio disconnection.

[0005] In a first aspect, embodiments of this application provide a method for controlling the flow of audio and video streams, the method comprising: The audio and video stream to be sent is acquired, and the audio and video stream is parsed to obtain the frame type and single frame data volume in the audio and video stream; Based on the frame type and the single frame data volume, determine the key frame data volume; Obtain a bandwidth threshold for the link used to send the audio and video streams, wherein the bandwidth threshold is used to characterize the upper limit of bandwidth that can be allocated to the link; Based on the keyframe data volume and the bandwidth threshold, determine whether the triggering conditions of the traffic control strategy are met. If the triggering conditions of the current traffic control strategy are met, each keyframe in the audio and video stream is split into multiple data sub-packets and sent.

[0006] Optionally, the step of splitting each keyframe in the audio / video stream into multiple data sub-packets for transmission includes: Obtain the current link status of the link; When the link is in an idle state, the multiple data sub-packets are sent sequentially and continuously in order of priority from high to low. When the link is busy, the multiple data sub-packets are sent sequentially and at intervals in order of priority from high to low.

[0007] Optionally, the plurality of data sub-packets include a first data sub-packet, a second data sub-packet, and a third data sub-packet, wherein the first data sub-packet has a higher priority than the second data sub-packet, and the second data sub-packet has a higher priority than the third data sub-packet; The step of sending the multiple data sub-packets sequentially and at intervals according to their priority from high to low includes: The first data sub-packet is sent, and after the first data sub-packet is sent, the second data sub-packet and the third data sub-packet are sequentially inserted into different transmission gaps of the normal frames of the audio and video stream for transmission.

[0008] Optionally, after splitting each keyframe in the audio / video stream into multiple data sub-packets for transmission, the method further includes: Acquire quality indicator data during the transmission of the audio and video streams, wherein the quality indicator data includes packet loss rate and average subjective score; Determine whether the packet loss rate is greater than a first preset threshold; If the packet loss rate is greater than the first preset threshold, the bandwidth threshold is reduced by a first preset step size; If the packet loss rate is less than or equal to the first preset threshold, determine whether the average subjective score is greater than the second preset threshold. If the average subjective score is greater than the second preset threshold, the bandwidth threshold is increased by the second preset step size; If the average subjective score is less than or equal to the second preset threshold, the bandwidth threshold remains unchanged.

[0009] Optionally, determining whether the triggering conditions for the traffic control strategy are met based on the keyframe data volume and the bandwidth threshold includes: Determine whether the amount of keyframe data is greater than the product of the bandwidth threshold and a preset coefficient; If the amount of keyframe data is greater than the product of the bandwidth threshold and the preset coefficient, determine the triggering condition that currently satisfies the traffic control strategy. If the amount of keyframe data is less than or equal to the product of the bandwidth threshold and the preset coefficient, it is determined that the triggering condition of the current traffic control strategy is not met.

[0010] Optionally, before determining whether the triggering conditions of the traffic control strategy are met based on the keyframe data volume and the bandwidth threshold, the method further includes: Determine whether the bandwidth threshold is less than a third preset threshold; If the bandwidth threshold is less than the third preset threshold, only the audio stream in the audio and video stream will be sent; If the bandwidth threshold is greater than or equal to the third preset threshold, the transmission code rate corresponding to the link is obtained, and it is determined whether the transmission code rate is greater than the bandwidth threshold. If the transmission bitrate is greater than the bandwidth threshold, the encoding parameters of the audio and video streams are reduced, wherein the encoding parameters include the encoding bitrate; If the transmission bit rate is less than the bandwidth threshold, the step of determining whether the triggering conditions of the traffic control strategy are met based on the key frame data volume and the bandwidth threshold is executed.

[0011] Optionally, obtaining the bandwidth threshold of the link used to send the audio and video streams includes: The bandwidth threshold is obtained from the server, wherein the bandwidth threshold is calculated by the server using Gaussian process regression based on the time-domain features of link parameters, the time-domain features of traffic, and the time-domain features of bandwidth. The time-domain features of link parameters are obtained by the server by feature extraction of link parameters within the most recently acquired first preset period. The time-domain features of traffic are obtained by the server by feature extraction of historical traffic data within the most recently acquired second preset period. The time-domain features of bandwidth are obtained by the server by feature extraction of historical bandwidth data within the most recently acquired second preset period.

[0012] Secondly, embodiments of this application also provide a traffic control device for audio and video streams, the device comprising: The acquisition and parsing module is used to acquire the audio and video stream to be sent, and to parse the audio and video stream to obtain the frame type and single frame data volume in the audio and video stream; The determination module is used to determine the amount of keyframe data based on the frame type and the amount of data in a single frame; The first acquisition module is used to acquire a bandwidth threshold of the link used to send the audio and video streams, wherein the bandwidth threshold is used to characterize the upper limit of bandwidth that can be allocated to the link; The first judgment module is used to determine whether the triggering conditions of the traffic control strategy are met based on the amount of key frame data and the bandwidth threshold. The sending module is used to split each keyframe in the audio and video stream into multiple data sub-packets for sending when the triggering conditions of the current traffic control strategy are met.

[0013] Thirdly, this application also provides an electronic device, which includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; The processor, when executing a program stored in memory, implements the method for controlling the flow of audio and video streams as described in the first aspect.

[0014] Fourthly, embodiments of this application also provide a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the audio / video stream flow control method as described in the first aspect.

[0015] Compared with the prior art, the technical solution provided in this application has the following advantages: The method provided in this application obtains the audio and video stream to be sent and parses the audio and video stream to obtain the frame type and single frame data volume in the audio and video stream; determines the key frame data volume based on the frame type and the single frame data volume; obtains the bandwidth threshold of the link used to send the audio and video stream, wherein the bandwidth threshold is used to characterize the upper limit of the bandwidth that can be allocated to the link; determines whether the triggering condition of the traffic control strategy is met based on the key frame data volume and the bandwidth threshold; and if the triggering condition of the traffic control strategy is met, splits each key frame in the audio and video stream into multiple data sub-packets for transmission. The above method can acquire the keyframe data volume in real time, and based on the keyframe data volume and the bandwidth threshold of the link, determine whether the triggering conditions of the traffic control strategy are met. When the triggering conditions of the traffic control strategy are met, the keyframes with large data volumes in the audio and video streams are split into multiple data sub-packets with smaller data volumes and sent. This achieves traffic control of the audio and video streams, avoiding the traffic peak of the audio and video streams from exceeding the bandwidth threshold of the link, which would lead to keyframe data loss and problems such as video screen tearing, stuttering, and audio disconnection. Attached Figure Description

[0016] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0019] Figure 1 A flowchart illustrating a method for controlling the flow of audio and video streams provided in an embodiment of this application; Figure 2 A flowchart illustrating another method for controlling the flow of audio and video streams provided in this application embodiment; Figure 3 A schematic diagram of an audio / video stream traffic control system provided in an embodiment of this application; Figure 4 A schematic diagram of the structure of an audio / video stream flow control device provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0021] The following disclosure provides numerous different embodiments or examples for implementing various structures of this application. To simplify the disclosure, specific examples of components and arrangements are described below. These are merely examples and are not intended to limit the scope of this application. Furthermore, reference numerals and / or letters may be repeated in different examples. Such repetition is for simplification and clarity and does not in itself indicate a relationship between the various embodiments and / or arrangements discussed.

[0022] To address the issues of inaccurate control over audio and video stream traffic in related technologies, which leads to problems such as video distortion, stuttering, and audio disconnection, this application provides a method, apparatus, electronic device, and storage medium for controlling audio and video stream traffic, enabling precise control over audio and video stream traffic.

[0023] See Figure 1 , Figure 1 This is a flowchart illustrating a method for controlling the flow of audio and video streams, as provided in an embodiment of this application. Figure 1 As shown, the method for controlling the flow of audio and video streams may include the following steps: Step S101: Obtain the audio and video stream to be sent, and parse the audio and video stream to obtain the frame type and single frame data volume in the audio and video stream.

[0024] Specifically, the aforementioned audio and video streams to be sent can refer to any audio and video stream that the sending end currently needs to send. When parsing the audio and video streams, the Real-time Transport Protocol (RTP) data of the audio and video streams encoded using a preset encoding method can be parsed to obtain the frame type and the amount of data per frame. The preset encoding method here refers to encoding using an inter-frame prediction mechanism, such as H.264 or H.265. The frame type here can include keyframes (i.e., I-frames) and ordinary frames (i.e., P-frames and B-frames). The amount of data per frame here refers to the amount of data contained in a single data frame.

[0025] Step S102: Determine the key frame data volume based on frame type and single frame data volume.

[0026] Specifically, key frames can be identified first based on frame type, and then the data volume of each key frame can be summed to obtain the key frame data volume.

[0027] Step S103: Obtain the bandwidth threshold of the link used to send audio and video streams, wherein the bandwidth threshold is used to characterize the upper limit of bandwidth that can be allocated to the link.

[0028] Specifically, the aforementioned bandwidth threshold can be a fixed value set based on link hardware conditions, or a dynamically changing value predicted based on historical data. This application embodiment does not impose any specific limitations.

[0029] Step S104: Based on the key frame data volume and bandwidth threshold, determine whether the triggering conditions of the traffic control strategy are met.

[0030] Specifically, the triggering condition for the above-mentioned traffic control strategy can be the difference between the key frame data volume and the bandwidth threshold and the preset constant, or the key frame data volume can be the product of the bandwidth threshold and the preset coefficient, etc. The embodiments of this application do not make specific limitations.

[0031] Step S105: If the triggering conditions of the current traffic control strategy are met, split each keyframe in the audio and video stream into multiple data sub-packets and send them.

[0032] Specifically, the number of data sub-packets into which each keyframe is split can be equal or unequal. When splitting each keyframe in the audio / video stream into multiple data sub-packets, the splitting can be based on the importance of each field in the keyframe, or it can be evenly split based on the number of each field in the keyframe. After obtaining multiple data sub-packets, these data sub-packets can be sent sequentially and continuously, or they can be sent sequentially and at intervals.

[0033] In this way, the amount of keyframe data can be obtained in real time, and based on the amount of keyframe data and the bandwidth threshold of the link, it can be determined whether the triggering conditions of the traffic control strategy are met. When the triggering conditions of the traffic control strategy are met, the keyframes with large amounts of data in the audio and video streams are split into multiple data sub-packets with smaller amounts of data and sent. This achieves traffic control of the audio and video streams and avoids the traffic peak of the audio and video streams from exceeding the bandwidth threshold of the link, which would cause keyframe data loss and result in problems such as video screen tearing, stuttering, and audio disconnection.

[0034] In an optional embodiment, step S105, splitting each keyframe in the audio / video stream into multiple data sub-packets for transmission, includes: Get the current link status; When the link is idle, multiple data sub-packets are sent sequentially in descending order of priority. When the link is busy, multiple data sub-packets are sent sequentially and at intervals in order of priority from high to low.

[0035] Specifically, the link status mentioned above refers to the link's idle / busy state. When obtaining the current link status, the receiving end can parse the timestamps, sequence numbers, and other information in the headers of each RTP packet in the currently received audio and video stream. Then, based on the timestamps, sequence numbers, and other information in each RTP packet header, the receiving end can determine the link latency and jitter. The receiving end then returns this data to the sending end, allowing the sending end to determine the current link status based on this data. It should be noted that the sending and receiving ends can use the Real-Time Transport Control Protocol (RTCP) for connection; that is, the receiving end can package this data into RTCP messages and transmit them to the sending end.

[0036] When sending multiple data sub-packets, the link status (idle or busy) can be used to determine if the link is in an idle state. If the link is idle, the data sub-packets can be sent sequentially in descending order of priority. For example, if a keyframe is divided into data sub-packet 1 (highest priority), data sub-packet 2 (middle priority), and data sub-packet 3 (lowest priority), these three data sub-packets can be sent sequentially in descending order of priority, i.e., data sub-packet 1, data sub-packet 2, and data sub-packet 3 are sent in that order, without any interval of ordinary frames between them. If the link is busy, the data sub-packets can be sent intermittently in descending order of priority. For example, if a keyframe is split into data sub-packet 1 (highest priority), data sub-packet 2 (middle priority), and data sub-packet 3 (lowest priority), then these three data sub-packets can be sent out sequentially at intervals according to their priorities. That is, data sub-packet 1, n ordinary frames, data sub-packet 2, m ordinary frames, and data sub-packet 3 can be sent in sequence, with ordinary frames between each data sub-packet.

[0037] By using the above method, multiple data sub-packets of each key frame can be sent in different ways according to the link status, thereby ensuring that the occurrence of frame-level traffic spikes caused by key frames can be reduced under different link statuses, achieving traffic "peak shaving and valley filling", reducing transmission packet loss rate and improving link utilization.

[0038] In one optional embodiment, the plurality of data sub-packets include a first data sub-packet, a second data sub-packet, and a third data sub-packet, wherein the first data sub-packet has a higher priority than the second data sub-packet, and the second data sub-packet has a higher priority than the third data sub-packet. The above steps involve sending multiple data sub-packets sequentially and at intervals according to their priority from high to low, including: The first data sub-packet is sent, and after the first data sub-packet is sent, the second and third data sub-packets are sequentially inserted into different transmission gaps of the normal frames of the audio and video stream for transmission.

[0039] Specifically, keyframes can be divided based on the importance of each field within them. The frame header, including information such as the Sequence Parameter Set (SPS), Picture Parameter Set (PPS), and Video Parameter Set (VPS), determines whether the image can be decoded. If this data is lost, the entire frame cannot be parsed correctly, thus it has the highest importance and priority. Next is the luminance component, which determines the basic image quality, thus its importance and priority are moderate. Finally, the chrominance component determines the image detail, thus its importance and priority are the lowest. Therefore, the frame header can be assigned to the first data sub-packet, the luminance component to the second, and the chrominance component to the third, and these three data sub-packets can be sent sequentially at intervals.

[0040] Keyframe hierarchical caching scheduling can be implemented using a red-black tree priority queue. For example, suppose a 1.2MB keyframe is split into three 400KB data sub-packets, marked as high, medium, and low priorities respectively, and then inserted into the red-black tree priority queue. When sending multiple data sub-packets in descending priority order, the main thread can prioritize sending high-priority data sub-packets, while background threads poll the link's idle status. When the link is busy, they trigger the transmission of medium and low-priority data sub-packets during the P-frame or B-frame transmission interval. The entire scheduling process can be thread-safe using mutex locks, and software timers precisely control the transmission timing, ensuring the entire frame transmission completion time is consistently within 16.7ms, without relying on hardware buffering.

[0041] This ensures that even when the link is busy, low-priority data sub-packets can be sent during the transmission gaps between P-frames or B-frames, further achieving the effect of "peak shaving and valley filling" of traffic.

[0042] In an optional embodiment, after step S105 above, in which each keyframe in the audio / video stream is split into multiple data sub-packets for transmission, the method further includes: Acquire quality indicator data during the transmission of audio and video streams, including packet loss rate and average subjective score; Determine if the packet loss rate is greater than a first preset threshold; If the packet loss rate is greater than the first preset threshold, the bandwidth threshold will be reduced by the first preset step size. If the packet loss rate is less than or equal to the first preset threshold, determine whether the average subjective score is greater than the second preset threshold. If the average subjective score is greater than the second preset threshold, the bandwidth threshold will be increased by the second preset step size. If the average subjective score is less than or equal to the second preset threshold, the bandwidth threshold remains unchanged.

[0043] Specifically, after splitting each keyframe in the audio and video stream into multiple data sub-packets for transmission, quality metric data, such as packet loss rate and Mean Opinion Scoring (MOS), can be obtained from the receiving end. Then, based on the packet loss rate and MOS, the bandwidth threshold can be adjusted. In this way, a bidirectional feedback link can be built based on the RTCP protocol, enabling real-time transmission of quality metric data from the receiving end and rapid iteration of the bandwidth threshold, achieving the goal of dynamically adjusting the bandwidth threshold.

[0044] Specifically, when adjusting the bandwidth threshold based on packet loss rate and average subjective score, we can first determine if the packet loss rate is greater than a first preset threshold. If the packet loss rate is greater than the first preset threshold, it indicates that there are still a certain number of traffic peaks exceeding the bandwidth threshold. In this case, the bandwidth threshold can be lowered by a first preset step size so that the sender can redetermine a new traffic control strategy based on the lowered bandwidth threshold. If the packet loss rate is less than or equal to the first preset threshold, it indicates that the current traffic peaks are basically less than or equal to the bandwidth threshold. In this case, we can further determine if the average subjective score is greater than a second preset threshold. If the average subjective score is greater than the second preset threshold, the bandwidth threshold can be increased by a second preset step size so that the sender can redetermine a new traffic control strategy based on the increased bandwidth threshold. If the average subjective score is less than or equal to the second preset threshold, the current bandwidth threshold can be maintained unchanged.

[0045] It should be noted that the first preset threshold, the second preset threshold, the first preset step size, and the second preset step size can be set according to actual needs, and the embodiments of this application do not impose specific limitations. As an optional implementation, the first preset threshold can be 1%, the second preset threshold can be 4.5 points, the first preset step size can be 10%, and the second preset step size can be 5%.

[0046] Using the above methods, the bandwidth threshold can be dynamically adjusted based on the packet loss rate and average subjective score, ensuring that the traffic fluctuation rate after shaping is controlled within 5%, thus meeting the requirements for high-quality and low-latency transmission.

[0047] In an optional embodiment, step S104, determining whether the triggering conditions of the traffic control strategy are met based on the key frame data volume and bandwidth threshold, includes: Determine whether the amount of keyframe data is greater than the product of the bandwidth threshold and the preset coefficient; If the amount of keyframe data is greater than the product of the bandwidth threshold and the preset coefficient, determine the triggering conditions that currently satisfy the traffic control strategy. If the amount of keyframe data is less than or equal to the product of the bandwidth threshold and the preset coefficient, the triggering condition for the current traffic control strategy is determined to be unsatisfactory.

[0048] Specifically, the above preset coefficients can be set according to actual needs, and no specific limit is made here, such as 70%.

[0049] When determining whether the triggering conditions for a traffic control strategy are met based on the keyframe data volume and bandwidth threshold, it can be determined whether the keyframe data volume is greater than the product of the bandwidth threshold and a preset coefficient. If the keyframe data volume is greater than the product of the bandwidth threshold and the preset coefficient, it can be determined that the triggering conditions for a traffic control strategy are met and traffic control needs to be performed. If the keyframe data volume is less than or equal to the product of the bandwidth threshold and the preset coefficient, it can be determined that the triggering conditions for a traffic control strategy are not met and traffic control does not need to be performed.

[0050] In this way, based on the amount of keyframe data and the bandwidth threshold, it is possible to accurately determine whether the triggering conditions of the traffic control strategy are met, and then determine whether traffic control is needed based on the judgment result.

[0051] In an optional embodiment, before step S104 above, which determines whether the triggering conditions of the traffic control strategy are met based on the key frame data volume and bandwidth threshold, the method further includes: Determine whether the bandwidth threshold is less than the third preset threshold; If the bandwidth threshold is less than the third preset threshold, only the audio stream in the audio and video stream will be sent; If the bandwidth threshold is greater than or equal to the third preset threshold, obtain the transmission code rate corresponding to the link and determine whether the transmission code rate is greater than the bandwidth threshold. When the transmission bitrate is greater than the bandwidth threshold, reduce the encoding parameters of the audio and video streams, where the encoding parameters include the encoding bitrate; If the transmission bit rate is less than the bandwidth threshold, the step of determining whether the triggering conditions of the traffic control strategy are met is executed based on the key frame data volume and the bandwidth threshold.

[0052] Specifically, we can first determine if the bandwidth threshold is less than the third preset threshold. If the bandwidth threshold is less than the third preset threshold, it means that the bandwidth available to the sender is very small. If audio and video streams are transmitted simultaneously, it will cause problems such as video screen tearing, stuttering, and audio disconnection. Therefore, in this case, we can abandon the transmission of video stream data and only transmit audio stream data (the amount of audio stream data is much smaller than that of video stream data) to ensure normal transmission of audio stream. If the bandwidth threshold is greater than or equal to the third preset threshold, it means that the bandwidth available to the sender is relatively large. In this case, we can further determine if the transmission bitrate is greater than the bandwidth threshold. If the transmission bitrate is greater than the bandwidth threshold, it means that the bandwidth threshold cannot meet the current transmission bitrate. In this case, we can reduce the encoding parameters (such as the encoding bitrate) under the current encoding type to reduce the amount of data transmitted in the audio and video streams to be sent. If the transmission bitrate is less than or equal to the bandwidth threshold, it indicates that the bandwidth threshold can meet the current transmission bitrate. In this case, it can be further determined whether the keyframe data volume is greater than the product of the bandwidth threshold and a preset coefficient. If the keyframe data volume is greater than the product of the bandwidth threshold and the preset coefficient, each keyframe can be split into multiple data sub-packets, and these sub-packets can be sent according to the link status to reduce frame-level traffic spikes caused by keyframes and make traffic peaks smoother. If the keyframe data volume is less than or equal to the product of the bandwidth threshold and the preset coefficient, the audio and video streams can be sent normally according to the current status. It should be noted that this third preset threshold can be set according to actual needs, and this application does not impose specific limitations.

[0053] The above methods allow for more sophisticated traffic control based on bandwidth thresholds and transmission bitrate, thereby enabling precise control of audio and video stream traffic.

[0054] In an optional embodiment, step S103, obtaining the bandwidth threshold of the link used to send audio and video streams, includes: The bandwidth threshold is obtained from the server. The bandwidth threshold is calculated by the server using Gaussian process regression based on the time-domain features of link parameters, the time-domain features of traffic, and the time-domain features of bandwidth. The time-domain features of link parameters are obtained by the server by extracting features from the link parameters within the latest first preset period. The time-domain features of traffic are obtained by the server by extracting features from the historical traffic data within the latest second preset period. The time-domain features of bandwidth are obtained by the server by extracting features from the historical bandwidth data within the latest second preset period.

[0055] Specifically, the aforementioned bandwidth threshold can be calculated by the server using Gaussian process regression based on the time-domain characteristics of link parameters, traffic, and bandwidth. The time-domain characteristics of the link parameters are obtained by extracting features from the link parameters within the most recently acquired first preset period. For example, this could involve performing short-window statistics on link parameters within a 100ms period (e.g., calculating the mean, variance, maximum value, volatility, etc., of a 5ms / 10ms sliding window) to transform high-frequency discrete sampling data into statistically significant time-domain features. The time-domain characteristics of the traffic are obtained by extracting features from historical traffic data within the most recently acquired second preset period on the server, and the time-domain characteristics of the bandwidth are obtained by extracting features from historical bandwidth data within the most recently acquired second preset period on the server. For example, this could involve performing time-series feature extraction on 24-hour historical traffic and bandwidth data (e.g., calculating the historical mean and peak values ​​of bandwidth thresholds, the correlation characteristics between link parameters and bandwidth, and daily / hourly traffic patterns for the same time period or network condition). Then, using the bandwidth threshold as the prediction target and incorporating time-domain features as input, a Gaussian Process Regression (GPR) probabilistic model is constructed. In this GPR probabilistic model, it is assumed that the predicted value y (i.e., the bandwidth threshold) follows a Gaussian distribution, i.e., y GP(m(x),k(x,x′)), where m(x) is the mean function (usually taken as 0 to simplify calculation), and k(x,x′) is the core function (characterizing the nonlinear relationship between features and target), and the bandwidth threshold is calculated accordingly.

[0056] In this way, edge nodes are responsible for collecting real-time link parameters, while the server is responsible for calculating bandwidth thresholds, enabling unified management and scheduling of millions of nodes.

[0057] In an optional embodiment, the audio and video stream traffic control process provided in this application embodiment can be as follows: Figure 2 As shown, it includes the following steps: Step S201: Receive RTP audio and video streams.

[0058] Here, RTP audio and video streams refer to encoded audio and video streams.

[0059] Step S202: Parse and extract features from the RTP audio and video streams.

[0060] This section primarily parses the timestamp, sequence number, and payload type in the RTP packet header, identifies the frame type (I-frame, P-frame, or B-frame) in the audio and video streams, and calculates the data volume of a single frame. A single frame of data is split into multiple RTP packets due to packet fragmentation. RTP packets within the same frame have the same timestamp, and the payload type can be used to determine whether it is an audio packet, video packet, or other type (such as a FEC packet). The sequence numbers of RTP packets in each stream are consecutive, and these numbers can be used to determine if there are any packet losses or out-of-order delivery. The sending end also needs to calculate the amount of data sent this time so that, after receiving the server's predicted bandwidth threshold, it can adjust the sending queue parameters by calculating whether the data volume per unit time exceeds the predicted bandwidth threshold.

[0061] Step S203: Dynamically calculate the bandwidth threshold.

[0062] The server also parses the RTP data, calculating the packet loss rate within a first preset time period (e.g., 100ms) based on the sequence number of RTP packets in the same stream. It calculates latency and jitter based on timestamps, and calculates the received traffic based on packet data volume. Then, based on the calculated packet loss rate and latency, it infers the current bandwidth and, combined with historical bandwidth and traffic data within the first preset time period (e.g., 24 hours), calculates a reasonable bandwidth threshold. The bandwidth threshold can be calculated every 100ms, and the previous threshold is recorded. If the threshold changes, it is fed back to the sender, which then uses the bandwidth threshold along with its own parameters to perform traffic shaping.

[0063] Step S204: Determine the flow control strategy.

[0064] The server calculates the bandwidth threshold based on link parameters and historical bandwidth and traffic data over 24 hours. For example, if the current stream's bandwidth threshold is 10M, the sender controls the peak data volume sent per unit time to not exceed 10M based on the current data volume in the buffer queue (this also needs to consider the current link latency and jitter; for example, if the latency is high, the peak of 10M needs to be multiplied by a coefficient such as 0.8). A memory pool can be used to manage the cache, and a red-black tree can be used to implement the priority queue. The shaping strategy switching response is ≤10ms, achieving "peak shaving and valley filling." I-frame hierarchical buffer scheduling can be accomplished using a red-black tree priority queue. For example, 1.2MB of I-frame data is split into three 400KB sub-packets, marked as high, medium, and low priority, and inserted into the queue. The main thread prioritizes sending high-priority sub-packets, while background threads poll the link's idle status and trigger the sending of medium and low-priority sub-packets during P / B frame transmission intervals. The entire scheduling process ensures thread safety through mutex locks, and software timers precisely control the timing of transmission. The transmission time of the entire frame is consistently within 16.7ms, without relying on hardware caches.

[0065] Step S205: Send RTP audio and video streams.

[0066] Step S206: Receive quality indicator feedback and iteratively update the bandwidth threshold.

[0067] By receiving quality metrics (such as packet loss rate and MOS score) from the receiver using RTCP messages, the traffic control strategy can adapt to changes in coding characteristics and link state, thus solving the problem of poor adaptability of traditional fixed strategies.

[0068] Figure 2 The audio and video stream traffic control process can be achieved through... Figure 3 The audio and video stream traffic control system shown is used to achieve this, and the audio and video stream traffic control system includes the following four modules: 1. RTP Stream Parsing Module: Responsible for parsing RTP packet header information and audio / video frame types, outputting frame-level data volume statistics results, with a processing latency of ≤1ms; 2. Bandwidth Awareness Module: Edge nodes are responsible for collecting real-time link parameters, while the server is responsible for historical data training and threshold calculation, supporting unified management and scheduling of millions of nodes; 3. Traffic Shaping Module: Integrates a probabilistic shaping module and a hierarchical cache scheduling module. It uses a memory pool to manage the cache, implements a priority queue using a red-black tree, and has a shaping strategy switching response of ≤10ms. It is the core module for achieving "peak shaving and valley filling". 4. Feedback Control Module: Based on the RTCP protocol, a bidirectional feedback link is constructed to realize the real-time transmission of quality indicators of the receiving end and the rapid iteration of shaping parameters.

[0069] To address the core shortcomings of existing RTP audio and video streaming technologies, such as poor adaptability of traffic shaping schemes to encoding characteristics and link states, leading to unstable transmission and low bandwidth utilization, this invention proposes a dynamic traffic control scheme that coordinates the entire link (encoding-transmission-feedback), offering the following advantages: 1. Achieve accurate perception of RTP stream characteristics, capture key information such as frame type and single frame data volume in real time, and provide data support for traffic regulation; 2. Construct a Gaussian process regression (GPR) probability model to dynamically calculate the bandwidth threshold, enabling the traffic adjustment strategy to be adjusted in real time according to the time-domain characteristics of link parameters, traffic, and bandwidth. 3. By combining the strategies of "probabilistic shaping + frame-level buffer scheduling", traffic "peak shaving and valley filling" can be achieved, reducing the transmission packet loss rate and improving link utilization. 4. Establish a two-way feedback mechanism to dynamically adjust the bandwidth threshold based on the quality indicator data of the receiving end (such as packet loss rate and average subjective score), and ensure that the traffic fluctuation rate after shaping based on the bandwidth threshold, transmission bit rate and key frame data volume is controlled within 5%, so as to meet the requirements of high image quality and low latency transmission. 5. Through RTP-based audio and video stream traffic control technology, it is particularly suitable for scenarios with extremely high requirements for transmission stability and low latency, such as 8K ultra-high-definition live streaming, remote medical consultation, cloud gaming, and enterprise-level video conferencing, and can be seamlessly adapted to existing audio and video transmission systems.

[0070] See Figure 4 , Figure 4 This is a schematic diagram of the structure of an audio / video stream flow control device provided in an embodiment of this application. Figure 4 As shown, the audio / video stream flow control device 400 includes: The acquisition and parsing module 401 is used to acquire the audio and video stream to be sent, and to parse the audio and video stream to obtain the frame type and single frame data volume in the audio and video stream; The determination module 402 is used to determine the key frame data volume based on the frame type and the single frame data volume; The first acquisition module 403 is used to acquire the bandwidth threshold of the link used to send audio and video streams, wherein the bandwidth threshold is used to characterize the upper limit of the bandwidth that can be allocated to the link. The first judgment module 404 is used to determine whether the triggering conditions of the traffic control strategy are met based on the key frame data volume and bandwidth threshold. The first sending module 405 is used to split each keyframe in the audio and video stream into multiple data sub-packets for sending when the triggering conditions of the current flow control strategy are met.

[0071] Furthermore, the first transmitting module 405 includes: The first acquisition submodule is used to acquire the current link status; The first sending submodule is used to send multiple data sub-packets sequentially in descending order of priority when the link is in an idle state. The second sending submodule is used to send multiple data sub-packets sequentially and at intervals in order of priority from high to low when the link is busy.

[0072] Furthermore, the multiple data sub-packets include a first data sub-packet, a second data sub-packet, and a third data sub-packet, with the first data sub-packet having a higher priority than the second data sub-packet, and the second data sub-packet having a higher priority than the third data sub-packet; the second sending submodule includes: The sending unit is used to send the first data sub-packet, and after the first data sub-packet is sent, to insert the second data sub-packet and the third data sub-packet into different transmission gaps of the normal frames of the audio and video stream for transmission.

[0073] Furthermore, the audio / video stream flow control device 400 also includes: The second acquisition module is used to acquire quality indicator data during the transmission of audio and video streams. The quality indicator data includes packet loss rate and average subjective score. The second judgment module is used to determine whether the packet loss rate is greater than the first preset threshold. The down-adjustment module is used to reduce the bandwidth threshold by a first preset step size when the packet loss rate is greater than a first preset threshold. The third judgment module is used to determine whether the average subjective score is greater than the second preset threshold when the packet loss rate is less than or equal to the first preset threshold. The adjustment module is used to increase the bandwidth threshold by a second preset step size when the average subjective score is greater than the second preset threshold. The maintenance module is used to maintain the bandwidth threshold unchanged when the average subjective score is less than or equal to a second preset threshold.

[0074] Furthermore, the first judgment module 404 includes: The judgment submodule is used to determine whether the amount of keyframe data is greater than the product of the bandwidth threshold and the preset coefficient; The first determining submodule is used to determine the triggering conditions that currently satisfy the traffic control strategy when the amount of key frame data is greater than the product of the bandwidth threshold and the preset coefficient. The second determining submodule is used to determine the triggering conditions that do not meet the traffic control strategy when the amount of key frame data is less than or equal to the product of the bandwidth threshold and the preset coefficient.

[0075] Furthermore, the audio / video stream flow control device 400 also includes: The fourth judgment module is used to determine whether the bandwidth threshold is less than the third preset threshold; The second sending module is used to send only the audio stream in the audio and video stream when the bandwidth threshold is less than the third preset threshold. The fifth judgment module is used to obtain the transmission code rate corresponding to the link when the bandwidth threshold is greater than or equal to the third preset threshold, and to determine whether the transmission code rate is greater than the bandwidth threshold. The reduction module is used to reduce the encoding parameters of the audio and video streams when the transmission bitrate is greater than the bandwidth threshold. The encoding parameters include the encoding bitrate. The execution module is used to perform steps based on the key frame data volume and bandwidth threshold to determine whether the triggering conditions of the traffic control strategy are met when the transmission bit rate is less than the bandwidth threshold.

[0076] Furthermore, the first acquisition module 403 includes: The second acquisition submodule is used to obtain the bandwidth threshold from the server. The bandwidth threshold is obtained by the server through Gaussian process regression calculation based on the time-domain features of link parameters, the time-domain features of traffic, and the time-domain features of bandwidth. The time-domain features of link parameters are obtained by the server through feature extraction of the link parameters within the latest acquired first preset period. The time-domain features of traffic are obtained by the server through feature extraction of the historical traffic data within the latest acquired second preset period. The time-domain features of bandwidth are obtained by the server through feature extraction of the historical bandwidth data within the latest acquired second preset period.

[0077] It should be noted that the audio and video stream flow control device 400 can implement the audio and video stream flow control method provided in any of the aforementioned method embodiments, and can achieve the same technical effect, which will not be described in detail here.

[0078] like Figure 5 As shown in the illustration, this application also provides an electronic device, including a processor 511, a communication interface 512, a memory 513, and a communication bus 514, wherein the processor 511, the communication interface 512, and the memory 513 communicate with each other via the communication bus 514. Memory 513 is used to store computer programs; In one embodiment of this application, the processor 511, when executing the program stored in the memory 513, implements the audio and video stream traffic control method provided in any of the foregoing method embodiments.

[0079] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the audio and video stream flow control method provided in any of the foregoing method embodiments.

[0080] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0081] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software plus a general-purpose hardware platform, or of course, using hardware. Based on this understanding, the above technical solutions, in essence or the parts that contribute to the related technology, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0082] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0083] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for controlling the flow of audio and video streams, characterized in that, The method includes: The audio and video stream to be sent is acquired, and the audio and video stream is parsed to obtain the frame type and single frame data volume in the audio and video stream; Based on the frame type and the single frame data volume, determine the key frame data volume; Obtain a bandwidth threshold for the link used to send the audio and video streams, wherein the bandwidth threshold is used to characterize the upper limit of bandwidth that can be allocated to the link; Based on the keyframe data volume and the bandwidth threshold, determine whether the triggering conditions of the traffic control strategy are met. If the triggering conditions of the current traffic control strategy are met, each keyframe in the audio and video stream is split into multiple data sub-packets and sent.

2. The method according to claim 1, characterized in that, The step of splitting each keyframe in the audio / video stream into multiple data sub-packets for transmission includes: Obtain the current link status of the link; When the link is in an idle state, the multiple data sub-packets are sent sequentially and continuously in order of priority from high to low. When the link is busy, the multiple data sub-packets are sent sequentially and at intervals in order of priority from high to low.

3. The method according to claim 2, characterized in that, The plurality of data sub-packets include a first data sub-packet, a second data sub-packet, and a third data sub-packet, wherein the first data sub-packet has a higher priority than the second data sub-packet, and the second data sub-packet has a higher priority than the third data sub-packet; The step of sending the multiple data sub-packets sequentially and at intervals according to their priority from high to low includes: The first data sub-packet is sent, and after the first data sub-packet is sent, the second data sub-packet and the third data sub-packet are sequentially inserted into different transmission gaps of the normal frames of the audio and video stream for transmission.

4. The method according to claim 1, characterized in that, After splitting each keyframe in the audio and video stream into multiple data sub-packets for transmission, the method further includes: Acquire quality indicator data during the transmission of the audio and video streams, wherein the quality indicator data includes packet loss rate and average subjective score; Determine whether the packet loss rate is greater than a first preset threshold; If the packet loss rate is greater than the first preset threshold, the bandwidth threshold is reduced by a first preset step size; If the packet loss rate is less than or equal to the first preset threshold, determine whether the average subjective score is greater than the second preset threshold. If the average subjective score is greater than the second preset threshold, the bandwidth threshold is increased by the second preset step size; If the average subjective score is less than or equal to the second preset threshold, the bandwidth threshold remains unchanged.

5. The method according to claim 1, characterized in that, The step of determining whether the triggering conditions for the traffic control strategy are met based on the keyframe data volume and the bandwidth threshold includes: Determine whether the amount of keyframe data is greater than the product of the bandwidth threshold and a preset coefficient; If the amount of keyframe data is greater than the product of the bandwidth threshold and the preset coefficient, determine the triggering condition that currently satisfies the traffic control strategy. If the amount of keyframe data is less than or equal to the product of the bandwidth threshold and the preset coefficient, it is determined that the triggering condition of the current traffic control strategy is not met.

6. The method according to claim 5, characterized in that, Before determining whether the triggering conditions of the traffic control strategy are met based on the keyframe data volume and the bandwidth threshold, the method further includes: Determine whether the bandwidth threshold is less than a third preset threshold; If the bandwidth threshold is less than the third preset threshold, only the audio stream in the audio and video stream will be sent; If the bandwidth threshold is greater than or equal to the third preset threshold, the transmission code rate corresponding to the link is obtained, and it is determined whether the transmission code rate is greater than the bandwidth threshold. If the transmission bitrate is greater than the bandwidth threshold, the encoding parameters of the audio and video streams are reduced, wherein the encoding parameters include the encoding bitrate; If the transmission bit rate is less than the bandwidth threshold, the step of determining whether the triggering conditions of the traffic control strategy are met based on the key frame data volume and the bandwidth threshold is executed.

7. The method according to claim 1, characterized in that, The step of obtaining the bandwidth threshold of the link used to send the audio and video streams includes: The bandwidth threshold is obtained from the server, wherein the bandwidth threshold is calculated by the server using Gaussian process regression based on the time-domain features of link parameters, the time-domain features of traffic, and the time-domain features of bandwidth. The time-domain features of link parameters are obtained by the server by feature extraction of link parameters within the most recently acquired first preset period. The time-domain features of traffic are obtained by the server by feature extraction of historical traffic data within the most recently acquired second preset period. The time-domain features of bandwidth are obtained by the server by feature extraction of historical bandwidth data within the most recently acquired second preset period.

8. A device for regulating the flow of audio and video streams, characterized in that, The device includes: The acquisition and parsing module is used to acquire the audio and video stream to be sent, and to parse the audio and video stream to obtain the frame type and single frame data volume in the audio and video stream; The determination module is used to determine the amount of keyframe data based on the frame type and the amount of data in a single frame; The first acquisition module is used to acquire a bandwidth threshold of the link used to send the audio and video streams, wherein the bandwidth threshold is used to characterize the upper limit of bandwidth that can be allocated to the link; The first judgment module is used to determine whether the triggering conditions of the traffic control strategy are met based on the amount of key frame data and the bandwidth threshold. The sending module is used to split each keyframe in the audio and video stream into multiple data sub-packets for sending when the triggering conditions of the current traffic control strategy are met.

9. An electronic device, characterized in that, The electronic device includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, when executing a program stored in memory, implements the method for controlling the flow of audio and video streams as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the audio and video stream flow control method according to any one of claims 1-7.

Images