Code stream variable time sensitive network audio and video data scheduling method and device

Abstract

The invention belongs to the technical field of network communication, and particularly discloses a code stream variable time sensitive network audio and video data scheduling method and device. Based on absolute priority time-sensitive data scheduling and credit shaping audio and video and best effort data scheduling mechanisms, resolution adjustment or code stream mapping conversion is performed by monitoring the switching node data buffer and the switching node audio and video frame rate of the time-sensitive network, the audio and video data volume in the time-sensitive network is optimized, and the time-sensitive network quality is improved. The audio and video data service quality of the time-sensitive network is improved; according to the method, the transmitted audio and video code stream is reduced or the transmitted audio and video code stream is improved according to the cache condition of the output queue, so that the phenomenon of network congestion when the audio and video data volume is large is relieved, and the audio and video service quality can be improved by effectively utilizing the network bandwidth when the network is idle. According to the invention, the audio and video data flow transmission in the time-sensitive network is optimized on the premise of not changing the existing time-sensitive network architecture.

Description

Technical Field

[0001] The present application relates to the field of network communication technology, and more specifically, to a method and device for scheduling time-sensitive network audio and video data with variable bitrate. Background Art

[0002] With the rapid development of networking and intelligent industrial control, the types of services carried by industrial communication networks are rapidly increasing, and the requirements for real-time and stable data transmission are gradually increasing. Time-sensitive networks based on standard Ethernet architecture, featuring precise time triggering, reliable data transmission, priority data transmission control, and efficient network bandwidth utilization, are becoming the development trend of future industrial communication networks.

[0003] Time-Sensitive Networking (TSN) is a data link layer protocol based on the standard Ethernet architecture that provides deterministic data transmission capabilities through mechanisms such as clock synchronization, enhanced scheduling, and traffic shaping. TSN provides microsecond-level time synchronization through precise clock synchronization. TSN uses absolute priority, enhanced transmission selection, credit shaping, asynchronous traffic shaping, and time-aware shaping to schedule network data. TSN offers a variety of traffic shaping mechanisms, including credit-based, time-aware, asynchronous traffic, and periodic queue forwarding, to limit the maximum rate of burst traffic, ensuring smooth network transmission and alleviating network congestion.

[0004] Industrial network traffic flows are categorized by their characteristics into three main types: time-triggered (TT) flows, audio and video bridging (AVB) flows, and best-effort (BE) flows. Each type of flow utilizes a different shaping mechanism and scheduling method. To ensure the real-time delivery of critical data, TSNs typically employ a preemptive strategy to prioritize the transmission of time-sensitive data flows. However, audio and video data can experience transmission delays due to lengthy queues. In severe cases, buffered audio and video data can exceed the buffer capacity of network switching equipment, leading to data loss.

[0005] Audio and video data streams typically occupy significant network bandwidth, and the quality of audio and video network service is negatively correlated with network transmission latency. Existing time-sensitive network equipment lacks the ability to change the bitrate of audio and video data streams. When time-sensitive data streams frequently occupy network channels, existing time-sensitive network equipment cannot promptly adjust the amount of audio and video data and transmit it in the time slots between the occupied data streams. This results in frame loss on the network, impacting service quality. Summary of the Invention

[0006] In response to the defects of the existing technology, the purpose of this application is to provide a time-sensitive network audio and video data scheduling method and device with variable bit rate, aiming to solve the problem that the existing time-sensitive network audio and video data traffic cannot be automatically adjusted according to the time-sensitive network traffic.

[0007] In a first aspect, the present application provides a method for scheduling time-sensitive network audio and video data with variable bitrates, the method comprising: Receiving time-sensitive network data; Classify the received time-sensitive network data and transmit them to the time-sensitive data queue, the original audio and video data queue and the best-effort data queue respectively; Copy the data in each original audio and video code stream data queue to the target audio and video code stream data queue; Monitor each target audio and video stream queue in real time, determine whether the stream conversion operation is to down-stream or up-stream based on the credit value of each target audio and video stream queue, and place the down-stream / up-stream audio and video data frames into the target audio and video stream queue; Based on the credit shaping scheduling method, the credit amount of each target audio and video data queue and the best-effort data queue is calculated separately; Gated selection transmission is performed through absolute priority-based time-sensitive data scheduling and credit-shaping audio and video and best-effort data scheduling mechanism. The absolute priority-based time-sensitive data scheduling adopts time-sensitive priority gating data shaping to prioritize time-sensitive data shaping transmission. For time-sensitive data queues, the frame preemption principle is followed and transmission is performed according to the time-sensitive data queue timing; the credit-shaping audio and video and best-effort data scheduling mechanism controls the transmission order of each target audio and video data queue and best-effort data queue based on the calculated credit amount.

[0008] Preferably, according to the credit value of each target audio and video stream queue, the stream conversion operation is determined to be down-stream processing, and the down-stream audio and video data frame is placed in the target audio and video stream queue, specifically: (1) If, within a certain time period, it is identified that the credit value of a target audio and video stream queue exceeds a first preset ratio of the queue credit limit, the decoded audio and video data of the multi-stream is subjected to bit rate reduction processing; (2) Copy the mainstream data of the audio and video data frame to the target audio and video code stream queue, and replace the data in the target audio and video code stream FIFO queue; (3) Detect whether the credit value of the transformed target audio and video stream queue does not exceed the first preset ratio of the queue credit limit in the next time period. If not, repeat steps (1)-(2).

[0009] Preferably, the code reduction coefficient and the ratio of the collection credit value of the queue on this path to the maximum credit value of the queue are complementary to each other.

[0010] Preferably, the maximum credit value AV credit max is determined by: .

[0011] Preferably, when processing the bitstream, it also includes first performing frame rate reduction encoding and / or resolution reduction encoding on the audio and video data, wherein the frame rate reduction encoding is to perform frame rate reduction encoding according to the frame rate after bitrate reduction, and replace the data in the target audio and video bitstream FIFO queue; the resolution reduction encoding is to perform frame rate reduction encoding according to the resolution after bitrate reduction, and replace the data in the target audio and video bitstream queue.

[0012] Preferably, according to the credit value of each target audio and video stream queue, the stream conversion operation is determined to be up-stream processing, and the up-stream audio and video data frame is placed in the target audio and video stream queue, specifically: (1) If, within a certain time period, it is identified that the credit value of a certain target audio and video stream FIFO queue exceeds the credit limit of the queue of the second preset ratio, the main stream data of the audio and video source of the channel is copied to the target audio and video stream queue to replace the data in the target audio and video stream queue; (2) Detect whether the credit value of the transformed target audio and video code stream FIFO queue does not exceed the first preset ratio of the queue credit limit in the next time period. If not, perform code stream reduction processing.

[0013] Preferably, the queue depths of the time-sensitive data queue, the original audio and video data queue, and the best-effort data queue are determined by:

[0014] The safety factor is a tolerance value set for the time-sensitive network, which is greater than 0 and less than 1.

[0015] Preferably, gated selective transmission is performed by scheduling time-sensitive data based on absolute priority and scheduling audio and video and best-effort data based on credit shaping, specifically: (1) When the time-sensitive data queue is empty, the corresponding queue data can be shaped and forwarded according to the credit of the audio and video data queue and the best-effort queue data; (2) If the time-sensitive data queue is not empty and the data shaper is transmitting time-sensitive data, the data shaper transmits the time-sensitive data in the first-in-first-out order of the queue; (3) If the time-sensitive data queue is not empty and the data shaper is transmitting non-time-sensitive data, the data shaper starts the time-sensitive data shaping transmission after completing the data transmission of the current queue; (4) When the audio and video code stream data frame arrives at the queue, if there is a low-priority data frame being transmitted, the audio and video code stream data frame will be queued and wait, and the credit value of the audio and video code stream queue will increase at the rate of IdleSlope. After the low-priority data frame is transmitted, the audio and video queue data with the largest credit value will start to be transmitted, and the credit value will decrease at the rate of SendSlope. (5) After the audio and video stream queue data frame transmission is completed, if the credit value is less than 0, it will increase to 0 at the rate of IdleSlope; if the credit value is greater than 0, it will be set to 0; (6) Data frames in the queue can be transmitted only when the credit value of the audio and video stream queue is greater than or equal to 0; when the credit value of the audio and video stream queue is less than 0, data frame transmission cannot be started, but data frames that were started before the credit value is reduced to 0 can continue to be transmitted.

[0016] Preferably, the video data of the audio and video data source has a single stream or multi-stream characteristic.

[0017] In the second aspect, the present application provides a time-sensitive network audio and video data scheduling device with variable bit rate, which includes: at least one memory for storing programs; at least one processor for entering the program stored in the memory, and when the program stored in the memory is entered, the processor is used to enter the audio and video data scheduling method as described in the first aspect.

[0018] It can be understood that the beneficial effects of the second aspect mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here.

[0019] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the existing technologies: The present application proposes a method and device for scheduling audio and video data in a time-sensitive network with variable bitrates. By means of absolute priority-based time-sensitive data scheduling and credit-shaping-based audio and video and best-effort data scheduling mechanisms, monitoring the data buffering of time-sensitive network switching nodes, adjusting the audio and video frame rate and resolution of switching nodes, or bitrate mapping conversion, the amount of audio and video data in the time-sensitive network is optimized, and the quality of service of audio and video data in the time-sensitive network is improved. According to the output queue cache situation, the transmitted audio and video bitrate is reduced or increased, which can not only alleviate the phenomenon of network congestion when the amount of audio and video data is large, but also effectively utilize the network bandwidth to improve the quality of audio and video services when the network is idle. The present application improves the quality of audio and video and other data transmission in the time-sensitive network under the premise of ensuring the deterministic transmission of strong real-time time-sensitive data; without changing the existing time-sensitive network architecture, the audio and video data traffic transmission in the time-sensitive network is optimized within the network switching node, and variable bitrate adjustment of audio and video data with multiple traffic characteristics or a single traffic characteristic is realized in the physical link transmission, thereby improving the overall network service quality of the time-sensitive network. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a schematic diagram of an embodiment of the present application providing a scheduling method using absolute priority and credit shaping to sequentially gate and select transmission of a time-sensitive data queue, an audio and video data queue, and a best-effort data queue.

[0021] Figure 2 This is a schematic diagram of the credit amount of the data shaper at each moment provided in an embodiment of the present application.

[0022] Figure 3 This is a flow chart of a time-sensitive network audio and video data scheduling method with variable bitstream applied to a single bitstream audio and video source provided by an embodiment of the present application.

[0023] Figure 4 This is a flow chart of a time-sensitive network audio and video data scheduling method with variable bitstreams applied to multi-bitstream audio and video sources provided by an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0025] The term "and / or" in this application describes an association relationship between associated objects, indicating that three relationships can exist. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, or B exists alone. The symbol " / " in this application indicates that the associated objects are in an "or" relationship, for example, A / B means A or B.

[0026] In this application, the terms "first" and "second" are used to distinguish different objects, rather than to describe a specific order of objects. For example, the terms "first response message" and "second response message" are used to distinguish different response messages, rather than to describe a specific order of response messages.

[0027] The term "electrical connection" in this application can be a direct circuit connection or signal transmission through a communication protocol.

[0028] In the embodiments of this application, words such as "exemplary" or "for example" are used to indicate examples, illustrations, or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of this application should not be interpreted as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "for example" is intended to present the relevant concepts in a concrete manner.

[0029] In the description of the embodiments of the present application, unless otherwise specified, "multiple" means two or more, for example, multiple processing units means two or more processing units, etc.; multiple elements means two or more elements, etc.

[0030] The present application provides a method for scheduling time-sensitive network audio and video data with variable bitrate, the method comprising: Receiving time-sensitive network data; Classify the received time-sensitive network data and transmit them to the time-sensitive data queue, the original audio and video data queue and the best-effort data queue respectively; Copy the data in each original audio and video code stream data queue to the target audio and video code stream data queue; Monitor each target audio and video stream queue in real time, determine whether the stream conversion operation is to down-stream or up-stream based on the credit value of each target audio and video stream queue, and place the down-stream / up-stream audio and video data frames into the target audio and video stream queue; Based on the credit shaping scheduling method, the credit amount of each target audio and video data queue and the best-effort data queue is calculated separately; Gated selection transmission is performed through absolute priority-based time-sensitive data scheduling and credit-shaping audio and video and best-effort data scheduling mechanism. The absolute priority-based time-sensitive data scheduling adopts time-sensitive priority gating data shaping to prioritize time-sensitive data shaping transmission. For time-sensitive data queues, the frame preemption principle is followed and transmission is performed according to the time-sensitive data queue timing; the credit-shaping audio and video and best-effort data scheduling mechanism controls the transmission order of each target audio and video data queue and best-effort data queue based on the calculated credit amount.

[0031] Correspondingly, the present application provides a time-sensitive network audio and video data scheduling device with variable bit rate, the device comprising: A data receiving port is used to receive time-sensitive network data from an input port or a data source; A data filter is used to classify the received time-sensitive network data and transmit it to a time-sensitive data queue, an original audio and video data queue, and a best-effort data queue respectively; The audio and video stream converter is configured to copy the data in each original audio and video stream data queue to the target audio and video stream data queue; monitor each target audio and video stream queue in real time, determine whether to perform a stream conversion operation, based on the credit value of each target audio and video stream queue, and place the down-streamed / up-streamed audio and video data frames into the target audio and video stream queue; and calculate the credit of each target audio and video data queue and the best-effort data queue based on a credit shaping scheduling method. A data shaper is used to perform gated selective transmission through absolute priority-based time-sensitive data scheduling and credit-shaped audio and video and best-effort data scheduling mechanisms. The absolute priority-based time-sensitive data scheduling uses time-sensitive priority gating data shaping to prioritize time-sensitive data shaping transmission. For time-sensitive data queues, the frame preemption principle is followed and transmission is performed according to the time-sensitive data queue timing; the credit-shaped audio and video and best-effort data scheduling mechanism controls the transmission order of each target audio and video data queue and best-effort data queue based on the calculated credit amount.

[0032] Data sending port, used to output gated shaping data to the target network node.

[0033] It should be noted that before all scheduling steps begin, a time synchronization mechanism, data receiving ports, data filters, time-sensitive data queues, original audio and video data queues, target audio and video data queues, best-effort data queues, credit-based scheduling, priority gating, data shaping, and a sending port are established in the time-sensitive network switching node. This time synchronization mechanism uses a time-sensitive network time synchronization algorithm to ensure that the switching node time is consistent with the time-sensitive network time. For each audio and video data stream, only one of these network switching nodes is required to manage it.

[0034] Example 1 This embodiment is applied to a time-sensitive network that transmits time-sensitive data, audio and video data, and best-effort data, wherein the video data of the audio and video data source has a single bitstream characteristic.

[0035] The embodiment of the present application provides a method for scheduling time-sensitive network audio and video data with variable bit rate, which is based on credit shaping scheduling and priority gate and data shaping. Figure 1 As shown in the figure, TT1-TTn represent time-sensitive network data from channels 1 to n, corresponding to queues T1-Tn; AVB1-AVBn represent audio and video network data from channels 1 to n, corresponding to queues AV1-AVn; and BE1-BEn represent best-effort network data from channels 1 to n, corresponding to queues BE1-BEn. After queues AV1-AVn and BE1-BEn transmit CBSs using credit control, they are then connected to a gated selective transmission. Queues T1-Tn are each connected to a gated selective transmission. The gated list controls (n+1) switches, and the outputs of these switches are sent to the output port via the shaping transmission port. Credit shaping scheduling sets a CBS based on the packet length of the shaping queue. The credit increases at a constant rate approximately equal to the input data rate and decreases at a constant rate approximately equal to the output data rate. Data in a queue is transmitted using credit shaping scheduling only when the credit is at its maximum. Credit shaping scheduling allows low-priority data queues to obtain a certain transmission time slot through credit accumulation.

[0036] Audio and video stream converters such as Figure 3 As shown, the method includes the following steps: Step 1: Configure the time-sensitive network switching node data filter, time-sensitive data queue, original audio and video data queue, best-effort data queue, and data shaping mechanism (gated output). The specific process is as follows: (1) The time-sensitive network switching node determines through tolerance analysis that the sum of the inlet traffic rates of each time-sensitive bridge controller cannot exceed the rate of the bridge outlet.

[0037] (2) By predicting the data type, data transmission cycle, data rate, and time-sensitive delay of the network port of the time-sensitive network switching node, the time-sensitive data queue, the original audio and video data queue, the best-effort data queue depth, and the data gating output logic configuration are set.

[0038]

[0039] Among them, the safety factor is greater than 0 and less than 1.

[0040] The data gating output logic is configured as follows: the time-sensitive data queue is configured to comply with frame preemption, while the audio and video data queue and the best-effort data queue are configured to control the transmission order based on the credit amount calculated by credit shaping scheduling, and the data is transmitted using gating selection.

[0041] Step 2: The audio and video stream converter copies the data of each original audio and video stream queue to the target audio and video stream queue; the transmission order is controlled by calculating the credit of multiple transmission queues based on credit shaping timing. The specific credit calculation formula is as follows:

[0042] ,x=1,2,3,...,maximum value of tolerance analysis

[0043] in, is the flow credit reduction rate, is the traffic credit growth rate (also represents the logical bandwidth reserved for this type of traffic), is the theoretical bandwidth of the physical link, For the Group target audio and video bitrate, For the The audio and video stream sending rate of the group principle, For best-effort data flow data rates, For best-effort data flow sending rate, The weight defined for best-effort traffic.

[0044] It should be noted that pre-transmission is required when starting work, so the data in the original audio and video code stream data queue is copied to the target audio and video code stream data queue, and subsequent judgment is made based on the congestion situation of the target audio and video code stream data queue.

[0045] Based on the IEEE 802.1 QAV queuing and forwarding regulations, multi-channel audio and video streams and best-effort data streams are transmitted using a scheduling method based on credit shaping (CBS). This method is equivalent to idle slope bandwidth and spreads out packets as much as possible, allowing low-priority queues to also obtain a certain amount of transmission opportunities.

[0046] The audio / video stream converter determines the primary / secondary stream characteristics of each audio source. Since the audio / video source in Example 1 is a single stream, the audio / video stream converter decodes and stores key frame data in a memory unit, obtaining information such as the audio / video stream frame rate, resolution, and encoding method for subsequent frame rate and resolution adjustments. The audio / video stream converter copies the data from each original audio / video stream queue to the corresponding target audio / video stream queue.

[0047] Step 3: The bridge controller completes the transmission management of each original audio and video stream queue and the target audio and video stream queue, calculates the original audio and video source data flow of each channel, and the audio and video data decoding cache in the original audio and video data queue. Based on the credit value of each target audio and video stream queue, it determines the stream conversion operation. The specific process is as follows: (1) Initially, the audio and video stream converter copies the data of each original audio and video stream queue to the target audio and video stream queue; (2) The audio and video stream converter decodes the encoded audio and video data of the multiple original audio data queues and caches the decoded audio and video data frames; (3) The audio and video stream converter calculates the original data volume based on the original audio and video decoding cache data rate of each channel. The credit value of each channel cannot exceed the maximum credit value. Based on the target audio and video stream queue capacity and audio and video data rate, the maximum credit value of the audio and video stream is calculated.

[0048] (4) The audio and video stream converter reads the credit value of each target audio and video stream queue.

[0049] Step 4: The data shaper of the bridge controller adopts an absolute priority and credit-based scheduling method to select and transmit the time-sensitive data queue, audio and video data queue, and best-effort data gate in turn. Since time-sensitive data is often the most critical and real-time data in the system, time-sensitive network priority gating and data shaping must give priority to ensuring the transmission of time-sensitive data. Figure 1 and Figure 2 The specific process is as follows: (1) When the time-sensitive data queue is empty, the corresponding queue data can be shaped and forwarded according to the audio and video data queue and the best-effort queue data credit; (2) If the time-sensitive data queue is not empty and the data shaper is transmitting time-sensitive data, the data shaper transmits the time-sensitive data in the first-in-first-out order of the queue; (3) If the time-sensitive data queue is not empty and the data shaper is transmitting non-time-sensitive data, the data shaper starts the time-sensitive data shaping transmission after completing the data transmission of the current queue; (4) When the audio and video code stream data frame arrives at the queue, if there is a low-priority data frame being transmitted, the audio and video code stream data frame will be queued and wait, and the credit value of the audio and video code stream queue will increase at the rate of IdleSlope. After the low-priority data frame is transmitted, the audio and video queue data with the largest credit value will start to be transmitted, and the credit value will decrease at the rate of SendSlope. (5) After the audio and video stream queue data frame transmission is completed, if the credit value is less than 0, it will increase to 0 at the rate of IdleSlope; if the credit value is greater than 0, it will be set to 0; (6) Data frames in the queue can only be transmitted when the credit value of the audio and video stream queue is greater than or equal to 0; when the credit value of the audio and video stream queue is less than 0, data frame transmission cannot be started, but data frames that started to be transmitted before the credit value is reduced to 0 can continue to be transmitted.

[0050] like Figure 2 As shown, the horizontal axis is the time axis, the vertical axis is the credit amount, the solid line corresponds to AVB1, and the dotted line corresponds to AVBn.

[0051] The physical meanings of each moment are as follows: The time indicates that there is AVB1 audio and video data stream input on the network, but the credit amount does not exceed the BE data credit amount, and the BE data continues to be transmitted. The time indicates that there is AVBn audio and video data stream input on the network, but the credit amount does not exceed the BE data credit amount, and BE data continues to be transmitted The moment indicates that the AVB1 audio and video credit exceeds the BE data credit, BE data transmission stops, and AVB1 audio and video data transmission starts; The moment indicates that time-sensitive network data TT has arrived, so the transmission of AVB1 audio and video data is stopped and the transmission of TT data is started. The moment indicates that TT data transmission is completed and AVB1 audio and video data transmission continues. The moment indicates that the AVB1 audio and video data transmission is completed and the AVBn audio and video data transmission begins. The moment indicates that time-sensitive network data TT has arrived, so the transmission of AVBn audio and video data is stopped and the transmission of TT data is started. The moment indicates that TT data transmission is completed and AVBn audio and video data transmission continues. The moment indicates that the AVBn audio and video data transmission is completed and the AVB1 audio and video data transmission continues. The moment indicates that the AVB1 audio and video data transmission is completed, and the time-sensitive network data TT arrives, and the transmission of TT data begins. The moment indicates that the TT data transmission is completed and the BE data transmission continues.

[0052] Step 5: The audio and video code stream converter detects whether the data entry data rate of each target audio and video code stream queue matches the data exit data rate. If the audio and video data entry is continuously greater than the audio and video data exit for a certain period of time, it will cause a queue overflow and result in the loss of audio and video data packets. The audio and video code stream converter performs code stream reduction processing on the audio and video data frames in the entire original queue, and places the reduced code stream audio and video data frames into the target audio and video code stream queue, which is gated and selected for transmission by the data shaper. The audio and video code stream converter continues to compare the data entry data rate and the data exit data rate of the target audio and video code stream queue, and it must meet the requirement that the data entry data rate is less than the data exit rate within a period of time; if it is insufficient, it is necessary to continue to reduce the code until the data entry data rate of the target audio and video code stream queue is less than the data exit data rate. The specific process is as follows: (1) If, in time period t, the audio and video code stream converter identifies that the credit value of a target audio and video code stream queue is greater than the credit limit of the queue × 0.75, the audio and video code stream converter will down-stream the audio and video decoded data of the target audio and video stream.

[0053]

[0054] (2) The audio and video code stream converter performs frame rate downcoding according to the frame rate after bit rate reduction and replaces the data in the target audio and video code stream FIFO queue.

[0055]

[0056] (3) The audio and video code stream converter performs frame rate encoding according to the resolution after bit rate reduction and replaces the data in the target audio and video code stream queue.

[0057] Resolution after bit rate reduction = (original resolution × bit rate reduction coefficient) and round up The original resolution ratio is a commonly used display resolution, such as 1920×1080, 1280×720, 640×480, etc.

[0058] (4) The audio and video code stream converter detects whether the credit value of the converted target audio and video code stream queue is less than or equal to the queue credit limit × 0.75 within the next time period t. If it is not satisfied, the operations (1) to (3) are repeated until the data entry rate of the target audio and video data queue is less than the data exit rate.

[0059] It should be noted that when the amount of network transmission data decreases and the network becomes less congested, the in-queue data rate is lower than the out-queue data rate, and the out-queue data rate can be cleared.

[0060] Step 6: The audio and video code stream converter detects whether the data entry rate of each target audio and video code stream queue matches the data exit rate. If the data entry rate of a target audio and video code stream queue is much lower than the data exit rate (for example, less than 10%) within a certain time period, network bandwidth resources may be wasted due to low bit rate reduction. The audio and video code stream converter copies the audio and video data frames in the original queue to the target audio and video code stream queue, and the data shaper performs gated selection for transmission; the audio and video code stream converter needs to continue to compare the data entry rate and the data exit rate of the target audio and video code stream queue; if the data entry rate of the audio and video data is continuously higher than the data exit rate within a certain time period, the code rate reduction operation in step 5 is performed. The specific process is as follows: (1) If, within the time period t, the audio and video code stream converter identifies that the credit value of a target audio and video code stream queue is greater than the credit limit of the queue × 0.2, the code stream converter will perform a bit rate recovery operation on the audio and video decoded data of the channel; (2) The audio and video code stream converter copies the audio and video data frames in the original queue to the target audio and video code stream queue, and replaces the data in the target audio and video code stream queue; (3) The audio and video code stream converter detects that the incoming data rate is close to the outgoing data rate in the next t period; if it is insufficient, step 6 is repeated until the incoming data rate and the outgoing data rate of the target audio and video data queue are in a reasonable range. If the credit value of the converted target audio and video code stream FIFO queue is less than the credit limit of the queue × 0.75, if it is not satisfied, the operations (1)-(3) of step 5 are performed.

[0061] It should be noted that if the in-queue data rate and the out-queue data rate are substantially equal, the in-queue data and the out-queue data of the network node are substantially equal.

[0062] Example 2 (Main and Auxiliary Stream Selection) This embodiment can be applied to a time-sensitive network that transmits time-sensitive data, audio and video data, and best-effort data, wherein the video data of the audio and video data source has a multi-stream characteristic.

[0063] The embodiment of the present application provides a method for scheduling time-sensitive network audio and video data with variable bit rate, which is based on credit shaping scheduling and priority gate and data shaping. Figure 1 As shown, the audio and video stream converter is as follows Figure 4 As shown, the method includes the following steps: Step 1: Configure time-sensitive networking. Configure data filters, time-sensitive data queues, original audio and video data queues, best-effort data queues, and data shaping mechanisms (gated output) through the bridge controller. The specific process is as follows: (1) Through tolerance analysis, it is determined that the sum of all traffic rates at the inlet of each time-sensitive bridge controller cannot exceed the rate at the outlet of the time-sensitive bridge.

[0064] (2) By predicting the data type, data transmission cycle, data volume, and time-sensitive delay of each network port of the bridge controller, the time-sensitive data queue, the original audio and video data queue, the best-effort data queue depth, and the data gating output logic configuration are realized.

[0065] Queue depth = safety factor × [maximum acceptable delay of data on this path × (maximum input rate - minimum output rate)], where the safety factor is a value greater than 0 and less than 1.

[0066] Step 2: Based on the IEEE 802.1 QAV queuing and forwarding specifications, IdleSlope is used to represent the traffic credit growth rate (which also represents the logical bandwidth reserved for this type of traffic), and SendSlope is used to represent the traffic credit reduction rate. For the transmission of multiple audio and video streams and best-effort data streams, a scheduling method based on credit shaping (CBS) is used. This method uses bandwidth equivalent to IdleSlope while distributing packets as much as possible, ensuring that even low-priority queues have a certain amount of transmission opportunities. The transmission order is controlled by calculating the credit amount of multiple transmission queues based on credit shaping. The specific process is as follows: The audio and video stream converter determines the main and auxiliary stream data characteristics of each audio source. Since the audio and video source in Example 1 is a single stream, the audio and video stream converter decodes and stores key frame data in the memory unit to obtain information such as the audio and video stream frame rate, resolution, and encoding method. The audio and video stream converter copies the data from each original audio and video stream queue to the target audio and video stream queue. The transmission order is controlled by calculating the credit of multiple transmission queues based on credit shaping timing. The specific credit calculation formula is as follows: SendSlope=IdleSlope- C Credit AV1 =n1× AV1 … Credit AVx =nx×AVx … Credit be =n BE × (BE) (n1≧n2≧…≧n BE ) Among them, C is the theoretical bandwidth of the physical link, Credit AVx is the target audio and video bit rate of group x, AV x The sending rate of the xth group of principle audio and video streams, Creditbe is the best-effort data flow data rate; BE is the best-effort data flow sending rate.

[0067] Step 3: The bridge controller manages the transmission of each original audio and video stream queue and the target audio and video stream queue, calculates the data flow of each original audio and video source, decodes and caches the audio and video data in the original audio and video data queue, and determines the stream conversion operation based on the credit value of each target audio and video stream queue. The specific process is as follows: (1) Initially, the audio and video stream converter copies the data of each original audio and video stream queue to the target audio and video stream queue; (2) The audio and video stream converter decodes the encoded audio and video data of the multiple original audio data queues and caches the decoded audio and video data frames; (3) The audio and video code stream converter calculates the original data volume based on the original audio and video decoding buffer data rate of each channel, and calculates the maximum credit volume of the audio and video code stream based on the target audio and video code stream queue capacity and audio and video data rate; AV credit max = FIFO capacity / audio and video data rate (4) The stream converter reads the credit value of each target audio and video stream queue.

[0068] Step 4: The data shaper of the bridge controller adopts the absolute priority and credit-based scheduling method to select the time-sensitive data queue, audio and video data queue and best-effort data for transmission. Since time-sensitive data is often the most critical and real-time data in the system, time-sensitive network priority gating and data shaping must give priority to ensuring the transmission of time-sensitive data. Figure 1 and Figure 2 The specific process is as follows: (1) Only when the time-sensitive data queue is empty can the corresponding queue data be shaped and forwarded according to the audio and video data queue and the best-effort queue data credit; (2) If the time-sensitive data queue is not empty and the data shaper is transmitting time-sensitive data, the data shaper transmits the time-sensitive data in the first-in-first-out order of the queue; (3) If the time-sensitive data queue is not empty and the data shaper is transmitting non-time-sensitive data, the data shaper starts the time-sensitive data shaping transmission after completing the data transmission of the current queue; (4) When the audio and video code stream data frame arrives at the queue, if there is a low-priority data frame being transmitted, the audio and video code stream data frame will be queued and wait, and the credit value of the audio and video code stream queue will increase at the rate of IdleSlope. After the low-priority data frame is transmitted, the audio and video queue data with the largest credit value will start to be transmitted, and the credit value will decrease at the rate of SendSlope. (5) After the audio and video stream queue data frame transmission is completed, if the credit value is less than 0, it will increase to 0 at the rate of IdleSlope; if the credit value is greater than 0, it will be set to 0; (6) Data frames in the queue can only be transmitted when the credit value of the audio and video stream queue is greater than or equal to 0; when the credit value of the audio and video stream queue is less than 0, data frame transmission cannot be started, but data frames that started to be transmitted before the credit value is reduced to 0 can continue to be transmitted.

[0069] Step 5: The audio and video code stream converter detects whether the data entry data rate of each target audio and video code stream queue matches the data exit data rate. If the audio and video data entry is continuously greater than the audio and video data exit within a certain period of time, it will cause a queue overflow and result in loss of audio and video data packets. The audio and video code stream converter performs code stream reduction processing on the audio and video data frames in the original queue, and places the auxiliary code stream audio and video data frames into the target audio and video code stream queue, which is gated and selected for transmission by the data shaper. The audio and video code stream converter continues to compare the data entry data rate and the data exit data rate of the target audio and video code stream queue, and it must meet the requirement that the data entry data rate is less than the exit data rate within a period of time; if it is insufficient, it is necessary to continue to reduce the code until the data entry data rate of the target audio and video code stream queue is less than the exit data rate. The specific process is as follows: (1) If, in time period t, the audio and video code stream converter identifies that the credit value of a target audio and video code stream queue is greater than the credit limit of the queue × 0.75, the audio and video code stream converter performs bit rate reduction processing on the multi-stream audio and video decoded data of the channel.

[0070] Bit rate reduction factor = 1-(collection credit value of the queue / maximum credit value of the queue) (2) The code stream converter copies the mainstream data of the audio and video data frame to the target audio and video code stream queue, and replaces the data in the target audio and video code stream FIFO queue.

[0071] (3) The code stream converter detects whether the credit value of the converted target audio and video code stream queue is less than the credit limit of the queue × 0.75 within the next t period. If not, perform steps (1)-(2) of step 5 according to Example 2.

[0072] Step 6: The audio and video stream converter detects whether the data entry rate of each target audio and video stream queue matches the data exit rate. If the data entry rate of a target audio and video stream queue is much lower than the data exit rate within a certain time period, network bandwidth resources may be wasted due to low bit rate reduction. The audio and video stream converter copies the mainstream data frames of the multi-stream audio and video source to the target audio and video stream queue, which is gated and selected for transmission by the data shaper. The audio and video stream converter needs to continue to compare the data entry rate and the data exit rate of the target audio and video stream queue. If the data entry rate of the audio and video data is continuously higher than the data exit rate within a certain time period, the bit rate reduction operation in step 5 is performed. The specific process is as follows: (1) If, within the time period t, the code stream converter identifies that the credit value of a target audio and video code stream FIFO queue is greater than the credit limit of the queue × 0.2, the code stream converter copies the main code stream data of the audio and video source to the target audio and video code stream queue, and replaces the data in the target audio and video code stream queue; (2) The audio and video code stream converter detects whether the credit value of the converted target audio and video code stream FIFO queue is less than the queue credit limit × 0.75 within the next t period. If not, the operation (1)-(3) of step 5 is performed.

[0073] The above-mentioned method for shaping and scheduling data in a time-sensitive network based on variable audio and video code streams monitors in real time the changes between the amount of data cached in the target audio and video code stream queue and the network transmission data bandwidth, and controls the amount of audio and video data transmitted by optimizing the audio and video source code stream selection, audio and video code stream conversion, or multi-code stream switching, thereby improving the quality of time-sensitive network data services.

[0074] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method and will not be repeated here.

[0075] Based on the methods in the above embodiments, embodiments of the present application provide an electronic device that may include: a processor, a communications interface, a memory, and a communication bus, wherein the processor, the communications interface, and the memory communicate with each other via the communication bus. The processor may invoke logic instructions in the memory to execute the methods in the above embodiments.

[0076] In addition, the logical instructions in the above-mentioned memory can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application.

[0077] Based on the method in the above embodiment, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program runs on a processor, the processor executes the method in the above embodiment.

[0078] Based on the method in the above embodiment, an embodiment of the present application provides a computer program product. When the computer program product runs on a processor, the processor executes the method in the above embodiment.

[0079] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA), other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0080] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be an integral part of the processor. The processor and storage medium can be located in an ASIC.

[0081] The above embodiments can be implemented in whole or in part using software, hardware, firmware, or any combination thereof. When implemented using software, they can be implemented in whole or in part in the form of a computer program product. The computer program product comprises one or more computer instructions. When loaded and executed on a computer, the computer program instructions fully or partially produce the processes or functions described in the embodiments of this application. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible by a computer or a data storage device such as a server or data center that integrates one or more available media. The available medium can be magnetic media (e.g., floppy disk, hard disk, tape), optical media (e.g., DVD), or semiconductor media (e.g., solid-state drive (SSD)).

[0082] It will be understood that the various numerical numbers involved in the embodiments of the present application are merely distinctions for the convenience of description and are not intended to limit the scope of the embodiments of the present application.

[0083] It is easy for those skilled in the art to understand that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A method for scheduling time-sensitive network audio and video data with variable bitrate, characterized in that: The method includes: Receiving time-sensitive network data; Classify the received time-sensitive network data and transmit them to the time-sensitive data queue, the original audio and video data queue and the best-effort data queue respectively; Copy the data in each original audio and video code stream data queue to the target audio and video code stream data queue; Monitor each target audio and video stream queue in real time, determine whether the stream conversion operation is to down-stream or up-stream based on the credit value of each target audio and video stream queue, and place the down-stream / up-stream audio and video data frames into the target audio and video stream queue; Based on the credit shaping scheduling method, the credit amount of each target audio and video data queue and the best-effort data queue is calculated separately; Gated selection transmission is performed through absolute priority-based time-sensitive data scheduling and credit-shaped audio and video and best-effort data scheduling mechanism. The absolute priority-based time-sensitive data scheduling adopts time-sensitive priority gating data shaping to prioritize time-sensitive data shaping transmission. For time-sensitive data queues, the frame preemption principle is followed and transmission is performed according to the time-sensitive data queue timing; the credit-shaped audio and video and best-effort data scheduling mechanism controls the transmission order of each target audio and video data queue and best-effort data queue based on the calculated credit amount.

2. The audio and video data scheduling method according to claim 1, wherein: According to the credit value of each target audio and video stream queue, the stream conversion operation is determined to be down-stream processing, and the down-stream audio and video data frames are placed into the target audio and video stream queue, specifically: (1) If, within a certain time period, it is identified that the credit value of a target audio and video stream queue exceeds a first preset ratio of the queue credit limit, the decoded audio and video data of the multi-stream is subjected to bit rate reduction processing; (2) Copy the mainstream data of the audio and video data frame to the target audio and video code stream queue, and replace the data in the target audio and video code stream FIFO queue; (3) Detect whether the credit value of the transformed target audio and video stream queue does not exceed the first preset ratio of the queue credit limit in the next time period. If not, repeat steps (1)-(2).

3. The audio and video data scheduling method according to claim 2, wherein: The bit rate reduction coefficient and the ratio of the collection credit value of the queue on this path to the maximum credit value of the queue are complementary to each other.

4. The audio and video data scheduling method according to claim 3, wherein: The maximum credit value AV creditmax is determined by: 。 5. The audio and video data scheduling method according to claim 2, wherein: When processing the bitstream, it also includes first performing frame rate reduction encoding and / or resolution reduction encoding on the audio and video data, wherein the frame rate reduction encoding is to perform frame rate reduction encoding according to the frame rate after bitrate reduction, and replace the data in the target audio and video bitstream FIFO queue; the resolution reduction encoding is to perform frame rate reduction encoding according to the resolution after bitrate reduction, and replace the data in the target audio and video bitstream queue.

6. The audio and video data scheduling method according to claim 1, wherein: According to the credit value of each target audio and video stream queue, the stream conversion operation is determined to be up-stream processing, and the up-stream audio and video data frames are placed into the target audio and video stream queue, specifically: (1) If, within a certain time period, it is identified that the credit value of a certain target audio and video stream FIFO queue exceeds the credit limit of the queue of the second preset ratio, the main stream data of the audio and video source of the channel is copied to the target audio and video stream queue to replace the data in the target audio and video stream queue; (2) Detect whether the credit value of the transformed target audio and video code stream FIFO queue does not exceed the first preset ratio of the queue credit limit in the next time period. If not, perform code stream reduction processing.

7. The audio and video data scheduling method according to claim 1, wherein: The queue depths of the time-sensitive data queue, the original audio and video data queue, and the best-effort data queue are determined as follows: The safety factor is a tolerance value set for the time-sensitive network, which is greater than 0 and less than 1.

8. The audio and video data scheduling method according to claim 1, wherein: Gated selective transmission is performed through absolute priority-based time-sensitive data scheduling and credit-shaped audio and video and best-effort data scheduling mechanisms. Specifically: (1) When the time-sensitive data queue is empty, the corresponding queue data can be shaped and forwarded according to the credit of the audio and video data queue and the best-effort queue data; (2) If the time-sensitive data queue is not empty and the data shaper is transmitting time-sensitive data, the data shaper transmits the time-sensitive data in the first-in-first-out order of the queue; (3) If the time-sensitive data queue is not empty and the data shaper is transmitting non-time-sensitive data, the data shaper starts the time-sensitive data shaping transmission after completing the data transmission of the current queue; (4) When the audio and video code stream data frame arrives at the queue, if there is a low-priority data frame being transmitted, the audio and video code stream data frame will be queued and wait, and the credit value of the audio and video code stream queue will increase at the rate of IdleSlope. After the low-priority data frame is transmitted, the audio and video queue data with the largest credit value will start to be transmitted, and the credit value will decrease at the rate of SendSlope. (5) After the audio and video stream queue data frame transmission is completed, if the credit value is less than 0, it will increase to 0 at the rate of IdleSlope; if the credit value is greater than 0, it will be set to 0; (6) Data frames in the queue can be transmitted only when the credit value of the audio and video stream queue is greater than or equal to 0; when the credit value of the audio and video stream queue is less than 0, data frame transmission cannot be started, but data frames that were started before the credit value is reduced to 0 can continue to be transmitted.

9. The audio and video data scheduling method according to any one of claims 1 to 8, wherein: The video data of the audio and video data source has a single stream or multiple stream characteristics.

10. A time-sensitive network audio and video data scheduling device with variable bit rate, characterized in that: The device includes: at least one memory for storing a program; At least one processor is used to enter the program stored in the memory, and when the program stored in the memory is entered, the processor is used to enter the audio and video data scheduling method as described in any one of claims 1 to 9.

Images