Delay-guaranteed adaptive bit rate transmission method and system for real-time video stream

Abstract

The invention relates to a delay guarantee adaptive bit rate transmission method and system for a real-time video stream, and the method comprises the steps: carrying out the coding of a video stream through a sender device according to a current target bit rate, so as to generate video data packages, transmitting the video data packages to a receiver device, and transmitting N video data packages in each control period; the receiver equipment stores the video data packets in a buffer area, and calculates the average end-to-end delay of the current control period when receiving N video data packets; estimating a relationship between the average end-to-end delay and the optimal target bit rate based on a time-varying pseudo partial derivative parameter; and the sender equipment receives the optimal target bit rate from the receiver equipment, takes the optimal target bit rate as the current target bit rate of the next control period and waits for next update. According to the invention, the throughput can be maximized while the delay is maintained under a specific delay demand.

Description

Technical Field

[0001] The present invention relates to the field of intelligent manufacturing, and more particularly, to a method and system for adaptively transmitting bit rate with guaranteed delay for real-time video streams. Background Art

[0002] With the development of intelligent manufacturing technology, visual sensing devices have been widely deployed in the industrial production field, giving rise to a large number of machine-centered video analysis applications. These applications not only require high throughput to improve video quality, but also require end-to-end delay guarantee for video data packets. To achieve this goal, in recent years, Adaptive Bit Rate (ABR) technology has received extensive attention, which can adjust the throughput of real-time video streams and affect the end-to-end delay of video data packets. Existing research on ABR technology mainly focuses on two ideas to balance the relationship between throughput and delay: The first type of technology dynamically adjusts the video bit rate to match the estimated link capacity. This type of technology can maximize throughput while avoiding network congestion, keeping the delay at a relatively low level. The second type of technology takes optimizing the Quality of Experience (QoE) of video streams as the core design goal. QoE usually involves throughput and delay, and this type of technology can find a balance point between the two on the premise of maximizing QoE. However, the above research is mainly applicable to human-centered video applications, so when applied to machine-centered video analysis applications, it usually has the following disadvantages.

[0003] Disadvantage 1: The existing technology cannot guarantee end-to-end delay. The existing technology often balances the internal trade-off between throughput and delay by avoiding congestion or optimizing QoE, and cannot guarantee the end-to-end delay requirements of machine-centered video analysis applications. Specifically, the ABR technology aiming at avoiding congestion aims to adjust the bit rate of the video stream to match the estimated link capacity. However, when the network environment changes dynamically, this type of technology is likely to misestimate the available link capacity, resulting in a sharp increase in delay. On the other hand, the ABR technology aiming at optimizing QoE finds a balance point between delay and throughput on the premise of maximizing QoE. However, this type of method generally sacrifices a certain degree of delay in exchange for higher video quality or less bit rate fluctuation, so it is difficult to guarantee the strict requirements of the application for end-to-end delay. All in all, these two types of technology cannot guarantee the end-to-end delay of machine-centered video analysis applications.

[0004] Disadvantage 2: The prior art fails to establish the relationship between bitrate and latency. To ensure end-to-end latency, it is crucial to establish the relationship between the video stream bitrate and latency. However, most of the prior art focuses on estimating the link capacity or designing and optimizing QoE metrics, without considering the relationship between bitrate and latency. Therefore, in an industrial network environment where the transmission delay and packet loss rate are unknown and non-stationary, it is a major challenge to characterize the relationship between bitrate and latency without relying on any prior knowledge or assumptions about the transmission delay and packet loss rate to meet the end-to-end latency requirements of applications. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a Latency-Guaranteed Adaptive Bitrate (LG-ABR) transmission method and apparatus for real-time video streams, which can maximize the throughput while maintaining the latency below a specific latency requirement in a highly dynamic industrial network.

[0006] The technical solution adopted by the present invention to solve its technical problems is to construct a latency-guaranteed adaptive bitrate transmission method for real-time video streams, including:

[0007] S1. The sender device encodes the video stream according to the current target bitrate to generate video data packets and sends the video data packets to the receiver device, where N video data packets are sent in each control period, and N is a positive integer;

[0008] S2. The receiver device stores the video data packets in a buffer and calculates the average end-to-end latency of the current control period when receiving N video data packets;

[0009] S3. Determine the optimal target bitrate for the next control period based on the average end-to-end latency, where the relationship between the average end-to-end latency and the optimal target bitrate is estimated based on time-varying pseudo partial derivative parameters;

[0010] S4. The sender device receives the optimal target bitrate from the receiver device and uses the optimal target bitrate as the current target bitrate for the next control period and returns to execute step S1.

[0011] In the latency-guaranteed adaptive bitrate transmission method for real-time video streams of the present invention, the step S1 further includes:

[0012] S11. The sender device encodes the video stream according to the current target bitrate to generate the video data packets;

[0013] S12. Push the video data packet into the FIFO queue and send it to the receiving device through the transmission network.

[0014] In the method for delay-guaranteed adaptive bitrate transmission of real-time video streams according to the present invention, in step S2, calculate the average end-to-end delay based on the following formula:

[0015]

[0016] where D k represents the average end-to-end delay of the k-th control period, T k,j represents the reception time of the j-th successfully received video data packet within the k-th control period, and t k,j represents the generation time of the j-th successfully received video data packet within the k-th control period; where k and j are positive integers, and j is less than or equal to N.

[0017] In the method for delay-guaranteed adaptive bitrate transmission of real-time video streams according to the present invention, step S3 further includes:

[0018] S31. Obtain the following expression based on simultaneously satisfying the end-to-end delay and maximizing the throughput:

[0019]

[0020] s.t. D k ≤δ; (2)

[0021]

[0022] where D k represents the average end-to-end delay of the k-th control period, T k,j represents the reception time of the j-th successfully received video data packet within the k-th control period, and t k,j represents the generation time of the j-th successfully received video data packet within the k-th control period; a k represents the optimal target bitrate of the k-th control period; K represents the total number of control periods, and δ represents the delay requirement; where K, k, and j are all positive integers, and j is less than or equal to N; expression (1) represents maximizing the optimal target bitrate of each control period; expression (2) represents that the average end-to-end delay D k is less than the delay requirement δ;

[0023] S32. Estimate the relationship between the average end-to-end delay D k and the optimal target bitrate a k based on the time-varying pseudo partial derivative parameter to obtain the calculation formula of the optimal target bitrate a k :

[0024]

[0025] Among them, v k represents the time-varying pseudo partial derivative parameter in the k-th control period; represents the estimated value of v k , D k-1 represents the average end-to-end delay in the (k - 1)-th control period; a k-1 represents the optimal target bit rate in the (k - 1)-th control period, and λ represents the weight parameter.

[0026] In the method for delay-guaranteed adaptive bitrate transmission of real-time video streams according to the present invention, the step S32 further includes:

[0027] S321. Represent the relationship between the average end-to-end delay D k and the optimal target bit rate a k in the k-th control period as the following dynamic linear data model:

[0028] ΔD k = v k Δa k , (5)

[0029] where, ΔD k = D k - D k-1 , Δa k = a k - a k-1 ;

[0030] S322. Estimate the time-varying pseudo partial derivative parameter v k in the k-th control period to estimate the estimation formula between the average end-to-end delay D k and the optimal target bit rate a k

[0031]

[0032] where μ represents the weight parameter, represents the time-varying pseudo partial derivative parameter in the (k - 1)-th control period;

[0033] S323. Obtain the calculation formula (4) based on the estimation formula (6).

[0034] In the method for delay-guaranteed adaptive bitrate transmission of real-time video streams according to the present invention, in the step S322,

[0035]

[0036] where, v0 represents the initial value of the time-varying pseudo partial derivative parameter, θ is a positive number and ensures​

[0037] In the method for delay-guaranteed adaptive bitrate transmission of real-time video streams according to the present invention, step S3 further includes:

[0038] S33. Determine the optimal target bitrate based on the optimal target bitrate, bitrate upper bound, and bitrate lower bound calculated in step S32.

[0039] Another technical solution adopted by the present invention to solve its technical problems is to construct a delay-guaranteed adaptive bitrate transmission system for real-time video streams, including a sender device, a transmission network, and a receiver device;

[0040] The sender device is used to encode the video stream according to the current target bitrate to generate video data packets and send the video data packets to the receiver device, where N video data packets are sent in each control period, and N is a positive integer;

[0041] The receiver device includes a buffer and a delay-guaranteed adaptive bitrate calculator;

[0042] The buffer is used to store the video data packets;

[0043] The delay-guaranteed adaptive bitrate calculator is used to calculate the average end-to-end delay of the current control period when receiving N video data packets, and determine the optimal target bitrate for the next control period based on the average end-to-end delay, where the relationship between the average end-to-end delay and the optimal target bitrate is estimated based on time-varying pseudo partial derivative parameters;

[0044] The sender device is further used to receive the optimal target bitrate from the receiver device and use the optimal target bitrate as the current target bitrate for the next control period and wait for the next update.

[0045] In the delay-guaranteed adaptive bitrate transmission system for real-time video streams according to the present invention, the delay-guaranteed adaptive bitrate calculator is further used for:

[0046] Calculate the average end-to-end delay based on the following formula:

[0047]

[0048] where D k represents the average end-to-end delay of the kth control period, T k,j represents the reception time of the jth successfully received video data packet in the kth control period, and t k,jdenote the generation time of the j-th successfully received video packet in the k-th control period; where k and j are positive integers, and j is less than or equal to N.

[0049] In the delay-guaranteed adaptive bitrate transmission system of the present invention, the delay-guaranteed adaptive bitrate calculator is further configured to:

[0050] Based on simultaneously satisfying the end-to-end delay and maximizing the throughput, the following expression is obtained:

[0051]

[0052] s.t.D k ≤δ; (2)

[0053]

[0054] where D k denotes the average end-to-end delay of the k-th control period, T k,j denotes the reception time of the j-th successfully received video packet in the k-th control period, t k,j denotes the generation time of the j-th successfully received video packet in the k-th control period; a k denotes the optimal target bitrate of the k-th control period; K denotes the total number of control periods, δ denotes the delay requirement; where K, k, and j are all positive integers, and j is less than or equal to N; Expression (1) represents maximizing the optimal target bitrate of each control period; Expression (2) represents that the average end-to-end delay D k is less than the delay requirement δ;

[0055] Estimate the average end-to-end delay D k and the relationship between the optimal target bitrate a k to obtain the calculation formula of the optimal target bitrate a k :

[0056]

[0057] where v k denotes the time-varying pseudo partial derivative parameter of the k-th control period; denotes the estimated value of v k , D k-1 denotes the average end-to-end delay of the (k - 1)-th control period; a k-1 denotes the optimal target bitrate of the (k - 1)-th control period, and λ denotes the weight parameter.

[0058] Implementing the method and system for delay-guaranteed adaptive bitrate transmission of real-time video streams according to the present invention can, in a highly dynamic industrial network, maintain the delay below a specific delay requirement while maximizing the throughput. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The present invention will be further described below in conjunction with the drawings and embodiments. In the drawings:

[0060] Figure 1 is a flowchart of the method for delay-guaranteed adaptive bitrate transmission of real-time video streams according to the present invention;

[0061] Figure 2 is a schematic block diagram of the system for delay-guaranteed adaptive bitrate transmission of real-time video streams according to the present invention;

[0062] Figure 3 shows the software algorithm of the method for delay-guaranteed adaptive bitrate transmission of real-time video streams according to the present invention;

[0063] Figures 4A - 4B respectively show the average throughput and delay of different technologies in each control cycle in highly dynamic and high packet loss rate scenarios. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0064] In order to make the objectives, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0065] Figure 1 is a flowchart of the method for delay-guaranteed adaptive bitrate transmission of real-time video streams according to the present invention. As Figure 1 shown, in step S1, the sender device encodes the video stream according to the current target bitrate to generate video data packets and sends the video data packets to the receiver device, where N video data packets are sent in each control cycle, and N is a positive integer.

[0066] In a preferred embodiment of the present invention, the sender device encodes the video stream according to the current target bitrate to generate the video data packets; pushes the video data packets into a FIFO queue and sends them to the receiver device through a transmission network.

[0067] In a preferred embodiment of the present invention, the sender device and the receiver device can be constructed using any known signal sending device and signal receiving device, such as a micro host, a microcontroller, etc. The transmission network can use any known wired or wireless network, such as the industrial Internet of Things.

[0068] These all fall within the protection scope of the present invention

[0069] In a preferred embodiment of the present invention, to cope with the dynamic changes of the transmission network, the video stream bit rate is adjusted in units of control cycles. Each control cycle contains N video data packets, and k is used to represent different control cycles. In a preferred embodiment of the present invention, the encoder of the sender device encodes the video stream according to the current target bit rate. Here, this encoding can be implemented by any encoding method known in the art. For example, this can be achieved by adjusting the encoding mode and quantization parameters. Subsequently, the video data packets are pushed into the FIFO queue and sent to the receiver device through the transmission network

[0070] In step S2, the receiver device stores the video data packets in the buffer, and when N video data packets are received, calculates the average end-to-end delay of the current control cycle. In a preferred embodiment of the present invention, when N data packets are received, the average end-to-end delay D of this control cycle is calculated k 。

[0071] In a preferred embodiment of the present invention, the average end-to-end delay is calculated based on the following formula

[0072]

[0073] where D k represents the average end-to-end delay of the k-th control cycle, T k,j represents the reception time of the j-th successfully received video data packet in the k-th control cycle, and t k,j represents the generation time of the j-th successfully received video data packet in the k-th control cycle; where k and j are positive integers, and j is less than or equal to N. It should be noted that the average end-to-end delay D k can be directly measured at the receiver device because the reception time T k,j is obtainable, and the generation time t k,j can be provided by the packet header of the video data packet

[0074] In this application, for the considered point-to-point link, the present invention defines the delay as the end-to-end delay experienced by the video data packet from generation to successful reception by the receiver device, including the queuing delay in the sending queue and the transmission delay in the communication network; in addition, the present invention defines the transmission delay as the delay experienced by the video data packet from leaving the sending queue to arriving at the receiving buffer, which is determined by the quality of service of the communication network; the packet loss rate represents the probability that a single data packet is lost during transmission

[0075] In step S3, the optimal target bitrate for the next control period is determined based on the average end-to-end delay, wherein the relationship between the average end-to-end delay and the optimal target bitrate is estimated based on time-varying pseudo partial derivative parameters.

[0076] In a preferred embodiment of the present invention, in order to simultaneously satisfy end-to-end delay and maximize throughput, the following expression is obtained:

[0077]

[0078] s.t.D k ≤δ; (2)

[0079]

[0080] wherein, D k represents the average end-to-end delay of the k-th control period, T k,j represents the reception time of the j-th successfully received video packet within the k-th control period, t k,j represents the generation time of the j-th successfully received video packet within the k-th control period; a k represents the optimal target bitrate of the k-th control period; K represents the total number of control periods, δ represents the delay requirement; where K, k, and j are all positive integers, and j is less than or equal to N; Expression (1) represents maximizing the optimal target bitrate of each control period; Expression (2) represents that the average end-to-end delay D k of each control period is less than the delay requirement δ; Expression (3) is the calculation formula for the average end-to-end delay D k of each control period.

[0081] Since in non-stationary transmission delay and packet loss rate, the prior art cannot establish the relationship between the optimal target bitrate a k and the average end-to-end delay D k Therefore, the present invention proposes a delay-guaranteed adaptive bitrate transmission technology for guaranteeing the end-to-end delay requirement of machine-centric video analysis applications in an unknown and non-stationary network environment. The core idea of this technology is to dynamically estimate the optimal target bitrate a k of the bitrate and the average end-to-end delay D kThe mathematical relationship between them. Based on the established model relationship, the bitrate can be adjusted in an optimal manner. To achieve this goal, the adaptive bitrate transmission method with latency guarantee for real-time video streams of the present invention (also known as the LG-ABR technology) first introduces a dynamic linearized data model with a time-varying pseudo-partial derivative parameter (PPD) to describe the mathematical relationship between the bitrate and the latency, then determines the data model by real-time estimating the PPD parameter, and finally decides the final bitrate based on the determined data model. That is, estimating the average end-to-end latency D based on the time-varying pseudo-partial derivative parameter k and the optimal target bitrate a k to obtain the optimal target bitrate a k The calculation formula of:

[0082]

[0083] where, v k represents the time-varying pseudo-partial derivative parameter of the k-th control period; represents the estimated value of v k , D k-1 represents the average end-to-end latency of the (k - 1)-th control period; a k-1 represents the optimal target bitrate of the (k - 1)-th control period, and λ represents the weight parameter.

[0084] The derivation process of this calculation formula is as follows.

[0085] Express the relationship between the average end-to-end latency D k and the optimal target bitrate a k in the k-th control period as the following dynamic linear data model:

[0086] ΔD k = v k Δa k , (5)

[0087] where, ΔD k = D k - D k-1 , Δa k = a k - a k-1 , v k represents the time-varying pseudo-partial derivative parameter of the k-th control period. Based on equation (5), the complex relationship implicit between the average end-to-end latency D k and the optimal target bitrate a k can be represented by ΔD k and Δa k through the time-varying pseudo-partial derivative parameter v kRevelation. Considering the non-stationary network characteristics in the video analysis system can be reflected as the dynamic change of the time-varying pseudo partial derivative parameter v k Estimate the time-varying pseudo partial derivative parameter v within the kth control period k To estimate the average end-to-end delay D k And the optimal target bit rate a k The estimation formula between them

[0088]

[0089] Where μ represents the weight parameter, Represents the time-varying pseudo partial derivative parameter in the (k - 1)th control period

[0090] Based on the estimation formula (6), the average end-to-end delay D in equation (5) can be established k And the optimal target bit rate a k The relationship between them; Next, based on this relationship, the control problem of the target bit rate in equations (1) to (3) will be further solved. Estimate the average end-to-end delay D based on the time-varying pseudo partial derivative parameter k And the optimal target bit rate a k The relationship between them to obtain the calculation formula of the optimal target bit rate a k :

[0091]

[0092] In a further preferred embodiment of the present invention, in order to further improve the estimation accuracy and ensure stability, the delay guarantee adaptive bit rate transmission method for real-time video streams of the present invention follows the following reset scheme to cope with the error estimation when the transmission delay suddenly changes:

[0093]

[0094] Where v0 represents the initial value of the time-varying pseudo partial derivative parameter, and θ is a very small positive number to ensure

[0095] In a further preferred embodiment of the present invention, the optimal target bit rate is determined based on the calculated optimal target bit rate, the bit rate upper bound, and the bit rate lower bound

[0096] When using the calculation formula (4) of the aforementioned optimal target bit rate a k In a further preferred embodiment of the present invention, the bit rate upper and lower bounds are adopted. Among them, the bit rate upper bound a ub,k Is to prevent queue overflow and infinite growth of the end-to-end delay, represents the link capacity, and can be estimated according to the prior art; the bit rate lower bound a lb,kThis is to avoid selecting an overly low bitrate. This is because when the bitrate is much lower than the link capacity (i.e., the queue is always empty), the end-to-end delay will approach the transmission delay. Therefore, further reducing the bitrate will not reduce the delay any further, but will only lead to a decrease in throughput, which can be calculated by the estimation method in Equation [6]. Finally, to ensure that a positive bitrate is always selected, the exponential function is used to map the updated value of the optimal target bitrate a k provided by the calculation formula (Equation (4)) of the optimal target bitrate a k to the actual bitrate. It should be noted that since the present invention makes no assumptions about time-varying and non-linear systems, other positive functions are also applicable to this mapping. Based on the above discussion, the actual bitrate used within each control period is given by the following formula:

[0097]

[0098] where a k ′ represents the actual bitrate, a k represents the optimal target bitrate a k provided by the calculation formula (Equation (4)) of the optimal target bitrate, a lb,k represents the lower bound of the bitrate, a ub,k represents the upper bound of the bitrate, exp represents the exponential function, and of course any other suitable positive function can be used to replace the exponential function.

[0099] Figure 3 shows the software algorithm of the delay-guaranteed adaptive bitrate transmission method for real-time video streams of the present invention. To deploy the aforementioned delay-guaranteed adaptive bitrate transmission method for real-time video streams in an actual system, the receiving device first updates its PPD estimate using the estimation formula (Equation (6)) at the beginning of each control period Next, the receiving device calculates the actual bitrate a′ k according to the calculation formulas (Equation (4)) and (8) of the optimal target bitrate a k , and transmits the updated bitrate to the sending device. Subsequently, since the receiving device used the exponential mapping when determining the actual bitrate previously, it needs to perform the inverse transformation a k ←log(a k ). After receiving N video packets, the receiving device calculates the average end-to-end delay using the following formula:

[0100]

[0101] Implementing the delay-guaranteed adaptive bitrate transmission method for real-time video streams of the present invention can maximize the throughput while maintaining the delay below a specific delay requirement in a highly dynamic industrial network.

[0102] Figure 2 is the principle block diagram of the delay-guaranteed adaptive bitrate transmission system for the real-time video stream of the present invention. As Figure 2 shown, the delay-guaranteed adaptive bitrate transmission system for the real-time video stream of the present invention includes a sender device, a transmission network, and a receiver device. The sender device is configured to encode a video stream according to a current target bitrate to generate video data packets and send the video data packets to the receiver device, where N video data packets are sent in each control period, and N is a positive integer. The receiver device includes a buffer and a delay-guaranteed adaptive bitrate calculator. The buffer is used to store the video data packets; the delay-guaranteed adaptive bitrate calculator is configured to calculate an average end-to-end delay of the current control period when receiving N video data packets, and determine an optimal target bitrate for the next control period based on the average end-to-end delay; the sender device is further configured to receive the optimal target bitrate from the receiver device and use the optimal target bitrate as the current target bitrate for the next control period and wait for the next update.

[0103] The delay-guaranteed adaptive bitrate transmission system for the real-time video stream of the present invention adjusts the video stream bitrate in units of control periods to cope with the dynamic changes of the transmission network, and each control period includes N video data packets. Use k to represent different control periods. In this system, the encoder of the sender device encodes the video stream according to the target bitrate, which can be achieved by adjusting the encoding mode and quantization parameters. Subsequently, the generated video data packets are pushed into the FIFO queue and sent to the receiver device through the transmission network. The buffer of the receiver device stores the arriving data packets, and when N data packets are received, the average end-to-end delay D of this control period is calculated k . Based on the measured D k , the LG-ABR controller determines the optimal target bitrate a for the next period k+1 , and feeds it back to the sender device through the transmission network. After receiving the feedback, the sender device adopts and waits for the next update using k to represent different control periods. The present invention focuses on the design of the LG-ABR controller to implement the ABR technology with guaranteed end-to-end delay.

[0104] The delay-guaranteed adaptive bitrate calculator is further configured to calculate the average end-to-end delay based on the following formula:

[0105]

[0106] where D k represents the average end-to-end delay of the k-th control period, T k,j represents the reception time of the j-th successfully received video data packet in the k-th control period, and t k,jdenotes the generation time of the j-th successfully received video data packet in the k-th control period; where k and j are positive integers, and j is less than or equal to N. It should be noted that the average end-to-end delay D k can be directly measured at the receiving device because the reception time T k,j is available, while the generation time t k,j can be provided by the packet header of the video data packet.

[0107] The present invention aims to maximize the throughput while ensuring the end-to-end delay requirement. The delay-guaranteed adaptive bitrate calculator is further used to obtain the following expression based on simultaneously satisfying the end-to-end delay and maximizing the throughput:

[0108]

[0109] s.t. D k ≤δ; (2)

[0110]

[0111] where D k denotes the average end-to-end delay in the k-th control period, T k,j denotes the reception time of the j-th successfully received video data packet in the k-th control period, t k,j denotes the generation time of the j-th successfully received video data packet in the k-th control period; a k denotes the optimal target bitrate in the k-th control period; K denotes the total number of control periods, δ denotes the delay requirement; where K, k, and j are all positive integers, and j is less than or equal to N; Expression (1) represents maximizing the optimal target bitrate for each control period; Expression (2) represents that the average end-to-end delay D k is less than the delay requirement δ.

[0112] Due to the non-stationary transmission delay and packet loss rate, the prior art cannot establish the relationship between the optimal target bitrate a k and the average end-to-end delay D k Therefore, the present invention proposes a delay-guaranteed adaptive bitrate transmission technology for ensuring the end-to-end delay requirement of machine-centric video analysis applications in an unknown and non-stationary network environment. The core idea of this technology is to dynamically estimate the optimal target bitrate a k and the average end-to-end delay D kThe mathematical relationship between them. Based on the established model relationship, the bitrate can be adjusted in an optimal manner. To achieve this goal, the adaptive bitrate transmission method for delay guarantee of real-time video streams according to the present invention first introduces a dynamic linearization data model with a time-varying pseudo-partial derivative parameter (PPD) to describe the mathematical relationship between the bitrate and the delay, then determines the data model by real-time estimating the PPD parameter, and finally decides the final bitrate based on the determined data model.

[0113] Express the average end-to-end delay D in the k-th control period k and the optimal target bitrate a k The relationship between them is expressed as the following dynamic linear data model:

[0114] ΔD k = v k Δa k , (5)

[0115] where, ΔD k = D k - D k-1 , Δa k = a k - a k-1 , v k represents the time-varying pseudo-partial derivative parameter in the k-th control period. Based on equation (5), the complex relationship implicit between the average end-to-end delay D k and the optimal target bitrate a k can be revealed by ΔD k and Δa k through the time-varying pseudo-partial derivative parameter v k . Considering that the non-stationary network characteristics in the video analysis system can be reflected as the dynamic change of the time-varying pseudo-partial derivative parameter v k . Estimate the time-varying pseudo-partial derivative parameter v k in the k-th control period to estimate the estimation formula between the average end-to-end delay D k and the optimal target bitrate a k :

[0116]

[0117] where μ represents the weight parameter, represents the time-varying pseudo-partial derivative parameter in the k -1 -th control period.

[0118] Based on the estimation formula (6), the average end-to-end delay D in equation (5) k and the optimal target bitrate a kThe relationship between them; Next, based on this relationship, the control problem of the target bit rate in formulas (1)-(3) will be further solved, and the average end-to-end delay D will be estimated based on the time-varying pseudo partial derivative parameter k and the optimal target bit rate a k The relationship between them is obtained to obtain the calculation formula of the optimal target bit rate a k :

[0119]

[0120] In a further preferred embodiment of the present invention, in order to further improve the estimation accuracy and ensure stability, the delay-guaranteed adaptive bit rate calculator is further used for the following reset scheme to cope with the error estimation when the transmission delay changes suddenly:

[0121]

[0122] wherein, v0 represents the initial value of the time-varying pseudo partial derivative parameter, and θ is a very small positive number to ensure

[0123] In a further preferred embodiment of the present invention, the delay-guaranteed adaptive bit rate calculator is further used to determine the optimal target bit rate based on the calculated optimal target bit rate, the upper bound of the bit rate, and the lower bound of the bit rate.

[0124] When using the calculation formula (4) of the aforementioned optimal target bit rate a k in a further preferred embodiment of the present invention, the upper and lower bounds of the bit rate are adopted. Among them, the upper bound of the bit rate a ub,k is to prevent queue overflow and infinite growth of the end-to-end delay, representing the link capacity, which can be estimated according to the prior art; the lower bound of the bit rate a lb,k is to avoid selecting too low a bit rate. This is because when the bit rate is much lower than the link capacity (i.e., the queue is always empty), the end-to-end delay will approach the transmission delay. Therefore, further reducing the bit rate cannot reduce the delay any further, but only results in a decrease in throughput, and it can be calculated by the estimation method in the estimation formula [6]. Finally, in order to ensure that a positive bit rate is always selected, the optimal target bit rate a k in the calculation formula (4) of the optimal target bit rate a k provided in the formula is mapped to the actual bit rate by using an exponential function. It should be noted that since the present invention does not make any assumptions about time-varying and non-linear systems, other positive functions are also applicable to this mapping. Based on the above discussion, the actual bit rate used in each control period is given by the following formula:

[0125]

[0126] wherein, a k′ represents the actual bitrate, a k represents the optimal target bitrate a k The optimal target bitrate provided in the calculation formula (4) of a lb,k represents the lower bound of the bitrate, a ub,k represents the upper bound of the bitrate, exp represents the exponential function, and of course any other suitable positive function can be used to replace the exponential function.

[0127] To deploy the aforementioned delay-guaranteed adaptive bitrate transmission method for real-time video streams in an actual system, the receiving device first updates its PPD estimate value using the estimation formula (6) at the beginning of each control period Next, the receiving device calculates the actual bitrate a′ according to the calculation formulas (4) and (8) of the optimal target bitrate a k and transmits the updated bitrate to the sending device. Subsequently, since the receiving device used an exponential mapping when determining the actual bitrate previously, an inverse transformation a k ←log(a k ) is required. After receiving N video data packets, the receiving device calculates the average end-to-end delay using the following formula: k )

[0128]

[0129] Implementing the delay-guaranteed adaptive bit transmission system for real-time video streams of the present invention can maximize the throughput while maintaining the delay below a specific delay requirement in a highly dynamic industrial network.

[0130] To verify the delay-guaranteed adaptive bitrate transmission method and system for real-time video streams of the present invention, we conducted the following simulations and comparisons, where the comparison objects were Google Congestion Control (GCC) technology and Stochastic Gradient Descent (SGD) technology. The GCC technology consists of a delay-based controller and a packet-loss-based controller. Its core is to use a Kalman filter to estimate the link capacity based on the one-way delay gradient, and at the same time combine the packet loss rate for rate adjustment. When the packet loss rate exceeds a certain threshold, the bitrate is decreased multiplicatively to avoid severe congestion. Experiments show that GCC can achieve high link utilization while maintaining low latency. The SGD technology models the bitrate control as an optimization problem and dynamically adjusts the bitrate by minimizing the weighted cost function of delay and link capacity. This technology adjusts the bitrate using the negative gradient direction based on the estimation of queue length and link capacity: once the bitrate exceeds the available capacity, SGD will quickly reduce the bitrate; while when it is below the capacity, SGD will gradually increase the bitrate in smaller steps. Experiments show that SGD can significantly reduce the end-to-end delay. However, this technology may lead to lower throughput and affect video quality.

[0131] In this simulation, the end-to-end delay requirement was considered to be 25 milliseconds and the video frame rate was 30 fps, which is commonly used in video analysis applications in the industrial production field. It was assumed that each control cycle contained N = 8000 data packets for evaluating the average throughput and delay. Without loss of generality, it was assumed that the transmission delay follows a lognormal distribution. The delay-guaranteed adaptive bitrate transmission method or system of the present invention (hereinafter referred to as the LG-ABR technology) does not require any prior assumptions about the distribution of transmission delay, so other distributions are equally applicable.

[0132] To comprehensively evaluate the performance of the proposed LG-ABR technology, this simulation considered two scenarios: a highly dynamic scenario and a high packet loss rate scenario, for performance comparison of the LG-ABR technology with the GCC and SGD technologies. In the highly dynamic scenario, the distribution of transmission delay randomly changes with a probability of 99.8% per second, and the packet loss rate is 0.02%. In the high packet loss rate scenario, the packet loss rate is further increased to 5%. It is worth emphasizing that the transmission delay reflects the quality of service of the communication network, different from the end-to-end delay, which targets the end-to-end experience of users. The goal of the present invention is to guarantee the end-to-end delay of the video stream under the premise of unknown dynamic changes in transmission delay. Table 1 lists the weight parameters used in the LG-ABR technology, and Table 2 summarizes the mean m, variance σ 2 and the range of the packet loss rate ε.

[0133] Table 1 LG-ABR Weight Coefficients

[0134]

[0135] Table 2 Simulation Setting Parameters

[0136]

[0137] Figures 4A - 4B respectively show the average throughput and latency of different technologies in each control cycle under highly dynamic and high packet loss rate scenarios. As Figure 4A shown, in the highly dynamic scenario, the average throughput and latency of the GCC technology in each control cycle may change significantly. This is mainly due to the inaccurate estimation of the link capacity by GCC in the dynamic environment. Although the SGD technology has a lower average latency, its throughput performance is poor. This is because the SGD technology reduces the bit rate when the average latency increases. In contrast, the proposed LG-ABR technology can control the latency below the target value in most cases while achieving high throughput. As Figure 4B ) shown, when the packet loss rate is high, the GCC technology may fail, and similar conclusions have also been pointed out in the existing literature. In the high packet loss rate scenario, the performance of the SGD technology is similar to that in the highly dynamic scenario. In addition, the proposed LG-ABR technology can achieve high throughput while ensuring the latency requirements.

[0138] Finally, Table 3 compares the performance of different technologies in terms of the latency violation probability and link utilization. The latency violation probability is defined as the probability that the latency exceeds the target value, and the link utilization is defined as the ratio of the average throughput to the link capacity. The results show that the proposed LG-ABR technology is significantly superior to the existing technologies in terms of performance. Compared with the GCC technology, the LG-ABR technology reduces the latency violation probability from 49.35% to 1.32% while only reducing the link utilization by 6.26%. Compared with the SGD technology, the LG-ABR technology increases the link utilization by 17 - 19% under a similar latency violation probability.

[0139] Table 3 Performance Comparison

[0140]

[0141] Although the present invention is illustrated by specific embodiments, those skilled in the art should understand that various transformations and equivalent substitutions can be made to the present invention without departing from the scope of the present invention. Additionally, various modifications can be made to the present invention for specific situations or materials without departing from the scope of the present invention. Therefore, the present invention is not limited to the specific embodiments disclosed, but should include all embodiments falling within the scope of the claims of the present invention.

[0142] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention shall be included within the protection scope of the present invention.

Claims

1. An adaptive bitrate transmission method for guaranteeing the delay of a real-time video stream, characterized in that, Including: S1. The sender device encodes the video stream according to the current target bitrate to generate video data packets and sends the video data packets to the receiver device, where N video data packets are sent in each control period, and N is a positive integer; S2. The receiver device stores the video data packets in a buffer, and when N video data packets are received, calculates the average end-to-end delay of the current control period; S3. Determines the optimal target bitrate for the next control period based on the average end-to-end delay, where the relationship between the average end-to-end delay and the optimal target bitrate is estimated based on time-varying pseudo-partial derivative parameters; S4. The sender device receives the optimal target bitrate from the receiver device, uses the optimal target bitrate as the current target bitrate for the next control period, and returns to execute step S1.

2. The adaptive bitrate transmission method for delay guarantee of real-time video streams according to claim 1, wherein, The step S1 further includes: S11. The sender device encodes the video stream according to the current target bitrate to generate the video data packets; S12. Pushes the video data packets into a FIFO queue and sends them to the receiver device through a transmission network.

3. The adaptive bitrate transmission method for delay guarantee of real-time video streams according to claim 1, wherein In the step S2, the average end-to-end delay is calculated based on the following formula: Among them, D k represents the average end-to-end delay of the k-th control period, T k,j represents the reception time of the j-th successfully received video packet within the k-th control period, t k,j represents the generation time of the j-th successfully received video packet within the k-th control period; where k and j are positive integers, and j is less than or equal to N.

4. The adaptive bitrate transmission method for delay guarantee of real-time video stream according to claim 2, characterized in that, The step S3 further includes: S31. Based on simultaneously satisfying the end-to-end delay and maximizing the throughput, obtains the following expression: Among them, D k represents the average end-to-end delay of the k-th control period, T k,j represents the reception time of the j-th successfully received video packet within the k-th control period, t k,j represents the generation time of the j-th successfully received video packet within the k-th control period; a k represents the optimal target bit rate of the k-th control period; K represents the total number of control periods, and δ represents the delay requirement; where K, k, and j are all positive integers, and j is less than or equal to N; Expression (1) represents maximizing the optimal target bit rate of each control period; Expression (2) represents that the average end-to-end delay D k of each control period is less than the delay requirement δ; S32. Estimate the average end-to-end delay D based on time-varying pseudo-derivative parameters k and the optimal target bit rate a k to obtain the calculation formula for the optimal target bit rate a k as follows: Among them, v k represents the time-varying pseudo partial derivative parameter in the k-th control period; represents the estimated value of v k , D k-1 represents the average end-to-end delay in the (k - 1)-th control period; a k-1 represents the optimal target bit rate in the (k - 1)-th control period, and λ represents the weight parameter.

5. The adaptive bitrate transmission method for delay guarantee of real-time video stream according to claim 4, characterized in that The step S32 further includes: S321. Represent the relationship between the average end-to-end delay D in the k-th control period k and the optimal target bit rate a k as the following dynamic linear data model: ΔD k = v k Δa k , (5) where, ΔD k = D k - D k-1 , Δa k = a k - a k-1 ; S322. Estimate the time-varying pseudo partial derivative parameter v in the k-th control period k to estimate the average end-to-end delay D k and the estimated formula between the optimal target bit rate a k ​ where μ represents the weight parameter, denotes the time-varying pseudo partial derivative parameter at the k -1 -th control period; S323. Obtains the calculation formula (4) based on the estimation formula (6).

6. The adaptive bitrate transmission method for delay guarantee of real-time video streams according to claim 5, wherein In the step S322, If or |Δa k-1 | ≤ θ or where v0 represents the initial value of the time-varying pseudo partial derivative parameter, θ is a positive number and ensures that 7. The method for adaptively transmitting bit rate with delay guarantee for real-time video stream according to claim 5, characterized in that The step S3 further includes: S33. Determines the optimal target bitrate based on the optimal target bitrate calculated in step S32, the bitrate upper bound, and the bitrate lower bound.

8. An adaptive bitrate transmission system for ensuring the delay of real-time video streams, characterized in that, Including a sender device, a transmission network, and a receiver device; The sender device is configured to encode a video stream according to a current target bitrate to generate video data packets and send the video data packets to the receiver device, where N video data packets are sent in each control period, and N is a positive integer; The receiver device includes a buffer and a delay-guaranteed adaptive bitrate calculator; The buffer is used to store the video data packets; The delay-guaranteed adaptive bitrate calculator is configured to, when N video data packets are received, calculate the average end-to-end delay of the current control period, and determine the optimal target bitrate for the next control period based on the average end-to-end delay, where the relationship between the average end-to-end delay and the optimal target bitrate is estimated based on time-varying pseudo-partial derivative parameters; The sender device is further configured to receive the optimal target bitrate from the receiver device, use the optimal target bitrate as the current target bitrate for the next control period, and wait for the next update.

9. The adaptive bitrate transmission system for latency guarantee of real-time video streams according to claim 8, characterized in that, The delay-guaranteed adaptive bitrate calculator is further configured to: Calculate the average end-to-end delay based on the following formula: Among them, D k represents the average end-to-end delay of the k-th control period, T k,j represents the reception time of the j-th successfully received video data packet within the k-th control period, t k,j represents the generation time of the j-th successfully received video data packet within the k-th control period; where k and j are positive integers, and j is less than or equal to N.

10. The delay guarantee adaptive bitrate transmission system for real-time video streams according to claim 9, characterized in that, The delay-guaranteed adaptive bitrate calculator is further configured to: Based on simultaneously satisfying the end-to-end delay and maximizing the throughput, obtain the following expression: s.t.D k ≤δ; (2) Among them, D k represents the average end-to-end delay of the k-th control period, T k,j represents the reception time of the j-th successfully received video packet within the k-th control period, t k,j represents the generation time of the j-th successfully received video packet within the k-th control period; a k represents the optimal target bit rate of the k-th control period; K represents the total number of control periods, and δ represents the delay requirement; where K, k, and j are all positive integers, and j is less than or equal to N; Expression (1) represents maximizing the optimal target bit rate of each control period; Expression (2) represents that the average end-to-end delay D k is less than or equal to the delay requirement δ; Estimate the average end-to-end delay D based on time-varying pseudo-derivative parameters k and the optimal target bit rate a k to obtain the calculation formula for the optimal target bit rate a k as follows: Among them, v k represents the time-varying pseudo partial derivative parameter in the k-th control period; represents the estimated value of v k , D k-1 represents the average end-to-end delay in the (k - 1)-th control period; a k-1 represents the optimal target bit rate in the (k - 1)-th control period, and λ represents the weight parameter.

Images