Video stream bit rate self-adaption method and system

Abstract

The invention relates to a video stream bit rate self-adaption method and system. The method comprises the following steps: a user requests video playing at a client; the BPTA module deployed on the client side receives information related to video playing from the server side; the BPTA module comprises a bandwidth prediction model and a bit rate selection model, and the bandwidth prediction model outputs a bandwidth prediction value at a next moment based on an improved transformer neural network and sends the bandwidth prediction value to the bit rate selection model; and the bit rate selection model adaptively selects a bit rate for playing the next video, and the client side sends the bit rate to the server side to download the video block. Compared with the prior art, the method has the advantages of high bandwidth prediction precision, fast decision making, good adaptive effect in a complex environment and the like.

Description

Technical Field

[0001] The present invention relates to the technical field of video processing, and in particular to a method and system for video stream bit rate self-adaptation. Background Art

[0002] Adaptive streaming over HTTP (standardized as DASH) is the dominant form of video delivery today. By using HTTP to deliver video, content providers are able to leverage existing CDN infrastructure and maintain a simplified backend. In addition, HTTP is compatible with a large number of client-side applications such as web browsers and mobile applications. In a DASH system, the video is stored on the server as multiple chunks, each of which represents a few seconds of the entire video playback. Each chunk is encoded at several discrete bit rates, where a higher bit rate means higher quality and therefore a larger chunk size.

[0003] In recent years, the traffic of video streaming over HTTP has increased rapidly. At the same time, users' requirements for video quality have also been rising steadily. Many studies have shown that users will quickly abandon video sessions if the quality is not enough, resulting in significant revenue losses for content providers. Despite this, content providers are still working hard to provide high-quality videos to viewers. Content providers mainly use adaptive bitrate (ABR) algorithms to optimize video quality, which run on the client-side video player and dynamically select the bitrate for each video chunk. ABR algorithms make bitrate decisions based on various observations, such as estimated network throughput and playback buffer occupancy. Their goal is to maximize the user's quality of experience (QoE) by adapting the video bitrate to the underlying network conditions.

[0004] However, most ABR algorithms have difficulty making correct decisions due to the following issues: (1) variability in network throughput; (2) conflicting video QoE requirements (such as high bitrate, minimal rebuffering, smoothness, etc.); and (3) the cascading effects of bitrate decisions (e.g., choosing a high bitrate may exhaust the playback buffer to dangerous levels and cause future rebuffering).

[0005] Currently, Pensieve, the most recognized algorithm in the field of video streaming ABR, is an automatically learned ABR algorithm that does not use any pre-programmed control rules or explicit assumptions about the operating environment. Pensieve uses modern reinforcement learning (RL) techniques to learn control policies for bitrate adaptation purely through experience. During training, Pensieve starts out knowing nothing about the task at hand. It then gradually learns to make better ABR decisions through reinforcement learning, with the form of reward signals reflecting the video QoE of past decisions. For example, Pensieve can learn how much playback buffer is needed to mitigate the risk of rebuffering in a specific network based on the network's inherent throughput variability. In contrast, methods that use fixed control rules or simplified network models cannot optimize their bitrate decisions based on all available information about the operating environment. Pensieve trains this neural network using the A3C algorithm, a state-of-the-art actor-critic RL algorithm.

[0006] However, research has found that the above algorithms have many shortcomings. For example, they are too conservative in using bandwidth and are not as good as some model-based methods in some cases. When using the A3C algorithm for training, there are disadvantages such as long training time and inability to make quick decisions in a short time.

[0007] After searching, Chinese invention patent publication number CN113596021B discloses a streaming media bit rate adaptation method, device and equipment supporting neural network, the method comprising: obtaining historical network throughput measurement values ​​of video blocks in the previous n time periods, a vector of preset available resolution size and current buffer occupancy information, inputting the historical network throughput measurement values, the vector of the resolution size and the current buffer occupancy information into a pre-constructed Bayesian neural network, and outputting the throughput prediction value of the next time period through the Bayesian neural network; obtaining the QoE information of the video block in the previous time period, constructing a model predictive control optimization model based on the throughput prediction value, the current buffer occupancy information and the QoE information with the preset QoE index optimization as the goal, solving the model predictive control optimization model to obtain the predicted download bit rate of the current video block; after executing the predicted download bit rate, obtaining a corresponding reward value based on the QoE index, and continuously training the Bayesian neural network based on the predicted download bit rate and the reward value; during the video playback process, adaptively obtaining the optimal bit rate of the downloaded video block based on the continuously trained Bayesian neural network and the model predictive control optimization model. This existing patent does not take bandwidth factors into consideration, and the model predictive control uses MPC, so there are problems of slow bit rate adaptation and insufficient accuracy.

[0008] How to achieve video stream bit rate adaptation based on accurate bandwidth prediction has become a technical problem that needs to be solved. Summary of the invention

[0009] The purpose of the present invention is to provide a video stream bit rate adaptive method and system in order to overcome the defects of the above-mentioned prior art.

[0010] The purpose of the present invention can be achieved by the following technical solutions:

[0011] According to one aspect of the present invention, a method for video stream bit rate adaptation is provided, the method comprising the following steps:

[0012] Step S1, the user requests video playback on the client;

[0013] Step S2, the BPTA module deployed on the client receives information related to video playback from the server, wherein the information related to video playback includes the throughput of the past block, the download time of the past block, the size of the next video block, the buffer information, the number of remaining video blocks, the bit rate of the previous block, and the bandwidth information;

[0014] Step S3, the BPTA module includes a bandwidth prediction model and a bit rate selection model, wherein the bandwidth prediction model outputs a bandwidth prediction value at the next moment according to the received bandwidth information and based on an improved transformer neural network, and sends the output to the bit rate selection model;

[0015] Step S4, the bit rate selection model combines the bandwidth information and bit rate information of the previous block and the buffer information of the past block to adaptively select the bit rate for playing the next video block, which is sent by the client to the server to download the video block.

[0016] Preferably, the bandwidth prediction model is based on an improved transformer neural network and specifically includes: adding an LSTM layer to both the encoder and the decoder of the transformer, wherein the LSTM layer is located between the feedforward neural network and the normalization layer.

[0017] Preferably, the bandwidth prediction model comprises an encoder and a decoder connected in sequence;

[0018] In the encoder, the attention score is first calculated through the multi-head attention mechanism, then handed over to the normalization layer for data normalization calculation, and then passed through the feedforward neural network and LSTM layer in sequence, and finally output to the decoder through the normalization layer;

[0019] The multi-head attention mechanism is the input part of the encoder, including an input embedding unit and a position encoding unit, wherein the input embedding unit uses a linear layer in a neural network to project the bandwidth data of the input video stream into a vector of a fixed size; then the position encoding unit is used to provide a vector representing its position in the sequence for each element in the bandwidth data; the outputs of the input embedding unit and the position encoding unit are added and input into the multi-head attention mechanism;

[0020] The output of the decoder passes through the linear layer and then outputs the prediction result.

[0021] More preferably, the bandwidth prediction model includes multiple encoders and multiple decoders.

[0022] Preferably, the bitrate selection model is based on the Actor-Critic framework of reinforcement learning, including an Actor network and a Critic network. The bandwidth prediction value at the next moment is input into the bitrate selection model. Combined with the information related to video playback in step S2, the Actor network outputs the bitrate of the next video block, and the Critic network outputs the value of the current state. By updating the network parameters, the optimal cumulative expected reward of the network is found, thereby finding the optimal bitrate action for the next video block.

[0023] More preferably, the Actor network and the Critic network are trained using the DC-PPO algorithm;

[0024] The loss function of the DC-PPO algorithm is specifically:

[0025]

[0026]

[0027] When the advantage function hour:

[0028]

[0029] In other cases:

[0030]

[0031] in, represents the advantage function, i.e., the part of the real reward r observed by the agent from the environment t and state estimates With the actual status value difference; is the discount factor; Represents the advantage function Take the expected operation; ratio t (St ,a t ) is the clipping function; (S t ,a t ) is each sampled state-action pair; r t (S t ,a t ) is the execution of action a under the new strategy and the old strategy t The probability ratio of .

[0032] More preferably, the gradient descent method is used to update the parameters θ of the Actor network, specifically:

[0033]

[0034] Among them, Η θ (S t ) indicates that in state S t Next strategy π θ The entropy of old is the parameter θ of the Actor network in the previous state, Indicates the gradient of parameter θ, L DPPO represents the loss function.

[0035] According to another aspect of the present invention, a video stream bit rate adaptation system is provided, the system includes a client and a server, the client deploys a BPTA module, the BPTA module includes a bandwidth prediction model and a bit rate selection model connected in sequence; the bandwidth prediction model outputs a bandwidth prediction value at the next moment based on an improved transformer neural network and sends it to the bit rate selection model; the bit rate selection model combines the bandwidth information and bit rate information of the previous block and the buffer information of the past block, adaptively selects the bit rate for playing the next video block, and is sent by the client to the server to download the video block; the information related to video playback includes the throughput of the past block, the download time of the past block, the size of the next video block, the buffer information, the number of remaining video blocks, the bit rate of the previous video block, and the bandwidth information.

[0036] Preferably, the bandwidth prediction model is based on an improved transformer neural network and specifically includes: adding an LSTM layer to both the encoder and the decoder of the transformer, wherein the LSTM layer is located between the feedforward neural network and the normalization layer.

[0037] Preferably, the bitrate selection model is based on an Actor-Critic framework of reinforcement learning, including an Actor network and a Critic network. The bandwidth prediction value at the next moment is input into the bitrate selection model. Combined with information related to video playback, the Actor network outputs the bitrate of the next video block, and the Critic network outputs the value of the current state. By updating network parameters, the optimal cumulative expected reward of the network is found, thereby finding the optimal bitrate action for the next video block.

[0038] The Actor network and Critic network are trained using the DC-PPO algorithm.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] 1) The present invention outputs the bandwidth prediction value of the next moment based on the improved transformer neural network and sends it to the bit rate selection model. Then the bit rate selection model adaptively selects the bit rate for playing the next video block in combination with the information related to the video playback. Compared with the traditional video stream bit rate selection algorithm, the bandwidth prediction accuracy is high, so the bit rate adaptation effect is still very good even in an environment with particularly harsh network conditions.

[0041] 2) The present invention adds an LSTM layer in the transformer neural network for bandwidth prediction, which is suitable for long sequence data of video streams and improves the prediction accuracy of time series. At the same time, due to the improvement of bandwidth prediction accuracy, the bit rate adaptation calculation time is greatly reduced without the need for too many agents.

[0042] 3) The bitrate selection model of the present invention is based on the Actor-Critic framework of reinforcement learning, which is trained using the DC-PPO algorithm. The reward is fixed within a certain range through double-end clipping. The two clipping ranges are used to process the positive and negative values ​​of the advantage function respectively, which can more finely control the step size of the strategy update and achieve a more stable optimal strategy in a shorter time. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 A schematic diagram of the process of the video stream bit rate adaptation method in the present invention;

[0044] Figure 2 It is a structural schematic diagram of the BPTA module in the present invention;

[0045] Figure 3 It is a structural schematic diagram of the TBP model in the present invention;

[0046] Figure 4 It is a structural schematic diagram of the BSM model in the present invention. DETAILED DESCRIPTION

[0047] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0048] Most existing ABR algorithms currently develop fixed control rules for making bitrate decisions based on estimated network throughput ("rate-based" algorithms), playout buffer size ("buffer-based" schemes), or a combination of both signals. These schemes require a lot of tuning and cannot be generalized to different network conditions and Quality of Experience (QoE) targets. Model Predictive Control (MPC) makes bitrate decisions by solving the QoE maximization problem over a few future blocks. By optimizing directly for the desired QoE target, MPC can perform better than methods using fixed algorithms. However, the performance of MPC depends on an accurate system dynamics model, especially the prediction of future network throughput. The MPC algorithm predicts the throughput as an input parameter through the system dynamics model, which has a certain loss of prediction accuracy. In addition, the MPC algorithm relies too much on prediction accuracy. When the prediction deviation is too large, the ABR results output by the model will also be poor. The Pensieve algorithm, the current best ABR algorithm, uses reinforcement learning to continuously interact with the agent and the environment to determine an optimal strategy. The Pensieve algorithm uses the latest A3C algorithm and uses 16 agents to interact with independent environments at the same time, which can find the current optimal strategy faster and more accurately. However, in experiments, it was found that when using 16 agents for training, it would take a long time. To find the optimal strategy, hundreds of thousands of iterations are required, and in some cases, the actual effect of the Pensieve algorithm is not as good as the MPC algorithm. The main reason is that the Pensieve algorithm is too conservative in its use of bandwidth and pursues a smoother video playback effect. Therefore, when the actual bandwidth is better, it dare not choose a higher video bit rate, which also leads to a decline in user viewing experience.

[0049] In view of the problems existing in the above-mentioned prior art, the present invention proposes a video stream bit rate adaptive algorithm based on transformer bandwidth prediction, namely BPTA (Bandwidth_Predict_based_Transformer_ABR). The BPTA algorithm estimates the bandwidth value at the next moment based on the existing bandwidth, and achieves precise control of the video stream bit rate by improving the accuracy of model prediction.

[0050] The present embodiment relates to a video stream bit rate adaptation method and system, wherein the method is implemented based on a video stream bit rate adaptation module (BPTA module) based on transformer bandwidth prediction.

[0051] like Figure 1 ,BPTA module includes two models connected in sequence: bandwidth prediction model (Transformer_Bandwidth_Prediction,TBP) and bitrate selection model (Bitrate_Selection_Model,BSM). The TBP model consists of transformer sub-models, which are used to output bandwidth prediction values; the BSM model consists of a reinforcement learning Actor-Critic framework, which is trained using the DC-PPO algorithm and is used to output bitrate actions.

[0052] like Figure 1 and Figure 2 ,The process of the video stream bit rate adaptation method based on bandwidth prediction includes the following steps:

[0053] Step S1, the user requests video playback on the client;

[0054] Step S2, after receiving the request, the server sends information related to video playback to the BPTA module deployed on the client; the information related to video playback includes the throughput of the past block, the download time of the past block, the size of the next video block, the buffer information, the number of remaining video blocks, the bit rate of the previous block, and the bandwidth information.

[0055] Step S3, the BPTA module first sends the bandwidth information to the bandwidth prediction model (TBP), and the bandwidth prediction model outputs the bandwidth prediction value of the next moment based on the transformer and sends it to the bit rate selection model (BSM);

[0056] Step S4, the bitrate selection model (BSM) combines the previous bandwidth information, bitrate information, and buffer information, etc., to adaptively select the bitrate for playing the next video block, which is sent by the client to the server to download the video block. The Actor network in the BSM model outputs the bitrate of the next block; the Critic network in the BSM model outputs the value of the current state, and by updating the network parameters, the optimal cumulative expected reward of the network is found, thereby finding the optimal action.

[0057] The two models are described in detail below.

[0058] 1. Bandwidth Prediction Model (TBP)

[0059] For complex and ever-changing network environments, it is difficult for ordinary network bandwidth estimation methods to achieve high prediction accuracy. This paper introduces a bandwidth prediction algorithm based on transformer. Transformer is a deep learning model based on the self-attention mechanism, which was originally proposed by Google researchers in the 2017 paper "Attention is All You Need". After research, people have found that the transformer's self-attention mechanism can simultaneously consider long-distance dependencies in time series, unlike traditional RNN or LSTM, which requires information to be passed through each element of the sequence one by one. This makes the transformer perform well in capturing long-term dependencies. In addition, the multi-head attention mechanism is also introduced in the transformer. The multi-head attention mechanism allows the model to capture different aspects and levels of information of the sequence data in different "heads", which helps to improve the accuracy and robustness of time series prediction. Based on the above findings, we combine the characteristics of the transformer and practical problems and propose the TBP model.

[0060] like Figure 3 , the TBP model consists of three parts, namely encoder (encode), decoder (decode) and output. In the encoder block, the main function of the input embedding unit and the position encoding unit is to process the input data (bandwidth data of the video stream). The multi-head attention mechanism is mainly used to calculate the attention score, which enables the model to effectively identify and utilize the long-distance dependencies in the input data. Compared with the 8 attention heads used in the original transformer, we use 3 attention heads for calculation. The Add&Norm layer is mainly used for data normalization to avoid overfitting. The feedforward neural network uses 2 fully connected layers and uses RELU as the activation function. In addition, it also contains an LSTM layer, which is mainly used to further obtain temporal features.

[0061] Mask in decode - Multi-head attention uses sequence masking to prevent the model from observing future data. The rest of the decode is similar to the encode. Finally, the prediction result is output through a linear layer with feature 1.

[0062] First, we take the bandwidth data as input data, and after passing through the input embedding unit, project the bandwidth data into a vector of fixed size. The input embedding here is replaced by the linear layer in the neural network. Then, the position encoding unit is used to provide each element in the bandwidth data with a vector representing its position in the sequence, so that the model can use this information to understand the order of the sequence. The calculation of the position encoding uses the sine and cosine function encoding used in the original transformer. It can be expressed as:

[0063]

[0064] Here, i represents the index in dimension d (starting from 0) and pos represents the position.

[0065] Then add the matrices calculated from these two parts to get the matrix X = {x1, x2, ..., x i}(Here each x i are all matrices), and X is input into the encoder module in the transformer. In the encoder module, the attention score is first calculated through a multi-head attention mechanism.

[0066] The self-attention mechanism is expressed as:

[0067]

[0068] Among them, Q, K, and V represent the query vector, key vector, and value vector respectively. These three vectors are obtained by multiplying the matrix X with the three weight matrices, namely:

[0069] X*W Q =Q

[0070] X*W K =K

[0071] X*W v =V

[0072] Among them, W Q , W K , W V are all weight matrices.

[0073] The so-called multi-head attention is to use multiple such weight matrices to multiply different x matrices in matrix X to obtain {Z0, Z1, ..., Z i}, after merging the above matrices, multiply them with the additional weight matrix W to get the final attention head. Note that the attention head obtained here is still a matrix. In this article, we use 3 attention heads for calculation.

[0074] We then pass the result to the next Add&Norm layer. In this layer, two parts are normalized, one is X and the other is Z. The formula is: i =LayerNorm(X i ,Z i ), Layer Normalization is used here. The purpose of normalization is mainly to speed up training and improve stability.

[0075] The obtained z is input into the feedforward neural network, which is mainly composed of two fully connected layers and uses ReLu as the activation function. In order to solve the practical problem of low prediction accuracy, this application has made some changes to the transformer structure. We add an LSTM layer after the feedforward neural network. Combined with the characteristics of LSTM being good at processing long sequence data, this is beneficial for the task. Set the number of LSTM neurons to 512. Finally, the encoding module result is output through the same Add&Norm layer as before.

[0076] The decode module has a similar structure to the encoder module. Two points need to be noted here: First, the input of the decode module is obtained by shifting the original data to the right by 5 positions, and it is named output to distinguish it from the input of the encoder. The subsequent output embedding operation is the same as the encoder. Second, the mask-multi-head attention, the so-called mask is a matrix used to indicate which positions should be masked. For prediction tasks, sequence masks are used, which are used to prevent the model from seeing future information when processing sequences. The sequence mask is an upper triangular matrix, where the elements of the upper triangular part are negative infinity to reduce the probability of these positions to almost zero when applying softmax. In addition, in the multi-head attention mechanism in the decode, the output of the encoder module is used as K and V, that is, the key vector and the value vector. The query vector Q is the output of the mask-multi-head attention after the Add&Norm layer. Similarly, the LSTM layer is also added to the decode module to capture time series information.

[0077] Multiple identical encoder and decoder modules can be used in the transformer. In this paper, three encoder and three decoder modules are used. Finally, the final output of the decoder module is passed through a linear layer with feature 1 to output the final prediction result.

[0078] 2. Bitrate Selection Network Model (BSM)

[0079] During the experiment, it was found that how to combine the bandwidth prediction model and the bit rate selection model is a challenge. In order to meet this challenge, we consider using a deep reinforcement learning algorithm and taking the predicted bandwidth value as the state value of the reinforcement learning model, thereby linking the bandwidth prediction result with the bit rate selection. Figure 4 As shown. The state S includes the throughput measurement of the past blocks, the download time and size, the bitrate selection of the previous block, the current buffer size, the number of remaining video blocks, and the predicted bandwidth value. The first layer of the actor network and the critic network both contain 3 1D convolutional layers and 4 fully connected layers, each containing 128 neurons. In the actor network, the second layer is a fully connected layer, that is, it is first merged and then output to the next layer using the fully connected layer. In practical applications, the bitrate selection should be more accurate, so a separate fully connected layer is set in the actor network to downsample the bitrate action to obtain better bitrate action. The results are output through a fully connected layer in the critic network.

[0080] Reinforcement Learning (RL) is a learning method in which an agent interacts with an environment to obtain the maximum cumulative reward. The agent observes the state S from the environment. t , and take action a t Give the environment, the environment gives the Agent reward r t , and transfer the state to state S t+1 The whole process follows the Markov decision process (MDP), and MDP uses tuples t , a t , r t > to describe. The parameters of RL are defined as follows:

[0081] Input (State S t ): After downloading each block t, the Agent enters the state Bring it to the BSM neural network, such as Figure 4 .in, is the network throughput measurement for the past k video chunks; is the download time of the past k video chunks, which represents the time interval for throughput measurement; is a vector of available sizes for the next video block; B t is the current buffer level; c t is the number of chunks remaining in the video; p​t is the bit rate of downloading the previous block; G t It indicates predicting the bandwidth value at the next time t+1 at time t.

[0082] Output (Action a t ): After receiving the state, the reinforcement learning agent needs to choose the action a of the bitrate of the next video chunk according to the environment t The agent selects actions based on the strategy, which is defined as: θ {S t ,a t}→[0,1], where π θ {S t ,a t} means in state S t Take action a t The probability of θ is an adjustable parameter, called the strategy parameter. It can be optimized by a neural network to automatically select a suitable value.

[0083] Reward (r t ): In the problem of bitrate adaptation, the linear formula of QoE is usually used to evaluate the quality of the algorithm. However, given the particularity of this performance, how to link the accuracy of bandwidth prediction with QoE is a challenge that needs to be solved. For the traditional linear formula of QoE, QoE is usually expressed as the user-perceived video quality q(R n ), the switching frequency between different video blocks |q(R n+1 -R n )| and rebuffer time T n The weighted values ​​of the three components and the bandwidth prediction accuracy I are introduced t , which is expressed as the ratio of the predicted bandwidth value to the actual bandwidth value at time t. At the same time, the weight coefficient ω is also considered to judge whether the user pays attention to the prediction error. Therefore, QoE is expressed as:

[0084]

[0085] Among them, α is a non-negative weight coefficient; q(R n ) represents the video quality perceived by the user, i.e., the bit rate R n Function mapped to QoE; I t It represents the ratio of the predicted bandwidth value to the actual bandwidth value at time t;

[0086] Rebuffering time, which means when the download time Greater than the buffer capacity B max The sum of the rebuffering times that occur when ω is the weight coefficient.

[0087] Download time u of the next video chunk t+1 , expressed as:

[0088]

[0089] Among them, u t Indicates the download time of the current video block, d t (R t ) represents the code rate R t The video block size at time Δu t Indicates the time from when the player sends a video request to the server to when the video encoding is completed; C t represents the real-time network throughput at time t, C n represents the real-time network throughput at time n, d n Represents the integral symbol in mathematics.

[0090] During video playback, the downloaded video blocks will be stored in the buffer, following the first-in-first-out principle. Buffer occupancy rate B t The dynamic change process of involves the video chunk being downloaded to the buffer and the video chunk being played in the buffer. When the video chunk n is downloaded, the buffer occupancy increases by L seconds and decreases as the user watches the video. Let B t It means that at time t, the player starts downloading video chunk n, and the buffer occupancy rate B t+1 The dynamic process is expressed as follows: Where L represents the duration of video block n.

[0091] Training algorithm: The actor-critic reinforcement learning network structure is adopted, and the improved algorithm DC-PPO (Dual-Clip PPO) of the proximal policy optimization algorithm (PPO) is used to train the neural network. The specific training method is as follows:

[0092] 1) Update of actor network strategy parameters θ:

[0093] (a) Clipping Importance Sampling Ratio: In PPO-Clip, the importance sampling ratio is used to measure the relative performance between the new strategy and the old strategy. Specifically, for each sampled state-action pair (S, a), the probability ratio of executing action a under the new strategy and the old strategy is calculated: Here π θ (a|S) and Represent the probability of executing action a in state S under the new and old strategies respectively. Introducing the clipping function ratio t(S,a)=clip(r t (S,a),1-ε,1+ε), directly limits the amplitude of policy updates by applying a clipping function on the policy ratio. Specifically, the policy ratio is fixed within a range to prevent drastic fluctuations in learning performance due to overly aggressive updates. Here, ε is a hyperparameter, generally set to 0.1.

[0094] (b) DC-PPO loss function: In DC-PPO, in order to ensure that the policy update does not deviate too far from the old policy, double-end pruning is used to further fix r t range.

[0095] When the advantage function hour:

[0096]

[0097] In other cases:

[0098]

[0099]

[0100] in represents the advantage function, which means the part of the real reward r observed by the agent from the environment t and state estimates With the actual status value difference; is the discount factor, generally Pick Express Take the desired action, that is, consider all possible values ​​and take a weighted average according to their probabilities.

[0101] Compared with A3C, DC-PPO uses two clipping ranges to handle positive and negative values ​​of the advantage function. This allows for more precise control of the step size of the policy update to avoid updates that are too large or too small. In contrast, A3C may cause policy instability due to asynchronous updates.

[0102] Compared with A3C, DC-PPO has higher sample efficiency because it can use more experience data to perform multiple policy updates, while A3C updates each thread once after collecting a certain amount of experience, which also leads to shorter training time for DC-PPO than A3C.

[0103] (c) Using the gradient descent method to update the parameters, the Actor network parameter θ is updated as:

[0104]

[0105] Among them, Η θ (S t ) indicates that in state S t Next strategy π θ entropy; β is a hyperparameter used to control the influence of the entropy term, that is, the degree of exploration. Increasing the value of β will give more weight to the entropy term and encourage more exploration behavior. In the experiment, β is set to 0.1. θ old is the value of the actor network parameter θ in the previous state, Indicates the gradient of parameter θ, L DPPO represents the loss function.

[0106] 2) Critic network parameters θ v Update: Use the mean squared error function (MSE) as the loss function, calculate the gradient through the loss function, and update the parameter θ v , in order to reduce the error of value prediction.

[0107] 3) Parallel training: To speed up the training of DC-PPO, 2 agents and 2 different environments are used simultaneously, along with 1 central agent, and these agents are trained in parallel.

[0108] This embodiment also relates to a video stream bit rate adaptation system, which includes a client and a server. The client deploys a BPTA module, which includes a video stream bit rate adaptation method for transformer bandwidth prediction.

[0109] like Figure 1 , the BPTA module includes a bandwidth prediction model and a bit rate selection model connected in sequence; the bandwidth prediction model outputs the bandwidth prediction value of the next moment based on the improved transformer neural network and sends it to the bit rate selection model. The bit rate selection model combines the bandwidth information and bit rate information of the previous block and the buffer information of the past block to adaptively select the bit rate for playing the next video block, which is sent by the client to the server to download the video block; the information related to video playback includes the throughput of the past block, the download time of the past block, the size of the next video block, the buffer information, the number of remaining video blocks, the bit rate of the previous video block, and the bandwidth information.

[0110] The bandwidth prediction model is based on the improved transformer neural network. Specifically, LSTM layers are added to both the encoder and decoder of the transformer, where the LSTM layer is located between the feedforward neural network and the normalization layer.

[0111] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A video stream bit rate adaptive method, characterized in that: The method comprises the following steps: Step S1, the user requests video playback on the client; Step S2, the BPTA module deployed on the client receives information related to video playback from the server, wherein the information related to video playback includes the throughput of the past block, the download time of the past block, the size of the next video block, the buffer information, the number of remaining video blocks, the bit rate of the previous block, and the bandwidth information; Step S3, the BPTA module includes a bandwidth prediction model and a bit rate selection model, wherein the bandwidth prediction model outputs a bandwidth prediction value at the next moment according to the received bandwidth information and based on an improved transformer neural network, and sends the output to the bit rate selection model; Step S4, the bit rate selection model combines the bandwidth information and bit rate information of the previous block and the buffer information of the past block to adaptively select the bit rate for playing the next video block, which is sent by the client to the server to download the video block.

2. A video stream bit rate adaptive method according to claim 1, characterized in that: The bandwidth prediction model is based on an improved transformer neural network and specifically includes: adding LSTM layers to both the encoder and decoder of the transformer, wherein the LSTM layer is located between the feedforward neural network and the normalization layer.

3. A video stream bit rate adaptive method according to claim 1, characterized in that: The bandwidth prediction model includes an encoder and a decoder connected in sequence; In the encoder, the attention score is first calculated through the multi-head attention mechanism, then handed over to the normalization layer for data normalization calculation, and then passed through the feedforward neural network and LSTM layer in sequence, and finally output to the decoder through the normalization layer; The multi-head attention mechanism is the input part of the encoder, including an input embedding unit and a position encoding unit, wherein the input embedding unit uses a linear layer in a neural network to project the bandwidth data of the input video stream into a vector of a fixed size; then the position encoding unit is used to provide a vector representing its position in the sequence for each element in the bandwidth data; the outputs of the input embedding unit and the position encoding unit are added and input into the multi-head attention mechanism; The output of the decoder passes through the linear layer and then outputs the prediction result.

4. A video stream bit rate adaptive method according to claim 3, characterized in that: The bandwidth prediction model includes multiple encoders and multiple decoders.

5. A video stream bit rate adaptive method according to claim 1, characterized in that: The bitrate selection model is based on the Actor-Critic framework of reinforcement learning, including an Actor network and a Critic network. The bandwidth prediction value at the next moment is input into the bitrate selection model. Combined with the information related to video playback in step S2, the Actor network outputs the bitrate of the next video block, and the Critic network outputs the value of the current state. By updating the network parameters, the optimal cumulative expected reward of the network is found, thereby finding the optimal bitrate action for the next video block.

6. A video stream bit rate adaptive method according to claim 5, characterized in that: The Actor network and Critic network are trained using the DC-PPO algorithm; The loss function of the DC-PPO algorithm is specifically: When the advantage function hour: In other cases: in, represents the advantage function, i.e., the part of the real reward r observed by the agent from the environment t and state estimates With the actual status value difference; is the discount factor; Represents the advantage function Take the expected operation; ratio t (S t ,a t ) is the clipping function; (S t ,a t ) is each sampled state-action pair; r t (S t ,a t ) is the execution of action a under the new strategy and the old strategy t The probability ratio of .

7. A video stream bit rate adaptive method according to claim 6, characterized in that: Use the gradient descent method to update the parameters θ of the Actor network, specifically: Among them, Η θ (S t ) indicates that in state S t Next strategy π θ The entropy of old is the parameter θ of the Actor network in the previous state, Indicates the gradient of parameter θ, L DPPO Represents the loss function.

8. A system using the video stream bit rate adaptation method according to any one of claims 1 to 8, the system comprising a client and a server, characterized in that: The client deploys a BPTA module, which includes a bandwidth prediction model and a bit rate selection model connected in sequence; the bandwidth prediction model outputs a bandwidth prediction value at the next moment based on an improved transformer neural network and sends it to the bit rate selection model; the bit rate selection model combines bandwidth information and bit rate information of a previous block and buffer information of a past block, adaptively selects a bit rate for playing a next video block, and sends it from the client to the server to download the video block; the information related to video playback includes the throughput of the past block, the download time of the past block, the size of the next video block, the buffer information, the number of remaining video blocks, the bit rate of the previous video block, and the bandwidth information.

9. The system according to claim 8, characterized in that The bandwidth prediction model is based on an improved transformer neural network and specifically includes: adding LSTM layers to both the encoder and decoder of the transformer, wherein the LSTM layer is located between the feedforward neural network and the normalization layer.

10. The system according to claim 8, characterized in that The bitrate selection model is based on the Actor-Critic framework of reinforcement learning, including an Actor network and a Critic network. The bandwidth prediction value at the next moment is input into the bitrate selection model. Combined with information related to video playback, the Actor network outputs the bitrate of the next video block, and the Critic network outputs the value of the current state. By updating the network parameters, the optimal cumulative expected reward of the network is found, thereby finding the optimal bitrate action for the next video block. The Actor network and Critic network are trained using the DC-PPO algorithm.

Images