Video stream bit rate self-adaption method and device based on bandwidth prediction
By deploying bandwidth prediction and reinforcement learning on the client, fast and accurate adaptation of video stream bit rate is achieved, the existing algorithms are solved in bandwidth utilization and response speed, the user experience quality is improved, and the user experience is improved in a variety of ways. Excellent performance in network environments.
Patent Information
- Application Number
- CN202411039787.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-31
- Publication Date
- 2024-12-03
AI Technical Summary
Existing video stream bit rate adaptive algorithms have shortcomings in terms of fast response and accuracy, especially because they are too conservative in bandwidth utilization, resulting in a decline in user experience and long training time, making it difficult to make decisions in a short period of time.
Adopting the video stream bit rate adaptive method based on bandwidth prediction, by deploying BPA modules on the client, using BiLstm network for bandwidth prediction, and combining the Actor-Critic framework of reinforcement learning, adaptively selecting the bit rate of video blocks, quickly and accurately Adjust the bit rate to optimize the user experience.
It realizes fast and accurate adaptation of video stream bit rate, improves user experience quality, reduces training time, and performs well in multiple network environments. It is better than the existing optimal algorithm Pensieve, and only requires 2 parallel agents. Achieved the performance of 16 agents.
Smart Images

Figure CN119071576A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of video processing, and in particular to a method and device for video stream bit rate self-adaptation based on bandwidth prediction. Background Art
[0002] Adaptive streaming over HTTP (standardized as DASH) is the dominant form of video delivery today. By using HTTP to deliver video, content providers are able to leverage existing CDN infrastructure and maintain a simplified backend. In addition, HTTP is compatible with a large number of client-side applications such as web browsers and mobile applications. In a DASH system, the video is stored on the server as multiple chunks, each of which represents a few seconds of the entire video playback. Each chunk is encoded at several discrete bit rates, where a higher bit rate means higher quality and therefore a larger chunk size.
[0003] In recent years, the amount of video streaming traffic over HTTP has increased rapidly. At the same time, user requirements for video quality have also been rising steadily. Many studies have shown that users will quickly abandon a video session if the quality is not adequate, resulting in significant revenue losses for content providers. Despite this, content providers continue to strive to deliver high-quality videos to viewers. Adaptive bitrate (ABR) algorithms are the primary tool used by content providers to optimize video quality. These algorithms run on the client-side video player and dynamically select a bitrate for each video chunk. ABR algorithms make bitrate decisions based on various observations, such as estimated network throughput and playback buffer occupancy. Their goal is to maximize the user's Quality of Experience (QoE) by adapting the video bitrate to the underlying network conditions. However, most ABR algorithms have difficulty making the right decisions due to the following factors: (1) variability in network throughput; (2) conflicting video QoE requirements (e.g., high bitrate, minimal rebuffering, smoothness, etc.); and (3) the cascading effects of bitrate decisions (e.g., selecting a high bitrate may exhaust the playback buffer to dangerous levels and cause future rebuffering).
[0004] Pensieve, currently recognized as the best algorithm in the field of video streaming ABR, is an automatically learned ABR algorithm that does not use any pre-programmed control rules or explicit assumptions about the operating environment. Pensieve uses modern reinforcement learning (RL) techniques to learn control strategies for bitrate adaptation purely through experience. During training, Pensieve starts out knowing nothing about the task at hand. It then gradually learns to make better ABR decisions through reinforcement learning, with the form of reward signals reflecting the video QoE of past decisions. For example, Pensieve can learn how much playback buffer is needed to mitigate the risk of rebuffering in a specific network based on the network's inherent throughput variability.
[0005] Most existing ABR algorithms develop fixed control rules for making bitrate decisions based on estimated network throughput ("rate-based" algorithms), playout buffer size ("buffer-based" schemes), or a combination of both signals. These schemes require extensive tuning and do not generalize to different network conditions and QoE objectives. In contrast, approaches that use fixed control rules or simplified network models are unable to optimize their bitrate decisions based on all available information about the operating environment. Pensieve trains this neural network using the A3C algorithm, a state-of-the-art actor-critic RL algorithm.
[0006] However, research has found that the above algorithms have many shortcomings. For example, they are too conservative in using bandwidth and are not as good as some model-based methods in some cases. When using the A3C algorithm for training, there are disadvantages such as long training time and inability to make quick decisions in a short time.
[0007] Model Predictive Control (MPC) makes bitrate decisions by solving the QoE maximization problem within the range of several future blocks. By directly optimizing for the desired QoE target, MPC can perform better than methods using fixed algorithms. However, the performance of MPC depends on an accurate system dynamic model, especially the prediction of future network throughput. The MPC algorithm predicts the throughput through the system dynamic model as an input parameter, which has a certain loss of prediction accuracy. In addition, the MPC algorithm is overly dependent on prediction accuracy. When the prediction deviation is too large, the ABR result output by the model will also be poor. The current best ABR algorithm, the Pensieve algorithm, uses reinforcement learning to continuously interact with the agent and the environment to determine an optimal strategy. The Pensieve algorithm uses the latest A3C algorithm and uses 16 agents to interact with independent environments at the same time, which can find the current optimal strategy faster and more accurately. However, in experiments, it was found that when 16 agents are used for training, it will take a long time. To find the optimal strategy, hundreds of thousands of iterations are required, and in some cases, the actual effect of the Pensieve algorithm is not as good as the MPC algorithm. The main reason is that the Pensieve algorithm is too conservative in its use of bandwidth and pursues a smoother video playback effect. As a result, when the actual bandwidth is better, it dares not choose a higher video bit rate, which also leads to a decline in user viewing experience.
[0008] After searching, Chinese invention patent publication number CN113596021B discloses a streaming media bit rate adaptation method, device and equipment supporting neural network, the method comprising: obtaining historical network throughput measurement values of video blocks in the previous n time periods, a vector of preset available resolution size and current buffer occupancy information, inputting the historical network throughput measurement values, the vector of the resolution size and the current buffer occupancy information into a pre-constructed Bayesian neural network, and outputting the throughput prediction value of the next time period through the Bayesian neural network; obtaining the QoE information of the video block in the previous time period, constructing a model predictive control optimization model based on the throughput prediction value, the current buffer occupancy information and the QoE information with the preset QoE index optimization as the goal, solving the model predictive control optimization model to obtain the predicted download bit rate of the current video block; after executing the predicted download bit rate, obtaining a corresponding reward value based on the QoE index, and continuously training the Bayesian neural network based on the predicted download bit rate and the reward value; during the video playback process, adaptively obtaining the optimal bit rate of the downloaded video block based on the continuously trained Bayesian neural network and the model predictive control optimization model. This existing patent does not take bandwidth factors into consideration, and the model predictive control uses MPC, so there are problems of slow bit rate adaptation and insufficient accuracy.
[0009] How to achieve fast and accurate adaptation of video stream bit rate has become a technical problem that needs to be solved. Summary of the invention
[0010] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a method and device for video stream bit rate adaptation based on bandwidth prediction.
[0011] The purpose of the present invention can be achieved by the following technical solutions:
[0012] According to one aspect of the present invention, a method for video stream bit rate adaptation based on bandwidth prediction is provided. The method is implemented based on a BPA module deployed on a client, wherein the BPA module is an ABR module based on bandwidth prediction. The method comprises the following steps:
[0013] Step S1, the user requests video playback on the client;
[0014] Step S2, after receiving the request, the server obtains information related to video playback and sends it to the BPA module, wherein the information related to video playback includes the throughput of the past block, the download time of the past block, the size of the next video block, the buffer information, the number of remaining video blocks, the bit rate of the previous video block, and the bandwidth information;
[0015] Step S3, the BPA module predicts the bandwidth of the next video block according to the received bandwidth information;
[0016] Step S4: The BPA module adaptively selects a bit rate for playing the next video block based on the predicted bandwidth of the next video block obtained in step S3 and the information related to video playback in step S2, and sends the bit rate to the server for downloading the video block.
[0017] Preferably, the BPA module includes a BPM model and a BSM model connected in sequence; wherein the BPM model is used to predict the bandwidth information of the next video block according to the bandwidth information and send it to the BSM model; the BSM model adaptively selects the optimal bit rate for the next video block based on the predicted bandwidth of the next video block and the received information related to the video playback.
[0018] More preferably, the step S3 comprises:
[0019] Step S301, after receiving the bandwidth information, the BPM pre-processes the bandwidth information data using a normalization method;
[0020] Step S302, input the data preprocessed in step S301 into the BiLstm network of the BPM model, use the BiLstm network to output the bandwidth prediction value at the next moment, and send it to the BSM model, wherein the BPM model includes multiple BiLstm networks.
[0021] More preferably, the BiLstm network uses a mean square error function as a loss function;
[0022] The input of the BiLstm network is G i =[b i-k ,b i-k+1 ,...,b i ], indicating the bandwidth information data b from time ik to time i; the output is G i+1 , represents the bandwidth prediction value at time i+1.
[0023] Preferably, the step S4 comprises:
[0024] Step S401, inputting the bandwidth prediction value at the next moment into the BSM model, the BSM model combines the information related to the video playback in step S2, and the Actor network of the BSM model outputs the bit rate of the next video block;
[0025] Step S402, the Critic network of the BSM model outputs the value of the current state, and finds the optimal cumulative expected reward of the network by updating the network parameters, thereby finding the optimal bit rate action for the next video block;
[0026] The BSM model described therein is based on the Actor-Critic framework of reinforcement learning, including an Actor network and a Critic network;
[0027] More preferably, the input of the reinforcement learning is the state S t , in is the throughput of the past k video blocks; is the download time of the past k video chunks, indicating the time interval for throughput measurement; is the vector of the available size of the next video block; B t is the current buffer level; c t is the number of chunks remaining in the video; p t is the bit rate of downloading the last block; G t It indicates the bandwidth value predicted at the next time t+1 at time t;
[0028] The download time u of the next video chunk t+1 It is expressed as:
[0029]
[0030] Among them, u t Indicates the download time of the current video block, d t (R t ) represents the code rate R t The video block size at time Δu t Indicates the time from when the player sends a video request to the server to when the video encoding is completed; C t represents the real-time network throughput at time t, C n represents the real-time network throughput at time n, d n Represents the integral symbol in mathematics.
[0031] More preferably, the Critic network uses QoE to evaluate the quality of the algorithm, and QoE is expressed as:
[0032]
[0033] Among them, α is a non-negative weight coefficient; q(R n ) represents the video quality perceived by the user, i.e., the bit rate R n Function mapped to QoE; I t It represents the ratio of the predicted bandwidth value to the actual bandwidth value at time t;
[0034] Rebuffering time, which means when the download time Greater than the buffer capacity B max The sum of the rebuffering times that occur when ; ω is the weight coefficient; N is the number of video blocks.
[0035] More preferably, the Actor network uses the PPO-Clip algorithm to update the strategy parameter θ, and the loss function L of the PPO-Clip algorithm is PPO for:
[0036] L PPO =min((r t (S,a)·A t ),ratio t (S,a)A t )
[0037] Among them, A t represents the advantage function, which is the difference between the reward obtained by taking action a in state S and the expected value of all actions in that state; ratio t (S,a) represents the clipping function; r t (S,a) represents the probability ratio of executing action a under the new strategy and the old strategy.
[0038] More preferably, the objective function of the strategy parameter θ of the Actor network is:
[0039]
[0040] in, represents the average estimate of all QoE values at time step t; θ (S t ) indicates that in state S t Next strategy π θ The entropy of ; λ is a hyperparameter used to control the influence of the entropy term.
[0041] According to another aspect of the present invention, a video stream bit rate adaptation device based on bandwidth prediction is provided, the device includes a client and a server, the client deploys a BPA module, the BPA module includes a BPM model and a BSM model connected in sequence; the BPM model is used to predict the bandwidth information of the next video block according to the bandwidth information, and send it to the BSM model; the BSM model adaptively selects the optimal bit rate for the next video block based on the predicted bandwidth of the next video block and the received information related to video playback, wherein the information related to video playback includes the throughput of the past block, the download time of the past block, the size of the next video block, the buffer information, the number of remaining video blocks, the bit rate of the previous video block and the bandwidth information.
[0042] Compared with the prior art, the present invention has the following beneficial effects:
[0043] 1) The present invention embeds a BPA module on the client side, and firstly performs bandwidth prediction through a BPM model including multiple BiLstm networks, which is fast and accurate. Then, the BSM model based on the reinforcement learning Actor-Critic framework is input. The BSM model combines information related to video playback to adaptively select the optimal bit rate for QoE for the next video block, thereby realizing fast and accurate video stream bit rate adaptation.
[0044] 2) Compared with the traditional video stream bitrate selection algorithm, the algorithm of the present invention still works well in a variety of different network environments, thanks to the bandwidth prediction of the BPA module; in terms of running time, the BPA algorithm converges faster, obtains higher QoE, and has better user experience quality. In addition, compared with the current optimal pensieve algorithm, the BPA algorithm only needs 2 parallel agents to achieve the same performance as the pensieve algorithm with 16 parallel agents. BRIEF DESCRIPTION OF THE DRAWINGS
[0045] Figure 1 A schematic diagram of the flow of a method for video stream bit rate adaptation based on bandwidth prediction in the present invention;
[0046] Figure 2 It is a structural schematic diagram of the BPA module in the present invention;
[0047] Figure 3 It is a structural diagram of the BPM model in the present invention;
[0048] Figure 4 It is a structural schematic diagram of the BSM model in the present invention;
[0049] Figure 5 Schematic diagram of performance comparison between the BPA algorithm in the present invention and other algorithms. DETAILED DESCRIPTION
[0050] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.
[0051] This embodiment relates to a video stream bit rate adaptation method based on bandwidth prediction.
[0052] Aiming at the problem that the current ABR algorithm requires a lot of adjustments and cannot be generalized to different network conditions and QoE targets, a video stream bitrate adaptation algorithm based on bandwidth prediction - BPA (Bandwidth_Predict_ABR) is proposed.
[0053] The BPA algorithm consists of two models. The first part is the bandwidth prediction model (Bandwidth_Predict_Model: BPM), which consists of BiLstm (bidirectional long short-term memory network) and is used to output bandwidth prediction values. The other part is the bitrate selection model (Bitrate_Selection_Model: BSM), which consists of a reinforcement learning Actor-Critic framework and is trained using the PPO algorithm to output bitrate actions. The BPA algorithm is deployed on the client.
[0054] like Figure 1 ,The process of the video stream bit rate adaptation method based on bandwidth prediction includes the following steps:
[0055] Step S1, the user requests video playback on the client;
[0056] Step S2, after receiving the request, the server sends information related to video playback to the BPA module deployed on the client; the information related to video playback includes the throughput of the past block, the download time of the past block, the size of the next video block; buffer information; the number of remaining video blocks; the bit rate of the previous block and bandwidth information.
[0057] Step S3, the BPA module first sends the bandwidth information to the bandwidth prediction network model (BPM), and the BPM outputs the predicted bandwidth of the next block and sends it to the bit rate selection network model (BSM);
[0058] Step S4, the BSM model combines the previous bandwidth information, bit rate information, buffer information, etc., and adaptively selects the bit rate for playing the next video block, which is sent by the client to the server to download the video block.
[0059] Step S3 includes:
[0060] Step S301, after receiving the bandwidth information, the BPM model pre-processes the bandwidth information data using a normalization method;
[0061] Step S302, input the pre-processed data into the BiLstm network of the BPM model, use the BiLstm network to output the bandwidth prediction value at the next moment, and send it to the BSM model;
[0062] Step S4 includes:
[0063] Step S401, input the bandwidth prediction value of the next moment into the BSM, the BSM combines the previous bit rate and buffer information, and the Actor network of the BSM outputs the bit rate of the next block;
[0064] In step S402, the Critic network of the BSM outputs the value of the current state, and finds the optimal cumulative expected reward of the network by updating the network parameters, thereby finding the optimal bit rate action for the next video block.
[0065] For complex and changeable network environments, it is difficult for ordinary network bandwidth estimation methods to achieve high prediction accuracy. Therefore, with the rapid development of neural networks, a bandwidth prediction algorithm based on bidirectional long short-term memory network (BPM) is proposed, such as Figure 3 .
[0066] Bidirectional Long Short-Term Memory (BiLstm) is an improved Recurrent Neural Network (RNN) designed specifically for processing sequence data. BiLstm can capture bidirectional dependencies in a sequence by combining the outputs of the forward and backward LSTM networks.
[0067] To understand the bidirectional long short-term memory network, you first need to know the long short-term memory network. The long short-term memory network (LSTM) is a special RNN that can learn long-term dependencies. LSTM overcomes the gradient vanishing and gradient exploding problems of traditional RNN when processing long sequences by introducing gate mechanisms (input gate, forget gate, and output gate). BiLstm improves on it. BiLstm runs two independent LSTMs at each time step, one from the beginning to the end of the sequence (forward LSTM) and the other from the end to the beginning of the sequence (backward LSTM). The outputs of these two LSTMs are combined to consider the context information at the same time. Compared with traditional LSTM, BiLstm can take into account the context information of the sequence at the same time, which is very beneficial for tasks that require understanding of the global context (such as time series prediction, machine translation, etc.).
[0068] Bidirectional long short-term memory neural network structure:
[0069] Input: G i =[b i-k ,b i-k+1 ,...,b i ], represents the bandwidth data b during the period from time ik to time i;
[0070] Output: G i+1 , represents the bandwidth prediction value at time i+1;
[0071] Loss function: Use MSE (mean square error function) as the loss function; the formula is:
[0072]
[0073] Where L t (G) is the loss function, G t represents the true value at time t, Represents the predicted value at time t, and N represents the number of samples;
[0074] Experimental process:
[0075] For our task, we need to predict a feature value in the next n seconds based on the data in the previous t seconds. In the experiment, t=3 is used. Because we need to obtain the bandwidth value at the next moment, n is set to 1, that is, the data in the first three seconds is used to predict the data in the last one.
[0076] Dataset: In the experiment, we first use the Internet traffic dataset (in bits) provided by the Internet service provider for large-scale training in order to find the optimal model. The total traffic of the UK academic network backbone, the data collection interval is 5 minutes, and a total of 19,888 network bandwidth data. Then we use the network small batch network bandwidth dataset for experiments, which are slow network paths (bit rate ≤ 6Mbps), medium network paths (6Mbps < bit rate ≤ 100Mbps) and fast network paths (bit rate > 100Mbps).
[0077] For the data set, data preprocessing is first performed using a normalization method and two standardization methods are used as follows:
[0078]
[0079] Among them, X max and X min They represent the maximum and minimum values of sample X, μ and σ represent the mean and variance of sample X respectively;
[0080] like Figure 3 , the number of BiLstm layers is set to 1, 2, and 3, and the number of Dropout layers (also called random inactivation layers, a regularization method to prevent overfitting) is set to 1, 2, and 3. At the same time, in order to verify the impact of different neurons on the accuracy of the model, the number of neurons in each layer is set to 32, 64, and 128 respectively.
[0081] During the experiment, it was found that the best effect was achieved when the number of BiLstm layers was 3, the number of Dropout layers was 1, and the number of neurons in each layer was 128. At the same time, the StandardScaler normalization function was required. During the experiment, it was found that the MinMaxScaler function was difficult to capture the differentiated information between the data.
[0082] Ablation experiment: In order to verify the benefits of BiLstm, the Lstm network with the same network structure and data set was also used. By observing the experimental results, it was found that Lstm has difficulty capturing feature information in data sets with a small number of features, resulting in inaccurate prediction results. However, BiLstm uses a bidirectional network structure, so even when the number of features is small, the prediction results are still very good.
[0083] During the experiment, it was found that how to combine the bandwidth prediction model and the bit rate selection model is a challenge. In order to meet this challenge, we consider using a deep reinforcement learning algorithm and taking the predicted bandwidth value as the state value of the reinforcement learning model, thereby linking the bandwidth prediction result with the bit rate selection. The structure of the BPA module is as follows: Figure 2 shown.
[0084] Reinforcement Learning (RL) is a learning method in which an agent interacts with an environment to obtain the maximum cumulative reward. The agent observes the state S from the environment. t , and take action a t Give the environment, the environment gives the Agent reward r t , and transfer the state to state S t+1 The whole process follows the Markov decision process (MDP), and MDP uses tuples t , a t , r t > to describe. The definitions of various RL parameters are as follows:
[0085] Input (State S t ): After downloading each block t, the Agent enters the state Bring it to the BSM neural network, such as Figure 4 .in, is the network throughput measurement for the past k video chunks; is the download time of the past k video chunks, which represents the time interval for throughput measurement; is a vector of available sizes for the next video block; B t is the current buffer level; c t is the number of chunks remaining in the video; p t is the bit rate of downloading the previous block; G t It indicates predicting the bandwidth value at the next time t+1 at time t.
[0086] Output (Action a t ): After receiving the state, the reinforcement learning agent needs to choose the action a of the bitrate of the next video chunk according to the environment t The agent selects actions based on the strategy, which is defined as: θ {S t ,a t}→[0,1], where π θ {S t ,a t} means in state S t Take action at The probability of θ is an adjustable parameter, called the strategy parameter. It can be optimized by a neural network to automatically select a suitable value.
[0087] Reward (r t ): In the problem of bitrate adaptation, the linear formula of QoE is usually used to evaluate the quality of the algorithm. However, given the particularity of this performance, how to link the accuracy of bandwidth prediction with QoE is a challenge that needs to be solved. For the traditional linear formula of QoE, QoE is usually expressed as the user-perceived video quality q(R n ), the switching frequency between different video blocks |q(R n+1 -R n )| and rebuffer time T n The weighted values of the three components and the bandwidth prediction accuracy I are introduced t , which is expressed as the ratio of the predicted bandwidth value to the actual bandwidth value at time t. At the same time, the weight coefficient ω is also considered to judge whether the user pays attention to the prediction error. Therefore, QoE is expressed as:
[0088]
[0089] Among them, α is a non-negative weight coefficient; q(R n ) represents the video quality perceived by the user, i.e., the bit rate R n Function mapped to QoE; I t It represents the ratio of the predicted bandwidth value to the actual bandwidth value at time t;
[0090] Rebuffering time, which means when the download time Greater than the buffer capacity B max The sum of the rebuffering times that occur when ω is the weight coefficient.
[0091] Download time u of the next video chunk t+1 , expressed as:
[0092]
[0093] Among them, u t Indicates the download time of the current video block, d t (R t ) represents the code rate R t The video block size at time Δu t Indicates the time from when the player sends a video request to the server to when the video encoding is completed; C t represents the real-time network throughput at time t, C n represents the real-time network throughput at time n, d nRepresents the integral symbol in mathematics.
[0094] During video playback, the downloaded video blocks will be stored in the buffer, following the first-in-first-out principle. Buffer occupancy rate B t The dynamic change process of involves the video chunk being downloaded to the buffer and the video chunk being played in the buffer. When the video chunk n is downloaded, the buffer occupancy increases by L seconds and decreases as the user watches the video. Let B t It means that at time t, the player starts downloading video chunk n, and the buffer occupancy rate B t+1 The dynamic process is expressed as follows: Where L represents the duration of video block n.
[0095] Training algorithm: The actor-critic reinforcement learning network structure is adopted, and the improved algorithm PPO-Clip (PPO-Clipping) of the proximal policy optimization algorithm (PPO) is used to train the neural network. The specific training method is as follows:
[0096] 1) Update of actor network strategy parameters θ:
[0097] (1) Clipping Importance Sampling Ratio: In PPO-Clip, the importance sampling ratio is used to measure the relative performance between the new strategy and the old strategy. Specifically, for each sampled state-action pair (S, a), the probability ratio of executing action a under the new strategy and the old strategy is calculated: Here π θ (a|S) and Represent the probability of executing action a in state S under the new and old strategies respectively. Introducing the clipping function ratio t (S,a)=clip(r t (S,a),1-ε,1+ε), directly limits the amplitude of policy updates by applying a clipping function on the policy ratio. Specifically, the policy ratio is fixed within a range to prevent drastic fluctuations in learning performance due to overly aggressive updates. Here, ε is a hyperparameter, generally set to 0.1.
[0098] (2) PPO-Clip loss function: L PPO =min((r t (S,a)·A t ),ratio t (S,a)A t ), where A trepresents the advantage function, which is the difference between the reward obtained by taking action a in state S and the expected value of all actions in that state; ratio t (S,a) represents the clipping function; r t (S,a) represents the probability ratio of executing action a under the new strategy and the old strategy.
[0099] (3) Objective function: in, represents the average estimate of the QoE value in all possible cases at time step t, that is, by extracting a small batch of data samples from the data set, and then calculating the average QoE on this small batch of samples as the estimate; θ (S t ) indicates that in state S t Next strategy π θ entropy; λ is a hyperparameter used to control the influence of the entropy term, that is, the degree of exploration. Increasing the value of λ will give more weight to the entropy term and encourage more exploration behavior. In the experiment, λ is set to 0.1.
[0100] 2) Critic network parameters θ v Update: Use the mean squared error function (MSE) as the loss function, calculate the gradient through the loss function, and update the parameter θ v , in order to reduce the error of value prediction.
[0101] 3) Parallel training: To speed up the training of PPO-Clip, 2 agents and 2 different environments are used simultaneously, with 1 central agent, and these agents are trained in parallel.
[0102] The algorithms selected in this experiment are representative algorithms in the current video stream ABR field, namely:
[0103] BBA: A normal buffer based approach.
[0104] BOLA: A standard ABR solution for the ABR problem via Lyapunov functions.
[0105] RobustMPC (RMPC): A method that predicts the bitrate using a combination of buffer occupancy and network throughput using an applied cybernetics approach (Model Predictive Control, MPC), but takes into account the error between the predicted and observed throughput by normalizing the throughput estimate by the maximum error seen in the past 5 blocks.
[0106] Pensive: An ABR solution based on common DRL, using a pre-trained model with 16 proxies.
[0107] QUETRA is a DASH rate adaptation algorithm based on queuing theory.
[0108] Rate-based algorithms, such as the Festive algorithm, use the harmonic mean method to predict the bitrate based on the network throughput download of the past few video blocks, and limit the predicted bitrate to a certain range.
[0109] like Figure 5 As shown in the figure, it can be clearly seen that the performance of the BPA algorithm is ahead of all the algorithms compared in the experiment, and is also better than the Pensieve algorithm in terms of performance. Among them, CDF represents the cumulative distribution function.
[0110] At the same time, in order to verify the necessity of bandwidth prediction and the superiority of training time, when using the same PPO-Clip algorithm to train the Pensieve algorithm, it was found that when using the 16 agents recommended in the original Pensieve text, at least 150,000 iterations are required to obtain the maximum cumulative reward. However, for the BPA algorithm, as shown in Table 1, only 2 agents and 70,000 iterations are required to find the maximum reward value that is better than the Pensieve algorithm.
[0111] Table 1
[0112] Number of rounds Cumulative expected rewards 74400 0.944973 108300 0.926846 90900 0.924297 76800 0.922925
[0113] The BPA algorithm estimates the bandwidth value in x seconds based on the existing bandwidth, and achieves precise control of the video stream bit rate by improving the prediction accuracy of the BSM and BPM models. Experiments show that the algorithm is superior to the Pensieve algorithm and the training time is significantly shorter than the Pensieve algorithm.
[0114] This embodiment also relates to a device for video stream bit rate adaptation based on bandwidth prediction, the device includes a client and a server, the client is deployed with a BPA module, such as Figure 1 .
[0115] The BPA module includes a BPM model and a BSM model which are connected in sequence.
[0116] like Figure 3 ,The BPM model is used to predict the bandwidth information of the next video block based on the bandwidth information and sent to the BSM model. ,At time T = i, the network bandwidth time series features are input to the BiLstm layer. After passing the BiLstm layer, it will be regularized through the Dropout layer to prevent overfitting. ,The parameter in the Dropout layer is set to 0.1, even if 0.1 neurons fail. ,After two rounds, the prediction result is output by the Linear layer, where the linear layer is a linear layer containing 128 neurons.
[0117] like Figure 4 ,The BSM model adaptively selects the optimal bitrate for the next video block based on the ,predicted bandwidth of the next video block and the received ,information related to video playback, where the information related to video playback includes the ,throughput of past blocks, the download time of past blocks, the size of the next video block, the buffer ,information, the number of remaining video blocks, the bitrate of the previous video block, and ,bandwidth information.
[0118] The state of the BSM model includes the throughput measurement of the past block, the download time and the size of the next block, the bitrate selection of the previous block, the current buffer size, the number of remaining video blocks, and the predicted bandwidth value. The first layer of the actor network and the critic network contains 3 1D convolutional layers and 4 fully connected layers. The 3 1D convolutional layers and 4 fully connected layers each contain 128 neurons. In the actor network, the second layer is a fully connected layer, that is, it is merged first and then output to the next layer using the fully connected layer. In practical problems, the bitrate selection should be more accurate, so a separate fully connected layer is set in the actor network to downsample the bitrate action to obtain better bitrate action. The result is output through a fully connected layer in the critic network.
[0119] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present invention, and these modifications or replacements should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.
Claims
1. A video stream bit rate adaptation method based on bandwidth prediction, It is characterized in that The method is implemented based on a BPA module deployed on a client, wherein the BPA module is an ABR module based on bandwidth prediction, and the method comprises the following steps: Step S1, the user requests video playback on the client; Step S2, after receiving the request, the server obtains information related to video playback and sends it to the BPA module, wherein the information related to video playback includes the throughput of the past block, the download time of the past block, the size of the next video block, the buffer information, the number of remaining video blocks, the bit rate of the previous video block, and the bandwidth information; Step S3, the BPA module predicts the bandwidth of the next video block according to the received bandwidth information; Step S4: The BPA module adaptively selects a bit rate for playing the next video block based on the predicted bandwidth of the next video block obtained in step S3 and the information related to video playback in step S2, and sends the bit rate to the server for downloading the video block.
2. The method for video stream bit rate adaptation based on bandwidth prediction according to claim 1, It is characterized in that The BPA module includes a BPM model and a BSM model connected in sequence; wherein the BPM model is used to predict the bandwidth information of the next video block according to the bandwidth information and send it to the BSM model; the BSM model adaptively selects the optimal bit rate for the next video block based on the predicted bandwidth of the next video block and the received information related to video playback.
3. The method for video stream bit rate adaptation based on bandwidth prediction according to claim 2, It is characterized in that The step S3 comprises: Step S301, after receiving the bandwidth information, the BPM pre-processes the bandwidth information data using a normalization method; Step S302, input the data preprocessed in step S301 into the BiLstm network of the BPM model, use the BiLstm network to output the bandwidth prediction value at the next moment, and send it to the BSM model, wherein the BPM model includes multiple BiLstm networks.
4. The method for video stream bit rate adaptation based on bandwidth prediction according to claim 3, It is characterized in that The BiLstm network uses the mean square error function as the loss function; The input of the BiLstm network is G i =[b i-k ,b i-k+1 ,...,b i ], indicating the bandwidth information data b from time ik to time i; the output is G i+1 , represents the bandwidth prediction value at time i+1.
5. The method for video stream bit rate adaptation based on bandwidth prediction according to claim 2, It is characterized in that The step S4 comprises: Step S401, inputting the bandwidth prediction value at the next moment into the BSM model, the BSM model combines the information related to the video playback in step S2, and the Actor network of the BSM model outputs the bit rate of the next video block; Step S402, the Critic network of the BSM model outputs the value of the current state, and finds the optimal cumulative expected reward of the network by updating the network parameters, thereby finding the optimal bit rate action for the next video block; The BSM model described therein is based on the Actor-Critic framework of reinforcement learning, including an Actor network and a Critic network.
6. The method for video stream bit rate adaptation based on bandwidth prediction according to claim 5, It is characterized in that The input of the reinforcement learning is the state S t , in is the throughput of the past k video blocks; is the download time of the past k video chunks, indicating the time interval for throughput measurement; is the vector of the available size of the next video block; B t is the current buffer level; c t is the number of chunks remaining in the video; p t is the bit rate of downloading the last block; G t It indicates the bandwidth value predicted at the next time t+1 at time t; The download time u of the next video chunk t+1 It is expressed as: Among them, u t Indicates the download time of the current video block, d t (R t ) represents the code rate R t The video block size at time Δu t Indicates the time from when the player sends a video request to the server to when the video encoding is completed; C t represents the real-time network throughput at time t, C n represents the real-time network throughput at time n, d n Represents the integral symbol in mathematics.
7. The method for video stream bit rate adaptation based on bandwidth prediction according to claim 5, It is characterized in that The Critic network uses QoE to evaluate the quality of the algorithm. QoE is expressed as: Among them, α is a non-negative weight coefficient; q(R n ) represents the video quality perceived by the user, i.e., the bit rate R n Function mapped to QoE; I t It represents the ratio of the predicted bandwidth value to the actual bandwidth value at time t; Rebuffering time, which means when the download time Greater than the buffer capacity B max The sum of the rebuffering times that occur when ; ω is the weight coefficient; N is the number of video blocks.
8. The method for video stream bit rate adaptation based on bandwidth prediction according to claim 5, It is characterized in that The Actor network uses the PPO-Clip algorithm to update the strategy parameter θ. The loss function L of the PPO-Clip algorithm is PPO for: L PPO =min((r t (S,a)·A t ),ratio t (S,a)A t ) Among them, A t represents the advantage function, which is the difference between the reward obtained by taking action a in state S and the expected value of all actions in that state; ratio t (S,a) represents the clipping function; r t (S,a) represents the probability ratio of executing action a under the new strategy and the old strategy.
9. The method for video stream bit rate adaptation based on bandwidth prediction according to claim 5, It is characterized in that The objective function of the strategy parameter θ of the Actor network is: in, represents the average estimate of all QoE values at time step t; θ (S t ) indicates that in state S t Next strategy π θ The entropy of ; λ is a hyperparameter used to control the influence of the entropy term.
10. A device using the video stream bit rate adaptive method based on bandwidth prediction according to claim 1, the device comprising a client and a server, It is characterized in that The client deploys a BPA module, which includes a BPM model and a BSM model connected in sequence; the BPM model is used to predict the bandwidth information of the next video block according to the bandwidth information and send it to the BSM model; the BSM model adaptively selects the optimal bit rate for the next video block based on the predicted bandwidth of the next video block and the received information related to video playback, wherein the information related to video playback includes the throughput of the past block, the download time of the past block, the size of the next video block, the buffer information, the number of remaining video blocks, the bit rate of the previous video block and the bandwidth information.
Citation Information
Cited By
Bandwidth prediction method based on buffer duration and stable segment segmentation
CN119583903A
A bandwidth prediction method based on buffer duration and stable segment segmentation
CN119583903B
Self-adaptive bit rate adjusting method and device
CN119854589A
An adaptive bit rate adjustment method and apparatus
CN119854589B
