Self-adaptive code rate control method based on Bayesian condition variation

Abstract

The invention relates to the technical field of video transmission, in particular to a self-adaptive code rate control method based on Bayesian condition variation. Comprising the following steps: S1, building a video stream transmission system; s2, video block downloading time is calculated; s3, calculating the buffer area occupancy rate of the client; s4, taking information in the client as an input Si; s5, inputting Si into the ABR decision network, and outputting code rate selection of the next video block; s6, setting a user experience quality function; s7, using the Robust MPC and the input state Si to automatically generate an expert strategy; s8, calculating a loss function and updating parameters of the neural network; s9, updating the state of the player to obtain Si + 1, and inputting the Si + 1 into the ABR decision network; and S10, repeating the steps S5-S9 until the video ends. A Bayesian condition variation decision maker and an uncertainty bandwidth prediction module are designed, the randomness and exploration space in the training process are increased, and the generalization of the ABR algorithm is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of video transmission, and in particular to an adaptive bit rate control method based on Bayesian conditional variation. Background Art

[0002] like Figure 1 As shown in Figure 1, adaptive bitrate (ABR) control is a method of dynamically controlling a video player to download video segments with different bitrates and use them for the next segment. High-performance bitrate adaptation algorithms are designed to dynamically adjust video quality (bitrate) based on past measurements (such as throughput and buffer occupancy) to maximize the use of limited bandwidth and thus maximize the quality of experience (QoE).

[0003] Research on ABR algorithms has been going on for many years. Rate-Based (RB) algorithms select the appropriate bit rate by predicting network throughput. The bit rate requested by the user must be lower than or equal to the estimated throughput to ensure smooth playback. Buffer-Based (BB) algorithms select the bit rate based on the real-time occupancy of the client buffer, aiming to reduce the risk of playback interruption. The model predictive control (MPC) method uses the harmonic mean of past throughput to predict future network throughput and uses control theory to solve the decoding rate adaptation problem.

[0004] However, these fixed rules or model methods cannot use all available information of the operating environment to comprehensively optimize decisions. In recent years, with the development of deep learning and neural networks, many researchers have begun to use the theories of reinforcement learning and neural networks to solve the problem of bitrate adaptation. The neural network-based method uses deep reinforcement learning (such as Pensieve) and imitation learning (such as Comyco) and other technologies to feed all available information into the neural network. With the powerful nonlinear fitting ability of the neural network, the effect of the ABR algorithm is further improved, and the quality of user experience is significantly improved.

[0005] Imitation Learning (IL) avoids inefficient random exploration in reinforcement learning by using expert examples to directly guide policy learning, thereby significantly improving training efficiency. However, the expert demonstrations used during training contain future real bandwidth information, and methods based on imitation learning may over-rely on expert strategies, resulting in overfitting, which weakens their generalization ability in real-world deployments. When the network conditions in the test phase differ significantly from the bandwidth distribution observed during training, the performance of adaptive bitrate (ABR) methods based on imitation learning may drop significantly.

[0006] Existing work directly uses behavior cloning (BC) technology to minimize the distance between expert demonstrations and learning strategies to improve learning strategies. However, expert demonstrations may be suboptimal decisions and contain future bandwidth information, which causes the decision maker to overfit future network conditions. During the testing and deployment phases, for unseen network conditions, the decision maker may not make reasonable decisions based on the current state, resulting in a significant decrease in user experience quality. Summary of the invention

[0007] The present invention provides an adaptive bit rate control method based on Bayesian conditional variation, aiming to solve the technical problems existing in the existing ABR algorithm in the research of video playback user experience.

[0008] The present invention provides an adaptive bit rate control method based on Bayesian conditional variation, comprising the following steps:

[0009] S1. Building a video streaming system: On the server side, the source video is divided into M video blocks, each of which contains video content of a fixed length of L seconds and is stored on the server; the video blocks are further encoded into K versions with different bit rates. The bit rate version set is make represents the bit rate of the i-th video block, where i∈M;

[0010] S2. When the client sends a download request for the i-th video block to the server, its download time τ is calculated i ;

[0011] S3. After the i-th video block is downloaded, the buffer occupancy B on the client is calculated i ;

[0012] S4.ABR algorithm takes three types of information in the client as input S i = {P i ,C i ,V i}, including the past video playback information Pi = {B i ,r i-1}, past network status information C i ={c i ,τ i}, video block information V i ={n i ,l i}, where c i The network throughput for past video chunks measures the video chunk information, τ i is the download time of the past video chunk, n i is the available block size for the next block, l i is the number of remaining blocks;

[0013] S5. i Input to the ABR decision network Output the next video block bit rate selection a i ;

[0014] S6. Set the user experience quality QoE function;

[0015] S7. Using RobustMPC as an expert guide, input state S i Automatically generate expert strategies

[0016] S8. Calculate the loss function and update the parameters of the neural network;

[0017] S9. According to (S i ,a i ) Update the player status to get S i+1 , S i+1 As the new state input ABR decision network

[0018] S10. Repeat steps S5 to S9 until the video ends, completing a process of adaptive video bit rate transmission.

[0019] As a further improvement of the present invention, in step S2, after the client sends a download request for the i-th video block to the server, its download time τ i It is expressed as:

[0020]

[0021] Among them, n i (r i ) represents the code rate r i The data size of the i-th video block, t i represents the time when the i-th video chunk starts downloading, c tRepresents the downstream bandwidth during downloading.

[0022] As a further improvement of the present invention, in step S3, after the i-th video block is downloaded, the buffer occupancy rate B on the client is i It is expressed as:

[0023] B i =[B i-1 -τ i ] + +L,[x] + =max{0,x}

[0024] Among them, B i ∈[0,B max ], B max is the maximum capacity of the client buffer, τ i is the download time of the current video block, L is the time length of each video block; if B i-1 -τ i If the item is a negative number, it means that the next video block has not been completely downloaded at this time, there is no video block left in the buffer, and a rebuffering event has occurred.

[0025] As a further improvement of the present invention, step S5 specifically includes:

[0026] S51. i Input feature extraction module θ 1 , get the state feature x i :

[0027] S52. The network status information C i Input to the uncertainty bandwidth prediction module Φ, Φ is implemented based on Bayesian neural network, network parameters are modeled by probability, and bandwidth prediction values ​​are obtained by multiple sampling and uncertainty u i ;

[0028]

[0029] In the formula, n is the total number of samples, is the bandwidth prediction value obtained at the kth sampling time, is the bandwidth prediction uncertainty obtained at the kth sampling, Φ (k) is the network weight at the kth sampling;

[0030] S53. Calculate state feature x i The posterior distribution p(z|x i ); use an independent multivariate Gaussian distribution family to approximate p(z|x i ), the state feature x iand bandwidth prediction information as conditional variables Input to the encoder φ to get the mean z of the multivariate Gaussian distribution μ and variance z σ :

[0031]

[0032] S54. Randomly sample the latent variable z and reparameterize z:

[0033] z=z μ +ε×z σ ,ε~N(0,I), where ε is a random variable sampled from a standard normal distribution;

[0034] S55. Combine the latent variable z with the bandwidth prediction information Input to the bit rate decision module θ 2 , get the bit rate of the next video block and select a i :

[0035] As a further improvement of the present invention, in step S51, the feature extraction module θ 1 The structure includes convolutional layer, fully connected layer, concat layer, and the input (c i ,τ i ,n i ) respectively use one-dimensional convolutional layers, with 4 convolution kernels and 128 channels; for the input (r i ,b i ,l i ) respectively use linear fully connected layers, the number of channels is 128, and the activation function uses the relu function. Different features are concatenated through the concat layer to obtain the state feature x i .

[0036] As a further improvement of the present invention, in step S53, the encoder φ includes two neural networks φ 1 and φ 2 , respectively, using a fully connected layer, the number of channels is 128, and the activation function uses the relu function to obtain the mean z of the multivariate Gaussian distribution μ and variance z σ :

[0037]

[0038] As a further improvement of the present invention, in step S55, the bit rate decision module θ 2 It contains a fully connected layer and a softmax activation function, outputs the rate selection probability of K versions, and selects the rate version corresponding to the maximum probability as the action selection a of the next video block.i .

[0039] As a further improvement of the present invention, in step S6, a user experience quality QoE function is set:

[0040]

[0041] Among them, q(r i ) indicates that the bit rate of the i-th video block is r i The mass function of Indicates that higher bit rates help improve user experience quality; the second Indicates the penalty of video freeze on QoE, the third item The video quality switching penalty is calculated; α 1 and α 2 are the preference weights for rebuffering and change penalty, respectively.

[0042] As a further improvement of the present invention, in step S8, the loss function is calculated and the parameters of the neural network are updated:

[0043]

[0044] In the formula, For expert strategies, is the strategy probability output by the ABR decision network, p θ (z|x) is the posterior distribution of the probability variable z, Used to approximate the prior distribution of z; D KL (p θ (z|x)||q(z)) represents the KL divergence between the two, which is used to describe the distance between two probability distributions; β>0 is a hyperparameter that controls the degree of random exploration of the strategy.

[0045] The beneficial effects of the present invention are as follows: the present invention is an adaptive bit rate control method with enhanced generalization, which no longer directly minimizes the distance with expert demonstration, designs a Bayesian conditional variational decision maker and an uncertainty bandwidth prediction module, increases the randomness and exploration space in the training process, and uses the predicted bandwidth and its uncertainty as conditional variables to improve the generalization of the ABR algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 It is a framework diagram of a bit rate adaptation system based on deep reinforcement learning in the background technology;

[0047] Figure 2 It is a system overview diagram of the adaptive bit rate control method based on Bayesian conditional variation of the present invention;

[0048] Figure 3It is the overall network structure diagram of the bit rate adaptation method based on Bayesian conditional variation in the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0050] like Figure 2 As shown, an adaptive bit rate control method based on Bayesian conditional variation of the present invention comprises the following steps:

[0051] S1. Building a video streaming system: On the server side, the source video is divided into M video blocks, each of which contains video content of a fixed length of L seconds and is stored on the server; the video blocks are further encoded into K versions with different bit rates. The set of available bit rate versions is make represents the bit rate size of the i-th video block, where i∈M.

[0052] S2. When the client sends a download request for the i-th video block to the server, its download time τ is calculated i It can be expressed as:

[0053]

[0054] Among them, n i (r i ) represents the code rate r i The data size of the i-th video block, t i represents the time when the i-th video chunk starts downloading, c t Represents the downstream bandwidth during downloading.

[0055] S3. After the i-th video block is downloaded, the buffer occupancy B on the current client is calculated. i It can be expressed as:

[0056] B i =[B i-1 -τ i ] + +L,[x] + =max{0,x}

[0057] Among them, B i ∈[0,B max ], B max is the maximum capacity of the client buffer, τ i is the download time of the current video block, L is the time length of each video block; if B i-1 -τ iIf the item is a negative number, it means that the next video block has not been completely downloaded at this time, there are no video blocks left in the buffer, and a rebuffering event has occurred.

[0058] S4.ABR algorithm takes three types of information in the client as input S i = {P i ,C i ,V i}, including the past video playback information P i = {B i ,r i-1}; Past network status information C i ={c i ,τ i}, including the network throughput measurement video block information of the past k video blocks and the download time τ of the past k video chunks i =(τ i-k+1 ,...,τ i );V i ={n i ,l i}, where n i is the available block size for the next block, l i is the number of remaining blocks.

[0059] S5. i Input to the ABR decision network Output the next video block bit rate selection a i ;

[0060] like Figure 3 As shown, step S5 specifically includes:

[0061] S51. i Input feature extraction module θ 1 , get the state feature x i :

[0062] In step S51, the feature extraction module θ 1 The structure includes convolutional layer, fully connected layer, and concat layer. Specifically, for the input (c i ,τ i ,n i ) respectively use one-dimensional convolutional layers, with 4 convolution kernels and 128 channels; for the input (r i ,b i ,l i ) respectively use a linear fully connected layer with 128 channels and the activation function uses the relu function. Then different features are concatenated through the concat layer to obtain the state feature x i .

[0063] S52. The network status information C i Input to the uncertainty bandwidth prediction module Φ, Φ is implemented based on Bayesian neural network, network parameters are modeled by probability, and bandwidth prediction values ​​are obtained by multiple sampling and uncertainty u i :

[0064]

[0065] In the formula, n is the total number of samples, is the bandwidth prediction value obtained at the kth sampling time, is the bandwidth prediction uncertainty obtained at the kth sampling, Φ (k) is the network weight at the kth sampling.

[0066] S53. Calculate state feature x i The posterior distribution p(z|x i ); Since p(z|x i ) is difficult to calculate directly, so we use the Bayesian variational inference method to approximate the distribution. Specifically, we use an independent multivariate Gaussian distribution family to approximate p(z|x i ), the state feature x i and bandwidth prediction information as conditional variables Input to the encoder φ to get the mean z of the multivariate Gaussian distribution μ and variance z σ :

[0067]

[0068] In step S53, the encoder φ includes two neural networks φ 1 and φ 2 , which are used to calculate the mean and variance of the above multivariate Gaussian distribution. 1 and φ 2 A fully connected layer is used, the number of channels is 128, and the activation function is the relu function to obtain the mean z of the multivariate Gaussian distribution μ and variance z σ :

[0069]

[0070] S54. Randomly sample the latent variable z. To facilitate the calculation of the neural network gradient, reparameterize z: z = z μ +ε×z σ ,ε~N(0,I). Among them, ε is a random variable sampled from a standard normal distribution.

[0071] S55. Combine the latent variable z with the bandwidth prediction information Input to the bit rate decision module θ 2 , get the bit rate of the next video block and select a i :

[0072] In step S55, the code rate decision module θ 2 It contains a fully connected layer and a softmax activation function, outputs the rate selection probability of K versions, and selects the rate version corresponding to the maximum probability as the action selection a of the next video block. i .

[0073] S6. Set the user experience quality QoE function:

[0074]

[0075] Among them, q(r i ) indicates that the bit rate of the i-th video block is r i The mass function of Indicates that higher bit rates help improve user experience quality. The second Indicates the penalty of video freeze on QoE, the third item The video quality switching penalty is calculated; α 1 and α 2 are the preference weights for rebuffering and change penalty, respectively.

[0076] S7. Using RobustMPC as an expert guide, input state S i , automatically generates expert strategies that maximize the user experience quality QoE function

[0077] S8. Calculate the loss function and update the parameters of the neural network:

[0078]

[0079] In the formula, For expert strategies, is the strategy probability output by the ABR decision network, p θ (z|x) is the posterior distribution of the probability variable z, Used to approximate the prior distribution of z; D KL (p θ (z|x)||q(z)) represents the KL divergence between the two, which is used to describe the distance between two probability distributions; β>0 is a hyperparameter that controls the degree of random exploration of the strategy.

[0080] S9. Repeat steps S2 to S4 to calculate the new player state S i+1 , Si+1 As the new state input ABR decision network

[0081] S10. Repeat steps S5 to S9 until the video ends, completing a process of adaptive video bit rate transmission.

[0082] The adaptive bit rate control method based on Bayesian conditional variation of the present invention has the following advantages:

[0083] (1) Uncertainty is introduced into the bandwidth prediction module to increase the robustness of the prediction and reduce rebuffering events;

[0084] (2) Randomly sample the potential probability variables of state features to increase the randomness of the training process and the exploration space of the strategy, thereby reducing overfitting;

[0085] (3) The uncertain bandwidth prediction results are combined with variational methods as conditional variables to control the ABR decision maker to explore strategies under different network dynamics, thereby improving the algorithm's adaptability to different network dynamics and enhancing its generalization ability.

[0086] The above contents are further detailed descriptions of the present invention in combination with specific preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, several simple deductions or substitutions can be made without departing from the concept of the present invention, which should be regarded as falling within the protection scope of the present invention.

Claims

1. An adaptive rate control method based on Bayesian conditional variation, characterized in that: The following steps are involved: S1. Building a video streaming system: On the server side, the source video is divided into M video blocks, each of which contains video content of a fixed length of L seconds and is stored on the server; the video blocks are further encoded into K versions with different bit rates. The bit rate version set is make represents the bit rate of the i-th video block, where i∈M; S2. When the client sends a download request for the i-th video block to the server, its download time τ is calculated i ; S3. After the i-th video block is downloaded, the buffer occupancy B on the client is calculated i ; S4.ABR algorithm takes three types of information in the client as input S i = {P i ,C i ,V i }, including the past video playback information P i = {B i ,r i-1 }, past network status information C i ={c i ,τ i }, video block information V i ={n i ,l i }, where c i The network throughput for past video chunks measures the video chunk information, τ i is the download time of the past video chunk, n i is the available block size for the next block, l i is the number of remaining blocks; S5. i Input to the ABR decision network Output the next video block bit rate selection a i ; S6. Set the user experience quality QoE function; S7. Using RobustMPC as an expert guide, input state S i Automatically generate expert strategies S8. Calculate the loss function and update the parameters of the neural network; S9. According to (S i ,a i ) Update the player status to get S i+1 , S i+1 As the new state input ABR decision network S10. Repeat steps S5 to S9 until the video ends, completing a process of adaptive video bit rate transmission.

2. The adaptive bit rate control method based on Bayesian conditional variation according to claim 1, characterized in that: In step S2, after the client sends a download request for the i-th video block to the server, its download time τ i It is expressed as: Among them, n i (r i ) represents the code rate r i The data size of the i-th video block, t i represents the time when the i-th video chunk starts downloading, c t Represents the downstream bandwidth during downloading.

3. The adaptive bit rate control method based on Bayesian conditional variation according to claim 1, characterized in that: In step S3, after the i-th video block is downloaded, the buffer occupancy rate B on the client is i It is expressed as: B i =[B i-1 -τ i ] + +L,[x] + =max{0,x} Among them, B i ∈[0,B max ], B max is the maximum capacity of the client buffer, τ i is the download time of the current video block, L is the time length of each video block; if B i-1 -τ i If the item is a negative number, it means that the next video block has not been completely downloaded at this time, there is no video block left in the buffer, and a rebuffering event has occurred.

4. The adaptive bit rate control method based on Bayesian conditional variation according to claim 1, characterized in that: The step S5 specifically includes: S51. i Input feature extraction module θ1 to obtain state feature x i : S52. The network status information C i Input to the uncertainty bandwidth prediction module Φ, Φ is implemented based on Bayesian neural network, network parameters are modeled by probability, and bandwidth prediction values ​​are obtained by multiple sampling and uncertainty u i ; In the formula, n is the total number of samples, The bandwidth prediction value is obtained at the kth sampling time, is the bandwidth prediction uncertainty obtained at the kth sampling, Φ (k) is the network weight at the kth sampling; S53. Calculate state feature x i The posterior distribution p(z|x i ); use an independent multivariate Gaussian distribution family to approximate p(z|x i ), the state feature x i and bandwidth prediction information as conditional variables Input to the encoder φ to get the mean z of the multivariate Gaussian distribution μ and variance z σ : S54. Randomly sample the latent variable z and reparameterize z: z=z μ +ε×z σ ,ε~N(0,I), where ε is a random variable sampled from a standard normal distribution; S55. Combine the latent variable z with the bandwidth prediction information Input to the bit rate decision module θ2 to obtain the bit rate selection a for the next video block i :

5. The adaptive rate control method based on Bayesian conditional variation according to claim 4, characterized in that: In step S51, the structure of the feature extraction module θ1 includes a convolution layer, a fully connected layer, and a concat layer. i ,τ i ,n i ) respectively use one-dimensional convolutional layers, with 4 convolution kernels and 128 channels; for the input (r i ,b i ,l i ) respectively use linear fully connected layers, the number of channels is 128, and the activation function uses the relu function. Different features are concatenated through the concat layer to obtain the state feature x i .

6. The adaptive rate control method based on Bayesian conditional variation according to claim 4, characterized in that: In step S53, the encoder φ includes two neural networks φ1 and φ2, which are used to calculate the mean and variance of the multivariate Gaussian distribution. φ1 and φ2 each use a fully connected layer with 128 channels and the activation function uses the relu function to obtain the mean z of the multivariate Gaussian distribution. μ and variance z σ :

7. The adaptive rate control method based on Bayesian conditional variation according to claim 4, characterized in that: In step S55, the bit rate decision module θ2 includes a fully connected layer and a softmax activation function, outputs K versions of bit rate selection probabilities, and selects the bit rate version corresponding to the maximum probability as the action selection a for the next video block. i .

8. The adaptive rate control method based on Bayesian conditional variation according to claim 1, characterized in that: In step S6, the user experience quality QoE function is set: Among them, q(r i ) indicates that the bit rate of the i-th video block is r i The mass function of Indicates that higher bit rates help improve user experience quality. The second Indicates the penalty of video freeze on QoE, the third item The video quality switching penalty is calculated; α1 and α2 are the preference weights of rebuffering and change penalties, respectively.

9. The adaptive rate control method based on Bayesian conditional variation according to claim 1, characterized in that: In step S8, the loss function is calculated and the parameters of the neural network are updated: In the formula, For expert strategies, is the strategy probability output by the ABR decision network, p θ (z|x) is the posterior distribution of the probability variable z, Used to approximate the prior distribution of z; D KL (p θ (z|x)||q(z)) represents the KL divergence between the two, which is used to describe the distance between two probability distributions; β>0 is a hyperparameter that controls the degree of random exploration of the strategy.

Images