VVC code rate control algorithm based on deep reinforcement learning

Abstract

The invention discloses a VVC code rate control algorithm based on deep reinforcement learning, and the algorithm comprises the following steps: importing a video sequence into an encoder, and enabling the video sequence to enter initial frame coding; after the encoder completes the default encoding of the first two frames, the subsequent frame prediction firstly extracts the encoding state information of the previous prediction frame, and a greedy strategy is adopted to perform action selection; an overall reward value is obtained through CTU-level code rate control and an actual coding process in sequence; observing the next state and the last reward, performing TD iteration on the Q value, and adding the Q value into the Q value network after TD iteration; setting an experience playback pool; after the capacity of the experience playback pool reaches a threshold value, randomly sampling from the experience playback pool in batches; resetting the Q value network; and carrying out coding test based on the Q value network obtained by training. The VVC code rate control algorithm based on deep reinforcement learning has a good capability of guiding code rate control coding, and compared with standard code rate control in VTM13.0, the VVC code rate control algorithm based on deep reinforcement learning can bring high rate distortion performance and improve code control precision.

Description

Technical Field

[0001] This invention belongs to the field of video coding technology, specifically relating to a VVC bitrate control algorithm based on deep reinforcement learning. Background Technology

[0002] With the rapid development of the information age, the number of video-related services such as short videos and video conferencing has been steadily increasing, with short video platforms like Douyin, Kuaishou, and Huoshan Video experiencing rapid growth. Related reports indicate that mobile network traffic has doubled in the past two years since the first quarter of 2020. In 2021, approximately 80% of all videos created were user-generated; in 2022, video resources accounted for approximately 82% of all IP traffic. This explosive growth in video data has brought enormous challenges to its storage and transmission management. For example, if a video has a resolution of 3840×2160, a sampling format of 4:2:0, a frame rate of 60 frames per second, and a pixel depth of 10 bits, then the amount of video data that needs to be transmitted per second is 3840×2160×3×8×60=1.5×10¹⁰ bits. Because uncompressed video is too large, it is unsuitable for storage and transmission as digital media, thus giving rise to video encoding and decoding technologies. In recent years, to achieve high-quality, high-resolution video storage and transmission, the Video Coding Experts Group (VCEG) of the International Telecommunication Union Telecommunication Standardization Sector (ITU-T) and the Moving Picture Experts Group (MPEG) of the International Organization for Standardization (ISO) / International Electrotechnical Commission (IEC) have released a series of international video coding standards: from the previous Advanced Video Coding (AVC) and High Efficiency Video Coding (HEVC) standards to the latest video coding standard, VVC. In addition, MPEG has also released its own MPEG-5 standard, and Google and the Chinese Video Coding Experts Group have developed the standards AV1 and AVS3. Supported resolutions range from Standard Definition (SD), High Definition (HD) to Ultra High Definition (UHD) and even 8K Ultra High Definition (FUHD). After decades of development, these standards have been repeatedly proven to improve coding efficiency.

[0003] Video typically consists of two parts: information and redundancy. Redundancy manifests as identical or similar repetitive information. Redundancy can be mainly categorized into spatial redundancy, temporal redundancy, structural redundancy, and knowledge redundancy. Video compression technology achieves this by removing visual and spatial redundancy. Data compression can be broadly classified into lossy compression and lossless compression. Lossless compression means that the compressed data can still be reconstructed and recovered from its original state; however, its compression ratio is generally low. Lossy compression, as the name suggests, loses some data during compression. Video images often contain a large amount of high-frequency information, which the human eye is not sensitive to. Therefore, within the acceptable range for the human eye, video compression typically uses lossy compression algorithms. The performance of these algorithms is closely related to both bitrate and distortion parameters.

[0004] In actual network transmission, the transmission bitrate is affected by differences in communication bandwidth across different networks, resulting in variations in video quality. Therefore, how to reasonably control the bitrate under limited network conditions to provide users with the clearest video becomes an unavoidable issue. If bandwidth is large, high bitrate video should be transmitted as much as possible; conversely, the compression ratio should be increased to reduce the bitstream. Otherwise, buffer overflow and underflow will occur, causing data loss and distortion. Bitrate control in video encoding technology aims to obtain the highest possible quality decoded image within a given bandwidth constraint. Bitrate control plays a crucial role in many video scenarios, such as online video-on-demand, live streaming, and real-time communication. Taking online video and live streaming as examples, due to the constant user bandwidth and limited buffered data, some instantaneous high bitrates can cause stuttering. Many large-scale live video streams are distributed to users through Content Delivery Networks (CDNs), which charge based on traffic. Inappropriate bitrate allocation can also lead to uncontrollable bandwidth costs. For each different business scenario, the various limitations on bitrate will also differ. Bitrate control aims to optimize a scene by controlling each frame or basic coding unit while ensuring encoding efficiency and video quality.

[0005] Due to the unique configuration of LD (Local Frame) in video coding, all frames other than I-frames are either B-frames or P-frames. Considering the rate-distortion dependency between frames (excluding I-frames), there is also a dependency between the QP (Queries Per Frame) of each frame. The VVC standard has significantly modified inter-frame prediction techniques (adding extended Merge mode, affine motion compensation prediction, symmetric MVD mode, and other modes). Therefore, the goal of inter-frame rate control optimization is to reasonably handle these dependencies to improve bitrate control performance.

[0006] Inter-frame prediction is primarily used to remove temporal dependencies in images. The process involves predicting reference blocks from neighboring already encoded images based on the block to be encoded in the current image. It is mainly used in P-frames and B-frames. Since temporal redundancy is often much greater than spatial redundancy, inter-frame prediction is more efficient than intra-frame prediction. The inter-frame prediction algorithm obtains motion information of each block in the current image within the reference image by using already encoded images as reference images.

[0007] Rate control algorithms based on the R-λ model are the algorithms currently used in the VVC standard reference software and are also the mainstream algorithms currently being researched. The R-λ model specifically refers to the R-λ-QP model. These algorithms cannot adaptively acquire dynamic information about video coding to dynamically select the optimal control parameters, resulting in poor coding control capabilities. Their rate-distortion performance and rate control accuracy need further improvement.

[0008] Therefore, this invention utilizes the dynamic programming characteristics of reinforcement learning to propose a VVC rate control algorithm based on deep reinforcement learning. Summary of the Invention

[0009] In view of this, the purpose of this invention is to provide a VVC rate control algorithm based on deep reinforcement learning. This invention aims to solve the problems of poor rate-distortion performance and poor adaptive rate control coding capability of existing methods.

[0010] To achieve the above objectives, this invention provides a VVC rate control algorithm based on deep reinforcement learning, comprising the following steps:

[0011] S1. Perform encoding training based on the DQN algorithm to obtain the Q-value network;

[0012] S1.1 imports the video sequence into the encoder, causing the video sequence to enter the initial frame encoding;

[0013] After the S1.2 encoder finishes encoding the first two frames by default, the subsequent video frame prediction first extracts the encoding state information of the previous prediction frame and uses an ε-greedy strategy for action selection.

[0014] After the S1.3 action selection is completed, it goes through CTU-level bitrate control and actual encoding process in sequence to obtain the overall reward value, i.e., Q value;

[0015] S1.4 Observe the next state and the previous reward, perform TD iteration on the Q value, and add it to the Q value network after TD iteration;

[0016] S1.5 Experience Replay: Set up an experience replay pool to save actions, reward values, previous frame state information and current frame state information;

[0017] S1.6 Once the data cached in the experience replay pool reaches the capacity threshold, random batches are sampled from the experience replay pool.

[0018] S1.7 Reset the Q-value network: Based on the sampling information in step S1.6, the parameters of the Q-value network are updated once each time the experience replay pool acquires data, so that the Q-value network is close to the regression value.

[0019] S2. Encode and test based on the Q-value network obtained from training;

[0020] S2.1 imports the video sequence into the encoder, causing the video sequence to enter the initial frame encoding;

[0021] S2.2 Extract the coding state information of the current prediction frame, query the Q-value network based on the state information, and the Q-value network gives the optimal action;

[0022] After determining the preferred QP value in S2.3, proceed with subsequent operations according to the coding process.

[0023] Furthermore, the state information includes: the MSE value of the Y component of the current frame, the MSE value of the U component of the current frame, the MSE value of the V component of the current frame, the QP value of the current frame, the PSNR value of the current frame, the SSIM value of the current frame, the target number of bits of the current frame, the actual number of bits encoded in the current frame, the number of bits remaining after actual encoding in the current frame, the number of frames actually encoded, the number of frames remaining to be encoded, the average gradient of the current frame, the information entropy of the current frame, and the current buffer size.

[0024] Furthermore, the calculation formula for the ε-greedy strategy is as follows:

[0025]

[0026] In the formula, 1-ε represents the probability that the encoder selects a large Q value; ε represents the probability that the encoder performs a random selection action.

[0027] Furthermore, the ε-greedy strategy is equipped with a decay factor, which enables ε to decay as the number of iterations increases.

[0028] Furthermore, the calculation formula for the TD iteration is as follows:

[0029] Q(s t ,a t )←Q(s t ,a t )+α[r t+1 +γmaxQ(s t+1 ,a)-Q 估 (s t ,a t )]

[0030] st ←s t+1

[0031] In the formula, s t Represents the current state; s t+1 Represents the next state; a t α represents the current action; 'a' represents the selected action; α represents the learning rate; r t+1 +γmaxQ(s t+1 a) represents the actual value; r t+1 Represents reward; γ represents discount factor; Q 估 (s t ,a t () represents the estimated value.

[0032] Furthermore, the reward is calculated using a reward function, which is defined as follows:

[0033]

[0034] In the formula, PSNR cur PSNR represents the PSNR value of the current frame; PSNR last Represents the PSNR value of the previous frame; buffer cur Represents the size of the buffer for the previous frame; buffer last This represents the size of the current frame buffer.

[0035] The beneficial effects of this invention are as follows:

[0036] This invention proposes a VVC rate control algorithm based on deep reinforcement learning. It is an improvement on the Q-learning algorithm, incorporating a neural network and an empirical buffer to address the poor generalization ability of Q-learning. Inter-frame coding experiments based on DQN demonstrate that, considering the rate-distortion relationship between frames, the powerful decision-making ability of deep reinforcement learning replaces the traditional R-λ model in selecting the QP values ​​for all subsequent predicted frames. Experiments show that deep reinforcement learning DQN has better guidance capabilities for rate control coding, delivering better rate-distortion performance and higher code control accuracy compared to the standard rate control in VTM13.0.

[0037] Other advantages, objectives, and features of the invention will be set forth in part in the description which follows, and in part will be apparent to those skilled in the art from the following examination, or may be learned from practice of the invention. The objectives and other advantages of the invention can be realized and obtained through the following description. Attached Figure Description

[0038] Figure 1 This describes the training process for deep reinforcement learning based on DQN.

[0039] Figure 2 This is the testing process for deep reinforcement learning based on DQN;

[0040] Figure 3 This is a diagram of the internal structure of the ANN network in DQN;

[0041] Figure 4 The actual bit allocation for each frame of BQSquare (QP=22);

[0042] Figure 5 for Figure 4 A magnified view of a portion of the image;

[0043] Figure 6 The actual bit allocation for each frame of BasketballDrill (QP=27);

[0044] Figure 7 for Figure 6 A magnified view of a portion of the image;

[0045] Figure 8 A comparison of the subjective quality of Cactus video sequences encoded using different methods. Detailed Implementation

[0046] To make the technical solutions, advantages, and objectives of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the described embodiments of the present invention without creative effort are within the protection scope of this application.

[0047] like Figure 1 and Figure 2 As shown, this invention provides a VVC rate control algorithm based on deep reinforcement learning, comprising the following steps:

[0048] S1. Perform encoding training based on the DQN algorithm to obtain the Q-value network;

[0049] Similar to Q-learning, DQN is also a value-based heterogeneous policy TD (True / False) algorithm. DQN doesn't use a single neural network, but rather two networks with identical structures but different parameters: a Q-estimation network and a Q-reality network, replacing the actual value and the estimated value, respectively. For learning stability, the weights of the Q-estimation network remain unchanged for a period of time; instead, the updated weights of the Q-reality network are copied to the Q-estimation network only after a certain period.

[0050] S1.1 imports the video sequence into the encoder, causing the video sequence to enter the initial frame encoding;

[0051] After the S1.2 encoder finishes encoding the first two frames by default, the subsequent video frame prediction first extracts the encoding state information of the previous prediction frame and uses an ε-greedy strategy to select actions, that is, selects the change in QP value of the previous frame.

[0052] The status information includes: the MSE value of the Y component of the current frame, the MSE value of the U component of the current frame, the MSE value of the V component of the current frame, the QP value of the current frame, the PSNR value of the current frame, the SSIM value of the current frame, the target number of bits of the current frame, the actual number of bits encoded in the current frame, the number of bits remaining after actual encoding in the current frame, the number of frames actually encoded, the number of frames remaining to be encoded, the average gradient of the current frame, the information entropy of the current frame, and the current buffer size;

[0053] The formula for calculating the ε-greedy strategy is as follows:

[0054]

[0055] In the formula, 1-ε represents the probability that the encoder selects a large Q value; ε represents the probability that the encoder performs a random selection action.

[0056] The ε-greedy strategy has a decay factor that causes ε to decrease as the number of iterations increases.

[0057] Here, the action is defined as the change in the previous predicted frame, ranging from [-2, +2], in order to balance the fluctuations between consecutive frames. In the Low Delay P (LDP) configuration, the QP value of the current frame is determined by adding the action to the previous frame and then used for encoding.

[0058] After the S1.3 action selection is completed, it goes through CTU-level bitrate control and actual encoding process in sequence to obtain the overall reward value, i.e., Q value;

[0059] The reward is calculated using a reward function, which is defined as follows:

[0060]

[0061] In the formula, PSNR cur PSNR represents the PSNR value of the current frame; PSNR last Represents the PSNR value of the previous frame; buffer cur Represents the size of the buffer for the previous frame; buffer last This represents the size of the current frame buffer.

[0062] Therefore, the criteria for the current frame to be better than the previous frame are: the PSNR value is higher than the previous frame and the buffer change is as small as possible.

[0063] S1.4 Observe the next state and the previous reward, perform TD iteration on the Q value, and add it to the Q value network after TD iteration;

[0064] Before exploring the environment, the Q-value network needs to be initialized. During the interaction between the encoder and the encoding environment, an ε-greedy strategy is first used to select action 'a', and the resulting reward 'r' is observed. Then, each time, an error estimate is made based on the actual value and the estimated value; this is one TD update. This process continues until all rounds are completed. Because each TD update occurs in the corresponding Q-value network, the final path information that maximizes the reward is saved through the remaining Q-value network.

[0065] The formula for calculating the TD iteration is as follows:

[0066] Q(s t ,a t )←Q(s t ,a t )+α[r t+1 +γmaxQ(s t+1 ,a)-Q 估 (s t ,a t )]

[0067] s t ←s t+1

[0068] In the formula, s t Represents the current state; s t+1 Represents the next state; a t α represents the current action; 'a' represents the selected action; α represents the learning rate; r t+1 +γmaxQ(s t+1 a) represents the actual value; r t+1 Represents reward; γ represents discount factor; Q 估 (s t ,a t () represents the estimated value.

[0069] S1.5 Experience Replay: Set up an experience replay pool to save actions, reward values, previous frame state information and current frame state information;

[0070] The experience replay pool is a buffer with a limited capacity. When the capacity is almost full, it will automatically delete the earliest experience that entered the buffer.

[0071] Experience replay pools allow us to learn from past strategies, reducing the number of interactions with the environment to some extent. On the other hand, the more diverse a batch of data is, the better it is for learning, thus improving iterative performance.

[0072] S1.6 Once the data cached in the experience replay pool reaches the capacity threshold, random batches are sampled from the experience replay pool.

[0073] S1.7 Reset the Q-value network: Based on the sampling information in step S1.6, the experience replay pool acquires data every C times and updates the parameters of the Q-value network once to make the Q-value network closer to the regression value.

[0074] S2. Encode and test based on the Q-value network obtained from training;

[0075] S2.1 imports the video sequence into the encoder, causing the video sequence to enter the initial frame encoding;

[0076] S2.2 Extract the coding state information of the current prediction frame, query the Q-value network based on the state information, and the Q-value network gives the optimal action;

[0077] After determining the preferred QP value in S2.3, proceed with subsequent operations according to the coding process.

[0078] Numerical Experiment

[0079] This embodiment uses the low-latency P configuration from the JVET common test condition (CTC). All test sequences are shown in Table 1.

[0080] Table 1 CTC Standard Test Sequence

[0081]

[0082] In the general test conditions, there are four test conditions: intra-frame, random access, low-latency B-frame, and low-latency P-frame. Since test sequences of classes A1 and A2 do not support LD type encoding, the video sequences in this experiment mainly consist of classes B, C, D, E, and F. A set of experiments using the same video sequence is tested with four QPs values ​​(22, 27, 32, 37). The corresponding bitrate control parameters in the specific encoding files are shown in Table 2.

[0083] Table 2 Bitrate Control Parameter Configuration

[0084]

[0085]

[0086] The VVC encoder version used in this experiment is VTM13.0. Experimental environment: CPU: AMD 5800X, GPU: NVIDIA RTX3070. DQN hyperparameters: state dimension 14, action dimension 7, discount factor 0.95, initial epsilon 0.95, final epsilon 0.01 in the e-greedy policy, decay rate 300, learning rate 0.001, experience buffer size 100,000, mini-batch size 32, two hidden layer neurons in the ANN network with 128 and 64 neurons respectively, ReLU activation function, target network update frequency 100, and MSE used to calculate the loss. The internal structure of the ANN network in DQN is as follows. Figure 3 As shown.

[0087] This example compares the standard inter-frame rate control algorithm in VTM software, which uses the R-λ model for QP decision-making, while the example experiment uses a deep reinforcement learning model for QP decision-making. The specific results are analyzed below:

[0088] (1) Rate-distortion performance

[0089] As shown in Table 3, the deep reinforcement learning algorithm proposed in this invention achieves an average BD-rate performance improvement of -0.445 and a BD-SSIM performance improvement of -1.234 compared to the traditional R-λ model rate control algorithm in VTM13.0. The algorithm of this invention improves both objective and subjective video quality.

[0090] Table 3 Comparison of Rate-Distortion Performance

[0091]

[0092]

[0093] (1) Bit allocation precision

[0094] The actual bitrate and the target bitrate are often not equal; perfect equality is the ideal situation for the algorithm. In bitrate control, higher bitrate accuracy indicates better algorithm performance. Bitrate accuracy (BRA) is typically calculated using the following formula:

[0095]

[0096] In the formula, R target R represents the target bitrate value. actalThe values ​​represent the actual bit rate, and the percentage difference between the two represents the bit rate error, i.e., the bit rate error (BRE). The sum of BRA and BRE is 1. Table 4 compares the bit rate control accuracy between the algorithm of this invention and the VTM bit rate control algorithm. It can be seen that the average bit allocation accuracy of the algorithm of this invention is slightly higher than that of the VTM13.0 algorithm across all sequences, with the most significant improvement for class F sequences.

[0097] Table 4 Comparison of Code Control Accuracy

[0098]

[0099]

[0100] (3) Fluctuation in each frame

[0101] like Figure 4-7 As shown, this invention compares the bitrate allocation of the deep reinforcement learning method and the VTM13.0 standard bitrate control method in the first 200 frames of the sequence. Since the given target bitrate allocation method remains unchanged, the actual bit rate variation trend remains consistent. However, careful observation reveals that the algorithm proposed in this invention mitigates some bitrate fluctuations.

[0102] (4) Time complexity comparison

[0103] Time complexity is a crucial factor to consider for almost all algorithms. A well-optimized algorithm will inevitably have low time overhead. Time complexity is primarily calculated using the following formulas:

[0104]

[0105] The time complexity of the deep reinforcement learning proposed in this invention is mainly due to the large proportion of the training part. After the model is saved, the time spent is mainly in calling the model and memory overhead. The time complexity of the algorithm is shown in Table 5. The average difference between the complexity and the original method in bitrate control is about 5%, which is almost negligible.

[0106] Table 5 Comparison of Time Complexity

[0107]

[0108]

[0109] (5) Comparison of subjective quality

[0110] like Figure 8 As shown, Figure 8 The image on the left shows the encoding using the VTM13.0 standard rate control algorithm. Figure 8 The right side shows the encoding of a deep reinforcement learning algorithm. Figure 8This is a frame from the video sequence Cactus. It's easy to see from the image that the texture in the right image is clearer than that in the left image. Experiments demonstrate that, under the same bitrate conditions, the proposed DQN-based bitrate control algorithm produces better subjective quality after encoding.

[0111] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the protection scope of the present invention.

Claims

1. A VVC rate control algorithm based on deep reinforcement learning, characterized in that, Includes the following steps: S1. Perform encoding training based on the DQN algorithm to obtain the Q-value network; S1.1 imports the video sequence into the encoder, causing the video sequence to enter the initial frame encoding; After the S1.2 encoder finishes encoding the first two frames by default, the subsequent video frame prediction first extracts the encoding state information of the previous prediction frame and uses an ε-greedy strategy for action selection. After the S1.3 action selection is completed, it goes through CTU-level bitrate control and actual encoding process in sequence to obtain the overall reward value, i.e., Q value; S1.4 Observe the next state and the previous reward, perform TD iteration on the Q value, and add it to the Q value network after TD iteration; S1.5 Experience Replay: Set up an experience replay pool to save actions, reward values, previous frame state information and current frame state information; S1.6 Once the data cached in the experience replay pool reaches the capacity threshold, random batches are sampled from the experience replay pool. S1.7 Reset the Q-value network: Based on the sampling information in step S1.6, the parameters of the Q-value network are updated once each time the experience replay pool acquires data, so that the Q-value network is close to the regression value. S2. Encode and test based on the Q-value network obtained from training; S2.1 imports the video sequence into the encoder, causing the video sequence to enter the initial frame encoding; S2.2 Extract the coding state information of the current prediction frame, query the Q-value network based on the state information, and the Q-value network gives the optimal action; After determining the preferred QP value in S2.3, proceed with subsequent operations according to the coding process.

2. The VVC rate control algorithm based on deep reinforcement learning according to claim 1, characterized in that: The state information includes: the MSE value of the Y component of the current frame, the MSE value of the U component of the current frame, the MSE value of the V component of the current frame, the QP value of the current frame, the PSNR value of the current frame, the SSIM value of the current frame, the target number of bits of the current frame, the actual number of bits encoded in the current frame, the number of bits remaining after actual encoding in the current frame, the number of frames actually encoded, the number of frames remaining to be encoded, the average gradient of the current frame, the information entropy of the current frame, and the current buffer size.

3. The VVC rate control algorithm based on deep reinforcement learning according to claim 1, characterized in that: The calculation formula for the ε-greedy strategy is as follows: In the formula, 1-ε represents the probability that the encoder selects a large Q value; ε represents the probability that the encoder performs a random selection action.

4. The VVC rate control algorithm based on deep reinforcement learning according to claim 3, characterized in that: The ε-greedy strategy is equipped with a decay factor that allows ε to decrease as the number of iterations increases.

5. The VVC rate control algorithm based on deep reinforcement learning according to claim 1, characterized in that: The calculation formula for the TD iteration is as follows: Q(s t ,a t )←Q(s t ,a t )+α[r t+1 +γmaxQ(s t+1 ,a)-Q 估 (s t ,a t )] s t ←s t+1 In the formula, s t Represents the current state; s t+1 Represents the next state; a t α represents the current action; 'a' represents the selected action; α represents the learning rate; r t+1 +γmaxQ(s t+1 a) represents the actual value; r t+1 Represents reward; γ represents discount factor; Q 估 (s t ,a t () represents the estimated value.

6. The VVC rate control algorithm based on deep reinforcement learning according to claim 1, characterized in that: The reward is calculated using a reward function, which is defined as follows: In the formula, PSNR cur PSNR represents the PSNR value of the current frame; PSNR last Represents the PSNR value of the previous frame; buffer cur Represents the size of the buffer for the previous frame; buffer last This represents the size of the current frame buffer.

Images