I-frame VVC code rate control optimization method based on neural network

Abstract

The invention provides an I-frame VVC code rate control optimization method based on a neural network. The method comprises the following steps: firstly, constructing a network model between video content features and prediction bits; secondly, inputting a to-be-coded CTU to the network to obtain a predicted bit after prediction; and finally, according to the predicted bits, guiding a bit allocation process of a CTU level and a frame level, and the process can explicitly change a bit allocation strategy. Experimental results show that compared with VVC and other advanced algorithms, the I-frame VVC code rate control optimization method based on the neural network disclosed by the invention has the advantage that indexes such as delta BRerror, MS-SSIM, BD-rate and a rate distortion curve are improved. The video sequence obtained on the basis of the method can better meet the requirements of human eyes, and can be greatly optimized in the aspect of subjective quality.

Description

Technical Field

[0001] The present invention relates to a rate control optimization technology problem in the field of video coding, and in particular to an I (Intra) frame Versatile Video Coding (VVC) rate control optimization method based on a neural network. Background Art

[0002] With the continuous development of intelligent terminal devices, video information has gradually become the focus of multimedia processing. Several video coding standards have been proposed, including H.264 / AVC, H.265 / HEVC, and H.266 / VVC. Among these standards, the new generation of video coding standards called Versatile Video Coding (VVC) was officially standardized in 2020. Compared with High Efficiency Video Coding (HEVC), VVC demonstrates the ability to reduce bit rate by nearly 50% while maintaining the same video quality. To achieve this, VVC extends the hybrid coding framework and introduces a series of innovative coding techniques.

[0003] As the compression performance of VVC gradually approaches the theoretical limit, the development direction of the next generation of video coding standards has become the focus of researchers. On the one hand, researchers continue to explore and optimize VVC-based technologies to obtain superior coding performance. On the other hand, some researchers transfer the concept of learning-based image processing and coding to the existing video coding framework. Using the learning ability of neural networks, research is conducted to achieve performance enhancement in specific modules. For example, learning-based methods have been proven to be effective in areas such as loop filtering, intra / inter prediction, entropy coding, and fast algorithms, showing great research potential.

[0004] As one of the core modules of the H.266 / VVC video coding standard, rate control can not only achieve the most accurate target bit allocation, but also has certain significance and value in minimizing the distortion of compressed video, maintaining the consistency of compressed video quality, and reducing system latency. VVC optimizes the rate control model in HEVC. For different profiles, it expands the size of the group of picture (GOP), introduces the concepts of skip area and non-skip area, proposes an adaptive lambda bit prediction algorithm, and optimizes the parameter update process. In addition, H.266 / VVC also proposes a rate control algorithm that depends on the quality factor and a rate control algorithm based on the loop filter module, which further improves the encoding performance. Therefore, introducing a learning-based method into the rate control module is an innovative technology.

[0005] However, in the existing learning-based rate control algorithms, obtaining the predicted coding parameters through fitting requires a lot of time and complexity in the training process. In addition, establishing a direct relationship between video content features and allocated bits remains a challenge. In order to solve these problems, this research group conducted exploratory research and introduced a rate control algorithm based on an efficient neural network. The algorithm uses a neural network to directly model the video content features and their corresponding predicted bits, greatly reducing the complexity of training. After making the above improvements to the rate control model in VVC, the subjective and objective performance of the encoded video can be improved while meeting the precise target bit allocation, thereby achieving the purpose of rate control optimization. This is exactly the task of the present invention. Summary of the invention

[0006] This invention proposes a learning-based rate control model for the I-frame rate control model of VVC. The specific process steps include:

[0007] (1) Construct a network model between video content features and predicted bits, as follows:

[0008] As attached Figure 2 As shown in Figure 1, the proposed network takes the original video frame X as input with a size of 128×128, and the output is the predicted bits as a scalar Y. The network consists of two key stages: feature extraction and bit prediction. In the feature extraction module, 10 convolutional submodules are used, each of which includes a convolutional layer, normalization layer, linear layer, and average pooling layer. In order to capture deeper feature information, five convolutional layers are used to gradually increase the output channels from 64 to 1024. At the same time, the last five convolutional layers gradually reduce the output channels to 1, ensuring the stability of the network and the refinement of the features. Subsequently, in the bit prediction module, the extracted features are expanded into one-dimensional features, then passed through three fully connected layers, interspersed with two linear layers, and finally the predicted number of bits is obtained.

[0009] (2) The predicted bits output by the above neural network are used to guide the bit allocation at the frame level and CTU level, as follows:

[0010] In frame-level bit allocation, the neural network can map each CTU in the frame to obtain the corresponding predicted bit Bit pre According to the predicted number of bits of each CTU, the number of bits of the current frame can be calculated. curframe for:

[0011]

[0012] Here, N CTUIndicates the number of CTUs contained in the current frame. At the same time, in the CTU-level bit allocation, the number of bits predicted by the network is directly used as the number of bits of each CTU, as shown in the following formula:

[0013] Bit curCTU =Bit pre (2)

[0014] Among them, Bit curCTU Indicates the number of bits in the current CTU.

[0015] The advantages and beneficial technical effects of the present invention compared with the prior art are as follows:

[0016] (1) The present invention proposes a neural network-based I-frame VVC rate control optimization method. Compared with the VVC default rate control algorithm, the coding performance of the rate control method proposed by the present invention is higher than the coding performance of the rate control model corresponding to the VVC standard algorithm;

[0017] (2) The present invention proposes a neural network-based I-frame VVC bit rate control optimization method, which utilizes the prior information of video content features to predict the coding bits of the I frame, control the bit error within a smaller range and obtain higher subjective quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is the overall flow chart of the I-frame VVC bit rate control optimization method based on neural network.

[0019] Figure 2 This is the network framework diagram of the bit prediction model based on neural network.

[0020] Figure 3 Graph showing rate-distortion curves of the sequences RitualDance and RaceHorses based on the method of the present invention.

[0021] Figure 4 This is a subjective comparison chart of the 80th frame of the sequence Cactus based on the method of the present invention and Zhao's method HPRC (the left side is HPRC and the right side is the optimization method of the present invention). DETAILED DESCRIPTION

[0022] The present invention is further described in detail below in conjunction with the embodiments. It is necessary to point out that the following embodiments are only used to further illustrate the present invention and cannot be understood as limiting the scope of protection of the present invention. Technical personnel familiar with the relevant field may make some non-essential improvements and adjustments to the present invention according to the above invention content for specific implementation, which should still fall within the scope of protection of the present invention.

[0023] (1) The method proposed in the present invention is implemented on the VVC standard test code VTM19.0 platform. The configuration file is encoder_lowdelay_P_main.cfg. In the rate control parameter configuration, RateControl is set to 1, indicating that rate control is enabled.

[0024] (2) The test sequences for verifying the coding performance of the method proposed in the present invention selected four official standard test sequences with different resolutions;

[0025] (3) In the VVC standard algorithm program, set RateControl to 0 in the configuration file encoder_lowdelay_P_main.cfg, indicating that rate control is turned off. Then input all the video standard sequences to be tested to obtain the target bit rate of each sequence when rate control is turned off.

[0026] (4) In the program of the algorithm proposed in the present invention, the parameters in (1) are set for the configuration file encoder_lowdelay_P_main.cfg, and TargetBitrate is set to the target bitrate corresponding to each sequence in (3), then all the video standard sequences to be tested are input to obtain the results of the algorithm proposed in the present invention, and the actual bitrate, peak signal-to-noise ratio (PSNR), and multi-scale structural similarity (MS-SSIM) are recorded and counted;

[0027] (5) In the VVC default algorithm program, set the parameters in (1) for the configuration file encoder_lowdelay_P_main.cfg, and set TargetBitrate to the target bitrate corresponding to each sequence in (3). Then input all the video standard sequences to be tested, obtain the results of the VVC default algorithm, and record and count the actual bitrate, peak signal-to-noise ratio (PSNR), and multi-scale structural similarity (MS-SSIM);

[0028] (6) The above three types of results are processed separately, where the unit of bit rate is Kbps; the bit error ΔBR of the present invention and VVC is calculated respectively using formula (3): error , the smaller the index, the better; using formula (4) to calculate the improvement of the structural complexity of the present invention compared with VVC, the larger the index, the better;

[0029]

[0030] ΔMS-SSIM=MS-SSIM pro -MS-SSIM VVC (4)

[0031] (7) Table 1 shows the performance comparison between VVC and the present invention. It can be seen from Table 1 that the average ΔBR of the proposed algorithm error It is only about 25% of the VVC default bitrate control algorithm, and the MS-SSIM is improved by an average of 0.005877, which is enough to prove the accuracy and rationality of the proposed algorithm in bit allocation strategy. At the same time, under the same PSNR condition, the proposed algorithm saves an average of 7.23% bitrate compared with the default algorithm. This shows that compared with the standard algorithm, the proposed algorithm has a certain degree of superiority in the BD-Rate evaluation index;

[0032] (8) Figure 3 It means that the rate-distortion curve of the proposed algorithm is higher than that of the VVC standard algorithm in both high and low bitrate segments, that is, under the same coding bitrate condition, the PSNR (MS-SSIM) of the reconstructed video after compression by the proposed algorithm is higher than that of other algorithms, and the coding performance in the full bitrate segment is better than that of the standard algorithm.

[0033] (9) Figure 4 In (a)(b), the left side shows the subjective effect of the compressed reconstructed video based on the HPRC algorithm, and the right side shows the subjective effect of the compressed reconstructed video based on the proposed algorithm. It can be clearly seen that in areas such as flower pots and cacti, the clarity of the proposed algorithm is better than that of the HPRC algorithm. Therefore, the proposed algorithm has a certain degree of improvement in the subjective quality of the compressed reconstructed video.

[0034] Table 1 Performance comparison between the present invention and VVC

[0035]

Claims

1. A neural network-based I-frame VVC rate control optimization method, characterized in that The overall framework of I-frame VVC bitrate control based on neural network: (1) Construct a network model between video content features and predicted bits, as follows: The proposed network takes the original video frame X as input with a size of 128×128, and the output is the predicted bit as a scalar Y. The network consists of two key stages: feature extraction and bit prediction. In the feature extraction module, 10 convolutional submodules are used, each of which includes a convolutional layer, a normalization layer, a linear layer, and an average pooling layer. In order to capture deeper feature information, five convolutional layers are used to gradually increase the output channels from 64 to 1024. At the same time, the last five convolutional layers gradually reduce the output channels to 1, ensuring the stability of the network and the refinement of the features. Subsequently, in the bit prediction module, the extracted features are expanded into one-dimensional features, and then passed through three fully connected layers, interspersed with two linear layers, and finally the predicted number of bits is obtained. (2) The predicted bits output by the above neural network are used to guide the bit allocation at the frame level and CTU level, as follows: In frame-level bit allocation, the neural network can map each CTU in the frame to obtain the corresponding predicted bit Bit pre ; According to the predicted number of bits of each CTU, the number of bits of the current frame can be calculated curframe for: Here, N CTU Indicates the number of CTUs contained in the current frame. At the same time, in the bit allocation at the CTU level, the number of bits predicted by the network is directly used as the number of bits of each CTU, as shown in the following formula: Bit curCTU =Bit pre (2) Among them, Bit curCTU Indicates the number of bits in the current CTU.

2. The overall framework of I-frame VVC rate control based on neural network as claimed in claim 1, characterized in that The process of establishing a relationship between the content characteristics of a video frame and the predicted bits.

3. The overall framework of I-frame VVC rate control based on neural network as claimed in claim 1, characterized in that The predicted bits output by the neural network guide the process of bit allocation at the frame level and CTU level.

Images