VVC block division fast prediction method and system based on convolutional neural network

Abstract

This invention discloses a convolutional neural network-based fast prediction method and system for VVC block partitioning. The method includes: preprocessing a 32×32 coding unit (CU) and its quantization parameter (QP) as input to the convolutional neural network. After a convolution operation, the method outputs a prediction of the current CU partitioning, predicting the most likely CU partitioning form: horizontal, vertical, no partitioning, or a complex partitioning form. This method effectively reduces the complexity of multi-type CU partitioning and reduces the required encoding time. It also improves encoding speed while ensuring video quality at the final decoder.

Description

Technical Field

[0001] The present invention belongs to the technical field of video coding and decoding, and in particular relates to a VVC block partition fast prediction method and system based on convolutional neural network. Background Art

[0002] Due to the continuous improvement of video resolution and the development of technologies such as VR video, video bit rates have increased dramatically. Therefore, ITU-T and ISO / IEC established the JVET working group and began the development of the H.266 / VVC coding standard. The first version of the VVC standard was officially released in 2020. VVC adopts the block-based hybrid coding framework and continues most of the coding tools of HEVC. Some new coding tools have been added to each module, or the coding tools have been optimized. In addition to quadtree partitioning, VVC can also be divided into multiple types of trees. Specifically, in VVC, the CTU is first divided into different CUs according to the quadtree. Then, the CUs of the quadtree child nodes can be divided according to multiple types of trees, including four types of partitioning: vertical binary tree partitioning (BV), horizontal binary tree partitioning (BH), vertical ternary tree partitioning (TV), and horizontal ternary tree partitioning (TH). The ternary tree is divided according to 1:2:1.

[0003] like Figure 1As shown, the coding block partitioning adopts a recursive design. Faced with coding blocks with obvious horizontal or vertical features, as well as smooth coding blocks, the complex multiple partitioning methods result in a lot of invalid redundant coding. Therefore, in order to reduce the complexity of the algorithm, quickly judging the partition type of the coding block is an effective way. The literature "Park SH, Kang J W. Context-based Ternary Tree Decision Method in Versatile Video Coding for Fast IntraCoding[J]. IEEE Access, 2019, PP(99):1-1." uses the VVC coding context to identify the ternary tree partitioning features, and therefore proposes a simple early decision technology to reduce the complexity of the ternary tree partitioning. The literature "Wu G, Huang Y, Zhu C, et al. SVM Based Fast CU Partitioning Algorithm for VVC Intra Coding[C] / / 2021IEEE International Symposium on Circuits and Systems(ISCAS). IEEE, 2021." proposes a VVC intra-frame coding fast CU partitioning algorithm based on support vector machine. By using texture information to predict the CU partition, redundant partitioning is terminated in advance, thereby improving coding efficiency. However, these methods all require pre-extraction of the features of the coding blocks, and the effectiveness of the algorithm is closely related to the selection of features.

[0004] In recent years, it has become increasingly difficult to continue using traditional methods to improve coding performance within the video coding framework, and neural network-based video coding has gradually become possible. Summary of the Invention

[0005] The purpose of the present invention is to provide a VVC block partition fast prediction method and system based on convolutional neural network, which reduces the computational complexity of multi-type partitioning of coding units and saves coding time while ensuring coding quality.

[0006] The technical solution to achieve the purpose of the present invention is: a VVC block partition fast prediction method based on convolutional neural network, comprising the following steps:

[0007] Step 1. Data collection: Select different videos, select one frame every M frames, and select N frames for each video. Encode them using the original VVC encoder under different quantization parameters QP, and collect the following data as the training set: the pixel values ​​of the 32×32 coding unit CU, the quantization parameter QP as input information, and the category of the optimal partitioning method of the CU during the encoding process as the corresponding label; the categories include horizontal partitioning, vertical partitioning, no partitioning, and quadtree partitioning;

[0008] Step 2: Data training: For the CUs in the training set, the CU brightness values ​​are preprocessed by removing the mean value and used as the input of the convolutional neural network. Then, four convolution calculations are performed to obtain 1×1 features. The QP value is used as a priori input to the fully connected layer, and the output features are superimposed with the CU features. The softmax is performed as the output of the partition result, which corresponds to the label. After training, a network for fast prediction of VVC block partitioning is obtained.

[0009] Step 3: Model deployment: During the actual VVC encoding process, for each 32×32 CU, the CU brightness value and QP value are input into the trained network to obtain the prediction of the optimal partition of the current CU, and subsequent encoding is performed based on the prediction.

[0010] A VVC block partition fast prediction system based on convolutional neural network, comprising:

[0011] The data collection module is used to implement the following: select different videos, select one frame every M frames, and select N frames in total for each video, encode them using the original VVC encoder under different quantization parameters (QP), and collect the following data as a training set: 32×32 coding units (CUs), quantization parameters (QP) as input information, and the category of the optimal partitioning method for the CU during the encoding process as the corresponding label; the optimal partitioning method categories include horizontal partitioning, vertical partitioning, no partitioning, and quadtree partitioning;

[0012] The data training module is used to achieve the following: for the CUs in the training set, the CU brightness values ​​are preprocessed by removing the mean value and used as the input of the convolutional neural network. Then, four convolution calculations are performed to obtain 1×1 features. The QP value is used as a prior input to the fully connected layer, and the output features are superimposed with the CU features. After softmax, the output is the partition result, which corresponds to the label. After training, a network is obtained for fast prediction of VVC block partitions.

[0013] The model deployment module is used to implement the following: During the actual VVC encoding process, for each 32×32 CU, the CU brightness value and QP value are input into the above-mentioned trained network to obtain the prediction of the optimal partition of the current CU, and subsequent encoding is performed based on the prediction.

[0014] An electronic device includes a memory, a processor, and a computer program stored in the memory and runnable on the processor. When the processor executes the program, the above-mentioned VVC block division fast prediction method based on convolutional neural network is implemented.

[0015] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the above-mentioned VVC block partition fast prediction method based on convolutional neural network.

[0016] Compared with the existing technology, the present invention has the following significant advantages: (1) the convolutional neural network can obtain a prediction model by training the original pixels, completing an end-to-end prediction process with pixels as input and partition types as output, without the need to manually extract features for judging the coding block partitioning; (2) the network model is relatively small, and the network prediction time accounts for about 3% of the total coding time, and its impact on the overall coding time can be ignored. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is the division structure of the coding unit in VVC.

[0018] Figure 2 This is a model training diagram of the VVC block division fast prediction method based on convolutional neural network of the present invention.

[0019] Figure 3 This is a coding flow chart of the VVC block partition fast prediction method based on convolutional neural network of the present invention. DETAILED DESCRIPTION

[0020] The present invention uses convolutional neural networks to determine the division method of 32×32 coding blocks in VVC. Figure 2 , the specific steps are as follows:

[0021] Step 1: Data collection: Select different videos, select one frame every M frames, and select N frames in total for each video. Encode them using the original VVC encoder at different quantization parameters (QP). Collect the following data as a training set: 32×32 coding units (CUs), quantization parameters (QP) as input information, and the category of the optimal partitioning method of the CU during the encoding process: horizontal partitioning, vertical partitioning, no partitioning, or quadtree partitioning as the corresponding label;

[0022] Step 2: Data training: For the CUs in the training set, the CU brightness values ​​are preprocessed by removing the mean value and used as the input of the convolutional neural network. Four convolution operations are then performed to obtain 1×1 features. The QP value is used as a priori input to the fully connected layer, and the output features are superimposed with the CU features. The softmax is performed to obtain the partitioning result, which corresponds to the label. After training, a network is obtained for fast prediction of VVC block partitioning.

[0023] Step 3: Model deployment: During the actual VVC encoding process, for each 32×32 CU, the CU brightness value and QP value are input into the trained network to obtain the optimal partitioning of the current CU, and subsequent encoding is performed based on the prediction.

[0024] Furthermore, the marking of the CU and its corresponding label in step 1 is selected according to the encoding result, and the CU block containing 32×H (H<32) in the encoding result is marked as horizontal division, and the CU block containing W×32 (W<32) in the encoding result is marked as vertical division. The CU that is not further divided in the encoding result is marked as not divided, and the CU that continues to be quadtree divided in the encoding result is marked as complex form.

[0025] Furthermore, in the four convolution calculations in step 2, the convolution kernel size is equal to the step size, and the convolution kernels of each convolution do not overlap.

[0026] Furthermore, step 2 connects the features obtained from the 32×32 CU and the features of the QP through a fully connected layer.

[0027] Furthermore, the judgment method of the convolutional neural network built in step 2 is as follows:

[0028] The features of the 32×32 CU after four layers of convolution are connected with the features of QP to obtain a 1×1 composite feature, and the division judgment of the current coding unit is output.

[0029] The present invention also provides a VVC block division fast prediction system based on convolutional neural network, comprising:

[0030] The data collection module is used to implement the following: select different videos, select one frame every M frames, and select N frames in total for each video, encode them using the original VVC encoder under different quantization parameters (QP), and collect the following data as a training set: 32×32 coding units (CUs), quantization parameters (QP) as input information, and the category of the optimal partitioning method for the CU during the encoding process as the corresponding label; the optimal partitioning method categories include horizontal partitioning, vertical partitioning, no partitioning, and quadtree partitioning;

[0031] The data training module is used to achieve the following: for the CUs in the training set, the CU brightness values ​​are preprocessed by removing the mean value and used as the input of the convolutional neural network. Then, four convolution calculations are performed to obtain 1×1 features. The QP value is used as a prior input to the fully connected layer, and the output features are superimposed with the CU features. After softmax, the output is the partition result, which corresponds to the label. After training, a network is obtained for fast prediction of VVC block partitions.

[0032] The model deployment module is used to implement the following: During the actual VVC encoding process, for each 32×32 CU, the CU brightness value and QP value are input into the above-mentioned trained network to obtain the prediction of the optimal partition of the current CU, and subsequent encoding is performed based on the prediction.

[0033] The implementation method of the above functional modules is partially the same as the aforementioned prediction method, and will not be repeated here.

[0034] The technical solution of the present invention is further specifically described below through examples.

[0035] Example

[0036] like Figure 2 、 Figure 3 As shown in FIG, a fast prediction method for VVC block division based on a convolutional neural network includes the following steps:

[0037] Step 1: Data collection: Select the video sequences BasketballDrill, FourPeople, and Johnny, select one frame every 8 frames, and select a total of 10 frames for each video, and collect the dataset using the above method;

[0038] Step 2: Input the data set into the network for training. After training, a network model for fast prediction of VVC block division is obtained.

[0039] Step 3: Model deployment: During the actual VVC encoding process, for each 32×32 CU, the CU brightness value and QP value are input into the trained network to obtain the optimal partitioning of the current CU, and subsequent encoding is performed based on the prediction. Specifically:

[0040] If the current CU is 32×32 in size, it is input into the network for partition prediction. If it is judged to be horizontally split, only the horizontal partition coding is performed; if it is judged to be vertically split, only the vertical partition coding is performed; if it is judged to be non-split, no subsequent partition coding is performed; if it is judged to be a complex form, it is encoded according to the original VTM partition method.

[0041] If the current CU size is not 32×32, it is encoded according to the original VTM division method.

[0042] The performance of the above method is compared with the original VTM6.0 model. Table 1 shows the encoding performance comparison in terms of encoding time and BDBR.

[0043] Table 1 Comparison of encoding results between the method of the present invention and the method of VTM6.0

[0044]

[0045] BDBR represents the bit rate savings compared to the original VTM 6.0 method under the same objective video quality conditions, while encoding time compares the encoding time savings compared to the original VTM 6.0 method.

[0046] The present invention is not limited to the contents of the claims and the above embodiments. Any invention created based on the concept of the present invention should fall within the protection scope of the present invention.

Claims

1. A fast prediction method for VVC block division based on convolutional neural network, characterized in that: The steps include: Step 1. Data Collection: Select different videos, select one frame every M frames, and select N frames for each video. Encode them using the original VVC encoder at different quantization parameters (QP). Collect the following data as a training set: 32×32 pixel values ​​of coding units (CUs) and quantization parameters (QP) as input information, and the category of the optimal partitioning method for the CU during the encoding process as the corresponding label; the categories include horizontal partitioning, vertical partitioning, no partitioning, and quadtree partitioning; Step 2: Data training: For the CUs in the training set, the CU brightness values ​​are preprocessed by removing the mean value and used as the input of the convolutional neural network. Then, four convolution calculations are performed to obtain 1×1 features. The QP value is used as a priori input to the fully connected layer, and the output features are superimposed with the CU features. The normalized exponential function is used as the output of the partition result, corresponding to the label. After training, a network for fast prediction of VVC block partitioning is obtained. Step 3, model deployment: During the actual VVC encoding process, for each 32×32 CU, the CU brightness value and QP value are input into the trained network to obtain the prediction of the optimal partition of the current CU, and subsequent encoding is performed based on the prediction.

2. The VVC block division fast prediction method based on convolutional neural network according to claim 1, characterized in that The CUs in step 1 and their corresponding labels are selected based on the encoding results. The CU blocks containing 32×H in the encoding results are marked as horizontal divisions, the CU blocks containing W×32 in the encoding results are marked as vertical divisions, the CUs that are not further divided in the encoding results are marked as undivided, and the CUs that continue to be quadtree divided in the encoding results are marked as complex forms; where H < 32 and W < 32.

3. The VVC block division fast prediction method based on convolutional neural network according to claim 1, characterized in that In the four convolution calculations in step 2, the convolution kernel size is equal to the step size, and the convolution kernels of each convolution do not overlap.

4. The VVC block division fast prediction method based on convolutional neural network according to claim 1, characterized in that Step 2 connects the features obtained from the 32×32 CU and the features of the QP through a fully connected layer.

5. A VVC block partition fast prediction system based on convolutional neural network, characterized in that: include: The data collection module is used to implement the following: select different videos, select one frame every M frames, and select N frames in total for each video, encode them using the original VVC encoder under different quantization parameters (QP), and collect the following data as a training set: 32×32 coding units (CUs), quantization parameters (QP) as input information, and the category of the optimal partitioning method for the CU during the encoding process as the corresponding label; the optimal partitioning method categories include horizontal partitioning, vertical partitioning, no partitioning, and quadtree partitioning; The data training module is used to achieve the following: for the CUs in the training set, the CU brightness values ​​are preprocessed by removing the mean value and used as the input of the convolutional neural network. Then, four convolution calculations are performed to obtain 1×1 features. The QP value is used as a priori input to the fully connected layer, and the output features are superimposed with the CU features. The output is a normalized exponential function and corresponds to the label as the partition result. After training, a network is obtained for fast prediction of VVC block partitions. The model deployment module is used to implement: During the actual VVC encoding process, for each 32×32 CU, the CU brightness value and QP value are input into the trained network to obtain the optimal partitioning of the current CU, and subsequent encoding is performed based on the prediction.

6. The VVC block partition fast prediction system based on convolutional neural network according to claim 5, characterized in that The category to which the optimal division method of the CU belongs is used as the corresponding label, specifically: according to the encoding result, the CU block containing 32×H in the encoding result is marked as horizontal division, the CU block containing W×32 in the encoding result is marked as vertical division, the CU that is not further divided in the encoding result is marked as not divided, and the CU that continues to be quadtree divided in the encoding result is marked as complex form; where H < 32, W < 32.

7. The VVC block partition fast prediction system based on convolutional neural network according to claim 5, characterized in that In the four convolution calculations, the convolution kernel size is equal to the step size, and the convolution kernels of each convolution do not overlap.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, it implements the VVC block partition fast prediction method based on convolutional neural network as described in any one of claims 1-4.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the VVC block partition fast prediction method based on a convolutional neural network as described in any one of claims 1 to 4 is implemented.

Images