VVC rapid CU division method based on ensemble learning

Abstract

The invention relates to a VVC fast CU division method based on ensemble learning, which belongs to the field of video coding and comprises the following steps of: initializing a given coding unit (CU), and starting a coding process from the deepest level; then, feature extraction is carried out; the feature attribute value of the current CU is calculated; and model prediction: loading a pre-trained integrated learning classifier (i = 0, 1, 2), carrying out division prediction on the current CU by using the pre-trained integrated learning classifier, and then carrying out decision processing. And once the coding of the current CU is completed, moving to the next CU in the list and repeating the steps. Through the series of steps, the algorithm can systematically perform division decision on each CU so as to optimize the video coding process. The method can be used for coding under VVC intra-frame coding configuration, the coding time of VVC is remarkably shortened under the condition that the coding efficiency and the coding quality are hardly reduced, and the method can be applied to video conferences and other application scenes with high requirements for coding real-time performance.

Description

Technical Field

[0001] The present invention belongs to the field of video coding, and more specifically, provides a CU fast partitioning method suitable for universal video coding VVC (Versatile Video Coding), which can be applied to video conferencing and other video coding scenarios with high real-time requirements. Background Art

[0002] With the rapid development of digital media and network technology, video has become an indispensable part of people's daily life. The explosive growth of video data has posed a huge challenge to storage and transmission, so efficient video coding technology is particularly important. High Efficiency Video Coding (HEVC) is a new generation of video coding standard launched by the International Organization for Standardization. Its goal is to double the video data compression efficiency while maintaining video quality. However, while HEVC improves compression efficiency, it also brings higher computational complexity, especially in the division of coding units (CU) and intra-frame prediction, its complexity directly affects the coding rate.

[0003] H.266 / VVC still follows the traditional hybrid coding structure. Compared with HEVC, VVC adjusts the structural division method of coding blocks, and adds many coding technologies in multiple modules such as quantization transformation, prediction coding, and entropy coding. In terms of coding blocks, the CU partition decision method of quadtree hybrid multi-tree (QTMT) is adopted. QTMT retains the quadtree (Quadtree, QT) division method, and on this basis adds binary tree (Binary Tree, BT) division method and ternary tree (Ternary Tree, TT) division method in the horizontal and vertical directions. In terms of residual data transformation, new technologies such as high-frequency zeroing large-size transformation and adaptive multi-core transformation (Multiple Transform Selection, MTS) are added, and the optimal transformation method can be adaptively selected during encoding. In terms of intra-frame mode prediction, VVC retains the original intra-frame prediction mode of HEVC, and on the basis of the original angle prediction mode, a new angle prediction mode is inserted between every two angle prediction modes, so that the angle prediction mode increases to 65, effectively improving the structural accuracy of the prediction coding block.

[0004] The fast CU partitioning algorithm based on the decision tree aims to reduce the computational complexity, improve the encoding speed, and maintain a high encoding efficiency. The present invention continues to use this method and is used as a submodule. With respect to the decision tree method, the model can quickly predict the CU partitioning scheme by learning the video frame features, greatly reducing the amount of calculation of the full search method and improving the encoding speed. Despite the reduction in the amount of calculation, the decision tree model can still maintain a high prediction accuracy, ensuring that the encoding quality is close to the original VVC standard. The model can adapt to different types of video content, and by training different video data sets, the splitting criteria and depth of the decision tree are optimized, further improving the applicability and performance of the model. The decision tree model has a simple structure, is easy to implement and integrate into the existing VVC coding framework, and has strong practicality. Summary of the invention

[0005] The present invention aims to solve the above problems of the prior art. A VVC fast CU division method based on ensemble learning is proposed. The technical solution of the present invention is as follows:

[0006] A VVC fast CU division method based on ensemble learning includes the following steps:

[0007] S1. For a given coding unit CU, start coding from the deepest level CuDepth=0, calculate the characteristic attribute value of the current CU, and enter step S2;

[0008] S2, determine the size and type of the current CU, if the current CU is a 64×64, 32×32 or 16×16 CU, go to step S3, otherwise, go to step S5;

[0009] S3, load the pre-trained integrated learning classifier and use it to predict the partition of the current CU. If the classifier output is 1, it is determined to be partitioned, and the current CU is subdivided into 4 sub-CUs, and then return to S1; If the output is -1, it is determined not to be partitioned, and the next CU of the current level is processed; If the output is 0, evaluate the rate distortion of the current CU and the sum of the rate distortions of the four potential sub-CUs to make the best partition selection;

[0010] S4, if it is decided to continue to divide the CU, CuDepth increases by 1, the current CU is divided into 4 sub-CUs, and then enters step S2. Otherwise, continue to process the next CU in the current level. When CuDepth reaches 3, it terminates;

[0011] S5, traverse the partitioning mode in the partitioning list, return to S1 to enter the partitioning of the next CU;

[0012] S6. End the current CU division process.

[0013] Furthermore, in step S1, the characteristic attribute values ​​of the CU are calculated, including entropy, contrast, inverse moment, angular second-order moment and correlation in four directions (0°, 45°, 90°, 135°), for a total of 20 features. The calculation formulas for entropy, contrast, inverse moment, angular second-order moment and correlation are as follows:

[0014] Ent=-∑ i,j P(i,j;d,θ)log2P(i,j;d,θ) (1)

[0015]

[0016] ASM=∑ i,j P(i,j;d,θ) (4)

[0017] COR=∑ i,j (i-μ)(j-μ)P(i,j;d,θ) (5)

[0018] Among them, P(i,j;d,θ) is the probability of co-occurrence of pixel values ​​i and j given direction θ and distance d, and μ is the mean of the GLCM matrix.

[0019] Furthermore, in step S2, judging the current CU size and type means that the CU of 8×8 size is the most basic unit in CU encoding and does not need to be further divided. For CUs of different sizes, three different DT-SVM models are designed, corresponding to CUs of sizes 64×64 (CuDepth=0), 32×32 (CuDepth=1) and 16×16 (CuDepth=2).

[0020] Furthermore, in step S3, a pre-trained integrated learning classifier is loaded, specifically combining the DT-SVM and CNN models as base learners, and the VVC coding database is trained.

[0021] Furthermore, in step S3, the training of the decision tree DT uses the C4.5 algorithm, and selects information gain as the indicator of node splitting. Recursive segmentation is applied to the training set to construct a decision tree. Pre-pruning technology is applied to avoid overfitting. The SVM training selects the RBF kernel function to perform high-dimensional mapping on the features. The training data is used to determine the parameters of the SVM. The results of the decision tree DT and the SVM are merged to create a DT-SVM model.

[0022] Furthermore, in step S3, the training of the CNN model is to determine the number and size of convolutional layers, the type and location of pooling layers, design the fully connected layer, and ensure that the number of neurons in the output layer matches the classification task. Use the back propagation algorithm to optimize the network weights. Adjust hyperparameters such as learning rate, batch size, number of iterations, etc.

[0023] Furthermore, in step S4, this is a recursive partitioning process, which continues until the block size reaches 8×8. The probability value is used to determine whether to continue partitioning: a probability value greater than 0.5 predicts that the partitioning will continue. A probability value less than 0.5 predicts that the partitioning will be terminated.

[0024] Furthermore, in step S5, entering the next CU partition specifically means that once the encoding of the current CTU is completed, move to the next CU in the list and repeat the above steps. Through this series of steps, the algorithm can systematically make partition decisions for each CU to optimize the video encoding process.

[0025] The advantages and beneficial effects of the present invention are as follows:

[0026] Aiming at the problem that the latest international video coding standard H.266 / VVC has high computational complexity and long encoding time, the present invention proposes a CU fast partitioning method based on ensemble learning. First, for a given current CU, the encoding process is started from the deepest level and the characteristic attribute value of the current CU is calculated, and then the pre-trained ensemble learning classifier is loaded and used to partition the current CU. The ensemble learning classifier includes DT-SVM and CNN models. The training of the decision tree DT uses the C4.5 algorithm and selects information gain as the indicator of node splitting. Recursive segmentation is applied to the training set to construct a decision tree. Pre-pruning technology is applied to avoid overfitting. The RBF kernel function is selected for SVM training to perform high-dimensional mapping of features. The parameters of the SVM are determined using training data. The training of the CNN model is to determine the number and size of the convolutional layers, the type and location of the pooling layer. The fully connected layer is designed to ensure that the number of neurons in the output layer matches the classification task. The network weights are optimized using the back propagation algorithm. Hyperparameters such as learning rate, batch size, number of iterations, etc. are adjusted. The present invention can be used for video coding under VVC inter-frame coding configuration, greatly reducing the VVC coding time without almost reducing the coding efficiency and coding quality, and can be applied to application scenarios such as video conferencing that have high requirements for real-time coding.

[0027] The innovation of the present invention mainly lies in claim 1. The method of claim 1 is adopted to divide the CU by loading the pre-trained classifier. In the prediction stage of the DT-SVM model, the algorithm calculates the SATD and GDS entropy values ​​of a given CU block, and predicts according to the path from the root node to the leaf node of the decision tree. If the output of the leaf node is 1, it is predicted that the CU block will be split; otherwise, the CU block will remain undivided. The features selected entropy, contrast, and inverse moment in four directions are used as the final input feature vector. The CNN model adopts a hierarchical prediction process. In order to reduce unnecessary calculations, special classifiers are trained for coding blocks of different sizes (64×64, 32×32, 16×16). These classifiers can predict whether further division is needed. The probability value is used to decide whether to continue the division. If the probability value is greater than 0.5, it is predicted to continue the division; if the probability value is less than 0.5, it is predicted to terminate the division. This method can greatly reduce the encoding time when the texture of the video sequence is not complex. Even in video sequences with complex textures, the encoding time can be significantly reduced by skipping the intermediate rate-distortion calculation. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 A flowchart of a VVC fast CU division method based on ensemble learning provided by the present invention;

[0029] Figure 2 It is the flow chart of texture feature extraction of CU based on gray level co-occurrence matrix (GLCM) in DT-SVM model;

[0030] Figure 3 is the training step of SVM in DT-SVM model;

[0031] Figure 4 This is the hierarchical prediction flow chart in the CNN model; DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention.

[0033] The technical solution of the present invention to solve the above technical problems is:

[0034] Figure 1 The present invention provides a VVC fast CU division method based on ensemble learning, and the method comprises the following steps:

[0035] S1. For a given coding unit CU, start coding from the deepest level CuDepth=0, calculate the characteristic attribute value of the current CU, and enter step S2;

[0036] S2, determine the size and type of the current CU, if the current CU is a 64×64, 32×32 or 16×16 CU, go to step S3, otherwise, go to step S5;

[0037] S3, load the pre-trained ensemble learning classifier and use it to predict the partition of the current CU. If the classifier output is 1, it is determined to be partitioned, and the current CU is subdivided into 4 sub-CUs, and then return to S1; If the output is -1, it is determined not to be partitioned, and the next CU of the current level is processed; If the output is 0, go to step S4;

[0038] S4, if it is decided to continue to divide the CU, CuDepth increases by 1, the current CU is divided into 4 sub-CUs, and then enters step S2. Otherwise, continue to process the next CU in the current level. When CuDepth reaches 3, it terminates;

[0039] S5, traverse the partitioning mode in the partitioning list, return to S1 to enter the partitioning of the next CU;

[0040] S6. End the current CU division process.

[0041] Preferably, in step S1, the characteristic attribute values ​​of the CU are calculated including entropy, contrast, inverse moment, angular second-order moment and correlation in four directions (0°, 45°, 90°, 135°), for a total of 20 features. The calculation formulas for entropy, contrast, inverse moment, angular second-order moment and correlation are as follows:

[0042] Ent=-∑ i,j P(i,j;d,θ)log2P(i,j;d,θ) (1)

[0043]

[0044] ASM=∑ i,j P(i,j;d,θ) (4)

[0045] COR=∑ i,j (i-μ)(j-μ)P(i,j;d,θ) (5)

[0046] Among them, P(i,j;d,θ) is the probability of co-occurrence of pixel values ​​i and j given direction θ and distance d, and μ is the mean of the GLCM matrix.

[0047] Preferably, in step S2, judging the current CU size and type means that the CU of 8×8 size is the most basic unit in CU encoding and does not need to be further divided. For CUs of different sizes, three different DT-SVM models are designed, corresponding to CUs of sizes of 64×64, 32×32 and 16×16, respectively.

[0048] Preferably, in step S3, a pre-trained integrated learning classifier is loaded, specifically combining DT-SVM and CNN models as base learners, and the VVC coding database is trained.

[0049] Preferably, in step S3, the training of the decision tree DT uses the C4.5 algorithm, and information gain is selected as the indicator of node splitting. Recursive segmentation is applied to the training set to construct a decision tree. Pre-pruning technology is applied to avoid overfitting. The SVM training selects the RBF kernel function to perform high-dimensional mapping on the features. The training data is used to determine the parameters of the SVM. The results of the decision tree DT and the SVM are merged to create a DT-SVM model.

[0050] Preferably, in step S3, the training of the CNN model is to determine the number and size of convolutional layers, the type and location of pooling layers, design fully connected layers, and ensure that the number of neurons in the output layer matches the classification task. Use the back propagation algorithm to optimize the network weights. Adjust hyperparameters such as learning rate, batch size, number of iterations, etc.

[0051] Preferably, in step S4, this is a recursive partitioning process, which continues until the block size reaches 8×8. The probability value is used to determine whether to continue partitioning: a probability value greater than 0.5 predicts that the partitioning will continue. A probability value less than 0.5 predicts that the partitioning will be terminated.

[0052] Preferably, in step S5, entering the next CU partition specifically means that once the encoding of the current CTU is completed, move to the next CU in the list and repeat the above steps. Through this series of steps, the algorithm can systematically make partition decisions for each CU to optimize the video encoding process.

[0053] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0054] The above embodiments should be understood to be only used to illustrate the present invention and not to limit the protection scope of the present invention. After reading the contents of the present invention, technicians can make various changes or modifications to the present invention, and these equivalent changes and modifications also fall within the scope defined by the claims of the present invention.

Claims

1. A VVC fast CU division method based on ensemble learning, characterized in that: The following steps are involved: S1. For a given coding unit CU, start encoding from the deepest level and calculate the characteristic attribute value of the current CU, and then go to step S2; S2, determine the size and type of the current CU, if the current CU is a 64×64, 32×32 or 16×16 CU, go to step S3, otherwise, go to step S5; S3. Load the pre-trained integrated learning classifier and use it to predict the partition of the current CU. If the classifier output is 1, it is determined to be a partition, and the current CU is subdivided into 4 sub-CUs, and then return to S1; If the output is -1, it is determined as not to be divided, and the next CU of the current level is processed; if the output is 0, go to step S4; S4, if it is decided to continue to divide the CU, CuDepth increases by 1, the current CU is divided into 4 sub-CUs, and then enters step S2. Otherwise, continue to process the next CU in the current level. When CuDepth reaches 3, it terminates; S5, traverse the partitioning mode in the partitioning list, return to S1 to enter the next depth CU partition; S6. End the current CU division process.

2. According to the VVC fast CU division method based on ensemble learning according to claim 1, it is characterized in that In step S1, starting encoding from the deepest level means starting the encoding process from CuDepth=0, and calculating the characteristic attribute values ​​of CU including entropy, contrast, inverse moment, angular second-order moment and correlation in four directions (0°, 45°, 90°, 135°), a total of 20 features. The calculation formulas for entropy, contrast, inverse moment, angular second-order moment and correlation are as follows: Ent=-∑ i,j P(i,j;d,θ)log2P(i,j;d,θ) (1) ASM=∑ i,j P(i,j;d,θ) (4) COR=∑ i,j (i-μ)(j-μ)P(i,j;d,θ) (5) Among them, P(i,j;d,θ) is the probability of co-occurrence of pixel values ​​i and j given direction θ and distance d, and μ is the mean of the GLCM matrix.

3. The VVC fast CU division method based on ensemble learning according to claim 1, characterized in that: In step S2, judging the current CU size and type means that the 8×8 CU is already the most basic unit in CU encoding and does not need to be further divided. For CUs of different sizes, three DT-SVM models are designed, corresponding to CUs of 64×64 (CuDepth=0), 32×32 (CuDepth=1) and 16×16 (CuDepth=2).

4. The VVC fast CU division method based on ensemble learning according to claim 1, characterized in that: In step S3, a pre-trained integrated learning classifier is loaded, specifically combining the DT-SVM and CNN models as base learners, and the VVC coding database is trained.

5. The VVC fast CU division method based on ensemble learning according to claim 4, characterized in that: In step S3, the training of the decision tree DT uses the C4.5 algorithm, and selects information gain as the indicator of node splitting. Recursive segmentation is applied to the training set to construct a decision tree. Pre-pruning technology is applied to avoid overfitting. The RBF kernel function is selected for SVM training to perform high-dimensional mapping of features. The training data is used to determine the parameters of the SVM. The results of the decision tree DT and the SVM are merged to create a DT-SVM model.

6. The VVC fast CU division method based on ensemble learning according to claim 4, characterized in that: In step S3, the training of the CNN model is to determine the number and size of convolutional layers, the type and location of pooling layers. Design the fully connected layer to ensure that the number of neurons in the output layer matches the classification task. Use the back propagation algorithm to optimize the network weights. Adjust hyperparameters such as learning rate, batch size, number of iterations, etc.

7. The VVC fast CU division method based on ensemble learning according to claim 1, characterized in that: In step S4, this is a recursive partitioning process, which continues until the block size reaches 8×8. The probability value is used to decide whether to continue the partitioning: a probability value greater than 0.5 predicts that the partitioning will continue. A probability value less than 0.5 predicts that the partitioning will be terminated.

8. The VVC fast CU division method based on ensemble learning according to claim 1, characterized in that: In step S5, entering the partition of the next CU specifically means that once the encoding of the current CTU is completed, move to the next CU in the list and repeat the above steps. Through this series of steps, the algorithm can systematically make partition decisions for each CU to optimize the video encoding process.

Images