A Lightweight Neural Network-Based Method for Early Skipping of VVC Inter-Frame CU Partitioning
By adopting a VVC inter-frame CU partitioning early skipping method based on a lightweight neural network, the problem of high CU partitioning complexity in the H.266/VVC coding standard is solved, achieving significant savings in coding time and a slight increase in bit rate while maintaining coding quality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-09-28
- Publication Date
- 2026-04-03
AI Technical Summary
The introduction of QTMT in the H.266/VVC coding standard increases the complexity of CU partitioning, requiring a fast and efficient method to reduce coding complexity while maintaining coding quality.
A method for skipping CU partitioning between VVC frames in advance is adopted based on a lightweight neural network. By training four neural network models, the feature values are used to predict whether CU partitioning can be skipped in advance, thereby reducing unnecessary CU partitioning patterns and reducing the computational complexity of encoding.
While maintaining essentially the same encoding quality, the encoding time was effectively reduced, saving an average of 39.60% of the encoding time, while the bitrate increased by only 1.6%, achieving a balance between encoding time and bitrate.
Smart Images
Figure CN115941960B_ABST
Abstract
Description
[Technical Field]
[0001] This invention relates to the field of video coding technology, and in particular to a method for early skipping of VVC inter-frame CU partitioning based on a lightweight neural network. [Background Technology]
[0002] With the rapid development of video technology, information disseminated in video format is playing an increasingly important role in people's communication, and video information has become a crucial part of life. The dramatic increase in the number of videos and the emergence of various new and complex videos have brought new challenges to video compression. In early 2020, JVET released a new generation video compression standard—H.266 / VVC. Compared to H.265 / HEVC, H.266 / VVC improves the main coding modules based on a hybrid coding framework and introduces many new coding technologies. VVC adopts a coding unit partitioning based on quadtree + binary tree + ternary tree, supporting more flexible partition shapes. The maximum coding unit size has increased from 64×64 to 128×128. To adapt to highly complex videos, a new motion vector prediction model has been added, which can be used to describe more complex motions and achieve better video compression. Compared to HEVC, VVC significantly improves coding performance, increasing coding efficiency by approximately 40% while maintaining essentially the same video quality.
[0003] The H.266 / VVC test model (VTM) introduces Quadtree with Nested Multi-type Tree (QTMT) partitioning, increasing the number of partitioning methods to six. Specifically, VVC's QTMT partitioning first starts with the current frame, dividing it into multiple CTUs of the same size (with a maximum size of 256×256). Then, each CTU is further partitioned, with each CTU first undergoing quadtree partitioning to obtain four leaf nodes (CUs). Each CU then begins recursively partitioning, including six partitioning methods: no partitioning (DT), quadtree partitioning (QT), horizontal binary tree partitioning (BTH), vertical binary tree partitioning (BTV), horizontal ternary tree partitioning (TTH), and vertical ternary tree partitioning (TTV). Figure 1Six methods for partitioning multiple tree types are presented. VVC sets some partition size constraints, such as a minimum leaf node size of 16×16 for quadtree partitioning, a maximum leaf node size of 64×64 for binary and ternary tree partitioning, and a minimum leaf node size of 4×4. It is also stipulated that quadtree partitioning cannot be performed after binary or ternary tree partitioning. The addition of multiple partitioning methods and rules leads to diverse partitioning results. To obtain the optimal CU partitioning for the current frame, all possible partitioning scenarios need to be traversed, and the RD-cost of each CU partitioning needs to be calculated. Finally, the partitioning method with the minimum RD-cost is selected as the optimal CU partitioning method.
[0004] The introduction of QTMT allows the H.266 / VVC coding standard to be applied to more complex video types, but it also significantly increases the time spent on block partitioning. Therefore, to reduce the complexity of QT partitioning and achieve faster CU partitioning, it is necessary to explore a simple yet efficient method to shorten this time overhead. While maintaining coding quality, skipping unnecessary CU partitioning patterns in advance can effectively reduce coding complexity. [Summary of the Invention]
[0005] This invention discloses a method for early skipping of VVC inter-frame CU partitioning based on a lightweight neural network, which effectively reduces the computational complexity of encoding and saves encoding time while ensuring that the encoding quality remains basically unchanged, thereby solving the technical problems involved in the background art.
[0006] To achieve the above objectives, the technical solution of the present invention is as follows:
[0007] A method for early skipping of VVC inter-frame CU partitioning based on a lightweight neural network includes the following steps:
[0008] Step 1: Data Collection. Encode 32 frames of the same video using the original VVC encoder under different quantization parameters (QP). Train four neural network models (BTH, BTV, TTH, and TTV) according to four partitioning modes: horizontal binary partitioning, vertical binary partitioning, horizontal four-equal partitioning, and vertical four-equal partitioning. Collect data for each neural network model as the training set.
[0009] Step 2, Data Training: Train four neural network models respectively. The dataset extracted from the CU is preprocessed and used as the input of the neural network. Then it goes through two fully connected layers and finally through the output layer to obtain a probability value corresponding to the label. After training, four networks are obtained for early termination of CU partitioning between frames in the VVC encoder.
[0010] Step 3, Model Deployment: Embed the neural network model into the VVC encoder. During the actual encoding process of the VVC encoder, for each CU, extract the corresponding 5 feature values according to the current partitioning mode and input them into the network trained in Step 2 to obtain a prediction of whether the current CU partitioning should be skipped in advance. Based on this prediction, perform subsequent encoding and set up operations to improve the prediction accuracy of the neural network model.
[0011] As a preferred improvement of the present invention, in step one, the data includes a comparison value of the difference between the variance values of sub-blocks, the maximum value of the variance values of sub-blocks, the QP value, the block shape ratio, and the depth.
[0012] As a preferred improvement of the present invention, in step one, data is collected as training sets for different neural network models, specifically including:
[0013] BTH Model: The current CU is divided into horizontal and vertical sub-partitions, and the variances of the residual values of the upper and lower sub-blocks and the left and right sub-blocks are calculated respectively. Then, the absolute values of the differences between the variance values of the upper and lower sub-blocks and the left and right sub-blocks are calculated. The comparison results of the current CU difference, the maximum value of the variance values of the upper and lower sub-blocks, the current QP value, the current CU block shape ratio height, and the current depth are extracted as input information. Whether the cost value of the current partition is the lowest cost value among all partitions is used as the corresponding label.
[0014] BTV model: The current CU is divided into horizontal and vertical sub-divisions, and the variances of the residual values of the upper and lower sub-blocks and the left and right sub-blocks are calculated respectively. Then, the absolute values of the differences between the variance values of the upper and lower sub-blocks and the left and right sub-blocks are calculated. The comparison results of the current CU difference, the maximum value of the variance values of the left and right sub-blocks, the current QP value, the current CU block shape ratio width, and the current depth are extracted as input information. Whether the cost value of the current division is the lowest cost value among all divisions is used as the corresponding label.
[0015] The TTH model divides the current CU into four equal horizontal and four equal vertical partitions, and calculates the variance of the residual values of the upper and lower four sub-blocks and the left and right four sub-blocks, respectively. Then, it calculates the absolute value of the difference between the variance values of the upper and lower four sub-blocks and the absolute value of the difference between the variance values of the left and right four sub-blocks. It extracts the comparison results of the current CU difference, the maximum value of the variance values of the upper and lower four sub-blocks, the current QP value, the current CU block shape ratio height, and the current depth as input information. Whether the cost value of the current partition is the lowest cost value among all partitions is used as the corresponding label.
[0016] TTV Model: The current CU is divided into four equal horizontal and four equal vertical partitions, and the variance values of the residual values of the upper and lower four sub-blocks and the left and right four sub-blocks are calculated respectively. Then, the absolute values of the differences between the variance values of the upper and lower four sub-blocks and the left and right four sub-blocks are calculated. The comparison results of the current CU difference, the maximum value of the variance values of the left and right four sub-blocks, the current QP value, the current CU block shape ratio width, and the current depth are extracted as input information. Whether the cost value of the current partition is the lowest cost value among all partitions is used as the corresponding label.
[0017] As a preferred improvement of the present invention, the substitution value includes six substitution values: quadtree partitioning, horizontal binary tree partitioning, vertical binary tree partitioning, horizontal ternary tree partitioning, and vertical ternary tree partitioning.
[0018] As a preferred improvement of the present invention, in step one, the quantization parameter QP is 22, 27, 32 and 37.
[0019] As a preferred improvement of the present invention, in step two, the number of neurons in each layer of the double fully connected layer is 20.
[0020] As a preferred improvement of the present invention, in step three, the operation of improving the prediction accuracy of the neural network model specifically includes setting a decision to reduce the loss of coding quality: if the current CU horizontal partitioning has been skipped by neural network prediction, the current CU vertical partitioning will no longer be subject to neural network prediction.
[0021] The beneficial effects of this invention are as follows:
[0022] 1. The network model is relatively simple, with four partitions corresponding to four models. The proportion of the total encoding time used for network prediction is negligible.
[0023] 2. The network structure consists of two fully connected layers, allowing the neural network to be directly embedded into VVC testing software, reducing the complexity of network calls;
[0024] 3. The decision threshold is adjustable. By adjusting the decision threshold, a balance can be achieved between saving encoding time and increasing the encoding rate. If a faster encoding speed is required, a larger decision threshold can be set. If a higher encoding quality is required, the decision threshold can be lowered to reduce the skipping of partitions. [Attached Image Description]
[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort, wherein:
[0026] Figure 1 This is a diagram showing six ways to partition a VVC multi-type tree;
[0027] Figure 2 This is the model training diagram of the VVC inter-frame CU partitioning early skipping method based on lightweight neural networks in this invention;
[0028] Figure 3 This is the encoding flowchart of the VVC inter-frame CU partitioning early skipping method based on lightweight neural networks of the present invention.
Detailed Implementation Methods
[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0030] Furthermore, the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are feasible for those skilled in the art. If the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.
[0031] This invention provides a method for early skipping of VVC inter-frame CU partitioning based on a lightweight neural network, comprising the following steps:
[0032] Step 1: Data Collection. Encode 32 frames of the same video using the original VVC encoder under different quantization parameters (QP). Train four neural network models (BTH, BTV, TTH, and TTV) according to four partitioning modes: horizontal binary partitioning, vertical binary partitioning, horizontal four-equal partitioning, and vertical four-equal partitioning. Collect data for each neural network model as the training set.
[0033] Specifically, the quantization parameter QP is 22, 27, 32, and 37, and the data includes the comparison value of the difference between sub-block variances, the maximum value of the sub-block variance, QP value, block size ratio, and depth.
[0034] Data is collected separately for different neural network models as training sets, specifically including:
[0035] BTH Model: The current CU is divided into horizontal and vertical sub-partitions, and the variances of the residual values of the upper and lower sub-blocks and the left and right sub-blocks are calculated respectively. Then, the absolute values of the differences between the variances of the upper and lower sub-blocks (SAD1) and the left and right sub-blocks (SAD2) are calculated. The comparison results of the current CU's differences are extracted (if SAD2 > SAD1, assign 0; otherwise, assign 1), the maximum value of the variances of the upper and lower sub-blocks / 60, the current QP value / 51, the current CU block shape ratio (height / (width + height)), and the current depth / 5 are used as input information. Whether the cost of the current partition is the lowest cost of all partitions is used as the corresponding label.
[0036] BTV Model: The current CU is divided into horizontal and vertical binary sub-partitions, and the variances of the residual values of the upper and lower sub-blocks and the left and right sub-blocks are calculated respectively. Then, the absolute values of the differences between the variances of the upper and lower sub-blocks (SAD1) and the left and right sub-blocks (SAD2) are calculated. The comparison results of the current CU's differences are extracted (if SAD1 > SAD2, assign 0; otherwise, assign 1), the maximum value of the variances of the left and right sub-blocks / 60, the current QP value / 51, the current CU block shape ratio width / (width + height), and the current depth / 5 as input information. Whether the cost of the current partition is the lowest cost of all partitions is used as the corresponding label.
[0037] The TTH model divides the current CU into four equal horizontal and four equal vertical partitions, calculating the variances of the residuals for the top and bottom four sub-blocks and the left and right four sub-blocks, respectively. Then, it calculates the absolute value of the difference between the variances of the top and bottom four sub-blocks (SAD1) and the absolute value of the difference between the variances of the left and right four sub-blocks (SAD2). The input information includes the comparison results of the current CU's differences (0 if SAD2 > SAD1, 1 otherwise), the maximum value of the variances of the top and bottom four sub-blocks divided by 60, the current QP value divided by 51, the current CU block aspect ratio (height / width + height), and the current depth divided by 5. The label is whether the cost of the current partition is the lowest cost among all partitions.
[0038] The TTV model involves dividing the current CU into four equal horizontal and four equal vertical partitions, calculating the variances of the residuals for the top and bottom four sub-blocks and the left and right four sub-blocks, respectively. Then, the absolute values of the differences between the variances of the top and bottom four sub-blocks (SAD1) and the left and right four sub-blocks (SAD2) are calculated. The following input information is extracted: the comparison result of the current CU's differences (0 if SAD1 > SAD2, 1 otherwise), the maximum value of the variances of the left and right four sub-blocks / 60, the current QP value / 51, the current CU block shape ratio (width + height) / 5, and the current depth / 5. Whether the cost of the current partition is the lowest cost among all partitions is used as the corresponding label.
[0039] It should be further explained that the cost values include six types: quadtree partitioning, horizontal binary tree partitioning, vertical binary tree partitioning, horizontal ternary tree partitioning, and vertical ternary tree partitioning.
[0040] Step 2, Data Training: Train four neural network models respectively. The dataset extracted from the CU is preprocessed and used as the input of the neural network. Then it goes through two fully connected layers and finally through the output layer to obtain a probability value corresponding to the label. After training, four networks are obtained for early termination of CU partitioning between frames in the VVC encoder.
[0041] Specifically, each layer of the double fully connected layer has 20 neurons.
[0042] Step 3, Model Deployment: Embed the neural network model into the VVC encoder. During the actual encoding process of the VVC encoder, for each CU, extract the corresponding 5 feature values according to the current partitioning mode and input them into the network trained in Step 2 to obtain a prediction of whether the current CU partitioning should be skipped in advance. Based on this prediction, perform subsequent encoding and set up operations to improve the prediction accuracy of the neural network model.
[0043] It should be noted that the operation to improve the prediction accuracy of the neural network model specifically includes setting a decision to reduce the loss of coding quality: if the current CU horizontal partition has already been skipped by neural network prediction, the current CU vertical partition will no longer be predicted by neural network.
[0044] The following detailed description of the VVC inter-frame CU partitioning early skipping method based on lightweight neural networks provided by the present invention will be based on specific embodiments.
[0045] Please combine Figure 2 and Figure 3 As shown, this embodiment provides a method for early skipping of VVC inter-frame CU partitioning based on a lightweight neural network, including the following steps:
[0046] Step 1: Data collection. Select the video sequence BlowingBubbles and run it for 32 frames using the method described above under four quantization parameters (QP) to collect four datasets corresponding to four neural network models.
[0047] Step 2: Input the four datasets into the neural network for training. After training, four neural network models for early skipping of VVC inter-frame CU partitioning are obtained.
[0048] Step 3: Model Deployment: During the actual VVC encoding process, for each CU, five corresponding feature values are extracted based on the current partitioning mode and input into the pre-trained neural network to obtain a prediction of whether the current CU partitioning should be skipped prematurely. Subsequent encoding is then performed based on this prediction. Specifically:
[0049] Based on the current partitioning method of the CU, extract five feature values of the corresponding neural network model. Before calculating the cost of the current partition, input the five feature values into the corresponding neural network model to obtain a probability value, which is compared with a pre-set decision threshold. If the probability value is greater than the threshold, continue encoding according to the original VTM process. If the probability value is less than the threshold, skip the calculation of the cost of the current partition. If the horizontal partitioning of the current CU (including binary trees and ternary trees) has been skipped by network prediction, the vertical partitioning will not be subject to network prediction and will be encoded according to the original VTM process.
[0050] Since skipping CU partitions that should not be skipped will cause huge losses to the encoding, in order to minimize the loss caused by prediction errors, this invention sets the training threshold and decision threshold of the four neural network models to 0.05.
[0051] The following simulation experiment verifies the coding performance of the VVC inter-frame CU partitioning early skipping method based on a lightweight neural network proposed in this invention.
[0052] To evaluate the feasibility and effectiveness of the proposed method, VTM14.2 was used as the test platform, and tests were conducted independently on a PC with an 11th GenInter® Core™ i7-11700F CPU and 16GB RAM. The test sequences included five resolutions: 416x240 (BlowingBubbles, BQSquare, BasketballPass), 832x480 (RaceHorsesC, PartyScene, BasketballDrill), 1280x720 (Johnny, KristenAndSara), 1920x1080 (Cactus, MarketPlace, BQTerrace), and 3840x2160 (Tango2). The encoding quantization parameters (QP) were set to (22, 27, 32, 37), and the encoding configuration was RA (Random Access). The bitrate variation was used... BD-Rate ) and coding time savings ( TS The performance of the algorithm is measured using ( ). TS Defined as:
[0053]
[0054] This indicates the encoding time of the original test model. This indicates the encoding time after the method of the present invention is applied to the original test model.
[0055] Table 1: Performance comparison results of the present invention and VTM14.2 (unit: %)
[0056]
[0057] Here, TS represents the saving in encoding time compared to the original VTM14.2 method. As can be seen from Table 1, the present invention saves an average of 39.60% in encoding time, while the BD-Rate only increases by 1.6%. In summary, the method provided by the present invention can effectively achieve a balance between saving encoding time and increasing bit rate within the range of acceptable encoding quality degradation for the human eye.
[0058] The beneficial effects of this invention are as follows:
[0059] 1. The network model is relatively simple, with four partitions corresponding to four models. The proportion of the total encoding time used for network prediction is negligible.
[0060] 2. The network structure consists of two fully connected layers, allowing the neural network to be directly embedded into VVC testing software, reducing the complexity of network calls;
[0061] 3. The decision threshold is adjustable. By adjusting the decision threshold, a balance can be achieved between saving encoding time and increasing the encoding rate. If a faster encoding speed is required, a larger decision threshold can be set. If a higher encoding quality is required, the decision threshold can be lowered to reduce the skipping of partitions.
[0062] Although embodiments of the present invention have been disclosed above, they are not limited to the applications listed in the specification and embodiments. They can be applied to various fields suitable for the present invention. Other modifications can be easily made by those skilled in the art. Therefore, without departing from the general concept defined by the claims and their equivalents, the present invention is not limited to the specific details and the illustrations shown and described herein.
Claims
1. A method for early skipping of VVC inter-frame CU partitioning based on a lightweight neural network, characterized in that, Includes the following steps: Step 1: Data Collection. Encode 32 frames of the same video using the original VVC encoder under different quantization parameters (QP). Train four neural network models (BTH, BTV, TTH, and TTV) according to four partitioning modes: horizontal binary partitioning, vertical binary partitioning, horizontal four-equal partitioning, and vertical four-equal partitioning. Collect data for each neural network model as the training set. This data includes comparisons of sub-block variance differences, maximum sub-block variance values, QP values, block size ratios, and depths. Specifically, the data collected for each neural network model as the training set includes: BTH Model: The current CU is divided into horizontal and vertical sub-partitions, and the variances of the residual values of the upper and lower sub-blocks and the left and right sub-blocks are calculated respectively. Then, the absolute values of the differences between the variance values of the upper and lower sub-blocks and the left and right sub-blocks are calculated. The comparison results of the current CU difference, the maximum value of the variance values of the upper and lower sub-blocks, the current QP value, the current CU block shape ratio height, and the current depth are extracted as input information. Whether the cost value of the current partition is the lowest cost value among all partitions is used as the corresponding label. BTV model: The current CU is divided into horizontal and vertical sub-divisions, and the variances of the residual values of the upper and lower sub-blocks and the left and right sub-blocks are calculated respectively. Then, the absolute values of the differences between the variance values of the upper and lower sub-blocks and the left and right sub-blocks are calculated. The comparison results of the current CU difference, the maximum value of the variance values of the left and right sub-blocks, the current QP value, the current CU block shape ratio width, and the current depth are extracted as input information. Whether the cost value of the current division is the lowest cost value among all divisions is used as the corresponding label. The TTH model divides the current CU into four equal horizontal and four equal vertical partitions, and calculates the variance of the residual values of the upper and lower four sub-blocks and the left and right four sub-blocks, respectively. Then, it calculates the absolute value of the difference between the variance values of the upper and lower four sub-blocks and the absolute value of the difference between the variance values of the left and right four sub-blocks. It extracts the comparison results of the current CU difference, the maximum value of the variance values of the upper and lower four sub-blocks, the current QP value, the current CU block shape ratio height, and the current depth as input information. Whether the cost value of the current partition is the lowest cost value among all partitions is used as the corresponding label. The TTV model divides the current CU into four equal horizontal and four equal vertical partitions, and calculates the variance of the residual values of the upper and lower four sub-blocks and the left and right four sub-blocks respectively. Then, it calculates the absolute value of the difference between the variance values of the upper and lower four sub-blocks and the absolute value of the difference between the variance values of the left and right four sub-blocks. It extracts the comparison result of the current CU difference, the maximum value of the variance values of the left and right four sub-blocks, the current QP value, the current CU block shape ratio width, and the current depth as input information. Whether the cost value of the current partition is the lowest cost value among all partitions is used as the corresponding label. Step 2, Data Training: Train four neural network models respectively. The dataset extracted from the CU is preprocessed and used as the input of the neural network. Then it goes through two fully connected layers and finally through the output layer to obtain a probability value corresponding to the label. After training, four networks are obtained for early termination of CU partitioning between frames in the VVC encoder. Step 3, Model Deployment: Embed the neural network model into the VVC encoder. During the actual encoding process of the VVC encoder, for each CU, extract the corresponding 5 feature values according to the current partitioning mode and input them into the network trained in Step 2 to obtain a prediction of whether the current CU partitioning should be skipped in advance. Based on this prediction, perform subsequent encoding and set up operations to improve the prediction accuracy of the neural network model.
2. The VVC inter-frame CU partitioning advance skipping method based on lightweight neural networks as described in claim 1, characterized in that, The cost values include six types: quadtree partitioning, horizontal binary tree partitioning, vertical binary tree partitioning, horizontal ternary tree partitioning, and vertical ternary tree partitioning.
3. The VVC inter-frame CU partitioning advance skipping method based on lightweight neural networks as described in claim 1, characterized in that, In step one, the quantization parameters QP are 22, 27, 32, and 37.
4. The VVC inter-frame CU partitioning advance skipping method based on lightweight neural networks as described in claim 1, characterized in that, In step two, each layer of the double fully connected layer has 20 neurons.
5. The VVC inter-frame CU partitioning advance skipping method based on lightweight neural networks as described in claim 1, characterized in that, In step three, the operation to improve the prediction accuracy of the neural network model specifically includes setting a decision to reduce the loss of coding quality: if the current CU horizontal partition has already been skipped by neural network prediction, the current CU vertical partition will no longer be subject to neural network prediction.