A method and system for fast decision of coding unit partitioning based on vvc

Abstract

The present application relates to the technical field of video coding, and especially relates to a coding unit division fast decision method and system based on multi-functional video coding, comprising constructing a quadtree division prediction network, using the network to judge whether a coding tree unit needs to perform quadtree division, and obtaining a coding unit after performing the quadtree division; constructing a mixed tree division prediction network, using the network to judge whether the coding unit needs to be divided, stopping division if the coding unit does not need to be divided, otherwise judging whether the coding unit adopts binary tree division or ternary tree division, and taking a smaller coding unit division prediction value in parallel branches as a final prediction result. The present application improves coding efficiency, and can balance the relationship between visual quality and coding efficiency by selecting different threshold values according to specific scene needs.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video coding, and particularly relates to a coding unit partitioning fast decision method and system based on versatile video coding (VVC). BACKGROUND

[0002] In the field of video compression, VVC significantly improves the coding efficiency, but the encoding complexity is also increased. There are many methods to speed up the encoding, and the prediction methods can be roughly divided into two categories: heuristic method and deep learning based method. The heuristic method extracts statistical features, establishes a correlation model, and is used to predict the possible coding unit (Coding Unit, CU) division, and excludes the less likely CU division, but it is difficult for the heuristic method to effectively extract statistical features. However, deep learning can solve this problem by automatically extracting effective features from data. The deep learning based algorithm can be divided into coding tree unit (Coding Tree Unit, CTU) based partition algorithm and CU based partition algorithm. The CTU based partition algorithm refers to predicting the partition structure of a CTU first, and then selecting the possible CU partition according to the partition structure, and skipping the impossible CU partition. The CU based partition method refers to predicting the possible partition mode of the CU, and skipping all impossible partition modes of the CU. In the CTU based method, the partition structure of the CTU is very complex. A CTU partition process contains two stages, in the first stage, each CTU is recursively divided by quad-tree (QT) partition. If QT partition is not used, it enters the second stage and recursively uses MT partition. Since the first stage only predicts whether to use QT partition, the prediction structure is very simple. However, the second stage recursively uses hybrid tree (MT) partition, which contains up to 5 splitting modes, resulting in a very complex partition structure. Therefore, predicting the quad-tree plus multi-type tree (QTMT) structure of a CTU is very complex. For the CU based partition method, in the first stage, a CU is checked by all six splitting modes to obtain the optimal partition mode. When the optimal partition mode is the QT partition mode, the child CU increases 6 partition modes; otherwise, the CU partition enters the second stage. In the second stage, each CU is checked by MT partition, and MT partition contains up to 5 partition modes to obtain the best partition mode. Since the CU based partition algorithm needs to select the candidate partition mode from all 6 modes, too many candidate partition modes may reduce the accuracy of prediction. Secondly, the CU based partition method needs to predict the partition mode of all CUs. However, many CUs will not be adopted, and predicting their partition will waste a lot of unnecessary encoding time. Thirdly, the existing method does not consider the distribution of the partition mode of the CU. Since the partition mode distribution is closely related to the CU partition, the prediction candidate partition mode without considering its distribution may not obtain the best performance. SUMMARY

[0003] In order to improve the efficiency of the coding unit division, the present application proposes a VVC based coding unit division fast decision method, which specifically comprises the following steps:

[0004] A quadtree partition prediction network is constructed to determine whether the coding unit needs to perform quadtree partition, and a coding unit is obtained after performing the quadtree partition;

[0005] A hybrid tree partition prediction network is constructed to determine whether the coding unit needs to be partitioned, if not, the partition is stopped, otherwise, it is determined whether the coding unit adopts binary tree partition or ternary tree partition, and the prediction value of the smaller coding unit in the parallel branch is taken as the final prediction result.

[0006] Further, the feature extraction unit of the quadtree partition prediction network includes six cascaded convolution modules, each convolution module adopts 3x3 convolution kernel, step 2, padding 1 convolution operation for feature mapping, and the features output by the last three convolution layers are respectively predicted by the corresponding prediction units whether to divide the 16x16 coding unit, 32x32 coding unit and 64x64 coding unit.

[0007] Preferably, each prediction module is composed of a cascaded flattening unit, a splicing operation and four fully connected layers, that is, the features output by the convolution module are flattened by the flattening unit, and then the flattened features are spliced with the quantization parameter QP and the size information of the video image to input the cascaded four fully connected layers.

[0008] Further, in the hybrid tree partition prediction unit, the coding unit is divided into multiple size sub-coding units, each sub-prediction unit extracts features of the corresponding size sub-coding unit, and then a sub-prediction unit composed of three binary classification networks predicts the division of each sub-coding unit according to the extracted features, that is, the first binary classification network predicts whether the sub-coding unit is divided, if so, the second binary classification network is input to determine whether to adopt vertical partition or horizontal partition, and the third binary classification network determines whether to adopt binary tree partition or ternary tree partition after performing horizontal partition or vertical partition.

[0009] Preferably, when the feature extraction module of the sub-prediction unit extracts the features of different sub-coding units, the square size sub-coding unit adopts the combination of 2x2, 3x3 and 5x5 convolution kernels for feature extraction, and the matrix size sub-coding unit adopts the combination of 2x2, 3x1 and 5x1 convolution kernels for feature extraction, wherein the step of 2x2 convolution is set to 2, and the steps of other convolutions are set to 1.

[0010] The application also provides a VVC-based coding unit division fast decision system for realizing a VVC-based fast coding unit division method, which comprises a pre-trained quadtree division prediction network and a mixed tree division prediction network.

[0011] Compared with the prior art, the application has the following beneficial effects:

[0012] 1. According to the CTC standard, 20 video sequences are selected from A to E for experiment, and the QP values of the sequences are 22, 27, 32 and 37 respectively. After coding, the TS (time save) is recorded, and the TS represents the time saving rate of coding relative to the original VTM to measure the reduction of complexity; in addition, The delta bit rate (BD-BR) is used to evaluate the RD performance, as shown in Table 1, the BD-BR of the method of the application is 2.68%, but the TS of the comment coding time is 69.55%, which greatly improves the coding efficiency;

[0013] 2. The application can make multi-threshold decision according to application, and is suitable for different types of images and various network conditions, such as Figure 7 and Figure 8 As shown in the table, the application can select different thresholds to balance the relationship between visual quality and coding efficiency, and also can select the best threshold.

[0014] Table 1 Performance evaluation of the algorithm proposed in the application

[0015]

[0016] BRIEF DESCRIPTION OF DRAWINGS

[0017] Figure 1 It is a framework diagram of the VVC-based coding unit division fast decision method of the application;

[0018] Figure 2 It is a structure diagram of the quadtree division prediction network based on CTU division of the application;

[0019] Figure 3 It is a structure diagram of the mixed tree division prediction network based on CU division of the application, wherein figure (a) is the overall structure of the mixed tree division prediction network, figure (b) is the network structure of each feature extraction branch in the mixed tree division prediction network, and figure (c) is the structure of the binary classification network in the mixed tree division prediction network;

[0020] Figure 4Ratio of the CU in the CTU in the QT split mode of the present application;

[0021] Figure 5 QT multi-threshold decision flowchart of the present application;

[0022] Figure 6 MT multi-threshold decision flowchart of the present application;

[0023] Figure 7 BD-BR diagram under QT threshold decision of the present application;

[0024] Figure 8 BD-BR diagram under MT threshold decision of the present application, wherein figure (a) is threshold decision of whether to split, and figure (b) is direction mode threshold decision. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0026] The present application proposes a coding unit division fast decision method based on VVC, specifically including the following steps:

[0027] A quadtree division prediction network is constructed, which is used to determine whether the coding tree unit needs to perform quadtree division, and the coding unit is obtained after performing quadtree division;

[0028] A hybrid tree division prediction network is constructed, which is used to determine whether the coding unit needs to be divided. If it does not need to be divided, the partition is stopped. Otherwise, it is determined whether the coding unit adopts binary tree division or ternary tree division, and the smaller coding unit division prediction value in the parallel branch is taken as the final prediction result.

[0029] In the present embodiment, the coding unit division fast decision method based on VVC includes two stages, wherein:

[0030] In the first stage, the CTU division algorithm only needs to predict whether to use the QT split mode, and its prediction process is very simple. The CU division algorithm needs to predict the candidate split mode from all six split modes, and its prediction process is very complex. Therefore, the present application uses the CTU division algorithm to predict the candidate split mode in this stage;

[0031] In the second stage, the prediction process based on the CTU method is very complex due to the highly complex partitioning structure of the CTU. However, the CU-based partitioning algorithm only needs to predict candidate partitioning patterns for a maximum of five CUs. Obviously, the prediction process based on the CU partitioning algorithm is much simpler than that based on the CTU partitioning algorithm. Therefore, this invention adopts the CU partitioning method to predict candidate partitioning patterns in this stage. Furthermore, since the distribution of partitioning patterns is closely related to the CU partitioning, this invention predicts candidate partitioning patterns based on their distribution.

[0032] In the VVC standard, CU partitioning uses the QTMT structure, which adds more flexibility compared to traditional QT partitioning. Dividing CUs into squares or rectangles makes them more suitable for complex texture structures in videos. This embodiment recursively partitions CUs using a quadtree approach. At each node, a CU can be divided into four smaller sub-blocks; then, MT partitioning can be further applied to the leaf nodes of the QT. In MT prediction, CUs are divided into five partitioning modes: non-split, horizontal binary tree (BTV), vertical binary tree (BTH), horizontal ternary tree (TTH), and vertical ternary tree (TTV). Each CU generates a different number of sub-CUs through MT, and the combination with the lowest RD cost is selected by checking the RD cost of all possible CU combinations, making the process of finding the optimal partitioning structure in MT partitioning very complex. To solve this problem, this invention performs QT partitioning first, followed by MT partitioning, based on the QTMT prediction process.

[0033] This embodiment represents the QT prediction process as a binary classification problem. In the MT prediction process, there are 5 partitioning modes for MT, making the prediction process relatively complex. To predict the partitioning mode for each CU, possible partitioning modes for CUs of different sizes were studied, intra-frame VVC CU partitioning labels were established, and CU partitioning was counted. The data studied in this embodiment comes from the DIV2K, RAISE, and DCID datasets. The image data comes from 204 original video sequences with different resolutions and contents. According to CTC (common test conditions), QP values ​​were set to 22, 27, 32, and 32. A fully inside (AI) configuration was used for testing. All video sequences and images were encoded using the VVC reference software VTM-17.0. The test results are shown in Table 2.

[0034] Table 2 shows the distribution of CU partitions ( / indicates no partitioning was performed).

[0035] Item N_Split BT_H BT_V TT_H TT_V 32×32 30.7％ 28.0％ 25.5％ 8.3％ 7.5％ 32×16 45.7％ 26.2％ 15.0％ 8.4％ 4.7％ 32×8 51.8％ 25.9％ 10.5％ 11.8％ / 32×4 76.3％ 14.6％ / 9.1％ / 16×32 48.0％ 15.9％ 23.4％ 5.0％ 7.7％ 16×16 42.5％ 21.6％ 22.1％ 6.9％ 6.8％ 16×8 64.5％ 19.4％ 10.2％ 5.9％ / 16×4 73.5％ 17.7％ / 8.8％ / 8×32 52.3％ 11.4％ 25.3％ / 11.0％ 8×16 67.3％ 10.6％ 16.5％ / 5.5％ 8×8 73.0％ 14.0％ 13.0％ / / 8×4 88.1％ 11.9％ / / / 4×32 78.8％ / 13.0％ / 8.2％ 4×16 73.6％ / 17.7％ / 8.7％ 4×8 88.5％ / 11.5％ / / average 64.0％ 14.0％ 14.0％ 4.0％ 4.0％

[0036] Based on the test results in Table 1, the following conclusions can be drawn:

[0037] 1. Different CU sizes have different proportions of partitioning patterns, and the choice of CU partitioning pattern is related to the specific CU size;

[0038] 2. The no-partitioning mode accounts for a large proportion of CU partitioning. For example, in most CUs, the no-partitioning mode is dominant, with an average proportion of 64%.

[0039] 3. In the partitioning mode, the proportion of CU partitioning in horizontal binary tree and vertical binary tree modes is much greater than that in vertical ternary tree and horizontal ternary tree modes.

[0040] Based on the above conclusions, the present invention proposes the following solution:

[0041] 1. The distribution of CU partitions is closely related to the size of the CUs. To improve the accuracy of CU partitioning, this invention designs a network structure for each CU. Furthermore, parallel prediction improves prediction efficiency.

[0042] 2. Since the proportion of non-partitioned mode is relatively large and the CUs in non-partitioned mode are relatively simple, it is possible to determine whether the current CU should be partitioned. If the current CU is in non-partitioned mode, the CU partitioning can be terminated in advance; otherwise, the CU partitioning can continue.

[0043] 3. In the partitioning method, the partitioning distribution of binary trees and ternary trees is significantly different. The partitioning of binary trees is much larger than that of ternary trees. Therefore, this invention considers representing the CU partitioning as a binary classification problem of binary trees and ternary trees.

[0044] 4. During the CU partitioning process, the texture differences in the horizontal and vertical directions are also very obvious. This invention divides the CU into horizontal mode and vertical mode.

[0045] Based on the solution proposed in this invention, this embodiment presents a specific implementation process of a fast decision-making method for coding unit partitioning based on VVC, as follows: Figure 1 First, QT prediction is used to partition the CTU into multiple CUs. Then, MT partitioning is used to perform split prediction on each CU. If the prediction result is no partitioning mode, the CU prediction partitioning is terminated early during partitioning. If it is a partitioning mode, the current CU partitioning mode is determined based on the CU direction prediction and binary tree or ternary tree prediction. The final partitioning mode is determined based on the smaller CU partitioning prediction in the parallel branches.

[0046] In this embodiment, a quadtree partitioning prediction network is constructed to determine whether a coding tree unit (CU) needs to be partitioned into a quadtree. After quadtree partitioning, the coding unit is obtained. Specifically, in a 64×64 CU, a set of 2×1×1 features is obtained through the two outputs of a convolutional network and three fully connected layers. These features are then mapped by the last fully connected layer to predict whether the 64×64 CU should be partitioned. In other words, the CU partitioning prediction is represented as a classification problem, where each node receives a field representing a 64×64 CU, used to predict whether the 64×64 CU should be partitioned. Furthermore, during the generation of the 1×1 feature map through convolution, 2×2 and 4×4 feature maps are obtained. The same method can then be used to determine whether to partition 32×32 and 16×16 CUs. Therefore, this invention can use a convolutional neural network to simultaneously obtain feature maps of three different sizes to predict CUs of different sizes.

[0047] Network design such as Figure 2 As shown, firstly, a luma CTU generates feature maps of different sizes through 6 convolutional layers, and divides and predicts CUs of different sizes. In each convolutional step, the number of channels and the dimension of the feature map are increased. The size of the convolutional kernel is 3×3, the stride is 2, and the padding is 1, so that each node of the last three layers of the convolutional feature network can fully reflect the local features of 16×16, 32×32, and 64×64 CUs respectively. Then, a fully connected layer is used to generate predicted labels based on the local features.

[0048] Specifically, the feature extraction unit of the quadtree partitioning prediction network includes six cascaded convolutional modules. Each convolutional module uses a 3×3 convolutional kernel, a stride of 2, and padding of 1 for feature mapping, wherein:

[0049] The prediction network is divided into quadtrees to map the luminance information coding tree unit with 1 channel and a size of 64×64.

[0050] The fourth convolutional module outputs a feature map with 256 channels and a size of 4×4. This feature map is used as the input to the 16×16 prediction module to predict whether to divide it into 16×16 coding units.

[0051] The fifth convolutional module outputs a feature map with 512 channels and a size of 2×2. This feature map is used as the input to the 32×32 prediction module to predict whether to divide it into 32×32 coding units.

[0052] The fifth convolutional module outputs a feature map with 1024 channels and a size of 1×1. This feature map is used as the input of the 64×64 prediction module to predict whether to divide it into 64×64 coding units.

[0053] Each prediction module of the quadtree partitioning prediction network consists of cascaded flattening units, a concatenation operation, and four fully connected layers. Specifically, the flattening unit flattens the features output by the convolutional module, and then the concatenation unit concatenates the flattened features with the quantization parameter QP and the size information of the video image and inputs them into the four cascaded fully connected layers.

[0054] In this embodiment, a hybrid tree partitioning prediction network is constructed, which divides the CTU unit into sub-blocks of multiple sizes, and then predicts all CU partitions in parallel. The hybrid tree partitioning prediction network includes a feature extraction sub-unit and a label prediction unit, and its overall structure is as follows: Figure 3 As shown in (a), the image is simultaneously cut into 13 different CUs of different sizes. These 13 different CUs have 9 different dimensions. These CUs are input into parallel feature extraction subunits. The output of each feature extraction subunit is used to predict the decision results of three modes through the corresponding label prediction unit. If the current prediction is the no-partition mode, the CU partitioning is terminated early at the current stage. Otherwise, the partitioning mode of the current CU is selected according to the directionality of the current CU and the binary tree and ternary tree prediction modes. The current CU is partitioned according to the smaller CU partitioning mode in the parallel branches to obtain the final partitioning mode.

[0055] During the input phase, the size of the CU (Computer Unit) varies significantly, and neural networks of different depths and structures perform better because feature extraction based on different depths and structures is often applicable to different CUs. Therefore, different fixed structures were used in the process of extracting texture structure features. For example... Figure 3 As shown in (b), this embodiment uses a combination of squared convolution kernels and non-squared convolution kernels for different CUs. Specifically, for all feature extraction subunits, this invention uses convolution kernels of size 2×2 and stride 2. Therefore, all kernels are non-overlapping. This non-overlapping convolution is suitable for the size and position of non-overlapping CUs in the final partitioning, because in a 2×2 convolution kernel, the receptive field of the convolution kernel and the possible CUs are the same. In square CUs with an aspect ratio of 1:1, in addition to 2×2 convolution kernels, this invention mainly uses 3×3 and 5×5 convolution kernels; in rectangular CUs with an aspect ratio not of 1:1, this invention mainly uses 3×1 and 5×1 convolution kernels. This is because using square convolution kernels and non-trivial convolution kernels can better extract the texture structure features of square CUs and rectangular CUs respectively. In addition, in order to deepen the network, this embodiment performs squared convolution on the rectangular CUs of M-Net2 and M-Net6 to fully extract texture features; then, the features are input into the fully connected layer to generate probabilities associated with the three modes, thus obtaining CU prediction candidate modes.

[0056] Specifically, such as Figure 3As shown in (b), the sizes of the sub-coding units include 32×32, 32×16, 32×8, 32×4, 16×16, 16×8, 16×4, 8×8, and 8×4, where:

[0057] The sub-coding unit has a size of 32×32. The feature extraction module of the sub-prediction unit M-Net 1 consists of seven concatenated convolutional modules. The convolutional kernels of the seven convolutional modules are 5×1, 2×2, 3×1, 2×2, 3×1, 3×3, and 2×2, respectively, and the strides are 1, 2, 1, 2, 1, 1, and 2, respectively.

[0058] The sub-coding unit has a size of 32×16 or 16×32. The feature extraction module of the sub-prediction unit M-Net 2 consists of seven concatenated convolutional modules. The convolutional kernels of the seven convolutional modules are 5×1, 2×2, 3×1, 2×2, 3×1, 3×3, and 2×2, respectively, and the strides are 1, 2, 1, 2, 1, 1, 2, respectively.

[0059] The sub-coding unit has a size of 32×8 or a size of 8×32. The feature extraction module of the sub-prediction unit is composed of seven concatenated convolutional modules. The convolutional kernels of the seven convolutional modules are 5×1, 2×2, 5×1, 3×1, 2×2, 3×1, and 2×2, and the strides are 1, 2, 1, 1, 2, 1, 2, respectively.

[0060] The sub-coding unit has a size of 32×4 or a size of 4×32. The feature extraction module of the sub-prediction unit M-Net 4 consists of six concatenated convolutional modules. The convolutional kernels of the six convolutional modules are 5×1, 2×2, 5×1, 5×1, 5×1, 2×2, and the strides are 1, 2, 1, 1, 1, 2, respectively.

[0061] The sub-coding unit has a size of 16×16. The feature extraction module of the sub-prediction unit is composed of six concatenated convolutional modules with convolutional kernels of 3×3, 3×3, 2×2, 3×3, 3×3, and 2×2, and strides of 1, 1, 2, 1, 1, and 2, respectively.

[0062] The sub-coding unit has a size of 16×8 or a size of 8×16. The feature extraction module of the sub-prediction unit is composed of five concatenated convolutional modules with convolutional kernels of 5×1, 2×2, 3×1, 3×3, and 2×2, and strides of 1, 2, 1, 1, and 2, respectively.

[0063] The sub-coding unit has a size of 16×4 or a size of 4×16. The feature extraction module of the sub-prediction unit is composed of six concatenated convolutional modules with convolutional kernels of 3×1, 3×1, 2×2, 3×1, 3×1, 2×2 and strides of 1, 1, 2, 1, 1, 2.

[0064] The sub-coding unit has a size of 8×8. The feature extraction module of the sub-prediction unit is composed of four concatenated convolutional modules with convolutional kernels of 3×3, 3×3, 3×3, and 2×2, and strides of 1, 1, 1, and 2, respectively.

[0065] The sub-coding units are either 8×4 or 4×8 in size. The feature extraction module of the sub-prediction unit is composed of four cascaded convolutional modules with convolutional kernels of 3×1, 3×1, 2×2, and 2×2 in sequence, and strides of 1, 1, 2, and 2 in sequence.

[0066] like Figure 3 (c) Each label prediction unit of the hybrid tree partitioning prediction network consists of three binary classification networks. The first binary classification network predicts whether the sub-encoding unit should be partitioned. If so, it inputs the second binary classification network to determine whether to use vertical or horizontal partitioning. After performing horizontal or vertical partitioning, the third binary classification network determines whether to use binary tree partitioning or ternary tree partitioning. Each binary classification network includes flattening operation, concatenation operation, and three fully connected layers. That is, the flattening unit flattens the features output by the convolution module, and then the concatenation unit concatenates the flattened features with the quantization parameter QP and the features transformed by the quantization parameter QP together and inputs them into the three cascaded fully connected layers.

[0067] In this invention, the quadtree partitioning prediction network and the hybrid tree partitioning prediction network can predict CTU partitions and CU partitions, respectively. Then, during RDO encoding, redundancy checks on CUs are skipped to reduce encoding complexity. However, the model's prediction output is not completely accurate; incorrect predictions of CU partitions lead to a decrease in encoding performance. Therefore, this invention also proposes a multi-threshold decision scheme to balance encoding complexity and the RDO process. The multi-threshold scheme consists of two stages: the QT threshold decision scheme and the MT threshold decision scheme. For the quadtree partitioning prediction network and the hybrid tree partitioning prediction network, the probabilities of candidate models can be obtained respectively. The higher the probability value, the more reliable the partitioning result. By setting a threshold, the prediction accuracy of the model can be improved. If the predicted probability is greater than the threshold, the candidate model is considered reliable; conversely, candidate models less than the threshold are unreliable and will not be adopted. However, if the threshold is too high, it will affect the encoding speed; if the threshold is too low, it will significantly reduce encoding efficiency.

[0068] In the QT threshold decision-making stage, such as Figure 4As shown, QT only has a partitioning mode and a non-partitioning mode, with the non-partitioning mode accounting for a large proportion. In CTU, only binary classification of CUs is performed. If the non-partitioning mode is correctly predicted, early termination will occur, which will greatly improve coding performance. Therefore, this invention uses a threshold to allow CUs in CTU to choose between partitioning or not partitioning. The threshold decision is only used for CUs of sizes 64×64, 32×32, and 16×16. The process of the QT threshold decision scheme is described as follows:

[0069] First, predict possible probabilities based on the network's output;

[0070] If the predicted probability does not meet the first-level partitioning conditions, skip the CU partitioning; otherwise, partition the current CU directly, and then partition the CU for the next stage.

[0071] As shown in Table 2, specifically, such as Figure 5 As shown, this embodiment assumes p qti (i = 1, 2, 3) This is the predicted probability at the CTU level, if p qt1 If the value is greater than the set threshold t, then the QT partition is considered reliable and is divided into 64×64 CUs; ​​otherwise, no partitioning is needed, and the current CU partitioning is skipped. Next, a threshold decision is made for the next level of CUs in the CTU, i.e., whether a 64×64 CU needs to be partitioned. If p qt2 If the value exceeds a set threshold, the 64×64 CU will be divided into four 32×32 CUs; ​​otherwise, the current CU division will be skipped. qt3 If the value exceeds the set threshold, the 32×32 CU will be divided into 16×16 CUs; ​​repeat the above operation until all QT partitioning decisions are completed in the CTU.

[0072] like Figure 6 During the MT threshold decision stage, the distribution of MT partitions differs significantly between CU partitioning and non-partitioning, both horizontally and vertically, although the horizontal distribution can be considered a transpose of the vertical distribution. Furthermore, the distributions of BT and TT partitions differ considerably. Therefore, this embodiment sets four thresholds s for CU partitioning. j (j = 1, 2, 3, 4), respectively determining whether the CU is segmented, orientation selection, and BT or TT mode selection. This embodiment assumes p mtk (k = 1, 2, 3, 4, 5) The probability of whether the current CU is divided, the horizontal division direction, the vertical division direction, and the BT and TT modes, specifically:

[0073] If the predicted probability p of CU being segmented is mt1 If the value is greater than the first threshold s1, then the MT partitioning of the current CU is reliable, and the current CU is partitioned; otherwise, the partitioning of the current CU is skipped.

[0074] Then, regarding the directional pattern, there are two candidate patterns, and the probability p of predicting the current CU in the horizontal direction is determined. mt2 If the value is greater than the second threshold s2, then the current CU's MT partition should be directly partitioned in the horizontal direction; otherwise, determine the probability p of predicting the current CU's partition in the vertical direction. mt3 If the value is greater than the second threshold s2, vertical division should be selected; if neither threshold is passed, both candidate modes will be selected, meaning that the current CU needs to be divided both horizontally and vertically.

[0075] Finally, determine the probability p of the current CU choosing the BT mode partitioning. mt4 If the value is greater than the third threshold s3, then the current CU selects the BT mode for MT partitioning; otherwise, determine the probability p of the current CU selecting the TT mode partitioning. mt5 If the value is greater than the fourth threshold s4, then the MT partitioning of the current CU should be selected as TT mode; otherwise, both BT mode partitioning and TT mode partitioning are required.

[0076] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A method for fast decision of VVC-based coding unit partitioning, characterized in that, Specifically comprising the following steps: The quadtree partition prediction network is constructed to determine whether the coding tree unit needs to perform quadtree partition, and the quadtree partition is performed according to the prediction result to obtain the coding unit; the feature extraction unit of the quadtree partition prediction network comprises six convolution modules in cascade, each convolution module adopts a 3*3 convolution kernel, a step of 2 and a padding of 1 to perform feature mapping, wherein: The quadtree partition prediction network is used to map the input luminance information coding tree unit with a channel number of 1 and a size of 64*64; The fourth convolution module outputs a feature map with a channel number of 256 and a size of 4*4, and the feature map is taken as the input of the 16*16 prediction module to predict whether to be divided into a 16*16 coding unit; The fifth convolution module outputs a feature map with a channel number of 512 and a size of 2*2, and the feature map is taken as the input of the 32*32 prediction module to predict whether to be divided into a 32*32 coding unit; The fifth convolution module outputs a feature map with a channel number of 1024 and a size of 1*1, and the feature map is taken as the input of the 64*64 prediction module to predict whether to be divided into a 64*64 coding unit; The hybrid tree partition prediction network is constructed to determine whether the coding unit needs to be divided, if not, the partition is stopped, otherwise, it is determined whether the coding unit adopts binary tree partition or ternary tree partition, and the prediction value of the smaller coding unit in the parallel branch is taken as the final prediction result; In the hybrid tree partition prediction unit, the coding unit is divided into a plurality of sub-coding units of different sizes, each sub-prediction unit extracts features of the corresponding size sub-coding unit, and then a label prediction unit composed of three binary classification networks predicts the division of each sub-coding unit according to the extracted features, that is, the first binary classification network predicts whether the sub-coding unit is divided, if yes, the second binary classification network is input to determine whether vertical division or horizontal division is adopted, and the third binary classification network determines whether binary tree partition or ternary tree partition is adopted after horizontal division or vertical division.

2. The fast decision method for VVC-based CU partition according to claim 1, wherein, Each prediction module of the quadtree partition prediction network comprises a cascaded flattening unit, a splicing operation and four fully connected layers, that is, the features output by the convolution module are flattened by the flattening unit, then the flattened features are spliced with the quantization parameter QP and the size information of the video image to input the cascaded four fully connected layers.

3. The fast decision method for VVC-based CU partition according to claim 1, wherein, When the feature extraction module of the sub-prediction unit extracts features of different sub-coding units, the square size sub-coding unit adopts a combination of 2*2, 3*3 and 5*5 convolution kernels for feature extraction, and the rectangular size sub-coding unit adopts a combination of 2*2, 3*1 and 5*1 convolution kernels for feature extraction, wherein the step of 2*2 convolution is set to 2, and the steps of other convolutions are set to 1.

4. The fast decision method for VVC-based CU partition according to claim 1, wherein, The three binary classification networks of the hybrid tree partition prediction network are provided with four division thresholds, and then it is determined whether to perform division according to the set thresholds, specifically comprising: When the sub-prediction network determines that the sub-coding unit needs to be divided and the prediction probability is greater than the set first threshold, the sub-coding unit is divided, otherwise the division of the coding unit is skipped; If the probability of adopting horizontal direction division is greater than the second threshold value, the sub-encoding unit adopts horizontal direction division; otherwise, if the probability of adopting vertical direction division is greater than the second threshold value, the sub-encoding unit adopts vertical direction division; otherwise, the sub-encoding unit needs to adopt both horizontal direction division and vertical direction division. If the probability of adopting binary tree division is greater than the third threshold value, the sub-encoding unit adopts binary tree division; otherwise, if the probability of adopting ternary tree division is greater than the fourth threshold value, the sub-encoding unit adopts ternary tree division; otherwise, the sub-encoding unit needs to adopt both binary tree division and ternary tree division.

5. The fast decision method for VVC-based CU partition according to claim 1 or 3, characterized in that, The size of the sub-encoding unit includes 32x32, 32x16, 32x8, 32x4, 16x16, 16x8, 16x4, 8x8, and 8x4. The feature extraction module of the sub-prediction unit of the sub-encoding unit with the size of 32x32 is composed of seven convolution modules in cascade, and the convolution kernels of the seven convolution modules are 5x1, 2x2, 3x1, 2x2, 3x1, 3x3, and 2x2 in sequence, and the steps are 1, 2, 1, 2, 1, 1, and 2 in sequence. The feature extraction module of the sub-prediction unit of the sub-encoding unit with the size of 32x16 is composed of seven convolution modules in cascade, and the convolution kernels of the seven convolution modules are 5x1, 2x2, 3x1, 2x2, 3x1, 3x3, and 2x2 in sequence, and the steps are 1, 2, 1, 2, 1, 1, and 2 in sequence. The feature extraction module of the sub-prediction unit of the sub-encoding unit with the size of 32x8 is composed of seven convolution modules in cascade, and the convolution kernels of the seven convolution modules are 5x1, 2x2, 5x1, 3x1, 2x2, 3x1, and 2x2 in sequence, and the steps are 1, 2, 1, 1, 2, 1, and 2 in sequence. The feature extraction module of the sub-prediction unit of the sub-encoding unit with the size of 32x4 is composed of six convolution modules in cascade, and the convolution kernels of the six convolution modules are 5x1, 2x2, 5x1, 5x1, 5x1, and 2x2 in sequence, and the steps are 1, 2, 1, 1, 1, and 2 in sequence. The feature extraction module of the sub-prediction unit of the sub-encoding unit with the size of 16x16 is composed of six convolution modules in cascade, and the convolution kernels of the six convolution modules are 3x3, 3x3, 2x2, 3x3, 3x3, and 2x2 in sequence, and the steps are 1, 1, 2, 1, 1, and 2 in sequence. The feature extraction module of the sub-prediction unit of the sub-encoding unit with the size of 16x8 is composed of five convolution modules in cascade, and the convolution kernels of the five convolution modules are 5x1, 2x2, 3x1, 3x3, and 2x2 in sequence, and the steps are 1, 2, 1, 1, and 2 in sequence. The feature extraction module of the sub-prediction unit of the sub-encoding unit with the size of 16x4 is composed of six convolution modules in cascade, and the convolution kernels of the six convolution modules are 3x1, 3x1, 2x2, 3x1, 3x1, and 2x2 in sequence, and the steps are 1, 1, 2, 1, 1, and 2 in sequence. The size of the sub-encoding unit is 8*8, the feature extraction module of the sub-prediction unit is composed of four convolution modules in cascade, the convolution kernels of the four convolution modules are 3*3, 3*3, 3*3 and 2*2 in turn, and the steps are 1, 1, 1 and 2 in turn; The size of the sub-encoding unit is 8*4, the feature extraction module of the sub-prediction unit is composed of four convolution modules in cascade, the convolution kernels of the four convolution modules are 3*1, 3*1, 2*2 and 2*2 in turn, and the steps are 1, 1, 2 and 2 in turn.

6. The fast decision method for VVC-based CU partition according to claim 1, wherein, The four-tree division prediction network is respectively provided with a division threshold when predicting whether the coding tree unit with the size of 16*16, 32*32 or 64*64 is divided, and when the prediction probability that the corresponding size coding tree unit needs to be divided is greater than the set threshold, the coding tree unit is divided.

7. A VVC-based coding unit partitioning fast decision system, characterized in that, A fast coding unit division method based on VVC is used to realize the method of claim 1, comprising a pre-trained four-tree division prediction network and a hybrid tree division prediction network, first using the pre-trained four-tree division prediction network to judge whether the coding tree unit needs to perform four-tree division, if yes, performing division to obtain the coding unit; the pre-trained hybrid tree division prediction network judges whether the coding unit needs to be divided, if not, the partitioning is stopped, otherwise, it is judged whether the coding unit adopts binary tree division or ternary tree division, and the smaller coding unit division prediction value in the parallel branch is taken as the final prediction result.

Images