CU partition mode prediction method for VVC coding unit and video coding device

Abstract

The invention provides a CU partition mode prediction method for a VVC coding unit and video coding equipment, and relates to the technical field of video coding. The prediction method comprises the steps that a CNN-RNN hybrid neural network model is constructed and trained, the CNN-RNN hybrid neural network model comprises a CNN module and an RNN module, the CNN module is used for extracting local texture features of a CU, and the RNN module is used for capturing a time sequence dependency relationship of the CU; inputting a to-be-coded CU into the CNN-RNN hybrid neural network model, and outputting a prediction result of a CU partition mode and a confidence score of the prediction result; based on the confidence score, the prediction result is selected as the final CU partition mode, or the RDO search is performed and the result of the RDO search is selected as the final CU partition mode. According to the method, the local texture features and the time sequence dependency relationship of the CU can be effectively extracted, and a final CU partition mode is selected according to the confidence score predicted by the model. On the premise of ensuring the coding efficiency, the calculation complexity is reduced, and the coding speed is improved.

Description

Technical Field

[0001] This invention relates to the field of video coding technology, and in particular to a CU partitioning mode prediction method and video coding device for VVC coding units. Background Technology

[0002] With the continuous improvement of video resolution and quality requirements, efficient compression of video coding technology under limited bandwidth and storage space has become particularly important. VVC (H.266), as the latest video coding standard, introduces a more flexible quadtree plus type-tree (QTMT) partitioning structure compared to its predecessor, HEVC (H.265), significantly improving video compression efficiency. However, this also leads to a significant increase in coding complexity, especially in the CU partitioning pattern prediction process. The traditional Rate-Distortion Optimization (RDO) method requires traversing all possible CU partitioning patterns, resulting in extremely high computational costs and becoming a bottleneck for improving coding efficiency.

[0003] In existing technologies, some deep learning-based methods attempt to predict CU partitioning patterns using convolutional neural networks (CNNs), thereby reducing the computational complexity of traversal search. However, pure CNN models are only good at capturing local texture information of images and cannot fully utilize the temporal dependencies between video frames, thus their performance in complex scenes remains limited.

[0004] Therefore, it is necessary to improve the existing CU partitioning pattern prediction technology to overcome its shortcomings. Summary of the Invention

[0005] To overcome the high computational complexity caused by complex CU partitioning pattern prediction in VVC video coding, one objective of this invention is to provide a CU partitioning pattern prediction method for VVC coding units. This method uses a CNN-RNN hybrid neural network model, combining local texture features and temporal dependencies of the CU to predict the CU partitioning pattern. The final CU partitioning pattern is selected based on the confidence score of the model prediction. This reduces computational complexity and increases coding speed while maintaining coding efficiency, thus overcoming the high computational complexity problem in existing technologies.

[0006] A method for predicting CU partitioning patterns for VVC coding units includes:

[0007] Step 100: Construct and train a CNN-RNN hybrid neural network model, which includes a CNN module and an RNN module. The CNN module is used to extract local texture features of the CU, and the RNN module is used to capture the temporal dependencies of the CU.

[0008] Step 200: Input the CU to be encoded into the CNN-RNN hybrid neural network model, and output the prediction result of the CU partitioning pattern and the confidence score of the prediction result;

[0009] Step 300: Based on the confidence score, select either the prediction result as the final CU partitioning mode, or perform an RDO search and use the result of the RDO search as the final CU partitioning mode.

[0010] In some embodiments, step 300 includes:

[0011] The confidence score is compared with a first preset threshold.

[0012] If the confidence score is greater than or equal to the first preset threshold, then the prediction result is selected as the final CU partitioning mode.

[0013] Otherwise, perform an RDO search and use the result of the RDO search as the final CU partitioning pattern.

[0014] A decision-making mechanism based on confidence scores is implemented by using a first preset threshold and a confidence score. When the confidence score of the CNN-RNN hybrid neural network model's prediction result is high, it indicates that the model is relatively certain about the prediction result, and the prediction result can be directly used as the final CU partitioning pattern, avoiding the execution of RDO search, thereby saving computational resources and improving encoding speed. When the confidence score is low, RDO search is performed to ensure the accuracy of the CU partitioning pattern, achieving a balance between encoding efficiency and computational complexity.

[0015] In some implementations, performing the RDO search includes:

[0016] The confidence score is compared with a second preset threshold.

[0017] If the confidence score is greater than or equal to the second preset threshold, then a fast RDO search is performed.

[0018] Otherwise, perform a full RDO search.

[0019] By comparing the confidence score and a second preset threshold, it can be determined whether to perform a fast RDO search. When the confidence score is high, performing a fast RDO search can reduce the search range and lower computational complexity; while when the confidence score is low, performing a full RDO search ensures the accuracy of the CU partitioning pattern, achieving a more refined allocation of computational resources and further improving coding efficiency.

[0020] In some implementations, training the CNN-RNN hybrid neural network model includes:

[0021] VVC test video sequences were obtained from publicly available datasets;

[0022] The VVC test video sequence was fully intra-frame encoded using the VTM source code to obtain the encoding information and the original reconstructed frames.

[0023] Extract CUs of different sizes and their corresponding partitioning patterns from the original reconstructed frames;

[0024] The CUs are classified according to their size, and a sample set of CU partitioning patterns is constructed.

[0025] The CU data in the sample set are normalized and resized to obtain the training set;

[0026] The CNN-RNN hybrid neural network model was trained using the training set.

[0027] By using publicly available VVC test video sequences and performing full intra-frame encoding using the VTM source code, a large number of character sets (CUs) and their corresponding partitioning patterns can be obtained, enabling the construction of a high-quality training dataset. Normalizing and resizing the CU data can improve the efficiency and stability of model training. Using the constructed training set to train a CNN-RNN model allows the model to learn the correlation between video content features and CU partitioning patterns, thereby achieving accurate CU partitioning pattern prediction.

[0028] In some implementations, the different sizes of CUs include CUs with dimensions of 16×16, 32×32, and 64×64, and the CU partitioning modes include no partitioning, quadtree partitioning, horizontal binary tree partitioning, vertical binary tree partitioning, horizontal ternary tree partitioning, and vertical ternary tree partitioning.

[0029] By specifying CU sizes of 16×16, 32×32, and 64×64, commonly used CU sizes in VVC are covered, allowing the model to learn features of CUs of different sizes. Specifying CU partitioning patterns, including no partitioning, quadtree partitioning, horizontal / vertical binary tree partitioning, and horizontal / vertical ternary tree partitioning, covers all possible CU partitioning patterns in VVC, enabling the model to predict all types of partitioning patterns, thereby improving the model's applicability and prediction accuracy.

[0030] In some implementations, the CNN module includes:

[0031] At least two 3×3 convolutional layers, followed by a ReLU activation function;

[0032] At least two 2×2 max pooling layers.

[0033] A 3×3 convolutional layer can effectively extract local texture features, and the subsequent ReLU activation function can enhance the non-linear expressive power of the model. The max pooling layer can reduce the dimensionality of the feature map and reduce the amount of computation.

[0034] In some implementations, the RNN module is a bidirectional LSTM network.

[0035] By using a bidirectional LSTM network, both past and future information of the Cu (Cumulative Component) can be considered simultaneously, capturing a longer range of temporal dependencies and thus more accurately predicting Cumulative partitioning patterns. In particular, for video sequences with intense motion, bidirectional LSTM can better capture motion information and improve prediction accuracy.

[0036] In some implementations, the CNN-RNN hybrid neural network model is trained using a cross-entropy loss function, which is L = (1 - ε)H(p,q) + εH(p,u), where p is the model's predicted logits output, q is the true label, u is a uniform distribution, and ε is a smoothing coefficient.

[0037] The cross-entropy loss function effectively measures the difference between the probability distribution predicted by the model and the true label. This allows the model to learn a more accurate probability distribution of CU partitioning patterns, thereby improving prediction accuracy.

[0038] In some implementations, the confidence score is the maximum probability value in the CU partitioning pattern probability distribution output by the CNN-RNN hybrid neural network model, and the first preset threshold and the second preset threshold are dynamically adjusted according to the complexity of the video sequence, and the first preset threshold and the second preset threshold are positively correlated with the complexity of the video sequence.

[0039] The confidence score is determined by selecting the maximum probability value from the probability distribution of the CU partitioning pattern output by the model, reflecting the model's certainty about the prediction results. Dynamically adjusting the first and second preset thresholds based on the complexity of the video sequence allows the model to better adapt to different types of video content. For more complex video sequences, the threshold can be lowered, favoring RDO search to ensure the accuracy of the CU partitioning pattern; for less complex video sequences, the threshold can be raised, favoring the model's prediction results to reduce computational complexity, achieving a good balance across video sequences of varying complexity.

[0040] A second objective of this invention is to provide a video encoding device, including a processor and a memory, wherein the memory stores instructions, and when the instructions are executed by the processor, they perform the CU partitioning mode prediction method for the VVC encoding unit as described above.

[0041] By implementing the CU partitioning pattern prediction method described above for the VVC coding unit, the video encoding device can achieve efficient CU partitioning pattern prediction, reduce the computational complexity of the VVC encoder, and improve coding efficiency.

[0042] The beneficial effects of this invention are as follows:

[0043] This invention provides a method for predicting CU partitioning patterns in VVC coding units. The method involves constructing and training a CNN-RNN hybrid neural network model, inputting the CU to be encoded into the trained CNN-RNN hybrid neural network model, and making a decision based on confidence scores to determine the CU partitioning pattern. By fusing the local texture features extracted by the CNN and the temporal dependencies captured by the RNN, a more comprehensive understanding of the video content can be achieved, thus predicting the CU partitioning pattern more accurately. Compared to traditional methods that rely entirely on RDO search to determine the CU partitioning pattern, this method can significantly reduce computational complexity and improve encoding speed while maintaining encoding efficiency, especially when processing video sequences with complex textures or rapid motion. By constructing and training a hybrid neural network model containing CNN and RNN modules, the local texture features and temporal dependencies of the CU can be effectively extracted. Based on the confidence score of the model prediction, the method adaptively selects to use the prediction result or perform an RDO search to determine the final CU partitioning pattern. When the confidence score is high, the model prediction result can be used directly, avoiding the time-consuming RDO search, thus achieving a balance between encoding speed and encoding efficiency. Attached Figure Description

[0044] Figure 1 This is a flowchart of the CU partitioning pattern prediction method for VVC coding units provided in this application;

[0045] Figure 2 This is an overall network architecture diagram of the CNN-RNN hybrid neural network model provided in the embodiments of this application.

[0046] Figure 3 This is a graph showing the relationship between the first and second preset thresholds and BDBR provided in the embodiments of this application. Detailed Implementation

[0047] The preferred embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. Although preferred embodiments of the present invention are shown in the accompanying drawings, it should be understood that the present invention can be implemented in various forms and should not be limited by the embodiments described herein. Rather, these embodiments are provided to make the present invention more thorough and complete and to fully convey the scope of the present invention to those skilled in the art.

[0048] Example

[0049] like Figure 1-Figure 2 As shown, this embodiment provides a CU partitioning pattern prediction method for a VVC coding unit, which includes the following steps:

[0050] Step 100: Construct and train a CNN-RNN hybrid neural network model, which includes a CNN module and an RNN module. The CNN module is used to extract local texture features of the CU, and the RNN module is used to capture the temporal dependencies of the CU.

[0051] Step 200: Input the CU to be encoded into the CNN-RNN hybrid neural network model, and output the prediction result of the CU partitioning pattern and the confidence score of the prediction result;

[0052] Step 300: Based on the confidence score, select either the prediction result as the final CU partitioning mode, or perform an RDO search and use the result of the RDO search as the final CU partitioning mode.

[0053] VVC stands for Versatile Video Coding. VVC is the latest generation video coding standard, also known as H.266, designed to provide more efficient video compression in various application scenarios.

[0054] CU stands for Coding Unit. The CU is the basic coding block in VVC, and it can be flexibly partitioned according to the complexity of the video content to achieve higher compression efficiency.

[0055] CU partitioning mode refers to the way a CU is divided into smaller sub-blocks;

[0056] The confidence score is a measure of how certain the model is about the predicted outcome.

[0057] RDO stands for Rate-Distortion Optimization. RDO search is a technique used to select the optimal coding parameters in video coding, aiming to minimize distortion at a given bit rate, or minimize bit rate under a given distortion constraint. RDO search typically requires traversing multiple possible combinations of coding parameters, thus resulting in high computational complexity.

[0058] Furthermore, step 300 includes:

[0059] Step 310: Compare the confidence score with the first preset threshold T1;

[0060] Step 320: If the confidence score is greater than or equal to the first preset threshold T1, then select the prediction result as the final CU partitioning mode;

[0061] Step 330: Otherwise, perform RDO search and use the result of the RDO search as the final CU partitioning mode.

[0062] Furthermore, the performing of the RDO search includes:

[0063] Step 331: Compare the confidence score with the second preset threshold T2;

[0064] Step 332: If the confidence score is greater than or equal to the second preset threshold T2, then select to perform fast RDO search;

[0065] Step 333: Otherwise, perform full RDO search.

[0066] Among them, full RDO search (Full RDO Search) will traverse all possible combinations of coding parameters. For the CU partitioning mode, it will try all possible splitting methods. It is necessary to evaluate all possible candidate solutions, so the computational complexity is the highest. Full RDO search can find the global optimal solution and ensure the best coding efficiency. However, the amount of calculation is large and the coding speed is slow, especially in high-resolution videos and complex scenarios.

[0067] Fast RDO search (Fast RDO Search) only searches a part of candidate coding parameter combinations. For example, it only considers a limited number of CU partitioning modes or uses a larger search step size. It reduces the number of candidate solutions that need to be evaluated, so the computational complexity is lower than that of full RDO search and the coding speed is faster. However, it cannot guarantee to find the global optimal solution, which may lead to a slightly lower coding efficiency than full RDO search.

[0068] When the confidence score is relatively high, it means that the model is relatively certain about the prediction result. Therefore, fast RDO search can be used to speed up the coding speed.

[0069] When the confidence score is relatively low, it means that the model is not very certain about the prediction result. Therefore, full RDO search needs to be used to ensure the coding efficiency.

[0070] Specifically, when the confidence score ≥ T1; directly adopt the CU partitioning mode predicted by the CNN-RNN model. At this time, any RDO search is skipped to reduce the amount of calculation.

[0071] When T2 ≤ similarity score < T1: Perform fast RDO search, which is suitable for use in cases where the video content complexity is medium.

[0072] When the similarity score < T2: Perform a full RDO search to ensure the accuracy of CU partitioning, especially suitable for complex scenarios.

[0073] Specifically, the confidence score is the maximum probability value in the probability distribution of the CU partitioning pattern output by the CNN-RNN hybrid neural network model. The first preset threshold and the second preset threshold are dynamically adjusted according to the complexity of the video sequence, and the first preset threshold and the second preset threshold are positively correlated with the complexity of the video sequence.

[0074] More specifically, in a video sequence with high complexity, the threshold adjustment tends to adopt a more refined RDO search strategy; in a video sequence with low complexity, the threshold adjustment makes more use of the prediction results of the CNN-RNN model to reduce the computational complexity.

[0075] To evaluate the complexity of a video sequence, the following methods can be used:

[0076] 1. Motion estimation: Evaluate the motion intensity of the video sequence through optical flow technology. Larger motion indicates that the changes between video frames are剧烈, and the complexity is higher.

[0077] 2. Texture analysis: The texture intensity of the image can be obtained by calculating the gradient or Laplacian operator of the image. For example, use the Sobel operator to perform edge detection on the CU, and the area with more edges is regarded as the part with higher complexity.

[0078] 3. Frequency domain analysis: Transform the video frame to the frequency domain through Fourier transform or wavelet transform, and analyze the frequency domain information. The larger the proportion of high-frequency components, the more complex the texture of the video, and the higher the complexity.

[0079] 4. Intra-frame change rate: Judge the complexity of the video sequence by analyzing the differences between adjacent frames. The greater the difference, the higher the motion complexity of the sequence.

[0080] Furthermore, the training of the CNN-RNN hybrid neural network model includes:

[0081] Step 110: Obtain VVC test video sequences from a public dataset;

[0082] Step 120: Perform full-frame intra coding on the VVC test video sequence using the VTM source code to obtain coding information and original reconstructed frames;

[0083] Step 130: Extract CUs of different sizes and their corresponding partitioning patterns from the original reconstructed frames;

[0084] Step 140: Classify the CUs according to the size of the CUs and construct a sample set of CU partitioning patterns;

[0085] Step 150: Normalize and resize the CU data in the sample set to obtain the training set;

[0086] Step 160: Train the CNN-RNN hybrid neural network model using the training set.

[0087] Specifically, the different sizes of CUs include CUs with sizes of 16×16, 32×32 and 64×64, and the CU partitioning modes include no partitioning, quadtree partitioning, horizontal binary tree partitioning, vertical binary tree partitioning, horizontal ternary tree partitioning and vertical ternary tree partitioning.

[0088] More specifically, the CNN module includes:

[0089] At least two 3×3 convolutional layers, followed by a ReLU activation function;

[0090] At least two 2×2 max pooling layers.

[0091] More specifically, the RNN module is a bidirectional LSTM network.

[0092] In this embodiment, the CNN-RNN model includes the following modules:

[0093] I. Local Texture Feature Extraction Layer: This layer extracts local spatial texture features of the CU (Cubic Texture) using a Convolutional Neural Network (CNN), including:

[0094] Convolutional layer 1: Uses 3×3 convolutional kernels, 64 filters, ReLU activation to extract preliminary texture features;

[0095] Pooling layer 1: 2×2 max pooling layer, which reduces feature map size and increases receptive field;

[0096] Convolutional layer 2: Uses 3×3 convolutional kernels, 128 filters, ReLU activation to extract deeper local texture features;

[0097] Pooling layer 2: 2×2 max pooling, further reducing the feature map size and computational complexity;

[0098] Output: The CU feature maps of different sizes are uniformly adjusted to 32×32 for subsequent temporal feature processing.

[0099] II. Global Temporal Feature Extraction Layer: This layer extracts the temporal dependency features of CUs (Complex Units) between video frames using a bidirectional LSTM (Bi-LSTM) network, including:

[0100] Bidirectional LSTM layer: Uses 128 hidden units to capture the temporal dependencies between video frames, and processes temporal information in both forward and backward directions;

[0101] Time step: The input is a sequence of video frames, and the time step is equal to the number of frames;

[0102] Output: Temporal feature representation of the bidirectional LSTM output.

[0103] Third, the feature fusion layer fuses the local texture features extracted by CNN and the temporal features extracted by RNN. The local texture features are combined with the global temporal features through feature concatenation. The ReLU activation function is used to enhance the non-linear expressive power of the fused features.

[0104] Fourth, after the feature fusion layer, there is a 1×1 convolutional layer: using 1×1 convolutional kernels to map the fused features to the number of CU partition patterns. The number of convolutional kernels is equal to the number of partition pattern categories, and the Softmax activation function is used to output the probability distribution of each partition pattern.

[0105] The model's output includes:

[0106] Output prediction layer: Predicts the partitioning pattern of each CU through the output of a 1×1 convolution;

[0107] Loss function: Cross-entropy loss is used to optimize the classification task.

[0108] More specifically, the CNN-RNN hybrid neural network model is trained using the cross-entropy loss function, which is L = (1 - ε)H(p,q) + εH(p,u), where p is the model's predicted logits output, q is the true label, u is uniformly distributed, and ε is the smoothing coefficient. During the training process of the CNN-RNN hybrid neural network model, data augmentation techniques are used to perform data augmentation operations on the sample set to improve the model's generalization ability.

[0109] After training, the CNN-RNN hybrid neural network model is compressed and converted to a hardware-friendly format for efficient deployment in the VTM inference module. The compressed model weights and parameters are then converted to a format loadable by the VTM inference framework. The converted CNN-RNN model is then ported and integrated into the inference module of the VVC reference software VTM to enable model inference and CU decision-making functions.

[0110] As shown in Table 1, by testing the encoding performance and comparing it with the original VTM, a trade-off can be achieved between encoding time savings and encoding efficiency by setting different thresholds. This reduces the complexity of VVC encoding while maintaining high encoding efficiency.

[0111] Table 1

[0112]

[0113] Among them, under the Fast Threshold (FT), the first preset threshold is 0.65, and the second preset threshold is 0.55; under the Faster Threshold (FsT), the first preset threshold is 0.50, and the second preset threshold is 0.30.

[0114] The working process in this embodiment:

[0115] 1. Offline training stage:

[0116] Obtain the VVC test video sequence from the public dataset, perform full intra-frame encoding using the VTM source code, and obtain the original reconstructed frames.

[0117] Extract CUs of different sizes (16×16, 32×32, 64×64) and their corresponding partition modes (no split, quadtree split, horizontal / vertical binary tree split, horizontal / vertical ternary tree split) from the reconstructed frames to construct a training sample set.

[0118] Perform data augmentation on the sample set, and perform normalization and size adjustment.

[0119] Use the augmented dataset to train the CNN-RNN hybrid neural network model. The CNN module extracts the local texture features of the CU, and the RNN module captures the temporal dependence relationship of the CU.

[0120] 2. Online encoding stage:

[0121] Input the CU to be encoded into the trained CNN-RNN hybrid neural network model to obtain the prediction result and confidence score of the CU partition mode.

[0122] Make a decision based on the confidence score and the preset thresholds T1 and T2: If the confidence score ≥ T1, directly adopt the prediction result of the CU partition mode predicted by the model and skip the RDO search; if T2 ≤ confidence score < T1, perform a fast RDO search. If the confidence score < T2, perform a full RDO search.

[0123] Use the finally determined CU partition mode for encoding.

[0124] The CU partitioning pattern prediction method for VVC coding units provided in this embodiment significantly reduces the computational cost of RDO search by utilizing a CNN-RNN hybrid neural network model to predict CU partitioning patterns and combining confidence scoring and a dynamic thresholding mechanism. This reduces the computational complexity of VVC coding. Especially when the confidence score is high, the RDO search can be skipped directly, saving significant computational resources. With high confidence, the CNN-RNN hybrid neural network model can accurately predict the optimal partitioning pattern of the CU, avoiding the possibility of the RDO search getting trapped in local optima, thus improving coding efficiency. The dynamic thresholding mechanism adaptively adjusts the RDO search strategy according to the complexity of the video content, ensuring coding quality in complex scenarios and improving coding speed in simple scenarios, achieving a balance between coding speed and efficiency. By reducing the computational cost of RDO search, VVC coding speed can be significantly improved, especially in high-resolution videos and complex scenarios.

[0125] Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of the components and steps set forth in these embodiments do not limit the scope of this application. Any specific values ​​in all examples shown and discussed herein should be interpreted as merely exemplary and not as limitations. Therefore, other examples of exemplary embodiments may have different values. It should be noted that similar reference numerals and letters in the following figures denote similar items; therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0126] Furthermore, it should be noted that the use of terms such as "first" and "second" is merely for ease of distinction, and unless otherwise stated, these terms have no special meaning and therefore should not be construed as limiting the scope of protection of this application.

[0127] The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that various modifications and variations of the present invention are possible. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention are intended to be within the scope of protection of the present invention.

Claims

1. A CU partition mode prediction method for a VVC coding unit, characterized in that: include: Step 100: construct and train a CNN-RNN hybrid neural network model, wherein the CNN-RNN hybrid neural network model includes a CNN module and an RNN module, wherein the CNN module is used to extract local texture features of the CU, and the RNN module is used to capture the temporal dependency of the CU; Step 200: inputting the CU to be encoded into the CNN-RNN hybrid neural network model, and outputting the prediction result of the CU partition mode and the confidence score of the prediction result; Step 300: Based on the confidence score, select the prediction result as the final CU partition mode, or perform an RDO search and use the result of the RDO search as the final CU partition mode.

2. The CU partition mode prediction method for a VVC coding unit according to claim 1, characterized in that: The step 300 includes: comparing the confidence score to a first preset threshold; If the confidence score is greater than or equal to the first preset threshold, selecting the prediction result as the final CU partition mode; Otherwise, an RDO search is performed, and the result of the RDO search is used as the final CU partition mode.

3. The CU partition mode prediction method for a VVC coding unit according to claim 2, characterized in that: The performing of the RDO search comprises: comparing the confidence score to a second preset threshold; If the confidence score is greater than or equal to the second preset threshold, selecting to perform a fast RDO search; Otherwise, a full RDO search is performed.

4. The CU partition mode prediction method for a VVC coding unit according to any one of claims 1 to 3, characterized in that: The training of the CNN-RNN hybrid neural network model includes: Get VVC test video sequences from public datasets; Perform full-frame intra-coding on the VVC test video sequence using the VTM source code to obtain coding information and original reconstructed frames; Extracting CUs of different sizes and their corresponding partition modes from the original reconstructed frame; Classify CUs according to their sizes and construct a sample set of CU partition patterns; Normalizing and resizing the CU data in the sample set to obtain a training set; The training set is used to train the CNN-RNN hybrid neural network model.

5. The method for predicting CU partition mode of a VVC coding unit according to claim 4, characterized in that: The CUs of different sizes include CUs of sizes 16×16, 32×32 and 64×64, and the CU partition modes include no partitioning, quadtree partitioning, horizontal binary tree partitioning, vertical binary tree partitioning, horizontal ternary tree partitioning and vertical ternary tree partitioning.

6. The method for predicting CU partition mode of a VVC coding unit according to claim 1, characterized in that: The CNN module includes: At least two 3×3 convolutional layers followed by a ReLU activation function; At least two 2×2 max pooling layers.

7. The method for predicting CU partition mode of a VVC coding unit according to claim 1, characterized in that: The RNN module is a bidirectional LSTM network.

8. The method for predicting CU partition mode of a VVC coding unit according to claim 1, characterized in that: The CNN-RNN hybrid neural network model is trained using a cross entropy loss function, which is L=(1-ε)H(p,q)+εH(p,u), where p is the model predicted logits output, q is the true label, u is a uniform distribution, and ε is a smoothing coefficient.

9. The method for predicting CU partition mode of a VVC coding unit according to claim 1, characterized in that: The confidence score is the maximum probability value in the CU partition mode probability distribution output by the CNN-RNN hybrid neural network model. The first preset threshold and the second preset threshold are dynamically adjusted according to the complexity of the video sequence. The first preset threshold and the second preset threshold are positively correlated with the complexity of the video sequence.

10. A video encoding device, comprising a processor and a memory, wherein the memory stores instructions, and when the instructions are executed by the processor, the method for predicting a CU partition mode for a VVC coding unit according to any one of claims 1 to 9 is executed.

Images