VVC intra-frame prediction mode pre-screening method based on low-level and high-level combined features

Abstract

The invention discloses a VVC intra-frame prediction mode pre-screening method based on low-level and high-level combined features, and belongs to the technical field of video coding. The technical problems that VVC intra-frame mode decision calculation is high in complexity, a traditional fast algorithm is insufficient in accuracy, and hardware deployment of a pure deep learning scheme is difficult are solved. According to the technical scheme, the method comprises the following steps: S1, extracting pixel-level low-level features of each 64 * 64 coding block in a YUV video by using a Sobel operator; s2, setting a dynamic threshold value according to a gradient amplitude median and a variance mean value, and quickly eliminating irrelevant modes; and S3, for the remaining candidate modes, calculating a semantic matching degree through a lightweight CNN to carry out secondary screening, inputting the screened modes into a VVC encoder to carry out rate distortion cost calculation, and determining final mode selection. The method has the beneficial effects that the coding complexity is obviously reduced while the rate distortion performance is kept basically unchanged.

Description

Technical Field

[0001] The present invention relates to the field of video coding technology, and particularly relates to a method for pre-screening intra prediction modes of VVC (Versatile Video Coding) based on combined low-level and high-level features, which is especially applicable to the intra prediction optimization in the H.266 / VVC standard. Background Art

[0002] The H.266 / VVC standard significantly increases the number of intra prediction modes from 35 in HEVC to 67, including Planar, DC, and 65 angular modes. Although this expansion improves the coding efficiency, it also brings a significant computational burden. During the actual coding process, the encoder needs to calculate the prediction residuals for all 67 modes for each coding unit (CU), perform operations such as transformation, quantization, and entropy coding, and finally compare the rate-distortion costs (RD Cost) of each mode. Experimental data shows that for 4K resolution video content, when using full-mode search, the coding time of VVC increases by 3 - 5 times compared to HEVC, which becomes the main bottleneck for realizing real-time coding.

[0003] The current mainstream acceleration schemes are mainly divided into two categories: traditional feature-based methods and deep learning-based methods. Although traditional feature-based methods are computationally fast, their high misjudgment rate in processing complex texture regions leads to a significant decline in rate-distortion performance. Another significant defect of this type of method is that it cannot effectively distinguish between modes that are semantically similar but slightly different in direction (such as mode 34 and mode 35). While deep learning-based methods have relatively high accuracy, they generally suffer from problems such as large model size and high computational latency, making it difficult to be deployed and applied in resource-constrained scenarios such as mobile devices.

[0004] The video coding industry currently faces several key challenges: First, the transformation and quantization operations in full-mode search consume 82% of the computational time, resulting in a significant increase in coding latency and a sharp rise in hardware power consumption; second, traditional CNN schemes need to cache the prediction blocks of all modes for each coding unit, which not only occupies a large amount of on-chip storage but also causes frequent DRAM access. These challenges severely restrict the popularization and application of the VVC standard in real-time coding scenarios.

[0005] By analyzing existing patented technologies, it can be found that some patents only use gradient histograms for mode pre-selection without considering semantic relevance, resulting in a relatively high BD-Rate loss in dynamic texture scenarios; while some patents use CNN to analyze spatio-temporal features, but their model parameter quantities are large and require dedicated AI accelerator support, with too high implementation costs. None of these schemes can meet the efficiency requirements of real-time coding while ensuring coding quality.

[0006] The present invention solves the above problems through three key innovations: First, a cascaded screening architecture is adopted. First, 90% of the obviously inappropriate patterns are quickly eliminated by using low-level features such as the Sobel operator, and then a lightweight CNN is used for fine screening. Then, a template library containing standard feature vectors of 67 patterns is constructed, significantly reducing the real-time calculation amount. This solution can greatly improve the coding efficiency while ensuring the coding quality. Summary of the Invention

[0007] The object of the present invention is to provide a method for pre-screening VVC intra prediction modes based on combined low-level and high-level features, which can solve the problem of high computational complexity of full-mode search in VVC intra prediction and balance the accuracy and real-time performance of mode screening.

[0008] The inventive concept of the present invention is as follows: The present invention provides a method for pre-screening VVC intra prediction modes based on combined low-level and high-level features. First, pixel-level low-level features of each 64×64 coding block in the YUV video are extracted by using the Sobel operator, including the average gradient magnitude, the main gradient direction angle, and the local variance. Then, dynamic thresholds are set according to the median of the gradient magnitude and the mean of the variance to quickly eliminate irrelevant modes. Finally, for the remaining candidate modes, the semantic matching degree is calculated by a lightweight CNN for secondary screening, and the screened candidate modes are input into the VVC encoder to perform rate-distortion cost calculation to determine the final mode selection.

[0009] In order to achieve the above object of the invention, the technical solution adopted by the present invention is specifically as follows: A method for pre-screening VVC intra prediction modes based on combined low-level and high-level features, comprising the following steps:

[0010] 1.1 Extract pixel-level low-level features of the YUV video by using the Sobel operator, including the average gradient magnitude, the main gradient direction angle, and the local variance;

[0011] 1.2 Perform the first intra prediction mode screening according to the average gradient magnitude, the main gradient direction angle, the mean of the variance, and the set threshold;

[0012] 1.3 For the remaining candidate modes, calculate the semantic matching degree by a lightweight CNN for secondary screening, and input the screened candidate modes into the VVC encoder to perform rate-distortion cost calculation to determine the final mode selection.

[0013] Further, the step 1.1 specifically comprises the following steps:

[0014] 2.1 Read the Y component of the input YUV video frame by frame, and divide each frame into non-overlapping coding blocks of size 64×64;

[0015] 2.2 Perform Sobel convolution on each coding block to calculate the horizontal gradient Gx and the vertical gradient G y , where the size of the Sobel kernel is 3×3, and the padding method uses the default BORDER_REFLECT_101 (symmetric padding);

[0016] 2.3. Based on G x and G y , calculate the gradient magnitude G and the gradient direction θ of each coding block, and the formula is Statistical average gradient magnitude and the main direction angle Take the median of the gradient magnitudes of all coding blocks as the set threshold T1;

[0017] 2.4. Calculate the variance of the pixel values for each 64×64 coding block one by one, and statistically calculate the variance σ of all blocks i 2 , and take the mean of the variances of all blocks as the set threshold T2.

[0018] Furthermore, step 1.2 specifically includes the following steps:

[0019] 3.1. For the current coding block, if G i < T1 and σ i 2 < T2, add the Planar and DC modes to the candidate mode set;

[0020] 3.2. For the current coding block, if G i ≥ T1 and add the angle modes that meet the conditions to the candidate mode set;

[0021] 3.3. For the current coding block, if none of the above situations hold, add all 67 intra prediction modes to the candidate mode set;

[0022] 3.4. Denote the candidate mode set after the initial screening as M primary = {m1, m2, …, m k}.

[0023] Furthermore, step 1.3 specifically includes the following steps:

[0024] 4.1. Normalize the pixel values of the Y component pixel matrix of the current 64×64 coding block from [0, 255] to [0, 1], and denote the processed data as B norm ;

[0025] 4.2. Input B norm into a lightweight CNN model composed of 3 layers of convolution and average pooling, and denote the feature vector output after flattening the features as F block ;

[0026] 4.3. For each candidate mode m j ∈M primary calculate the cosine similarity, and its formula is Retain the mode with the highest similarity, where is the feature vector corresponding to the candidate mode in the mode feature library;

[0027] 4.4. Input the filtered candidate modes into the VVC encoder to perform rate-distortion cost calculation to determine the final mode selection.

[0028] Furthermore, the specific steps of step 4.3 include the following steps:

[0029] 5.1. Use the VTM encoder to perform full-mode encoding on the standard test sequences (Class B-E), and use the optimal prediction mode number m∈[0,66] and the corresponding prediction block P of each encoded block after encoding m as the data set;

[0030] 5.2. Use the above data set to train and test the lightweight CNN model composed of 3 layers of convolution and average pooling, so that each mode m∈[0,66] can find the prediction block P that best matches it mb ;

[0031] 5.3. Input the prediction block P corresponding to each mode m∈[0,66] mb into the CNN model, store the extracted features as mode feature vectors, and establish a mode feature library

[0032] Compared with the prior art, the present invention surpasses the prior art solutions in key indicators such as computing efficiency, coding quality, and resource occupancy through an innovative cascaded screening architecture, providing reliable technical support for the industrial application of the VVC standard. Specifically, the beneficial effects of the present invention are as follows:

[0033] (1) The cascaded screening architecture of the present invention fundamentally reconstructs the decision-making process of the VVC coding unit through a multi-stage progressive mode elimination mechanism. The traditional gradient preselection scheme needs to perform full-traversal calculations on all possible coding modes (67 intra modes), while the present invention uses a low-complexity filter in the first stage to eliminate obviously unreasonable modes, and combines CNN semantic analysis in the second stage for further screening. Finally, only a few high-probability modes need to be subjected to complete rate-distortion optimization. This hierarchical processing significantly reduces the number of modes that need to be calculated for each coding unit (CU). Compared with the traditional gradient preselection scheme, the computing efficiency is further improved;

[0034] (2)Regarding the misjudgment problem of the traditional scheme in complex texture regions (such as fine stripes and irregular edges), the present invention innovatively embeds CNN semantic matching into the screening process. By extracting the texture features of CUs through a pre-trained network, similarity matching is performed with typical patterns in the pattern feature library, and the priority of candidate patterns is dynamically adjusted. By introducing CNN semantic matching, the misjudgment rate is reduced, and the subjective visual quality is effectively improved;

[0035] (3)The pattern feature library constructed by the present invention reduces the memory occupancy by 94% compared with the traditional CNN scheme, and the model volume is small. In addition, the traditional fixed candidate set (67 patterns of VVC) is prone to cause resource waste. The present invention constructs a dynamic candidate set based on scene characteristics by analyzing the spatio-temporal correlation of videos. The introduction of the dynamic candidate pattern set further reduces the demand for peak memory. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] The drawings are used to provide a further understanding of the present invention, and constitute a part of the specification. They are used together with the embodiments of the present invention to explain the present invention, and do not constitute a limitation to the present invention.

[0037] Figure 1 It is a schematic flowchart of the method of the present invention;

[0038] Figure 2 It is a schematic diagram of Sobel gradient calculation in the present invention;

[0039] Figure 3 It is a schematic diagram of the lightweight CNN model architecture in the present invention;

[0040] Figure 4 It is a schematic diagram of the system structure in the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0041] In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the drawings and embodiments. Of course, the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0042] Embodiment 1

[0043] See Figures 1 to 3 , the technical solution provided by this embodiment is a VVC intra prediction mode pre-screening method based on low-level and high-level joint features, including the following steps:

[0044] 1. Use the Sobel operator to extract pixel-level low-level features of the YUV video, including the average gradient magnitude, the main gradient direction angle, and the local variance;

[0045] 2. Perform the first intra-prediction mode screening according to the average gradient magnitude, the main gradient direction angle, the mean of the variances, and the set threshold;

[0046] 3. For the remaining candidate modes, perform a secondary screening by calculating the semantic matching degree through a lightweight CNN, and input the screened candidate modes into the VVC encoder to perform rate-distortion cost calculation to determine the final mode selection.

[0047] Specifically, in step 1, referring to Figure 2 , use the Sobel operator to extract pixel-level low-level features of the YUV video, including the average gradient magnitude, the main gradient direction angle, and the local variance, including the following steps:

[0048] 1) Read the Y component of the input YUV video frame by frame, and divide each frame into non-overlapping coding blocks of size 64×64;

[0049] 2) Perform Sobel convolution on each coding block to calculate the horizontal gradient G x and the vertical gradient G y , where the size of the Sobel kernel is 3×3, and the padding method uses the default BORDER_REFLECT_101 (symmetric padding);

[0050] 3) Based on G x and G y , calculate the gradient magnitude G and the gradient direction θ of each coding block, and the formula is Statistically calculate the average gradient magnitude and the main direction angle Take the median of all coding block gradient magnitudes as the set threshold T1;

[0051] 4) Calculate the variance of the pixel values block by block for each 64×64 coding block, and statistically calculate the variance σ i 2 of all blocks, and take the mean of all block variances as the set threshold T2.

[0052] Specifically, in step 2, perform the first intra-prediction mode screening according to the average gradient magnitude, the main gradient direction angle, the mean of the variances, and the set threshold, including the following steps:

[0053] 1) For the current coding block, if G i < T1 and σ i 2 < T2, add the Planar and DC modes to the candidate mode set;

[0054] 2) For the current coding block, if G i ≥ T1 and Add the angle patterns that meet the conditions to the candidate pattern set;

[0055] 3) For the current coding block, if none of the above cases hold, add all 67 intra prediction patterns to the candidate pattern set;

[0056] 4) Denote the candidate pattern set after the initial screening as M primary ={m1, m2, …, m k}.

[0057] Specifically, in step 3, referring to Figure 3 , for the remaining candidate patterns, perform secondary screening by calculating the semantic matching degree through a lightweight CNN, and input the screened candidate patterns into the VVC encoder to perform rate-distortion cost calculation to determine the final mode selection, including the following steps:

[0058] 1) Use the VTM encoder to perform full-mode encoding on the standard test sequences (Class B - E), and use the optimal prediction mode number m ∈ [0, 66] and the corresponding prediction block P m of each coding block after encoding as the data set;

[0059] 2) Use the above data set to train and test a lightweight CNN model composed of three layers of convolution and average pooling, so that each mode m ∈ [0, 66] can find the prediction block P mb that best matches it;

[0060] 3) Input the prediction block P mb corresponding to each mode m ∈ [0, 66] into the CNN model, store the extracted features as mode feature vectors, and establish a mode feature library

[0061] 4) When encoding the remaining standard test sequences (Class A), perform pixel value normalization on the Y-component pixel matrix of the current 64×64 coding block from [0, 255] to [0, 1], and denote the processed data as B norm ;

[0062] 5) Input B norm into a lightweight CNN model composed of three layers of convolution and average pooling, and denote the feature vector output after flattening the features as F block ;

[0063] 6) Calculate the cosine similarity for each candidate mode m j ∈M primary , and its formula is Retain the mode with the highest similarity, where is the feature vector corresponding to the candidate mode in the mode feature library;

[0064] 7) Input the filtered candidate patterns into the VVC encoder to perform rate-distortion cost calculation and determine the final mode selection.

[0065] In the experiment, 12 typical sequences of VVC standard test sequences Class B - E were selected (including video contents with different texture characteristics such as "ParkRunning3", "BasketballDrive", "Cactus", etc.). On the hardware platform composed of 13th Gen Intel(R) Core(TM) i9 - 13900K 3.00GHz and NVIDIA RTX 3080 GPU, a test environment was built based on the VTM - 23.3 reference software and PyTorch 1.13.1 framework. The evaluation system includes three key indicators: BD - Rate reflecting the coding efficiency loss, ΔT characterizing the change in coding time, and the mode screening accuracy measuring the pre - selection quality.

[0066] Three types of comparison schemes were selected in the experiment: full mode search as the baseline scheme (VTM baseline), traditional gradient pre - selection scheme, and CNN scheme. As shown in the test data in Table 1, the present invention reduces the coding time by an average of 68.2% on Class A (4K) sequences, and the BD - Rate only increases by 0.08%, significantly superior to the traditional gradient pre - selection scheme (BD - Rate increases by 1.83%, time saving of 62.4%) and the CNN scheme (BD - Rate increases by 0.12%, time saving of 45.7%). On the "Tango2" sequence with complex texture, the traditional scheme causes a BD - Rate loss of up to 2.5% due to misjudgment, while the present invention only loses 0.3% through CNN semantic matching, demonstrating strong adaptability to complex textures.

[0067] Table 1

[0068] Method BD-Rate(%) ΔT(%) Accuracy rate(%) Full-mode search 0.00 +0.0 100.0 Traditional gradient preselection +1.83 -62.4 78.2 CNN +0.12 -45.7 95.6 The present invention +0.08 -68.2 96.8

[0069] Example 2

[0070] As Figure 4 shown, to achieve the above - mentioned purpose, the present invention discloses a VVC intra - prediction mode pre - screening system based on combined low - level and high - level features, including:

[0071] 1. Video pre - processing module 11, used to perform frame segmentation on the input YUV video, and then perform division of 64×64 coding blocks and Y - component extraction on the segmented video frames;

[0072] 2. Low - level feature extraction module 12, used to perform Sobel gradient calculation and variance calculation on the processed coding blocks, and generate a dynamic threshold based on the calculation results;

[0073] 3. Prediction mode initial screening module 13, which is used for the judgment of Planar and DC modes and the screening of angle modes, and then generates a candidate mode set;

[0074] 4. Advanced feature processing module 14, which is used to generate a feature vector corresponding to the encoding block for the input lightweight CNN model, and then perform feature vector comparison and pattern feature library query;

[0075] 5. VVC encoder interface module 15, which is used to calculate the rate-distortion cost for the screened mode and determine the final mode selection.

[0076] The above are only the preferred embodiments of the present invention, and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A VVC intra prediction mode pre-screening method based on combined low-level and high-level features, characterized in that, It includes the following steps: S1. Extract the pixel-level low-level features of the YUV video using the Sobel operator, including the average gradient magnitude, the main gradient direction angle, and the local variance; S2. Perform the first intra-prediction mode screening according to the average gradient magnitude, the main gradient direction angle, the mean of the variance, and the set threshold; S3. For the remaining candidate modes, perform secondary screening by calculating the semantic matching degree through a lightweight CNN, and input the screened candidate modes into the VVC encoder to perform rate-distortion cost calculation to determine the final mode selection.

2. The method for pre-screening VVC intra prediction modes based on combined low-level and high-level features according to claim 1, wherein The step S1 includes the following steps: S21. Read the Y component of the input YUV video frame by frame, and divide each frame into non-overlapping coding blocks with a size of 64×64; S22. Perform Sobel convolution on each coding block to calculate the horizontal gradient G x and the vertical gradient G y , where the size of the Sobel kernel is 3×3 and the padding method uses the default symmetric padding BORDER_REFLECT_101; S23. Based on G x and G y , calculate the gradient magnitude G and the gradient direction θ of each coding block. The formula is Statistical average gradient magnitude and the main direction angle Take the median of the gradient magnitudes of all coding blocks as the set threshold T1; S24. Calculate the variance of pixel values for each 64×64 coding block one by one, and statistically calculate the variance σ of all blocks i 2 , and take the mean value of all block variances as the set threshold T2.

3. The VVC intra prediction mode pre-screening method based on combined low-level and high-level features according to claim 1, characterized in that The step S2 includes the following steps: S31. For the current coding block, if G i < T1 and σ i 2 < T2, add the Planar and DC modes to the candidate mode set; S32. For the current coding block, if G i ≥ T1 and Add the angle modes that meet the conditions to the candidate mode set; S33. For the current coding block, if none of the above situations hold, add all 67 intra-prediction modes to the candidate mode set; S34. Denote the candidate pattern set after the initial screening as M primary = {m1, m2, …, m k}.

4. The VVC intra prediction mode pre-screening method based on combined low-level and high-level features according to claim 1, characterized in that The step S3 includes the following steps: S41. Normalize the pixel values of the Y - component pixel matrix of the current 64×64 coding block from [0, 255] to [0, 1], and denote the processed data as B norm ; S42. Input B norm into a lightweight CNN architecture consisting of 3 layers of convolution and average pooling. The feature vector output after flattening the features is denoted as F block ; S43. For each candidate pattern m j ∈M primary calculate the cosine similarity, and its formula is retain the pattern with the highest similarity, where is the feature vector corresponding to the candidate pattern in the pattern feature library; S44. Input the screened candidate modes into the VVC encoder to perform rate-distortion cost calculation to determine the final mode selection.

5. The VVC intra prediction mode pre-screening method based on combined low-level and high-level features according to claim 4, wherein The mode feature library described in the step S43 includes the following steps: S51. Use the VTM encoder to perform full-mode encoding on the standard test sequences Class B-E, and use the optimal prediction mode number m ∈ [0, 66] of each encoded block after encoding and the corresponding prediction block P m as the dataset; S52. Use the above dataset to train and test a CNN model composed of three layers of convolution and average pooling, so that for each pattern m ∈ [0, 66], the most matching prediction block P can be found mb ; S53. Input the prediction block P corresponding to each pattern m ∈ [0, 66] mb into the CNN model, store the extracted features as pattern feature vectors, and establish a pattern feature library

Images