A VVC Inter-Frame Coding Acceleration Method and Device Based on Lightweight Neural Network

Abstract

The present invention discloses a VVC inter-frame coding acceleration method and device based on a lightweight neural network, relating to the field of video coding technology. The method includes the following steps: obtaining real-time feature data and residual information before starting the CU partition mode test; feeding the feature data and residual information into the MFLCNN network to calculate the probability of each candidate partition mode; selecting several partition modes with the highest probability based on a pre-set trade-off configuration, and actively skipping several modes with lower probabilities during the encoding process, thereby achieving encoding acceleration; the MFLCNN network uses a residual processing module to extract residual features from the residual information, uses a feature processing module to extract variable features from the feature data, and finally uses a feature fusion module to integrate the residual features and variable features and output the probability value of the possible partition mode. The present invention uses the MFLCNN network for feature analysis and mode prediction to achieve VVC inter-frame coding acceleration, which is simple to implement and has significant effects.

Description

Technical Field

[0001] The present invention relates to the field of video coding technology, and in particular to a VVC inter-frame coding acceleration method and device based on a lightweight neural network. Background Art

[0002] Video coding acceleration technology reduces the computational complexity and time consumption of video encoding by optimizing algorithms while maintaining the rate-distortion performance of video encoding to improve efficiency and real-time performance. This technology is widely used in video streaming, video conferencing, video surveillance, video storage, and other fields, and is crucial for reducing video processing costs and improving user experience.

[0003] Currently, research on VVC video coding acceleration technologies focuses on using three features: original pixels, residual pixels, and motion vectors. These methods reduce encoding time by prematurely terminating pattern testing during the encoding process. These methods can reduce the complexity of VVC inter-frame coding to a certain extent. However, a deeper analysis of these existing methods reveals several significant drawbacks. First, the features selected by these methods have low correlation with the final partitioning pattern. While this low correlation may save some time during the encoding process, it significantly reduces the encoder's rate-distortion performance. This reduced rate-distortion performance directly increases the bitstream data size, which is disadvantageous for video storage and transmission, increasing storage costs and bandwidth requirements. Second, some methods consume significant additional time performing motion search to obtain the motion vector data required by the decision maker. Motion search itself is a computationally intensive process, and this additional time consumption significantly reduces the encoding acceleration effect, making it difficult to effectively achieve the desired acceleration goals. Third, some methods often require extremely complex neural network models to achieve better pattern prediction for the decision maker. While complex neural network models may improve pattern prediction accuracy to a certain extent, they also incur significant time consumption. In some cases, even GPU acceleration is required to achieve encoding acceleration, which undoubtedly increases hardware costs and system complexity, limiting the widespread adoption of these methods in practical applications. Therefore, designing a lightweight, fast VVC inter-frame algorithm has become a pressing challenge in the field of video coding, requiring more innovative and targeted technologies to address this issue. Summary of the Invention

[0004] The purpose of the present invention is to solve the problems in the prior art such as low correlation between the selected features and the final division mode and unclear acceleration effect.

[0005] The technical solution adopted by the present invention to solve the technical problem is: to provide a VVC inter-frame coding acceleration method based on a lightweight neural network, comprising the following steps:

[0006] Before the CU partitioning mode test begins, obtain the real-time feature data and residual information;

[0007] The feature data and residual information are fed into the MFLCNN network to calculate the probability of each candidate partition mode;

[0008] The encoder selects several partitioning modes with the highest probability based on a pre-set trade-off configuration, and then tests the mode with the lowest rate-distortion cost.

[0009] The MFLCNN network uses a residual processing module to extract residual features from residual information, uses a feature processing module to extract variable features from feature data, and finally uses a feature fusion module to integrate residual features and variable features and output the probability value of possible division modes.

[0010] Preferably, before the CU split mode test is started, the real-time feature data and the motion search residual image are obtained, specifically: after all non-split mode tests are completed in the encoder and before the split mode test is started, the motion search residual image and the auxiliary features are collected;

[0011] The motion search residual image is a result of subtracting a predicted image obtained in a non-partitioning mode from an original image, and the auxiliary features include intermediate variables in an optimal non-partitioning mode selected by an encoder.

[0012] Preferably, the auxiliary features include: quantization parameter QP value , QP value of the current frame , the total distortion of the motion compensated image in the best non-partitioning mode , the overall rate-distortion cost of the best non-partitioning mode , the number of bits encoded , Regular merge mode flag variable , merge mode flag variable , Merge mode flag variable with motion vector difference , CU no residual flag , whether the merge mode with motion vector difference uses the residual-free method , affine mode flag , geometric partitioning mode , the depth of the current CU partition , the depth of quadtree division , the depth of the binary tree partition and the partition depth of the multi-tree method , providing information about the upper-level partitioning mode.

[0013] Preferably, the auxiliary features need to be preprocessed, which can be expressed as:

[0014] ;

[0015] ;

[0016] ;

[0017] ;

[0018] .

[0019] Preferably, the residual processing module of the MFLCNN network includes several different processing structures for compressing residual images of different sizes into a uniform data size; each processing structure includes several downsampling submodules and a flattening operation, and the downsampling submodule includes a Conv2D layer, a LayerNorm layer, and a ReLU layer connected in sequence; the processing process of the residual processing module includes the following steps:

[0020] Receive the absolute value generated by the residual image conversion and input it into the corresponding processing unit according to the CU size;

[0021] The corresponding processing unit uses the downsampling submodule to compress residual images of different sizes into a uniform data size.

[0022] Preferably, the feature processing module of the MFLCNN network includes three fully connected layers, each of which is followed by a LeakyReLU layer; the FPM accepts an auxiliary feature group with an input dimension of 16 and outputs variable features of the same dimension.

[0023] Preferably, the processing of the downsampling submodule of the MFLCNN network includes the following steps:

[0024] The structure of feature integration is selected according to the CU size; for larger CUs, a common structure is used to integrate features. The common structure includes an input layer with a dimension of 32, a hidden layer with a dimension of 16, a hidden layer with a dimension of 8, and an output layer with a dimension of 6; for smaller CUs, a simplified structure is used to integrate features. The simplified structure includes an input layer with a dimension of 32, a hidden layer with a dimension of 16, and an output layer with a dimension of 4.

[0025] The probability value of each partition mode is generated by the Softmax layer, and the output dimension depends on the number of candidate partition modes for a given CU size.

[0026] The present invention also provides a VVC inter-frame coding acceleration device based on a lightweight neural network, comprising:

[0027] The data collection module obtains the real-time feature data and residual information of the CU before the CU partitioning pattern test begins;

[0028] The pattern prediction module feeds the feature data and residual information into the MFLCNN network to calculate the probability of each candidate partition pattern;

[0029] The mode selection module and the encoding module select several partitioning modes with the highest probability based on the pre-set trade-off configuration, and the encoder then tests the mode with the lowest rate-distortion cost;

[0030] The MFLCNN network uses a residual processing module to extract residual features from residual information, uses a feature processing module to extract variable features from feature data, and finally uses a feature fusion module to integrate residual features and variable features and output the probability value of possible division modes.

[0031] The present invention has the following beneficial effects:

[0032] (1) This paper designs an extremely lightweight neural network for feature analysis and pattern prediction, which realizes VVC inter-frame coding acceleration with simple implementation and significant effect;

[0033] (2) The present invention proposes an auxiliary feature group that is more correlated with the final segmentation pattern to make up for the information loss of commonly used residual features, original pixels, and motion vector features.

[0034] The present invention will be further described in detail below with reference to the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is a diagram showing the steps of a VVC inter-frame coding acceleration method based on a lightweight neural network according to an embodiment of the present invention;

[0036] Figure 2 This is a diagram of the MFLCNN network structure of a VVC inter-frame coding acceleration method based on a lightweight neural network according to an embodiment of the present invention;

[0037] Figure 3 This is a structural diagram of a residual processing module of a VVC inter-frame coding acceleration method based on a lightweight neural network according to an embodiment of the present invention;

[0038] Figure 4 This is a schematic structural diagram of a downsampling submodule of a VVC inter-frame coding acceleration method based on a lightweight neural network according to an embodiment of the present invention;

[0039] Figure 5 This is a structural diagram of a VVC inter-frame coding acceleration device based on a lightweight neural network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to effectively solve the problem of insufficient acceleration of the current VVC video inter-frame coding method and a large decrease in encoder rate-distortion performance, the present invention provides a VVC inter-frame coding acceleration method based on a lightweight neural network. Figure 1 As shown, the following steps are included:

[0041] S101, before starting the CU division mode test, obtaining the real-time feature data and residual information therein;

[0042] S102, sending the feature data and residual information to the MFLCNN network to calculate the probability of each candidate partition mode;

[0043] S103, selecting several partitioning modes with the highest probability according to a preset trade-off configuration, and then the encoder tests and finds a mode with the lowest rate-distortion cost;

[0044] The MFLCNN network uses a residual processing module to extract residual features from residual information, uses a feature processing module to extract variable features from feature data, and finally uses a feature fusion module to integrate residual features and variable features to output probability values of possible partitioning modes. Several partitioning modes with the highest probability are selected, and the encoder then tests a mode with the lowest rate-distortion cost. By narrowing the mode traversal range of the encoder during the encoding process, encoding acceleration is achieved.

[0045] Specifically, the MFLCNN network (lightweight neural network) uses a residual processing module to extract residual features from residual information, uses a feature processing module to extract variable features from feature data, and finally uses a feature fusion module to integrate residual features and variable features to output probability values of possible division patterns.

[0046] Specifically, in S101, after the non-partitioning mode tests, such as the fusion mode, motion search mode, and affine mode, are completed, and before the CU partitioning mode test begins, the test results for the best non-partitioning mode are extracted. These features include the motion search residual image and a series of auxiliary variables; the residual image is the result of subtracting the predicted image obtained in the non-partitioning mode from the original image; the auxiliary features include 16 variables, described as follows:

[0047] Quantization parameter QP value , QP value of the current frame , the total distortion of the motion compensated image in the best non-partitioning mode , the overall rate-distortion cost of the best non-partitioning mode , the number of bits encoded , Regular merge mode flag variable , merge mode flag variable , Merge mode flag variable with motion vector difference , CU no residual flag , whether the merge mode with motion vector difference uses the residual-free method , affine mode flag , geometric partitioning mode ; and four division stage indicator variables, 、 、 、 , these variables represent the depth of the current CU partition, the partition depth of the quadtree method, the partition depth of the binary tree method and the partition depth of the multitree method, respectively, to provide information about the upper-level partition mode.

[0048] represents the total distortion of the motion compensated image of the best non-partitioned mode; represents the overall rate-distortion cost of this non-partitioning mode. Indicates the number of bits used for encoding. Higher values for these three parameters generally indicate lower prediction accuracy for the non-split mode, increasing the likelihood that the CU will be split. These three variables have a very wide range of values in the encoder and cannot be directly input into the neural network for training. Therefore, these variables need to be preprocessed. The preprocessing method is as follows:

[0049] ;

[0050] ;

[0051] ;

[0052] and The preprocessing method is the same as above:

[0053] ;

[0054] ;

[0055] Specifically, in S102, the residual image and the pre-processed auxiliary features are fed into the MFLCNN network for feature analysis and calculation of the probability of segmentation patterns. Figure 2As shown in Figure 1, MFLCNN consists of three main components: the residual processing module (RPM), which extracts features from residual pixels; the feature processing module (FPM), which processes the auxiliary feature set; and the feature fusion module (FFM), which integrates the information from the first two modules and outputs the probability value of the possible partitioning mode. The details of each module are as follows:

[0056] RPM is used to analyze the matching degree of motion compensated images in different regions of a CU. Before inputting into RPM, the residual image is converted to its absolute value. Figure 3 As shown in Figure 1, RPM consists of 14 different structures, each designed for a specific size of CU. Each structure includes several downsampling submodules (DSMs) and a flattening operation to compress residual images of different sizes into a uniform data size. The structure of DSM can be found in Figure 4 As shown in Figure 1, it consists of three layers: Conv2D layer, LayerNorm layer and ReLU layer. Five parameters are designed specifically for DSM and provided to the Conv2D layer: Indicates the number of input channels of the Conv2D layer, Indicates the number of output channels, represents the kernel size, represents the kernel stride, Indicates padding. After the Conv2D layer, the LayerNorm layer normalizes the data across all dimensions except the batch dimension. For example, if the feature map generated by the Conv2D layer has The dimension of LayerNorm will be The data is normalized within the dimension. After normalization, the data is passed through the ReLU activation function to introduce nonlinearity in the network.

[0057] The FPM module analyzes auxiliary feature groups. It consists of three fully connected layers, each followed by a LeakyReLU layer. FPM accepts an auxiliary feature group with an input dimension of 16 and outputs features of the same dimension for subsequent use. The FFM integrates and analyzes the output features of the first two modules and generates probabilities for possible partition patterns through a Softmax layer. The output dimension of the FFM depends on the number of candidate partition patterns for a given CU size. The FFM has two architectures: a standard architecture and a simplified architecture. For some larger CUs, the FFM module uses a standard architecture to more comprehensively analyze the information from the first two modules. For smaller CUs, the FFM uses a simplified architecture to save more time and reduce the risk of network overfitting. Finally, testing of partition patterns with lower probabilities is skipped, improving encoding speed.

[0058] Specifically, the experimental platform was Ubuntu 20.24.2 and the CPU was an AMD EPYC 9654 96 Core processor. In the time comparison test, both the original encoder and the accelerated encoder were tested using this experimental platform. This solution achieved an average time savings of 47.57%, while only increasing the bitrate by 2.15% at the same quality. The method proposed by Pan et al. in the paper "A CNN-based fast inter coding method for VVC" achieved a time savings of 30.63% and increased the bitrate by 3.18% at the same quality. The method proposed by Tissier et al. in the paper "Machine learning based efficient QT-MTT partitioning for VVC inter coding" achieved a time savings of 43.4% and increased the bitrate by 2.33% at the same quality. This solution outperformed the comparison solutions in both time savings and bitrate increase suppression.

[0059] See also Figure 5 FIG. 1 is a schematic diagram of a VVC inter-frame coding acceleration device based on a lightweight neural network according to an embodiment of the present invention, comprising:

[0060] The data collection module 501 obtains the real-time feature data and residual information of the CU before the CU partition mode test begins;

[0061] The mode prediction module 502 feeds the feature data and residual information into the MFLCNN network to calculate the probability of each candidate partition mode;

[0062] The model selection module 503 selects several partitioning modes with the highest probability according to a preset trade-off configuration, and the encoder then tests a mode with the lowest rate-distortion cost;

[0063] The device achieves encoding acceleration by reducing the mode traversal range of the encoder during the encoding process; the implementation of each module in the device is the same as the modules in the VVC inter-frame encoding acceleration method based on a lightweight neural network, and will not be repeated here.

[0064] It can be seen that the VVC inter-frame coding acceleration method and device based on lightweight neural network proposed in the present invention obtains real-time feature data and residual information of the CU before the division mode test begins, inputs the designed MFLCNN network to calculate the probability of each candidate division mode, and then selects several division modes with the highest probability according to the preset trade-off configuration, and actively skips several modes with lower probability during the encoding process, thereby achieving encoding acceleration.

[0065] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A VVC inter-frame coding acceleration method based on a lightweight neural network, characterized in that: The following steps are involved: Before the CU partitioning mode test begins, obtain the real-time feature data and residual information; The feature data and residual information are fed into the MFLCNN network to calculate the probability of each candidate partition mode; The encoder selects several partitioning modes with the highest probability based on a pre-set trade-off configuration, and then tests the mode with the lowest rate-distortion cost. The MFLCNN network uses the residual processing module to extract residual features from the residual information, uses the feature processing module to extract variable features from the feature data, and finally uses the feature fusion module to integrate the residual features and variable features and output the probability value of the possible division mode; Before the CU partition mode test begins, real-time feature data and a motion search residual image are obtained, specifically: after all non-partition mode tests in the encoder are completed and before the partition mode test begins, the motion search residual image and auxiliary features are collected; The motion search residual image is a result of subtracting the predicted image obtained in the non-partitioning mode from the original image, and the auxiliary features include intermediate variables in the best non-partitioning mode selected by the encoder; The auxiliary features include: quantization parameter QP value f QP_cfg , the QP value of the current frame f QP_current , the total distortion f of the motion compensated image in the best non-partitioning mode dist , the overall rate-distortion cost f of the best non-partitioning mode Cost , the number of bits of coding f fracBits , conventional merge mode flag variable f RegularMergeFlag , merge mode flag variable f MergeFlag , merge mode flag variable f with motion vector difference MmvdMergeFlag , CU no residual flag f CUSkip 、Whether the merge mode with motion vector difference uses the no-residual method f MmvdSkip , affine mode flag f Affine , geometric partitioning mode f Geo , the depth f of the current CU partition CUDepth , the quadtree partition depth f CUQtDepth , the partition depth f of the binary tree method CUBtDepth And the partition depth f of the multi-tree method CUMtDepth ,provides information about the upper layer partitioning mode; The residual processing module of the MFLCNN network includes several different processing structures for compressing residual images of different sizes into a unified data size; The feature processing module of the MFLCNN network includes three fully connected layers; The processing of the downsampling submodule of the MFLCNN network includes the following steps: The structure of feature integration is selected according to the CU size. For larger CUs, a common structure is used to integrate features. The common structure includes an input layer with a dimension of 32, a hidden layer with a dimension of 16, a hidden layer with a dimension of 8, and an output layer with a dimension of 6. For smaller CUs, a simplified structure is used to integrate features. The simplified structure includes an input layer with a dimension of 32, a hidden layer with a dimension of 16, and an output layer with a dimension of 4. The probability value of each partition mode is generated by the Softmax layer, and the output dimension depends on the number of candidate partition modes for a given CU size.

2. The VVC inter-frame coding acceleration method based on a lightweight neural network according to claim 1, characterized in that: The auxiliary features need to be preprocessed, which can be expressed as: f' QP_cfg =(f QP_cfg / 51)+0.5; f QP_current =(f QP_current / 51)+0.5; f' cost =f cost / 10 9 ; f' dist =f dist / 10 7 ; f' fracBits =f fracBits / 10 7 。 3. The VVC inter-frame coding acceleration method based on lightweight neural network according to claim 1, characterized in that In the residual processing module, each processing structure includes several downsampling submodules and a flattening operation, and the downsampling submodule includes a Conv2D layer, a LayerNorm layer, and a ReLU layer connected in sequence. The processing process of the residual processing module includes the following steps: Receive the absolute value generated by the residual image conversion and input it into the corresponding processing unit according to the CU size; The corresponding processing unit uses the downsampling submodule to compress residual images of different sizes into a uniform data size.

4. The VVC inter-frame coding acceleration method based on lightweight neural network according to claim 1, characterized in that In the feature processing module, each fully connected layer is followed by a LeakyReLU layer; FPM receives an auxiliary feature group with an input dimension of 16 and outputs variable features of the same dimension.

5. A VVC inter-frame coding acceleration device based on a lightweight neural network, used to implement the VVC inter-frame coding acceleration method based on a lightweight neural network according to any one of claims 1 to 4, characterized in that: include: The data collection module obtains the real-time feature data and residual information of the CU before the CU partitioning pattern test begins; The pattern prediction module feeds the feature data and residual information into the MFLCNN network to calculate the probability of each candidate partition pattern; The mode selection module and the encoding module select several partitioning modes with the highest probability based on the pre-set trade-off configuration, and the encoder then tests the mode with the lowest rate-distortion cost; The MFLCNN network uses a residual processing module to extract residual features from residual information, uses a feature processing module to extract variable features from feature data, and finally uses a feature fusion module to integrate residual features and variable features and output the probability value of possible division modes.

Images