A fast VVC intra coding method for machine-oriented video coding

Abstract

The application belongs to the technical field of video compression coding, and particularly relates to a fast VVC (Versatile Video Coding) intra-frame coding method for machine-oriented video coding. The application adopts a neural network technology, and connects the encoder and the decoder of a convolutional neural network (CNN) block and a deep transformer block in series; high-efficiency and accurate coding is realized, so that the segmentation decision of a coding block is more accurate. The application also guides block division through retraining of a CNN network, and reduces the complexity of a rate-distortion optimization (RDO) process. The method is particularly suitable for a new generation of a general video coding standard VVC. The application can reduce coding complexity while significantly improving the performance of video coding and decoding, especially in a machine-oriented application scenario.

Description

Technical Field

[0001] The present invention belongs to the technical field of video compression coding and decoding, and particularly relates to a VVC intra-frame coding method for machine video coding. Background Art

[0002] Advanced video coding technologies such as High Efficiency Video Coding (HEVC) and Next Generation Video Coding (VVC) have emerged to meet the growing demand for high-resolution and high-frame-rate videos. However, these codecs are mainly designed to meet the needs of human vision and are not suitable for machine vision tasks such as image classification, object detection, and semantic segmentation. In recent years, many research works have begun to explore the use of coding blocks (CUs) to extract features from machine video coding. For example, in 2020, Gao Wen et al. proposed a set of machine-oriented coding and decoding frameworks and solutions for collaborative compression and intelligent perspective. In 2021, Ahonin et al. developed a learning post-processing filter that significantly improved the efficiency of machine vision tasks applied to VVC-encoded video restoration images. However, existing machine-oriented video coding and decoding technologies are mainly based on convolutional neural networks (CNNs) to extract features, which is insufficient in capturing and extracting global features. To this end, the present invention combines CNN and Transformer networks to integrate global and local feature information, replacing the traditional rate-distortion (RDO) process, thereby improving feature representation capabilities and better adapting to the needs of machine video coding.

[0003] In the VVC standard, intra-frame coding introduces binary and ternary tree divisions, which adds five block division modes compared to the quadtree of the HEVC standard, greatly increasing the complexity of coding. In order to cope with the additional coding complexity brought by the machine video coding neural network and accelerate the speed of intra-frame coding under the VVC standard, the present invention proposes a machine-oriented fast intra-frame block division method. This method retrains the CNN network designed by the present invention, and can directly determine the block division pattern without going through the RDO process of traditional coding and decoding, effectively improving the coding and decoding speed for machines. Summary of the Invention

[0004] The present invention aims to propose a fast VVC intra-frame coding method for machine video coding to meet the needs of machine video coding, which can efficiently process video content, ensure coding quality, and significantly improve processing speed.

[0005] The fast VVC intra-frame coding method for machine video coding proposed in this invention is based on neural network. Figure 1 Specifically, it includes two network branches, one is a convolutional neural network (CNN) branch, and the other is a deep transformer branch; wherein:

[0006] A convolutional neural network (CNN) branch includes an encoder module and a decoder module, which are connected in series. The encoder module includes two identical encoders connected in sequence, and the decoder module includes two identical decoders connected in sequence. In the CNN branch, a single encoder includes 1×1 convolution, batch normalization, and 3×3 convolution; a single decoder includes 1×1 deconvolution, batch normalization, and 3×3 deconvolution.

[0007] The deep transformer branch also includes an encoder module and a decoder module, which are connected in series. The encoder module consists of two identical encoders connected in sequence, and the decoder module consists of two identical decoders connected in sequence. In the transformer branch, a single encoder consists of layer normalization and multi-head self-attention. The decoder structure is the same as the encoder, except that the number of multi-head self-attention is doubled.

[0008] The input video frame first extracts features through the two encoders of the CNN branch and the Transformer branch respectively, and then the features are added (concat) and the results are sent to the decoding ends of the CNN and Transformer respectively. The decoding ends of the CNN and Transformer are independent of each other. After further feature extraction and restoration processes at the two decoding ends, the output of the CNN decoding end is connected to a 1*1 convolution and batch normalization (Norm), and then further pixel-wise summed with the information output by the Transformer decoding end to obtain the expected output feature information.

[0009] The neural network structure of the present invention is a residual network (resnet) architecture, such as Figure 2 shown.

[0010] The encoder and decoder of the CNN branch have symmetrical structures. The CNN encoder has two convolutional layers. The CNN branch uses a residual network (ResNet) architecture to integrate encoder features with decoder features of the same size.

[0011] The encoder of the Transformer branch controls the feature size through average pooling. The Transformer block with upsampling operation is the decoder of the Transformer block. The outputs of the CNN and Transformer encoders are connected to the CNN and Transformer decoder respectively. In this way, both global and local data are introduced into the dual branches, thereby improving the description of image features. Figure 3 As shown in Figure 1, the CNN branch consists of an encoder with two downsampling blocks and a decoder with two upsampling blocks. Convolution and deconvolution operations are used for feature extraction in the encoder and decoder architectures. ResNet connections pass feature information from the encoder to the decoder.

[0012] In the CNN, both an encoder and a decoder are included. The encoder has a symmetric structure, including two down-sampling blocks, denoted as Ei (i = 1, 2), each designed to halve the width and height of the input at each stage. These down-sampling blocks form the core of the CNN branch encoder, mainly consisting of 3 x 3 convolution operations to achieve efficient feature extraction. The activation function in these blocks is a rectified linear unit (ReLU), supplemented by reflective padding to maintain feature size, ensuring a balance between feature extraction efficiency and size consistency.

[0013] The decoder has a symmetric structure, containing two up-sampling blocks, denoted as Di (i = 1, 2). The up-sampling blocks use convolution operations with a kernel size of 2. The feature size is doubled in each block. In addition, ReLU is also used as the activation function, and the feature size is supplemented by reflective padding.

[0014] The encoder output and the decoder up-sampling block D 3-i The input of Ei (i = 1, 2) has the same size, E1 (Encoder Block 1) and D2 (Decoder Block 2) are symmetric, and E2 (Encoder Block 2) and D1 (Decoder Block 1) are symmetric.

[0015] The CNN branch also introduces a ResNet architecture to obtain rich feature information and better network convergence performance. The features of the encoder are connected to the decoder of the same size, promoting the flow of feature information in the deep layer. The decoder and the encoder are symmetric structures for image fusion feature extraction. In this way, the high-resolution output of the CNN branch is the same as the input image.

[0016] To meet the global and local feature extraction capabilities, a global and local loss function is used to train the neural network, i.e., the loss function contains global and local information. The local information loss contains two parts: pixel loss and structural similarity loss.

[0017] The pixel loss Lp is calculated as the variance of the input image I(x, y) and the output image O(x, y):

[0018] L p = (O(x, y) - I(x, y)) 2 , (1)

[0019] The structural similarity index (SSIM) is used to calculate the structural similarity between the input image I(x, y) and the output image O(x, y). The higher the value, the higher the similarity. The formula for the SSIM loss Lssim is:

[0020] L SSIM = 1 - SSIM(O(x, y) - I(x, y)). (2)

[0021] The local loss is the sum of the pixel loss Lp and the SSIM loss Lssim:

[0022] L local =L p +L SSIM , (3)

[0023] In order to enhance the global information acquisition capability of the image, the present invention adopts perceptual loss as a global loss factor. This loss is evaluated by comparing the convolutional features of the generated image with the baseline real image to ensure the similarity in high-level information. The main purpose of perceptual loss is to promote generalization, thereby making the architecture more versatile and task-agnostic. In terms of feature extraction, perceptual loss utilizes the VGG-16 model pre-trained on the ImageNet dataset [1]. However, due to the large parameter set of VGG-16, there are certain limitations on the training speed. To solve this problem, only the first five layers of VGG-16 are used for feature extraction, which effectively speeds up the training process without compromising the effectiveness of the model. The perceptual loss of the system is expressed as:

[0024]

[0025] Among them, MSE stands for mean square error, F i (x gt ) refers to the value of the given input label, The features extracted by the VGG-16 model are used for s. The loss function aims to achieve convergence of the CTF network and optimize three contributions.

[0026] These losses are used as the loss function for training the network in the first contribution. The total loss function value is:

[0027] L=α*L local +β*L global , (5)

[0028] α, β are weight coefficients.

[0029] Further, in video coding, complex rate-distortion optimization (RDO) process is needed when block partitioning. This process is very time-consuming. In order to replace the traditional RDO process, reduce the encoding time, and give accurate block partition mode, the present application proposes an innovative fast block partition mode decision method. Specifically, a fully connected layer is set after the CNN branch, the input of the fully connected layer is the feature information output by the CNN branch, and the output of the fully connected layer is the block partition mode: the fully connected layer between the input layer and the output layer is five block partition modes: quad tree (QT), horizontal binary tree (BH), vertical binary tree (BV), vertical ternary tree (TV), and horizontal ternary tree (TH). Therefore, the CNN branch and the following fully connected layer can directly give the block partition mode of the current CU block, thereby avoiding the complex RDO process. This method is designed for the new generation of Versatile Video Coding (VVC) standard, aiming to significantly shorten the processing time in the video coding process.

[0030] As shown in Figure 2 The present application adopts a fine-tuned convolutional neural network (CNN) branch, specifically, the branch is retrained (fine-tuned) using the VVC standard reference test condition CTC. It is integrated into the classification network designed in the present application. This network aims to guide the block partition mode decision process, and can effectively select from the five modes of block partition mode in the VVC coding standard: quad tree (QT), horizontal binary tree (BH), vertical binary tree (BV), vertical ternary tree (TV), and horizontal ternary tree (TH). The key of this method is that it significantly reduces the need for complex and time-consuming rate-distortion optimization (RDO) process. Through this innovative technology, the present application not only speeds up the video coding, but also maintains the quality of the coding with high-accuracy block partition mode decision.

[0031] In the present application, the candidate values of the weight parameter α are 1 / 4, 1 / 3, 1 / 2, 1, 2, 3, and 4, and the candidate values of the parameter β are 1 / 16, 1 / 8, 1 / 4, 1, 4, 8, and 16. By training for 100 epochs, the values of α and β that minimize the convergence value are selected. The finally determined value of α is 1 / 3, and the value of β is 4.

[0032] In the rate-distortion optimization (RDO) process in video coding, neural networks are used for feature extraction, and CNN branch fine-tuning is used to obtain fast block partition results. Through these two aspects, the intra prediction mode selection process is greatly accelerated, and the machine vision task performance can be improved. Compared with the original VVC coding standard VTM, the method proposed in the present application can reduce the encoding time by more than 30.01% and improve the target detection accuracy by 0.32. BRIEF DESCRIPTION OF DRAWINGS

[0033] Figure 1 This is the overall system diagram proposed by the present invention.

[0034] Figure 2 A block partition decision diagram is proposed for the present invention.

[0035] Figure 3 The AP50 value of the detection performance is measured under the same bit rate. DETAILED DESCRIPTION

[0036] The present invention is further described below through embodiments in conjunction with the accompanying drawings.

[0037] The fast VVC intra-frame coding method for machine video coding proposed in the present invention is specifically divided into the following steps:

[0038] (1) Train the neural network for the target machine task (e.g. Figure 1 ), the loss function is shown in formulas 1, 2, 3, and 4. In addition, the convolutional neural network (CNN) branch (such as Figure 2 ) to train the block partitioning network.

[0039] (2) Using the trained network (such as Figure 1 ) is forward passed, and before the rate-distortion optimization (RDO) of the video encoding process, the input coding block (CU) is subjected to a convolution operation. Specifically, the encoders of CNN and Transformer are connected in series to the decoder. The single encoder of the CNN block includes 1×1 convolution, batch normalization, and 3×3 convolution. The single decoder of the CNN block includes 1×1 deconvolution, batch normalization, and 3×3 deconvolution. The single encoder of the Transformer consists of layer normalization and multi-head self-attention. This neural network structure (residual network (resnet) architecture). As Figure 2 As shown, the number of output channels of the convolutional layer is 64, 128, 256, and 512 respectively;

[0040] (3) First, a convolutional neural network is trained using the VVC standard reference test conditions (CTC). CTC is a standard set of test conditions used to evaluate the performance of video coding technologies. In this process, the existing CNN model will be fine-tuned to adapt to the specific video coding task. The fine-tuning process involves training using the video dataset in CTC, which enables the CNN to extract features that are more suitable for video coding.

[0041] (4) When performing video encoding, the features of each coding unit (CU) are first extracted through a convolutional neural network. These features can effectively represent the information of the image block. In the RDO (rate-distortion optimization) process, these feature values ​​are used to optimize the encoding parameters to achieve the best encoding efficiency. The RDO process here does not include block division, which means that there is no need to re-divide the image blocks when determining how to compress the data. After RDO optimization, the video encoding is completed and a compressed video stream is generated. During video decoding, the decoder decodes the received compressed video stream and restores the video frame. The frame obtained by decoding should be as consistent as possible with the frame before encoding to ensure high-quality video reconstruction.

[0042] (5) Using the VVC standard official reference code VTM10.0, the proposed CTF network, and the combination of CTF and fast algorithm (fast), the segmentation accuracy (AP50) under six different bitrate data is obtained as the final performance measurement comparison. Using these data to draw Figure 3 , you can get the corresponding line chart.

[0043] The method of the present invention is embedded in the VVC official code VTM10.0 of the coding standard. After testing, it can reduce the time taken for intra-frame coding by 35.21% compared with the official standard. In addition, when testing the segmentation task on the public COCO dataset, an average accuracy improvement of 0.26 can be obtained.

[0044] References

[0045] [1] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint arXiv:1409.1556, 2014.

Claims

1. A fast VVC intra-frame coding method for machine video coding, characterized in that: It is based on neural network technology. Specifically, it constructs two network branches, one is the convolutional neural network CNN branch, and the other is the deep transformer transformer branch; among them: The convolutional neural network (CNN) branch includes an encoder module and a decoder module, which are connected in series. The encoder module includes two identical encoders connected in sequence, and the decoder module includes two identical decoders connected in sequence. In the CNN branch, a single encoder includes 1×1 convolution, batch normalization, and 3×3 convolution; a single decoder includes 1×1 deconvolution, batch normalization, and 3×3 deconvolution. The deep transformer transformer branch also includes an encoder module and a decoder module, which are connected in series. The encoder module consists of two identical encoders connected in sequence, and the decoder module consists of two identical decoders connected in sequence. In the transformer branch, a single encoder consists of layer normalization and multi-head self-attention. The decoder structure is the same as the encoder, except that the number of multi-head self-attention is doubled. The input video frame first passes through the two encoders of the CNN branch and the Transformer branch to extract features, and then the features are added (concat) and the results are sent to the decoding ends of the CNN and Transformer respectively. The decoding ends of the CNN and Transformer are independent of each other. After further feature extraction and restoration processes at the two decoding ends, the output of the CNN decoding end is connected to a 1*1 convolution and batch normalization (Norm), and then further pixel-wise summed with the information output by the Transformer decoding end to obtain the expected output feature information.

2. The fast VVC intra-frame coding method for machine video coding according to claim 1, characterized in that In the CNN branch: The encoder has a symmetric structure consisting of two downsampling blocks, labeled Ei (i=1,2), each designed to halve the width and height of the input at each stage; the activation function in these blocks is the rectified linear unit (ReLU), supplemented by reflective padding to maintain feature size; The decoder has a symmetrical structure and consists of two upsampling blocks, denoted as Di(i=1,2). The upsampling blocks use convolution operations with a kernel size of 2. The feature size in each block is doubled. In addition, ReLU is also used as the activation function, and the feature size is supplemented by reflection padding; Encoder output and decoder upsampling block D 3-i The inputs of (i=1,2) have the same size, E1 and D2 are symmetric, and E2 and D are symmetric.

3. The fast VVC intra-frame coding method for machine video coding according to claim 2, characterized in that The neural network training adopts the following loss functions: global information loss function and local information loss function, where: Local information loss consists of two parts: pixel loss and structural similarity loss; The pixel loss Lp is calculated by computing the variance of the input image I(x,y) and the output image O(x,y): L p =(O(x,y)-I(x,y)) 2 , (1) The structural similarity index (SSIM) is used to calculate the structural similarity between the input image I(x,y) and the output image O(x,y). The SSIM loss L SSIM The formula is: L SSIM =1-SSIM(O(x,y)-I(x,y)). (2) The local loss is the pixel loss Lp and the SSIM loss L SSIM The sum of: L local =L p +L SSIM , (3) The global information loss adopts perceptual loss, which uses the VGG-16 model pre-trained on the ImageNet dataset and only uses the first five layers of VGG-16 for feature extraction; the perceptual loss is expressed as: Among them, MSE stands for mean square error, F i (x gt ) refers to the value of the given input label, Use the features extracted by the VGG-16 model for s; The total loss function value is: L=α*L local +β*L global , (5) α, β are weight coefficients.

4. The fast VVC intra-frame coding method for machine video coding according to claim 3, characterized in that A fully connected layer is set after the CNN branch. The input of the fully connected layer is the feature information output by the CNN branch, and the output of the fully connected layer is the block partitioning mode: the fully connected layer between the input layer and the output layer implements five block partitioning modes: quadtree QT, horizontal binary tree BH, vertical binary tree BV, vertical ternary tree TV, and horizontal ternary tree TH; In addition, the CNN branch is fine-tuned. Specifically, the VVC standard reference test condition CTC is used to retrain the CNN branch, that is, fine-tune it; to guide the decision process of the block partition mode, and to make a selection among the five modes in the VVC coding standard: quadtree QT, horizontal binary tree BH, vertical binary tree BV, vertical ternary tree TV, and horizontal ternary tree TH.

Images