Fast VVC intra-frame coding method for machine video coding

Abstract

The invention belongs to the technical field of video compression coding and decoding, and particularly relates to a fast VVC intra-frame coding method for machine video coding. According to the method, a neural network technology is adopted, and encoders and decoders of a convolutional neural network (CNN) block and a deep former block are connected in series; and efficient and accurate coding is realized, so that the segmentation judgment of the coding block is more accurate. According to the method, block division is guided by retraining the CNN network, the complexity of a rate distortion optimization (RDO) process is reduced, and the method is particularly suitable for a new generation of universal video coding standard VVC. According to the method, the coding complexity can be reduced, and meanwhile, the video coding and decoding performance is remarkably improved, especially in a machine-oriented application scene.

Description

Technical Field

[0001] The present invention belongs to the technical field of video compression coding and decoding, and in particular relates to a VVC intra-frame coding method for machine video coding. Background Art

[0002] Advanced video coding technologies such as high-efficiency video coding (HEVC) and new generation video coding (VVC) have emerged to meet the growing demand for high-resolution and high-frame-rate videos. However, these codecs are mainly designed to meet the needs of human vision and are not suitable for machine vision tasks such as image classification, object detection, and semantic segmentation. In recent years, many research works have begun to explore the use of coding blocks (CUs) to extract features from machine video coding. For example, in 2020, Gao Wen et al. proposed a set of machine-oriented coding and decoding frameworks and solutions for collaborative compression and intelligent perspective. In 2021, Ahonen et al. developed a learning post-processing filter that significantly improved the efficiency of machine vision tasks applied to VVC encoded video restoration images. However, existing machine-oriented video coding and decoding technologies are mainly based on convolutional neural networks (CNNs) to extract features, which is insufficient in capturing and extracting global features. To this end, the present invention combines CNN and Transformer networks to integrate global and local feature information, replacing the traditional rate-distortion (RDO) process, thereby improving feature representation capabilities and better adapting to the needs of machine video coding.

[0003] In the VVC standard, intra-frame coding introduces binary tree and ternary tree divisions, and compared with the quadtree of the HEVC standard, five block division modes are added, which greatly increases the complexity of coding. In order to cope with the additional coding complexity brought by the machine video coding neural network and accelerate the speed of intra-frame coding under the VVC standard, the present invention proposes a machine-oriented fast intra-frame block division method. This method retrains the CNN network designed by the present invention, and can directly determine the block division mode without going through the RDO process of traditional encoding and decoding, effectively improving the encoding and decoding speed for machines. Summary of the invention

[0004] The present invention aims to propose a fast VVC intra-frame coding method for machine video coding to meet the needs of machine video coding, which can efficiently process video content, ensure coding quality and significantly improve processing speed.

[0005] The fast VVC intra-frame coding method for machine video coding proposed in the present invention is based on a neural network. Figure 1 Specifically, it includes two network branches, one is a convolutional neural network (CNN) branch, and the other is a deep transformer branch; wherein:

[0006] A convolutional neural network (CNN) branch includes an encoder module and a decoder module, which are connected in series; wherein the encoder module includes two identical encoders connected in sequence, and the decoder module includes two identical decoders connected in sequence; in the CNN branch, a single encoder includes 1×1 convolution, batch normalization, and 3×3 convolution; a single decoder includes 1×1 deconvolution, batch normalization, and 3×3 deconvolution.

[0007] The deep transformer branch also includes an encoder module and a decoder module, which are connected in series. The encoder module includes two identical encoders connected in sequence, and the decoder module includes two identical decoders connected in sequence. In the transformer branch, a single encoder consists of layer normalization and multi-head self-attention. The decoder structure is the same as the encoder, except that the number of multi-head self-attention is doubled.

[0008] The input video frame first extracts features through the two encoders of the CNN branch and the Transformer branch respectively, and then the features are added (concat) and the results are sent to the decoding ends of the CNN and Transformer respectively. The decoding ends of the CNN and Transformer are independent of each other. After further feature extraction and restoration by the two decoding ends, the output of the CNN decoding end is connected to a 1*1 convolution and batch normalization (Norm), and then further pixel-wise summed with the information output by the Transformer decoding end to obtain the expected output feature information.

[0009] The neural network structure of the present invention is a residual network (resnet) architecture, such as Figure 2 shown.

[0010] The encoder and decoder of the CNN branch are symmetrical structures. The encoder of the CNN has two convolutional layers. The CNN branch uses the residual network (ResNet) architecture to integrate the encoder features with the decoder features of the same size.

[0011] The encoder of the Transformer branch controls the feature size through average pooling. The Transformer block with upsampling operation is the decoder of the Transformer block. The outputs of the CNN and Transformer encoders are connected to the CNN and Transformer decoders respectively. In this way, both global and local data are introduced into the dual branches, thereby improving the description of image features. Figure 3 As shown in Figure 1, the CNN branch consists of an encoder with two downsampling operation blocks and a decoder with two upsampling blocks. Convolution and deconvolution operations are used for feature extraction in the encoder and decoder architectures. ResNet connections pass feature information from the encoder to the decoder.

[0012] In CNN, there are encoders and decoders. The encoder has a symmetrical structure and includes two downsampling blocks, labeled Ei (i=1,2), each of which is designed to halve the width and height of the input at each stage. These downsampling blocks form the core of the CNN branch encoder, which mainly consists of 3×3 convolution operations to achieve efficient feature extraction. The activation function in these blocks is the rectified linear unit (ReLU), supplemented by reflection padding to maintain the feature size, ensuring a balance between feature extraction efficiency and size consistency.

[0013] The decoder has a symmetric structure and contains two upsampling blocks, denoted as Di(i=1,2). The upsampling blocks use convolution operations with a kernel size of 2. The feature size is doubled in each block. In addition, ReLU is also used as an activation function, and the feature size is supplemented by reflection padding.

[0014] Encoder output and decoder upsampling block D 3-i The inputs of (i=1,2) have the same size, E1(EncoderBlock1) and D2(Decoder Block 2) are symmetric, and E2(Encoder Block 2) and D1(Decoder Block 1) are symmetric.

[0015] This CNN branch also introduces the ResNet architecture to obtain rich feature information and better network convergence performance. The features of the encoder are connected to the decoder of the same size, which promotes the flow of feature information in the deep layer. The decoder and encoder are symmetrical structures for image fusion feature extraction. In this way, the high-resolution output of the CNN branch is the same as the input image.

[0016] In order to meet the global feature and local feature extraction capabilities, the global and local loss functions are used to train this neural network, that is, the loss function contains global information and local information. The local information loss consists of two parts: pixel loss and structural similarity loss.

[0017] The pixel loss Lp is calculated by calculating the variance of the input image I(x,y) and the output image O(x,y):

[0018] L p =(O(x,y)-I(x,y)) 2 , (1)

[0019] The structural similarity index (SSIM) is used to calculate the structural similarity between the input image I(x,y) and the output image O(x,y). The higher the value, the higher the similarity. The formula for the SSIM loss Lssim is:

[0020] L SSIM =1-SSIM(O(x,y)-I(x,y)). (2)

[0021] The local loss is the sum of the pixel loss Lp and the SSIM loss Lssim:

[0022] L local =L p +L SSIM , (3)

[0023] In order to enhance the global information acquisition capability of the image, the present invention adopts perceptual loss as a global loss factor. The loss is evaluated by comparing the convolutional features of the generated image with the baseline real image to ensure the similarity in high-level information. The main purpose of perceptual loss is to promote generalization, thereby making the architecture more versatile and task-agnostic. In terms of feature extraction, perceptual loss uses the VGG-16 model pre-trained on the ImageNet dataset [1]. However, due to the large parameter set of VGG-16, there are certain limitations on the training speed. To solve this problem, only the first five layers of VGG-16 are used for feature extraction, which effectively speeds up the training process without compromising the effectiveness of the model. The perceptual loss of the system is expressed as:

[0024]

[0025] Among them, MSE stands for mean square error, F i (x gt ) refers to the value of a given input label, The features extracted by the VGG-16 model are used for s. The loss function aims to achieve convergence of the CTF network and optimize three contributions.

[0026] These losses are used as the loss function for training the network in the first contribution. The total loss function value is:

[0027] L=α*L local +β*L global , (5)

[0028] α, β are weight coefficients.

[0029] Furthermore, in video encoding and decoding, a complex rate-distortion optimization (RDO) process is required when performing block division, which is very time-consuming. In order to replace the traditional RDO process, reduce the encoding time, and provide an accurate block division mode. The present invention proposes an innovative fast block division mode judgment method. Specifically, a fully connected layer is set after the CNN branch, the input of the fully connected layer is the feature information output by the CNN branch, and the output of the fully connected layer is the block division mode: the fully connected layer between the input layer and the output layer is five block division modes: quadtree (QT), horizontal binary tree (BH), vertical binary tree (BV), vertical ternary tree (TV), horizontal ternary tree (TH). Therefore, this CNN branch and the subsequent fully connected layer can directly give the block division mode of the current CU block, thereby avoiding the complex RDO process. This method is designed for the new generation of universal video coding standard (VVC), aiming to significantly shorten the processing time during video encoding.

[0030] like Figure 2 As shown, the present invention adopts a fine-tuned convolutional neural network (CNN) branch, specifically using the VVC standard reference test condition CTC, and the current branch is retrained (fine-tuned). It is integrated into the classification network designed by the present invention. This network is intended to guide the decision process of the block partition mode, and can effectively make choices among the five modes of block partition modes quadtree (QT), horizontal binary tree (BH), vertical binary tree (BV), vertical ternary tree (TV), and horizontal ternary tree (TH) in the VVC coding standard. The key to this method is that it significantly reduces the need for complex and time-consuming rate-distortion optimization (RDO) processes. Through this innovative technology, the present invention not only speeds up the speed of video encoding, but also makes high-accuracy block partition mode decisions while maintaining the quality of encoding.

[0031] The weights are determined in the present invention, and the candidate values ​​of the parameter α are 1 / 4, 1 / 3, 1 / 2, 1, 2, 3 and 4, and the candidate values ​​of the parameter β are 1 / 16, 1 / 8, 1 / 4, 1, 4, 8 and 16. By training for 100 epochs, the α and β values ​​that minimize the convergence value are selected. The final determined α value is 1 / 3, and the β value is 4.

[0032] In the process of rate-distortion optimization (RDO) in video encoding and decoding, neural networks are used for feature extraction, and CNN branches are fine-tuned to obtain fast block division results. These two aspects greatly accelerate the mode selection process of intra-frame prediction and improve the performance of machine vision tasks. Compared with the original VVC coding standard VTM in different data sets, the method proposed in this invention can reduce the encoding time by more than 30.01% and improve the target detection accuracy by 0.32. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 This is the overall system diagram proposed by the present invention.

[0034] Figure 2 A block partition decision diagram is proposed for the present invention.

[0035] Figure 3 It is the value of AP50 of the detection performance under the premise of the same bit rate. DETAILED DESCRIPTION

[0036] The present invention is further described below by way of embodiments in conjunction with the accompanying drawings.

[0037] The fast VVC intra-frame coding method for machine video coding proposed in the present invention is specifically divided into the following steps:

[0038] (1) Train the neural network for the target machine task (e.g. Figure 1 ), the loss function is shown in formulas 1, 2, 3, and 4. In addition, the convolutional neural network (CNN) branch (such as Figure 2 ) to train the block partitioning network.

[0039] (2) Using the trained network (such as Figure 1 ) is forward passed, and before the rate-distortion optimization (RDO) of the video encoding process, the input coding block (CU) is used for convolution operation. Specifically, the encoders of CNN and Transformer are connected in series to the decoder. The single encoder of the CNN block includes 1×1 convolution, batch normalization, and 3×3 convolution. The single decoder of the CNN block includes 1×1 deconvolution, batch normalization, and 3×3 deconvolution. The single encoder of the Transformer consists of layer normalization and multi-head self-attention. The neural network structure (residual network (resnet) architecture). Figure 2 As shown, the number of output channels of the convolutional layer is 64, 128, 256, and 512 respectively;

[0040] (3) First, the convolutional neural network is trained using the VVC standard reference test conditions (CTC). CTC is a standard set of test conditions used to evaluate the performance of video coding technologies. In this process, the existing CNN model will be fine-tuned to adapt to specific video coding tasks. The fine-tuning process involves training using a video dataset in CTC, which enables the CNN to extract features that are more suitable for video coding.

[0041] (4) When performing video encoding, the features of each coding unit (CU) are first extracted through a convolutional neural network. These features can effectively represent the information of the image block. In the RDO (rate-distortion optimization) process, these feature values ​​are used to optimize the encoding parameters to achieve the best coding efficiency. The RDO process here does not include block division, which means that there is no need to re-divide the image blocks when determining how to compress the data. After RDO optimization, the video encoding is completed and a compressed video stream is generated. During video decoding, the decoder decodes the received compressed video stream and restores the video frame. The frame obtained by decoding should be as consistent as possible with the frame before encoding to ensure high-quality video reconstruction.

[0042] (5) Using the VVC standard official reference code VTM10.0, the proposed CTF network, and the combination of CTF and fast algorithm (fast), the segmentation accuracy (AP50) under six different bitrate data is obtained as the final performance measurement comparison. Using these data, we draw Figure 3 , you can get the corresponding line chart.

[0043] The method of the present invention is embedded into the VVC official code VTM10.0 of the coding standard. After testing, it can reduce the time taken by 35.21% compared with the official standard intra-frame coding. In addition, when the segmentation task is tested on the public COCO dataset, an average accuracy improvement of 0.26 can be obtained.

[0044] References

[0045] [1] Simonyan K, Zisserman A. Very deep convolutional networks for large-scale image recognition[J]. arXiv preprint arXiv:1409.1556, 2014.

Claims

1. A fast VVC intra-frame coding method for machine video coding, characterized in that: It is based on neural network technology. Specifically, two network branches are constructed, one is the convolutional neural network CNN branch, and the other is the deep transformer transformer branch; among them: A convolutional neural network (CNN) branch includes an encoder module and a decoder module, which are connected in series; wherein the encoder module includes two identical encoders connected in sequence, and the decoder module includes two identical decoders connected in sequence; in the CNN branch, a single encoder includes 1×1 convolution, batch normalization, and 3×3 convolution; a single decoder includes 1×1 deconvolution, batch normalization, and 3×3 deconvolution; The deep transformer transformer branch also includes an encoder module and a decoder module, which are connected in series; the encoder module includes two identical encoders connected in sequence, and the decoder module includes two identical decoders connected in sequence; in the transformer branch, a single encoder is composed of layer normalization and multi-head self-attention; the decoder structure is the same as the encoder, except that the number of multi-head self-attention is doubled; The input video frame is firstly extracted features by two encoders of CNN branch and Transformer branch respectively, and then the features are added (concat) and the results are sent to the decoding ends of CNN and Transformer respectively. The decoding ends of CNN and Transformer are independent of each other. After further feature extraction and restoration process of the two decoding ends, the output of CNN decoding end is connected with a 1*1 convolution and batch normalization (Norm), and then further pixel-wise summed with the information output by Transformer decoding end to obtain the expected output feature information.

2. The fast VVC intra-frame coding method for machine video coding according to claim 1, characterized in that: In the CNN branch: The encoder has a symmetrical structure, consisting of two downsampling blocks, labeled Ei (i=1,2), each of which is designed to halve the width and height of the input at each stage; the activation function in these blocks is the rectified linear unit (ReLU), supplemented by reflective padding to maintain the feature size, ensuring a balance between feature extraction efficiency and size consistency; The decoder has a symmetrical structure and contains two upsampling blocks, denoted as Di(i=1,2); the upsampling block uses a convolution operation with a kernel size of 2; the feature size in each block is doubled; In addition, ReLU is also used as an activation function, and the feature size is supplemented by reflection padding; Encoder output and decoder upsampling block D 3-i The inputs of (i=1,2) have the same size, E1 and D2 are symmetric, and E2 and D are symmetric.

3. The fast VVC intra-frame coding method for machine video coding according to claim 2, characterized in that: The neural network training adopts the following loss functions: global information loss function and local information loss function, where: Local information loss consists of two parts: pixel loss and structural similarity loss; The pixel loss Lp is calculated by calculating the variance of the input image I(x,y) and the output image O(x,y): L p =(O(x,y)-I(x,y)) 2 , (1) The structural similarity index (SSIM) is used to calculate the structural similarity between the input image I(x, y) and the output image O(x, y). The formula of SSIM loss Lssim is: L SSIM =1-SSIM(O(x,y)-I(x,y)). (2) The local loss is the sum of the pixel loss Lp and the SSIM loss Lssim: L local =L p +L SSIM , (3) The global information loss adopts perceptual loss, which uses the VGG-16 model pre-trained on the ImageNet dataset and only uses the first five layers of VGG-16 for feature extraction; the perceptual loss is expressed as: Among them, MSE stands for mean square error, F i (x gt ) refers to the value of a given input label, Use the features extracted by the VGG-16 model for s; The total loss function value is: L=α*L lpcal +β*L global , (5) α, β are weight coefficients.

4. The fast VVC intra-frame coding method for machine video coding according to claim 3, characterized in that: In order to reduce the encoding time in video encoding and decoding and give an accurate block partitioning mode, a fully connected layer is set after the CNN branch. The input of the fully connected layer is the feature information output by the CNN branch, and the output of the fully connected layer is the block partitioning mode: the fully connected layer between the input layer and the output layer implements five block partitioning modes: quadtree QT, horizontal binary tree BH, vertical binary tree BV, vertical ternary tree TV, horizontal ternary tree TH; And fine-tune the CNN branch, specifically, use the VVC standard reference test condition CTC to retrain the CNN branch, i.e., fine-tune it; to guide the decision process of the block division mode, and make a selection among the five modes of quadtree QT, horizontal binary tree BH, vertical binary tree BV, vertical ternary tree TV, and horizontal ternary tree TH in the VVC coding standard.

Images