Video compression method based on feature domain and deformable convolution

Abstract

The invention provides a video compression method based on a feature domain and deformable convolution. The video compression method mainly comprises the following steps that a current frame xt and a reference frame # imgabs0 # firstly obtain motion information mt through a motion estimation network based on motion refinement; performing motion compensation on reconstructed motion information # imgabs1 # reference frames # imgabs2 # and # imgabs3 # through deformable convolution on the reconstructed motion information # imgabs1 # reference frames # imgabs2 # reference frames # imgabs2 # reference frames # imgabs3 # reference frames # imgabs2 # reference frames # imgabs3 # reference frames # imgabs2 # reference frames # imgabs3 # reference frames # imgabs2 # reference frames # imgabs3 # reference frames # imgabs2 # reference frames # imgabs3 # reference frames The reconstructed residual feature # imgabs5 # and the predicted feature # imgabs6 # are added to obtain a reconstructed feature # imgabs7 #, the reconstructed feature # imgabs8 # is converted into a reconstructed frame # imgabs9 # through a frame reconstruction network, and the experimental result shows that the method can effectively improve the compression effect, so that the original information is reserved as far as possible after the video is compressed.

Description

Technical Field

[0001] The present invention relates to video coding technology, and in particular to a video compression method based on deep learning, and belongs to the field of image communication. Background Art

[0002] In recent years, with the widespread popularity of high-definition video applications and the emergence of new video formats, the scale of video data has grown rapidly and reached an unprecedented peak. At the same time, with the rapid development of mobile Internet, users can upload high-definition videos anytime and anywhere, which brings severe challenges to video transmission and storage systems. Due to the increasing demand for visual content, the research on image and video compression has gradually attracted widespread attention from researchers. At present, traditional coding standards such as JPEG, JPEG2000, BPG, and AVC / H.264, HEVC / H.265, and VVC / H.266 for video compression all adopt a block-related approach, and each block is executed sequentially. Traditional image and video compression methods have achieved good performance through carefully designed artificial techniques and complex processing, but introduced block effects, which hindered the improvement of overall compression efficiency. Although each module is usually designed and optimized independently, this local optimization limits the compression performance and cannot be jointly optimized in an end-to-end manner. In recent years, with the resurgence of deep learning technology and its widespread success in the field of computer vision, video coding technology has ushered in a new research field of end-to-end automatic modeling. Unlike traditional video coding compression standards, learning-based video compression methods can more efficiently extract and utilize data features while avoiding block artifacts, thereby achieving higher quality reconstructed frames. This method breaks away from the limitations of traditional local optimization and opens up new possibilities for comprehensive improvement of video compression performance.

[0003] Recently, various learning-based video compression models have been proposed and achieved remarkable performance. Most of these models are based on a hybrid coding framework, following the process of transformation, quantization, entropy coding, and inverse transformation. Currently, most works rely only on pixel-level operations (such as motion estimation or motion compensation) to reduce redundancy. However, such pixel-level operations are difficult to generate accurate pixel-level optical flow information, which reduces the accuracy of prediction, especially for videos with complex non-rigid motion patterns. When performing motion compensation and predictive coding in the pixel domain, only shallow pixel information visible to the human eye is used, and the deep feature information learned by the neural network is not used. Therefore, how to obtain accurate optical flow information is extremely important for subsequent motion compensation and motion and residual compression.

[0004] Traditional depth video coding methods mainly establish corresponding models by analyzing the content characteristics of depth videos, thereby improving video coding efficiency. With the in-depth research of machine learning and deep learning, their data-driven parameter learning ability has achieved remarkable results in various image and video task applications, and the research on video compression coding methods based on deep learning has also received extensive attention in the academic community. Different from traditional image or video coding methods, the method based on convolutional neural network designs a deep neural network structure specifically according to the specific problems involved, and enables the network to complete a specific goal within the coding system through appropriate training strategies. Related research shows that the method based on convolutional neural network can handle more complex tasks and obtain better performance. Inspired by this, researching how to integrate convolutional neural network into the depth video coding system, optimizing the convolutional neural network model end-to-end, and giving full play to the advantages of deep learning to improve video compression ability have important research significance and broad application prospects. Summary of the Invention

[0005] Video compression technology mainly utilizes the correlation in video data and the particularity of human visual system. Then, it removes the redundancy in video data (such as RGB, YUV formats) to reduce the amount of video data. The present invention first systematically studies and summarizes the key algorithms of deep learning compression technology and their improvements, focusing on operations such as motion estimation and motion compensation in the feature domain. By adopting a deep learning-based motion optical flow estimation network, we can obtain the motion information between video frames more accurately, and refine the motion information with deformable convolution, which helps to generate motion vector prediction, thereby reducing the coding cost of motion vectors. These motion information are compressed through an autoencoder network and motion compensation is performed with the help of deformable convolution, and finally a predicted frame is generated.

[0006] The present invention proposes a video compression method based on the feature domain and deformable convolution. First, the feature representations corresponding to the original frame and the reference frame are obtained through a feature extraction network, and accurate optical flow information is obtained through motion estimation based on motion refinement. Then, deformable convolution is introduced to align the current frame to obtain a predicted frame. The overall framework of the present invention is as shown in the appendix Figure 1 as follows.

[0007] The main operation steps include:

[0008] (1) For the current frame x t and the reference frame first, motion information m is obtained through a motion estimation network based on motion refinement t ;

[0009] (2) Use a motion codec to compress the motion information to obtain the reconstructed motion information

[0010] (3) Use deformable convolution on the reference frame and to perform motion compensation to obtain the predicted features

[0011] (4) Use a residual encoder-decoder to compress the residual feature r between the input feature and the predicted feature t ;

[0012] (5) The reconstructed residual feature and the predicted feature are added together to obtain the reconstructed feature

[0013] (6) The reconstructed feature passes through a frame reconstruction network to transform the final reconstructed feature into a reconstructed frame

[0014] The feature extraction network in step (1) consists of a convolution, an activation function, and a residual block as shown in the appendix Figure 2 shown. The residual block consists of a convolution and an activation function. In this algorithm, to make the motion estimation more accurate, a motion estimation based on motion vector refinement is proposed as shown in the appendix Figure 3 shown. Motion estimation is performed in the feature domain and an optical flow generation network is used to obtain the optical flow between two frames and deformable convolution is used for alignment, and then the initial pixel-level optical flow information is refined to obtain the final motion information for motion compensation. First, the current frame x t and the reference frame obtain the motion information M t through the optical flow estimation network, and this pixel-level motion vector is transformed to the feature domain The current frame x t and the reference frame are fed into the feature extraction network to obtain the corresponding feature representations F t , Use deformable convolution to align the input feature and to obtain the predicted feature After feature alignment, use the motion vector reconstruction network to refine the feature domain to obtain the final motion vector m t ;

[0015] The specific steps of motion compensation in step (4) are as shown in the appendix Figure 4 shown. First, use the feature extraction network to transform the reference frame into the input feature, then use deformable convolution to and the decoded for alignment to obtain the predicted feature, and finally obtain a more accurate predicted feature through a predicted feature frame refinement network The rate - distortion function of the training algorithm of the present invention is as follows:

[0016]

[0017] d(·) represents the distortion measurement value between the reconstructed frame and the original frame x. Different distortion functions are adopted according to different final performance indicators of the model. Peak Signal - to - Noise Ratio (PSNR) is often used as a measurement method for signal reconstruction quality in fields such as image compression. It is often simply defined by the Mean Squared Error (MSE). And because PSNR has a negative - correlation functional relationship with MSE, when PSNR is used as the image distortion evaluation in the rate - distortion evaluation, MSE is used as the distortion term in the loss function. respectively represent the number of bits of the encoded residual information and the motion information. By adjusting the Lagrange constant λ, compressed images under different compression conditions can be obtained. In the present invention, when calculating the distortion term with MSE, let λ ∈ {256, 512, 1024, 2048}.

[0018] Advantages and beneficial technical effects of the present invention compared with the prior art:

[0019] (1) The present invention proposes a video compression method based on the feature domain and deformable convolution. This method performs all operations in the feature domain, making motion estimation and motion compensation more accurate, rather than following the pixel - level operations in traditional video codecs.

[0020] (2) The present invention applies deformable convolution to motion estimation and motion compensation. The deformable convolution is used to refine the motion vectors after optical flow estimation and is used in motion compensation. The deformable convolution can better adapt to the differences between frames, thus achieving more accurate alignment.

[0021] (3) Experimental results show that this method can obtain better compression effects at the objective index level. Brief Description of the Drawings

[0022] Figure 1 is a block diagram of a video compression method based on deformable convolution of the present invention.

[0023] Figure 2 is the network structure diagram of the feature extractor in the present invention.

[0024] Figure 3 is the network structure diagram of motion estimation based on motion vector refinement in the present invention.

[0025] Figure 4 is the network structure diagram of motion compensation based on deformable alignment in the present invention.

[0026] Figure 5Rate-distortion curves of different methods on the HEVC Class B test set.

[0027] Figure 6 Rate-distortion curves of different methods on the HEVC Class C test set. Detailed implementation manners

[0028] The present invention will be further described in detail below with reference to embodiments. It should be noted that the following embodiments are only used to further illustrate the present invention and should not be construed as limiting the protection scope of the present invention. Those skilled in the art can make some non-essential improvements and adjustments to the present invention according to the above invention content and implement it specifically, which should still fall within the protection scope of the present invention.

[0029] (1) The present invention proposes a video compression method based on deformable convolution. To verify the effectiveness of the proposed algorithm, the HEVC standard test sequences, which are often used for testing video compression effects, are selected as the experimental test set. The average value of the compression results is taken as the final result, and its compression effect is measured. The metric used for the compression rate is the average bits per pixel (bpp), which is calculated by dividing the total number of bits of the encoded image by the total number of pixels. The Vimeo-90K dataset is used in the training phase, which contains 89,800 video clips, each video has 7 frames, and the resolution is 448×256. The video sequences are randomly cropped to a resolution of 256×256 before training. The Adam optimization algorithm is used for training, and the initial learning rate is set to 0.0001, and the size of a single training group is 32.

[0030] (2) The training set is input into our network to train four best models optimized with MSE. Each best model is used for the test set to obtain the compressed pictures. We use metrics such as MSE and bpp to measure the compression effects of the method of the present invention and other methods.

[0031] (3) Figure 5 、 6 The rate-distortion curves of different methods on the HEVC Class B and HEVC Class C test sets are given respectively. The present invention is not only superior to traditional coding methods and deep learning-based methods DVC and DVC_Pro in the full bitrate range, but also shows better performance than MOVI-Codec in the medium and high bitrate ranges. This indicates that under the condition of the same bitrate, the reconstructed video PSNR of the method proposed by us is higher.

Claims

1. A video compression method based on the feature domain and deformable convolution, characterized in that Perform operations such as motion estimation and motion compensation in the feature space, and use deformable convolution for motion estimation and motion compensation. The steps are as follows: (1) Current frame x t and the reference frame First, the motion information m is obtained through a motion estimation network based on motion refinement t , the current frame x t and the reference frame The motion information M is obtained through an optical flow estimation network t , and this pixel-level motion vector is feature-transformed into the feature domain The current frame x t and the reference frame are fed into a feature extraction network to obtain the corresponding feature representation F t , Using deformable convolution to align the input features and to obtain the predicted features After feature alignment, the motion vector reconstruction network is used to refine the feature domain to obtain the final motion vector m t , the process of motion estimation can be expressed by formula (1) as follows: Among them, FlowNet is an optical flow estimation network, FT is a feature transformation network, FE is a feature extraction network, Deform is a deformable convolution, and D re is a motion vector reconstruction network; (2) Compress the motion information using a motion codec to obtain the reconstructed motion information (3) Use deformable convolution for the reference frame and perform motion compensation to obtain the predicted features (4) Use a residual encoder-decoder to compress the residual feature r between the input feature and the predicted feature t ; (5) The reconstructed residual features and the predicted features are added together to obtain the reconstructed features (6) Reconstructed features Through the frame reconstruction network, the final reconstructed features are transformed into reconstructed frames

Images