End-to-end video compression method and system based on self-attention mechanism

Abstract

The present invention discloses an end-to-end video compression method and system based on a self-attention mechanism, which relates to the field of video coding, and the method includes: extracting shallow features of a current frame, a motion reference frame and a reconstructed frame at the previous moment; extracting motion information features between the current frame and the motion reference frame; compressing the motion information features to obtain compressed motion information features; compensating the compressed motion information features on the shallow features of the reconstructed frame at the previous moment to obtain context information features at the current moment; compressing to obtain compressed context information features; encoding the compressed context information features and the shallow features of the current frame to fuse features to obtain the reconstructed frame at the current moment; upsampling the reconstructed frame at the current moment to obtain a compressed viewpoint image; correcting the motion information accumulated error in the compressed viewpoint image to obtain the motion reference frame of the next frame. The present invention can improve the accuracy of motion estimation while improving the efficiency of coding.

Description

Technical Field

[0001] The present invention relates to the field of video coding technology, and in particular to an end-to-end video compression method and system based on a self-attention mechanism. Background Art

[0002] In recent years, about 82% of Internet traffic is provided by video, and almost half of the devices support video functions. Therefore, the study of efficient video compression algorithms is crucial for the storage and transmission of video data. Due to the strong temporal correlation between frames, the quality of the reference frame directly affects the encoding quality of subsequent frames. High-quality reference frames can ensure more stable encoding quality, which is particularly important for low-latency video encoding. The traditional coding technology H.265 uses loop filtering to reduce block effects and noise to effectively reduce the reference frame cumulative error and improve compression efficiency. However, the optimization space of traditional loop filters is limited, and its performance improvement mainly depends on manual design and empirical parameter adjustment. In recent years, many deep learning-based video compression algorithms have shifted from manual design to automatic learning, aiming to improve compression efficiency by improving encoder internal algorithms such as motion estimation and coding frames. However, these methods usually directly use the reconstructed frame as the reference frame for the next frame without further processing. This practice may cause cumulative errors in the motion information in the reference frame, fail to fully utilize the temporal and spatial correlation between frames, and reduce compression efficiency.

[0003] Therefore, how to further reduce the motion information error in the reference frame to achieve high-quality video compression has become an urgent problem to be solved. Summary of the invention

[0004] The purpose of the present invention is to provide an end-to-end video compression method and system based on a self-attention mechanism, which improves the accuracy of motion estimation and the efficiency of encoding while correcting the calculation of motion errors in the encoding process based on a convolutional neural network.

[0005] The present invention adopts the following technical solution:

[0006] On the one hand, an end-to-end video compression method based on a self-attention mechanism includes:

[0007] A shallow feature extraction step is to extract shallow features of the current frame, the motion reference frame and the reconstructed frame at the previous moment;

[0008] The motion estimation step extracts motion information features between the current frame and the motion reference frame based on the shallow features of the current frame and the shallow features of the motion reference frame;

[0009] A motion compression step, compressing the motion information features to obtain compressed motion information features;

[0010] The motion compensation step compensates the compressed motion information features on the shallow features of the reconstructed frame at the previous moment to obtain the context information features at the current moment;

[0011] A conditional compression step, compressing the context information features to obtain compressed context information features;

[0012] The conditional encoding step encodes the compressed context information features and the shallow features of the current frame to obtain the reconstructed frame at the current moment;

[0013] The video reconstruction step upsamples the reconstructed frame at the current moment to obtain a reconstructed compressed viewpoint image;

[0014] The motion information correction step corrects the accumulated error of the motion information in the reconstructed compressed viewpoint image to obtain a motion reference frame for the next frame.

[0015] Preferably, the shallow features of the current frame are extracted , which is expressed as follows:

[0016] ;

[0017] in, represents the current frame; c represents the number of output channels; t represents the time of the input frame; Represents the shallow features of the current frame at time t; Represents a shallow feature extraction network function;

[0018] Shallow features of extracted motion reference frames , which is expressed as follows:

[0019] ;

[0020] in, represents the motion reference frame;

[0021] Extract the shallow features of the reconstructed frame at the previous moment , which is expressed as follows:

[0022] ;

[0023] in, Represents the reconstructed frame at the previous time t-1.

[0024] Preferably, shallow feature extraction network function , which is expressed as follows:

[0025] ;

[0026] in, Represents the convolution kernel as , a convolutional layer function with output channels c; The convolution kernel is , the number of output channels is The residual block.

[0027] Preferably, the motion information feature is expressed as follows:

[0028] ;

[0029] in, represents the motion information characteristics at time t, represents the height of the feature map, Indicates the width of the feature map; Indicates that the features are concatenated in the channel dimension; The convolution kernel is , a convolutional layer function with output channels c; Represents the shallow features of the frame at time t; Shallow features representing motion reference frames.

[0030] Preferably, the motion compression step specifically comprises: compressing the motion information features by an entropy coding function to obtain compressed motion information features;

[0031] The compressed motion information features are expressed as follows:

[0032] ;

[0033] in, Represents the compressed motion information features, represents the height of the feature map, Indicates the width of the feature map; represents the entropy coding function; Represents the quantization process; Represents the motion information features at time t.

[0034] Preferably, the motion compensation step is specifically as follows: compensating the compressed motion information features on the shallow features of the reconstructed frame at the previous moment through deformable convolution and res blocks to obtain the context information features at the current moment;

[0035] The context information features at the current moment are expressed as follows:

[0036] ;

[0037] in, represents the context information feature at the current time t, represents the height of the feature map, Indicates the width of the feature map; The convolution kernel is , a convolutional layer function with output channels c; The convolution kernel is , the number of output channels is The residual block of Represents a variable convolution block; The convolution kernel is , a convolutional layer function with output channels c; Represents the compressed motion information characteristics; Represents the shallow features of the reconstructed frame at the previous moment.

[0038] Preferably, the conditional encoding step is specifically as follows: encoding the compressed context information features and the shallow feature fusion features of the current frame through a conditional encoder to obtain a reconstructed frame at the current moment;

[0039] The reconstructed frame at the current moment is expressed as follows:

[0040] ;

[0041] in, represents the reconstructed frame at the current moment, represents the height of the feature map, Indicates the width of the feature map; represents the conditional encoding function; Represents the shallow features of the current frame; Represents the context information features at the current moment.

[0042] Preferably, the obtained reconstructed compressed viewpoint image is expressed as follows:

[0043] ;

[0044] in, represents the reconstructed compressed viewpoint image, represents the height of the feature map, Indicates the width of the feature map; The convolution kernel is , a deconvolution layer function with the number of output channels being c; The convolution kernel is , the number of output channels is The residual block of Represents the reconstructed frame at the current moment.

[0045] Preferably, the motion compensation reference frame at the current moment is represented as follows:

[0046] ;

[0047] in, Represents the motion reference frame for the next moment; Represents the convolution kernel as , the function of the convolutional layer with output channels c; Indicates that the features are concatenated in the channel dimension; Represents the aggregation and reconstruction network function; represents the full receptive field network function; Represents a multi-scale feature extraction network function; represents the reconstructed compressed viewpoint image; Indicates the current frame.

[0048] On the other hand, an end-to-end video compression system based on a self-attention mechanism includes:

[0049] A shallow feature extraction module is used to extract shallow features of the current frame, the motion reference frame and the reconstructed frame at the previous moment;

[0050] A motion estimation module, used for extracting motion information features between the current frame and the motion reference frame based on shallow features of the current frame and shallow features of the motion reference frame;

[0051] A motion compression module, used for compressing motion information features to obtain compressed motion information features;

[0052] A motion compensation module is used to compensate the compressed motion information features on the shallow features of the reconstructed frame at the previous moment to obtain the context information features at the current moment;

[0053] A conditional compression module, used for compressing context information features to obtain compressed context information features;

[0054] The conditional encoding module is used to encode the compressed context information features and the shallow features of the current frame to obtain the reconstructed frame at the current moment;

[0055] A video reconstruction module is used to upsample the reconstructed frame at the current moment to obtain a reconstructed compressed viewpoint image;

[0056] The motion information correction module is used to correct the accumulated error of motion information in the reconstructed compressed viewpoint image to obtain a motion reference frame for the next frame.

[0057] Compared with the prior art, the present invention has the following beneficial effects:

[0058] (1) The present invention combines the advantages of deformable convolution and conditional coding to improve the compression performance of video;

[0059] (2) The present invention improves the compression performance of the video by correcting the accumulated error of the motion information, thereby improving the accuracy of the motion information in the reference frame. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 A brief flowchart of an end-to-end video compression method based on a self-attention mechanism according to an embodiment of the present invention;

[0061] Figure 2 A schematic diagram of a network model of a method according to an embodiment of the present invention;

[0062] Figure 3 A schematic diagram of condition generation for an embodiment of the present invention;

[0063] Figure 4 A schematic diagram of motion information correction according to an embodiment of the present invention;

[0064] Figure 5 This is a structural block diagram of the end-to-end video compression device based on the self-attention mechanism of the present invention. DETAILED DESCRIPTION

[0065] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0066] See also Figure 1 and Figure 2 As shown, the present invention provides an end-to-end video compression method based on a self-attention mechanism, comprising the following steps.

[0067] In the shallow feature extraction step S101, shallow features of the current frame, the motion reference frame and the reconstructed frame at the previous moment are extracted.

[0068] Specifically, the shallow features of the current frame are extracted , which is expressed as follows:

[0069] ;

[0070] in, represents the current frame; c represents the number of output channels; t represents the time of the input frame; Represents the shallow features of the current frame at time t; Represents a shallow feature extraction network function;

[0071] Shallow features of extracted motion reference frames , which is expressed as follows:

[0072] ;

[0073] in, represents the motion reference frame;

[0074] Extract the shallow features of the reconstructed frame at the previous moment , which is expressed as follows:

[0075] ;

[0076] in, Represents the reconstructed frame at the previous time t-1.

[0077] The shallow feature extraction network function , using 3 downsampling convolutions with c output channels and 3 residual blocks, expressed as follows:

[0078] ;

[0079] in, The convolution kernel is , the function of the up-sampled convolutional layer with channel c; The convolution kernel is , the number of output channels is The residual block.

[0080] Residual Block It is expressed as follows:

[0081] ;

[0082] in, Represents the function of the ReLU unit; Represents input features; The convolution kernel is , the function of the convolutional layer with c output channels;

[0083] The motion estimation step S102 extracts motion information features between the current frame and the motion reference frame based on shallow features of the current frame and shallow features of the motion reference frame.

[0084] See also Figure 3 FIG. 1 is a schematic diagram of motion estimation, motion compression and motion compensation of the present invention. Specifically, the motion information feature is represented as follows:

[0085] ;

[0086] in, represents the motion information characteristics at time t, represents the height of the feature map, Indicates the width of the feature map; Indicates that the features are concatenated in the channel dimension; The convolution kernel is , a convolutional layer function with output channels c; Represents the shallow features of the frame at time t; Shallow features representing motion reference frames.

[0087] In the motion compression step S103, the motion information features are compressed to obtain compressed motion information features.

[0088] Specifically, the motion information feature is compressed by an entropy coding function to obtain the compressed motion information feature;

[0089] The compressed motion information features are expressed as follows:

[0090] ;

[0091] in, Represents the compressed motion information characteristics; represents the entropy coding function; Represents the motion information characteristics at time t; Represent the quantization process as follows:

[0092] ;

[0093] in, Indicates the range of uniform distribution.

[0094] In the motion compensation step S104, the compressed motion information features are compensated on the shallow features of the reconstructed frame at the previous moment to obtain the context information features at the current moment.

[0095] Specifically, the compressed motion information features are compensated on the shallow features of the reconstructed frame at the previous moment through deformable convolution and residual blocks to obtain the context information features at the current moment;

[0096] The context information features at the current moment are expressed as follows:

[0097] ;

[0098] in, represents the context information feature at the current time t, represents the height of the feature map, Indicates the width of the feature map; Represents the convolution kernel as , a convolutional layer function with output channels c; Represents the convolution kernel as , the number of output channels is The residual block of Represents a variable convolution block; Represents the convolution kernel as , a convolutional layer function with output channels c; Represents the compressed motion information characteristics; Represents the shallow features of the reconstructed frame at the previous moment.

[0099] The conditional compression step S105 compresses the context information features to obtain compressed context information features.

[0100] Specifically, the context information feature is compressed by an entropy coding function to obtain the compressed context information feature.

[0101] In the conditional encoding step S106, the compressed context information features and the shallow features of the current frame are fused and encoded to obtain a reconstructed frame at the current moment.

[0102] Specifically, the compressed context information features and the shallow features of the current frame are fused and encoded through a conditional encoder to obtain a reconstructed frame at the current moment;

[0103] The reconstructed frame at the current moment is expressed as follows:

[0104] ;

[0105] in, represents the reconstructed frame at the current moment, represents the height of the feature map, Indicates the width of the feature map; represents the conditional encoding function; Represents the shallow features of the current frame; Represents the context information features at the current moment.

[0106] In the video reconstruction step S107, up-sampling is performed on the reconstructed frame at the current moment to obtain a reconstructed compressed viewpoint image.

[0107] Specifically, the obtained reconstructed compressed viewpoint image is expressed as follows:

[0108] ;

[0109] in, represents the reconstructed compressed viewpoint image, represents the height of the feature map, Indicates the width of the feature map; Represents the convolution kernel as , a deconvolution layer function with the number of output channels being c; Represents the convolution kernel as , the number of output channels is The residual block of Represents the reconstructed frame at the current moment.

[0110] The motion information correction step S108 corrects the accumulated error of the motion information in the reconstructed compressed viewpoint image to obtain a motion reference frame for the next frame. Figure 4 shown.

[0111] Specifically, the motion compensation reference frame at the current moment is expressed as follows:

[0112] ;

[0113] in, Represents the motion reference frame for the next frame; Represents the convolution kernel as , the function of the convolutional layer with output channels c; Indicates that the features are concatenated in the channel dimension; Represents the aggregation and reconstruction network function; represents the full receptive field network function; Represents a multi-scale feature extraction network function; represents the reconstructed compressed viewpoint image; Indicates the current frame.

[0114] In this embodiment, the number of channels c can be set to 64.

[0115] In the specific implementation, the Vimeo-90K dataset containing 89,800 video clips is used to train the model of the proposed method, and the images are randomly cropped from the original resolution of 448 × 256 to 256 × 256. Furthermore, six test datasets, namely VTL, UVG, MCL-JCV, HEVC Class B, HEVC Class C, and HEVC Class D, are used to evaluate the performance of the proposed method. The details are shown in Table 1.

[0116] Table 1 Comparison of BDBR with other methods on six datasets;

[0117]

[0118] It can be seen from Table 1 that, in terms of the overall results of the benchmark dataset, the end-to-end video compression method based on the self-attention mechanism of this embodiment saves a certain bit rate (the negative sign in Table 1 indicates the bit rate saving). Therefore, the end-to-end video compression method based on the self-attention mechanism of this embodiment is superior to DVC and other video compression methods.

[0119] In addition, this embodiment uses the traditional coding platform HEVC as the base point and uses BDBR to measure the average bit rate difference between the two rate-distortion curves. These test videos are widely used in the evaluation of traditional and neural video codecs. And in order to improve the model calculation efficiency, the video resolution in the test set is cropped to a multiple of 64 in this embodiment.

[0120] See also Figure 5 As shown, the present invention also discloses an end-to-end video compression system based on a self-attention mechanism, comprising:

[0121] A shallow feature extraction module 501 is used to extract shallow features of the current frame, the motion reference frame and the reconstructed frame at the previous moment;

[0122] A motion estimation module 502, configured to extract motion information features between the current frame and the motion reference frame based on shallow features of the current frame and shallow features of the motion reference frame;

[0123] A motion compression module 503, used to compress the motion information features to obtain compressed motion information features;

[0124] The motion compensation module 504 is used to compensate the compressed motion information features on the shallow features of the reconstructed frame at the previous moment to obtain the context information features at the current moment;

[0125] A conditional compression module 505, used to compress the context information feature to obtain the compressed context information feature;

[0126] The conditional encoding module 506 is used to encode the compressed context information features and the shallow features of the current frame to obtain the reconstructed frame at the current moment;

[0127] The video reconstruction module 507 is used to upsample the reconstructed frame at the current moment to obtain a reconstructed compressed viewpoint image;

[0128] The motion information correction module 508 is used to correct the accumulated error of the motion information in the reconstructed compressed viewpoint image to obtain a motion reference frame for the next frame.

[0129] The specific implementation of each module of an end-to-end video compression device based on a self-attention mechanism is the same as the end-to-end video compression method based on a self-attention mechanism, and this embodiment will not be repeated.

[0130] Although the specific embodiments of the present invention are described above, those skilled in the art should understand that the specific embodiments described in this application are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. An end-to-end video compression method based on self-attention mechanism, characterized in that: include: A shallow feature extraction step is to extract shallow features of the current frame, the motion reference frame and the reconstructed frame at the previous moment; The motion estimation step extracts motion information features between the current frame and the motion reference frame based on the shallow features of the current frame and the shallow features of the motion reference frame; A motion compression step, compressing the motion information features to obtain compressed motion information features; The motion compensation step compensates the compressed motion information features on the shallow features of the reconstructed frame at the previous moment to obtain the context information features at the current moment; A conditional compression step, compressing the context information features to obtain compressed context information features; The conditional encoding step encodes the compressed context information features and the shallow features of the current frame to obtain the reconstructed frame at the current moment; The video reconstruction step upsamples the reconstructed frame at the current moment to obtain a reconstructed compressed viewpoint image; A motion information correction step, correcting the accumulated error of motion information in the reconstructed compressed viewpoint image to obtain a motion reference frame for the next frame; The shallow feature F of the current frame extracted t , which is expressed as follows: F t =Extractor (c) (X t ); Among them, X t ∈R 3×H×W represents the current frame, H represents the height of the feature map, W represents the width of the feature map; c represents the number of output channels; t represents the time of the input frame; F t Represents the shallow features of the current frame at time t; Extracor (c) (·) represents the shallow feature extraction network function; Shallow features of extracted motion reference frames It is expressed as follows: in, represents the motion reference frame; Extract the shallow features of the reconstructed frame at the previous moment It is expressed as follows: in, Represents the reconstructed frame at the previous time t-1.

2. The end-to-end video compression method based on the self-attention mechanism according to claim 1, characterized in that: Shallow feature extraction network function Extractor (c) (·), which means as follows: Extractor (c) (·)=Conv (5,c) (res (3,c) (Conv (5,c) (res (3,c) (Conv (5,c) (·))))); Among them, Conv (5,c) (·) represents a convolutional layer function with a convolution kernel of 5×5 and c output channels; res (3,c) (·) represents a residual block with a convolution kernel of 3×3 and c output channels.

3. The end-to-end video compression method based on the self-attention mechanism according to claim 1 is characterized in that: The motion information features are expressed as follows: Among them, O t ∈R 64×H×W represents the motion information feature at time t, H represents the height of the feature map, and W represents the width of the feature map; concat(·) represents concatenating the features in the channel dimension; Conv (3,c) represents a convolutional layer function with a convolution kernel of 3×3 and c output channels; F t Represents the shallow features of the frame at time t; Shallow features representing motion reference frames.

4. The end-to-end video compression method based on the self-attention mechanism according to claim 1, characterized in that: The motion compression step specifically comprises: compressing the motion information features by an entropy coding function to obtain compressed motion information features; The compressed motion information features are expressed as follows: in, represents the compressed motion information feature, H represents the height of the feature map, and W represents the width of the feature map; HyperCoding (·) represents the entropy coding function; Q (·) represents the quantization process; O t Represents the motion information features at time t.

5. The end-to-end video compression method based on the self-attention mechanism according to claim 1, characterized in that: The motion compensation step is specifically as follows: compensating the compressed motion information features on the shallow features of the reconstructed frame at the previous moment through deformable convolution and res blocks to obtain the context information features at the current moment; The context information features at the current moment are expressed as follows: in, Represents the context information feature at the current time t, H represents the height of the feature map, and W represents the width of the feature map; Conv (c,c) (·) represents a convolutional layer function with a convolution kernel of 3×3 and c output channels; res (3,c) (·) indicates a residual block with a convolution kernel of 3×3 and c output channels; deformconv(·) indicates a deformable convolution block; Conv (3,c) (·) represents a convolutional layer function with a convolution kernel of 3×3 and c output channels; Represents the compressed motion information characteristics; Represents the shallow features of the reconstructed frame at the previous moment.

6. The end-to-end video compression method based on the self-attention mechanism according to claim 1, characterized in that: The conditional encoding step is specifically as follows: encoding the compressed context information features and the shallow feature fusion features of the current frame through a conditional encoder to obtain a reconstructed frame at the current moment; The reconstructed frame at the current moment is expressed as follows: in, represents the reconstructed frame at the current moment, H represents the height of the feature map, and W represents the width of the feature map; f context (·) represents the conditional coding function; F t Represents the shallow features of the current frame; Represents the context information features at the current moment.

7. The end-to-end video compression method based on the self-attention mechanism according to claim 1, characterized in that: The reconstructed compressed viewpoint image obtained is expressed as follows: in, represents the reconstructed compressed viewpoint image, H represents the height of the feature map, and W represents the width of the feature map; deconv (5,c) (·) represents the deconvolution layer function with a convolution kernel of 5×5 and c output channels; res (3,c) (·) represents a residual block with a convolution kernel of 3×3 and c output channels; Represents the reconstructed frame at the current moment.

8. The end-to-end video compression method based on the self-attention mechanism according to claim 1, characterized in that: The motion compensation reference frame at the current moment is expressed as follows: in, Represents the motion reference frame of the next moment; Conv (3,c) (·) represents the function of the convolution layer with a convolution kernel of 3×3 and c output channels; concat(·) represents concatenation of features in the channel dimension; FR(·) represents the aggregation reconstruction network function; GRF(·) represents the full receptive field network function; FE(·) represents the multi-scale feature extraction network function; represents the reconstructed compressed viewpoint image; X t Indicates the current frame.

9. An end-to-end video compression system based on self-attention mechanism, characterized in that: include: A shallow feature extraction module is used to extract shallow features of the current frame, the motion reference frame and the reconstructed frame at the previous moment; A motion estimation module, used for extracting motion information features between the current frame and the motion reference frame based on shallow features of the current frame and shallow features of the motion reference frame; A motion compression module, used for compressing motion information features to obtain compressed motion information features; A motion compensation module is used to compensate the compressed motion information features on the shallow features of the reconstructed frame at the previous moment to obtain the context information features at the current moment; A conditional compression module, used for compressing context information features to obtain compressed context information features; The conditional encoding module is used to encode the compressed context information features and the shallow features of the current frame to obtain the reconstructed frame at the current moment; A video reconstruction module is used to upsample the reconstructed frame at the current moment to obtain a reconstructed compressed viewpoint image; A motion information correction module, used to correct the accumulated error of motion information in the reconstructed compressed viewpoint image to obtain a motion reference frame for the next frame; The shallow feature F of the current frame extracted t , which is expressed as follows: F t =Extractor (c) (X t ); Among them, X t ∈R 3×H×W represents the current frame, H represents the height of the feature map, W represents the width of the feature map; c represents the number of output channels; t represents the time of the input frame; F t Represents the shallow features of the current frame at time t; Extracor (c) (·) represents the shallow feature extraction network function; Shallow features of extracted motion reference frames It is expressed as follows: in, represents the motion reference frame; Extract the shallow features of the reconstructed frame at the previous moment It is expressed as follows: in, Represents the reconstructed frame at the previous time t-1.

Images