Video compression method based on multi-scale optical flow and multi-reference frame conditional coding

Abstract

The invention discloses a video compression method based on multi-scale optical flow and multi-reference frame condition coding. The video compression method comprises the following steps: (1) outputting an optical flow field, namely motion information, between two adjacent frames through a multi-scale optical flow estimation network; and (2) inputting the motion information into a motion encoder for compression to generate a motion code stream. And (3) performing motion compensation on the previous reconstructed frame by using the motion information to generate a prediction frame. And (4) extracting the features of the first three reconstructed frames from the reference frame buffer area, and generating an enhanced context condition by using deformable convolution alignment and attention mechanism fusion. And (5) inputting the context condition and the current frame into a condition encoder together, and outputting a potential representation of the current frame. And (6) inputting the motion potential representation and the current frame potential representation into a joint entropy model together, performing accurate probability estimation by taking an enhanced context condition as prior, and generating a final compressed code stream. And (7) the decoding process is a symmetric process of a coding end, motion is recovered, and the current frame is reconstructed.

Description

Technical Field

[0001] This invention relates to the field of video coding technology, and specifically to a video compression method based on multi-scale optical flow and multi-reference frame conditional coding. Background Technology

[0002] In recent years, with the rapid development of the Internet and multimedia technologies, video content has increasingly become the mainstream carrier of information dissemination and entertainment. The rapid popularization of high-definition and ultra-high-definition (such as 4K and 8K) videos, as well as immersive application scenarios such as virtual reality (VR) and augmented reality (AR), has led to an explosive growth in video data volume. Efficient video compression technology has become key to alleviating network bandwidth pressure and storage bottlenecks.

[0003] Over the past few decades, a series of traditional video coding standards, such as H.264 / AVC, H.265 / HEVC, and the next-generation H.266 / VVC, have achieved a good balance between compression efficiency and reconstruction quality by employing a block-based hybrid coding framework and combining complex intra-frame prediction, inter-frame prediction, transform quantization, and entropy coding techniques. These standards have been widely used in video streaming, digital television, video conferencing, and storage. However, these traditional standards heavily rely on complex, manually designed rules and heuristic algorithms. With the continuous improvement of video resolution and frame rate, and the increasing complexity of video content, existing coding standards face significant challenges in terms of both compression efficiency and computational complexity. Especially when dealing with high dynamic range and complex motion scenes (such as fast motion, occlusion, and nonlinear deformation), traditional methods struggle to fully exploit the spatiotemporal redundancy in video data, and compression performance improvements are gradually approaching their limits.

[0004] In recent years, breakthroughs in deep learning technology have injected new vitality into the field of video coding. Learning-based video compression methods have gradually become a research hotspot, mainly divided into two categories: one is hybrid coding methods combined with traditional coding frameworks, which utilize neural networks to enhance or replace specific modules in traditional encoders (such as intra-frame prediction, inter-frame prediction, loop filtering, etc.). These methods can bring certain performance gains on the basis of existing standards, but the overall architecture is still limited by the traditional hybrid coding paradigm. The other category is end-to-end video coding methods based entirely on neural networks. It abandons the manually designed modules in traditional coding standards and attempts to automatically learn the mapping relationship from the original video to the compressed bitstream through neural networks. In addition, there are also researches exploring video compression methods based on probabilistic modeling (such as hierarchical VAEs) or Transformer architectures. Although the above-mentioned deep learning methods have shown great potential, they still have obvious limitations and challenges in practical applications: existing end-to-end methods mostly rely on CNNs for optical flow estimation. However, CNNs have inherent limitations in capturing long-distance spatiotemporal dependencies and global motion patterns, especially when dealing with large displacements, complex occlusions, and subtle motions, the estimation accuracy is difficult to meet the requirements of efficient inter-frame prediction. While advanced optical flow algorithms such as RAFT and GMFlow have improved accuracy, their computational complexity and overall optimization with the coding process remain challenges. Current leading conditional coding frameworks (such as DCVC) primarily rely on features from a single reference frame as contextual conditions. This limits the model's ability to utilize long-term dependencies and richer spatiotemporal information in video sequences. In regions with complex scene transitions, periodic motion, or requiring multiple frames for accurate prediction (such as regions that reappear after occlusion), the contextual information provided by a single reference frame is insufficient, leading to compromised compression efficiency. While the pure Transformer architecture excels at modeling global relationships, the computational complexity of its self-attention mechanism increases quadratically with sequence length, posing significant computational and memory challenges when processing high-resolution video, making it difficult to meet the speed requirements of practical applications. How to introduce powerful model capabilities while controlling computational overhead remains a pressing problem.

[0005] Therefore, there is an urgent need in this field for a new video coding solution that can effectively overcome the above-mentioned shortcomings of existing deep learning methods. By designing a more accurate and efficient motion estimation network, making full use of the contextual information of multiple reference frames, and achieving good collaborative optimization among modules, the compression efficiency and reconstruction quality of video can be significantly improved while maintaining low computational complexity. Summary of the Invention

[0006] Objective: To address the problems of insufficient optical flow estimation accuracy, inadequate utilization of contextual information, and difficulty in balancing model complexity and computational efficiency in existing end-to-end video compression methods, this invention proposes a video compression method based on multi-scale optical flow and multi-reference frame conditional coding. Specifically, this invention aims to: improve motion estimation accuracy in complex motion and large displacement scenes by using a hybrid optical flow estimation network combining global matching and local refinement, providing more accurate predicted frames for motion compensation; generate a richer enhanced contextual condition by introducing multi-reference frame features and performing spatiotemporal alignment and attention fusion to compensate for the lack of single-reference frame conditional information, thereby improving coding efficiency and reconstruction quality; and simultaneously apply the enhanced contextual condition to the conditional encoder and joint entropy model to achieve comprehensive utilization of contextual information in the transform coding and entropy coding stages, thereby optimizing rate-distortion performance overall.

[0007] Technical Solution: To achieve the above-mentioned objectives and optimize the end-to-end video compression method to improve rate-distortion performance, the technical solution of this invention is as follows:

[0008] (1) Input the current frame and the previous reconstructed frame into the multi-scale optical flow estimation network and output the final optical flow field, which is the motion information.

[0009] (2) The motion information is input into the motion encoder for compression to generate a motion bitstream. The decoding end recovers the motion information from the motion bitstream through the motion decoder.

[0010] (3) Use motion information to perform motion compensation on the previous reconstructed frame to generate a prediction frame.

[0011] (4) Extract features from the first three reconstructed frames from the reference frame buffer. Use deformable convolution to align these historical features with the predicted frames in space and time, and then fuse them through channel and spatial attention mechanisms to generate an enhanced contextual condition rich in space and time information.

[0012] (5) Input the context conditions along with the current frame into the conditional encoder. Under the guidance of strong context conditions, the encoder directly and efficiently compresses the information of the current frame and outputs the latent representation of the current frame.

[0013] (6) The motion latent representation (from the motion encoder) and the current frame latent representation (from the conditional encoder) are fed into the joint entropy model. This entropy model uses enhanced contextual conditions as prior information to more accurately estimate the probability distribution of these representations, and finally the entropy encoder generates the final compressed bitstream.

[0014] (7) The decoding process is a symmetrical process at the encoding end: First, the entropy decoder uses the same enhanced context conditions to parse the latent representations of motion and the current frame from the compressed bitstream. Next, the motion decoder recovers the optical flow field and generates a predicted frame, while the conditional decoder, guided by the enhanced context conditions, directly reconstructs the current frame from its latent representation. Finally, the features of the reconstructed frame are updated in the reference frame buffer for decoding subsequent frames.

[0015] Preferably, in step (1), the current frame and the previous reconstructed frame are input into the multi-scale optical flow estimation network, and the final optical flow field is output. This optical flow field is the motion information, and specifically includes the following steps:

[0016] (a) Use a weight-shared convolutional structure to downsample the input video frames by 8 times to extract dense features;

[0017] (b) At a 1 / 8 resolution scale, the downsampled features are enhanced and matched with global motion using the SwinTransformer to preliminarily estimate the optical flow.

[0018]

[0019] in, This represents the features of the first frame (current frame) after being enhanced by the Swin Transformer. This represents the features of the second frame (the previous reconstructed frame) after enhancement by the Swin Transformer, where G represents the grid coordinate tensor.

[0020] (c) Introduce a self-attention layer to propagate high-quality optical flow predictions from matched pixels to unmatched pixels by calculating the self-similarity of features;

[0021]

[0022] in denoted as the initial optical flow estimate, and V represents the output optical flow field after propagation through the self-attention layer.

[0023] (d) At the 1 / 4 and 1 / 2 resolution scales, a CNN-based pyramid structure is used to refine the optical flow layer by layer: first, the optical flow of the previous layer is upsampled, then the upsampled optical flow is used to perform reverse deformation on the features of the second frame, and finally the motion residual is estimated through CNN to optimize the optical flow field.

[0024]

[0025] in, This represents the optical flow after upsampling the optical flow at the previous scale at scale t; Upsample indicates the upsampling operation with a scaling factor of 2. This represents the optical flow estimate at the previous scale. This represents the features of the second frame after being distorted (deformed) by the upsampled optical flow at scale t. This represents the optical flow residual estimated by CNN at scale t. This represents the optical flow after upsampling.

[0026] Preferably, in step (2), the motion information is input into the motion encoder for compression to generate a motion bitstream, and the decoding end recovers the motion information from the motion bitstream through the motion decoder, specifically including the following steps:

[0027] (a) The motion encoder consists of several convolutional layers that transform and downsample the input optical flow field, and output the latent representation of motion y. mv ;

[0028] (b) The motion decoder is structurally symmetrical to the motion encoder, consisting of several deconvolutional or upsampled convolutional layers, and is responsible for retrieving the motion latent representation. Recovering motion information

[0029] Preferably, in step (3), motion compensation is performed on the previous reconstructed frame using motion information to generate a predicted frame. The specific steps are as follows:

[0030] (a) Using the motion information recovered from decoding in step (2) The previous reconstructed frame was obtained through bilinear interpolation. Perform a warping operation to generate a predicted frame for the current frame.

[0031] (b) The motion-compensated predicted frame will serve as a benchmark for comparison with the current frame, providing the effect of motion information compensation.

[0032] Preferably, in step (4), features from the first three reconstructed frames are extracted from the reference frame buffer. These historical features are then spatiotemporally aligned with the predicted frames using deformable convolution, and fused through a channel and spatial attention mechanism to generate an enhanced contextual condition rich in spatiotemporal information. The specific steps are as follows:

[0033] (a) Extract features from the first 3 reconstructed frames from the reference frame buffer, and extract features from the predicted frames at the same time;

[0034] (b) For each reference frame feature, a deformable convolution is used to spatiotemporally align it with the prediction frame feature, wherein the offset of the deformable convolution is calculated by a small neural network whose input is the prediction frame feature and the corresponding reference frame feature.

[0035]

[0036] θ=f θ (F t F t-N )

[0037] in, This represents the features of the tNth frame after alignment. dc This indicates a deformable convolution operation. Let θ represent the features of the tNth frame. θ represents the offset of the deformable convolution.

[0038] (c) Concatenate the aligned features of the three reference frames with the features of the predicted frame along the channel dimension;

[0039]

[0040] Among them, F cat This represents the concatenated features; Concat indicates a concatenation operation along the channel dimension. This represents the features of the predicted frame.

[0041] (d) The concatenated features are processed sequentially through the channel attention module and the spatial attention module. The channel attention module performs adaptive weighting on different feature channels, and the spatial attention module evaluates the importance of different spatial locations.

[0042] F attn_cat =C attn (F cat )·F cat

[0043] F cat_conv =ReLU(Conv 1×1 (F attn_cat ))

[0044] F attn_sp =S attn (F cat_conv )·F cat_conv

[0045] Among them, F attn_cat This represents the features after channel attention weighting. (C) attn This represents the channel attention module, which outputs an attention weight vector with the same number of input feature channels. ReLU represents the modified linear unit activation function. F attn_sp This represents the features after spatial attention weighting. attn This represents the spatial attention module, which outputs an attention map with the same spatial size as the input features and a single channel. · represents position-wise multiplication.

[0046] (e) The features enhanced by the dual attention mechanism are residually fused with the original predicted frame features to generate the final enhanced contextual condition, which will be provided to both the conditional encoder and the joint entropy model.

[0047]

[0048] Preferably, in step (5), the context conditions are input into the conditional encoder along with the current frame. Guided by strong context conditions, this encoder directly and efficiently compresses the information of the current frame, outputting the latent representation of the current frame. The specific steps are as follows:

[0049] (a) The features of the current frame, along with the context conditions, are input into the conditional encoder;

[0050] (b) The conditional encoder encodes the current frame based on contextual conditions, extracting deeper feature representations. The encoding process includes using convolutional layers, fully connected layers, etc., to extract effective features, and guiding the encoder to perform effective information compression based on contextual conditions;

[0051] (c) The final output of the conditional encoder is the latent representation (latent vector) of the current frame. This representation is compressed image information and is used for subsequent compression and decoding processes.

[0052] Preferably, in step (6), the motion latent representation (from the motion encoder) and the current frame latent representation (from the conditional encoder) are input together into the joint entropy model. This entropy model uses enhanced contextual conditions as prior information to more accurately estimate the probability distribution of these representations. Finally, the entropy encoder generates the final compressed bitstream, specifically including the following steps:

[0053] (a) The motion latent representation (from the motion encoder) and the latent representation of the current frame (from the conditional encoder) are fed into the joint entropy model. By combining the latent representations of both, the joint entropy model can utilize contextual conditions to improve compression efficiency;

[0054] (b) The joint entropy model uses contextual conditions as prior information to estimate the probability distribution of the input motion potential representation and the current frame potential representation;

[0055] (c) Finally, based on the estimated probability distribution, the entropy encoder encodes these latent representations into the final compressed bitstream.

[0056] Preferably, in step (7), the decoding process is a symmetrical process at the encoding end: First, the entropy decoder uses the same enhanced context conditions to parse the latent representation of motion and the current frame from the compressed bitstream. Next, the motion decoder recovers the optical flow field and generates a predicted frame, while the conditional decoder, guided by the enhanced context conditions, directly reconstructs the current frame from the latent representation of the current frame. Finally, the features of the reconstructed frame are updated in the reference frame buffer for decoding subsequent frames. The specific steps are as follows:

[0057] (a) The decoding end first performs the same operation as step (4) at the encoding end to generate the enhanced context condition.

[0058] (b) Entropy decoder utilizes As a priori, the latent representation of motion and the latent representation of the current frame are decoded from the received compressed bitstream;

[0059] (c) Motion decoder pair Decode the motion information to recover it, and then use... right Perform motion compensation to generate predicted frames;

[0060] (d) Conditional decoder in enhanced context conditions Under the guidance of the algorithm, the latent representation of the current frame is decoded to reconstruct the current frame.

[0061] (e) Reconstruct the frame Features are extracted and updated in the reference frame buffer, replacing the oldest frame features, in preparation for decoding the next frame. Attached Figure Description

[0062] Figure 1 This is a detailed structural diagram of the present invention;

[0063] Figure 2 This is a structural diagram of the optical flow estimation part;

[0064] Figure 3 It is a fusion structure diagram of contextual features. Detailed Implementation

[0065] The technical solution of the present invention will be further described below with reference to the accompanying drawings:

[0066] Step (1) Input the current frame and the previous reconstructed frame into the multi-scale optical flow estimation network, and output the final optical flow field, which is the motion information. The specific steps are as follows:

[0067] Step (1a) uses a weight-shared convolutional structure to downsample the input video frames by 8 times to extract dense features;

[0068] Step (1b) uses the Swin Transformer to enhance the downsampled features and match them with global motion at a 1 / 8 resolution scale to initially estimate the optical flow;

[0069]

[0070] in, This represents the features of the first frame (current frame) after being enhanced by SwinTransformer. This represents the features of the second frame (the previous reconstructed frame) after enhancement by the Swin Transformer, where G represents the grid coordinate tensor.

[0071] Step (1c) introduces a self-attention layer, which propagates high-quality optical flow predictions from matched pixels to unmatched pixels by calculating the self-similarity of features.

[0072]

[0073] in denoted as the initial optical flow estimate, and V represents the output optical flow field after propagation through the self-attention layer.

[0074] Step (1d) refines the optical flow layer by layer using a CNN-based pyramid structure at 1 / 4 and 1 / 2 resolution scales: first, the optical flow of the previous layer is upsampled, then the upsampled optical flow is used to perform reverse deformation on the features of the second frame, and finally the motion residual is estimated through CNN to optimize the optical flow field.

[0075]

[0076] in, This represents the optical flow after upsampling the optical flow at the previous scale at scale t; Upsample indicates the upsampling operation with a scaling factor of 2. This represents the optical flow estimate at the previous scale. This represents the features of the second frame after being distorted (deformed) by the upsampled optical flow at scale t. This represents the optical flow residual estimated by CNN at scale t. This represents the optical flow after upsampling.

[0077] Step (2) The motion information is input into the motion encoder for compression to generate a motion bitstream. The decoding end recovers the motion information from the motion bitstream through the motion decoder. The specific steps are as follows:

[0078] Step (2a) The motion encoder consists of several convolutional layers that transform and downsample the input optical flow field, outputting the latent representation of motion y. mv ;

[0079] Step (2b) The motion decoder is structurally symmetrical to the motion encoder, consisting of several deconvolutional layers or upsampled convolutional layers, and is responsible for extracting the motion latent representation. Recovering motion information

[0080] Step (3) Use motion information to perform motion compensation on the previous reconstructed frame to generate the prediction frame. The specific steps are as follows:

[0081] Step (3a) utilizes the motion information recovered from decoding The previous reconstructed frame was obtained through bilinear interpolation. Perform a warping operation to generate a predicted frame for the current frame.

[0082] The predicted frame after motion compensation in step (3b) will serve as a benchmark for comparison with the current frame, providing the effect of motion information compensation.

[0083] In step (4), features from the first three reconstructed frames are extracted from the reference frame buffer. Deformable convolution is used to align these historical features with the predicted frames spatiotemporally, and then they are fused using a channel and spatial attention mechanism to generate an enhanced contextual condition rich in spatiotemporal information. The specific steps are as follows:

[0084] Step (4a) extracts features from the first three reconstructed frames from the reference frame buffer, and simultaneously extracts features from the predicted frames;

[0085] Step (4b) For each reference frame feature, a deformable convolution is used to spatiotemporally align it with the predicted frame feature, wherein the offset of the deformable convolution is calculated by a small neural network whose input is the predicted frame feature and the corresponding reference frame feature.

[0086]

[0087] θ=f θ (F t F t-N )

[0088] in, This represents the features of the tNth frame after alignment. dc This indicates a deformable convolution operation. Let θ represent the features of the tNth frame. θ represents the offset of the deformable convolution.

[0089] Step (4c) concatenates the aligned features of the three reference frames with the features of the predicted frame along the channel dimension;

[0090]

[0091] Among them, F cat This represents the concatenated features; Concat indicates a concatenation operation along the channel dimension. This represents the features of the predicted frame.

[0092] Step (4d) processes the stitched features sequentially through the channel attention module and the spatial attention module. The channel attention module adaptively weights different feature channels, while the spatial attention module evaluates the importance of different spatial locations.

[0093] F attn_cat =C attn (F cat )·F cat

[0094] F cat_conv =ReLU(Conv 1×1 (F attn_cat ))

[0095] F attn_sp =S attn (F cat_conv )·F cat_conv

[0096] Among them, F attn_cat This represents the features after channel attention weighting. (C) attn This represents the channel attention module, which outputs an attention weight vector with the same number of input feature channels. ReLU represents the modified linear unit activation function. F attn_sp This represents the features after spatial attention weighting. attn This represents the spatial attention module, which outputs an attention map with the same spatial size as the input features and a single channel. · represents position-wise multiplication.

[0097] Step (4e) performs residual fusion between the enhanced features obtained through the dual attention mechanism and the original predicted frame features to generate the final enhanced contextual condition, which will be provided to both the conditional encoder and the joint entropy model.

[0098]

[0099] Step (5) inputs the context conditions along with the current frame into the conditional encoder. Guided by powerful context conditions, this encoder efficiently compresses the information of the current frame and outputs its latent representation. The specific steps are as follows:

[0100] Step (5a): The features of the current frame, along with the context conditions, are input into the conditional encoder:

[0101] Step (5b) involves the conditional encoder encoding the current frame based on contextual conditions to extract deeper feature representations. The encoding process includes using convolutional layers, fully connected layers, etc., to extract effective features and guiding the encoder to perform effective information compression based on contextual conditions.

[0102] The final output of step (5c) of the conditional encoder is the latent representation (latent vector) of the current frame. This representation is the compressed image information used for subsequent compression and decoding processes.

[0103] Step (6) inputs the motion latent representation (from the motion encoder) and the current frame latent representation (from the conditional encoder) into the joint entropy model. This entropy model utilizes enhanced contextual conditions as prior information to more accurately estimate the probability distribution of these representations. Finally, the entropy encoder generates the final compressed bitstream, specifically including the following steps:

[0104] In step (6a), the motion latent representation (from the motion encoder) and the latent representation of the current frame (from the conditional encoder) are fed into the joint entropy model. By combining the latent representations of both, the joint entropy model can utilize contextual conditions to improve compression efficiency;

[0105] Step (6b) The joint entropy model uses contextual conditions as prior information to estimate the probability distribution of the input motion latent representation and the current frame latent representation;

[0106] Step (6c) involves the entropy encoder encoding these latent representations into the final compressed bitstream based on the estimated probability distribution.

[0107] Step (7) is the decoding process: First, the entropy decoder uses the same enhanced context conditions to parse the latent representations of motion and the current frame from the compressed bitstream. Next, the motion decoder recovers the optical flow field and generates a predicted frame, while the conditional decoder, guided by the enhanced context conditions, directly reconstructs the current frame from its latent representation. Finally, the features of the reconstructed frame are updated in the reference frame buffer for decoding subsequent frames. The specific steps are as follows:

[0108] In step (7a), the decoding end first performs the same operation as step (4) at the encoding end to generate the enhanced context condition.

[0109] Step (7b) The entropy decoder utilizes As a priori, the latent representation of motion and the latent representation of the current frame are decoded from the received compressed bitstream;

[0110] Step (7c) Motion decoder pair Decode the motion information to recover it, and then use... right Perform motion compensation to generate predicted frames;

[0111] Step (7d) Conditional decoder in enhanced contextual conditions Under the guidance of the algorithm, the latent representation of the current frame is decoded to reconstruct the current frame.

[0112] Step (7e) will reconstruct the frame Features are extracted and updated in the reference frame buffer, replacing the oldest frame features, in preparation for decoding the next frame.

[0113] This invention addresses the problems of insufficient optical flow estimation accuracy, inadequate utilization of contextual information, and difficulty in balancing model complexity and computational efficiency in existing end-to-end video compression methods. It proposes a video compression method based on multi-scale optical flow and multi-reference frame conditional coding. Specifically, this invention aims to: improve motion estimation accuracy in complex motion and large displacement scenes by using a hybrid optical flow estimation network combining global matching and local refinement, providing more accurate predicted frames for motion compensation; generate a richer enhanced contextual condition by introducing multi-reference frame features and performing spatiotemporal alignment and attention fusion to compensate for the shortcomings of single-reference frame conditional information, thereby improving coding efficiency and reconstruction quality; and simultaneously apply the enhanced contextual condition to the conditional encoder and joint entropy model to achieve comprehensive utilization of contextual information in the transform coding and entropy coding stages, thus optimizing rate-distortion performance overall.

Claims

1. A video compression method based on multi-scale optical flow and multi-reference frame conditional coding, characterized in that, Includes the following steps: (1) Input the current frame and the previous reconstructed frame into the multi-scale optical flow estimation network and output the final optical flow field, which is the motion information. (2) The motion information is input into the motion encoder for compression to generate a motion bitstream. The decoding end recovers the motion information from the motion bitstream through the motion decoder. (3) Use motion information to perform motion compensation on the previous reconstructed frame to generate a prediction frame. (4) Extract features from the first three reconstructed frames from the reference frame buffer. Use deformable convolution to align these historical features with the predicted frames in space and time, and then fuse them through channel and spatial attention mechanisms to generate an enhanced contextual condition rich in space and time information. (5) Input the context conditions along with the current frame into the conditional encoder. Under the guidance of strong context conditions, the encoder directly and efficiently compresses the information of the current frame and outputs the latent representation of the current frame. (6) The motion latent representation (from the motion encoder) and the current frame latent representation (from the conditional encoder) are fed into the joint entropy model. This entropy model uses enhanced contextual conditions as prior information to more accurately estimate the probability distribution of these representations, and finally the entropy encoder generates the final compressed bitstream. (7) The decoding process is a symmetrical process at the encoding end: First, the entropy decoder uses the same enhanced context conditions to parse the latent representations of motion and the current frame from the compressed bitstream. Next, the motion decoder recovers the optical flow field and generates a predicted frame, while the conditional decoder, guided by the enhanced context conditions, directly reconstructs the current frame from its latent representation. Finally, the features of the reconstructed frame are updated in the reference frame buffer for decoding subsequent frames.

2. The video compression method based on multi-scale optical flow and multi-reference frame conditional coding according to claim 1, characterized in that, Step (1) includes: (a) Use a weight-shared convolutional structure to downsample the input video frames by 8 times to extract dense features; (b) At a 1 / 8 resolution scale, the downsampled features are enhanced and matched with global motion using the Swin Transformer to preliminarily estimate optical flow. in, This represents the features of the first frame (current frame) after being enhanced by the Swin Transformer. This represents the features of the second frame (the previous reconstructed frame) after enhancement by the Swin Transformer, where G represents the grid coordinate tensor. (c) Introduce a self-attention layer to propagate high-quality optical flow predictions from matched pixels to unmatched pixels by calculating the self-similarity of features; in denoted as the initial optical flow estimate, and V represents the output optical flow field after propagation through the self-attention layer. (d) At the 1 / 4 and 1 / 2 resolution scales, a CNN-based pyramid structure is used to refine the optical flow layer by layer: first, the optical flow of the previous layer is upsampled, then the upsampled optical flow is used to perform reverse deformation on the features of the second frame, and finally the motion residual is estimated through CNN to optimize the optical flow field. in, This represents the optical flow at scale t after upsampling the optical flow at the previous scale; Upsample indicates the upsampling operation with a scaling factor of 2. ； This represents the optical flow estimate at the previous scale. This represents the features of the second frame after being distorted (deformed) by the upsampled optical flow at scale t. This represents the optical flow residual estimated by CNN at scale t. This represents the optical flow after upsampling.

3. The video compression method based on multi-scale optical flow and multi-reference frame conditional coding according to claim 1, characterized in that, Step (4) includes: (a) Extract features from the first 3 reconstructed frames from the reference frame buffer, and extract features from the predicted frames at the same time; (b) For each reference frame feature, a deformable convolution is used to spatiotemporally align it with the prediction frame feature, wherein the offset of the deformable convolution is calculated by a small neural network whose input is the prediction frame feature and the corresponding reference frame feature. θ=f θ (F t ,F t-N ) in, This represents the features of the tNth frame after alignment. dc This indicates a deformable convolution operation. Let represent the features of the tNth frame. θ represents the offset of the deformable convolution. (c) Concatenate the aligned features of the three reference frames with the features of the predicted frame along the channel dimension; Among them, F cat This represents the concatenated features; Concat indicates a concatenation operation along the channel dimension. This represents the features of the predicted frame. (d) The concatenated features are processed sequentially through the channel attention module and the spatial attention module. The channel attention module performs adaptive weighting on different feature channels, and the spatial attention module evaluates the importance of different spatial locations. F attn_cat =C attn (F cat )·F cat F cat_conv =ReLU(Conv 1×1 (F attn_cat )) F attn_sp =S attn (F cat_conv )·F cat_conv in, This represents the features after channel attention weighting. (C) attn This represents the channel attention module, which outputs an attention weight vector with the same number of input feature channels. ReLU represents the modified linear unit activation function. This represents the features after spatial attention weighting. attn This represents the spatial attention module, which outputs an attention map with the same spatial size as the input features and a single channel. · represents position-wise multiplication. (e) The features enhanced by the dual attention mechanism are residually fused with the original predicted frame features to generate the final enhanced contextual condition, which will be provided to both the conditional encoder and the joint entropy model.

Images