End-to-end video compression method based on temporal context mining and frame enhancement reconstruction

Abstract

This invention relates to the field of video coding technology, specifically to an end-to-end video compression method based on temporal context mining and reconstructed frame enhancement. The method includes: extracting the decoded frame of the previous video frame; inputting the current video frame and the decoded frame into a motion estimation module to output a motion vector; compressing and reconstructing the motion vector in a lossy manner using a motion vector encoder and a motion vector decoder to obtain a reconstructed motion vector; inputting the reconstructed motion vector and features of the previous video frame into an enhanced context mining module to obtain a context; inputting the current video frame and its context into a conditional encoder and a conditional decoder to obtain a reconstructed frame; and inputting the reconstructed motion vector, features, and reconstructed frame into a reconstructed frame enhancement module to obtain a decoded frame. This invention improves the overall quality of the final output frame.

Description

Technical Field

[0001] This invention relates to the field of video coding technology, and more specifically to an end-to-end video compression method based on temporal context mining and reconstructed frame enhancement. Background Technology

[0002] As video data continues to account for a growing proportion of internet traffic, efficient video compression technology is crucial for reducing transmission and storage costs. Current mainstream traditional video coding standards (such as H.264 / AVC and H.265 / HEVC) are built on hybrid coding frameworks. While these technologies are mature and offer high coding efficiency, their modular design paradigm somewhat limits further performance optimization. In recent years, end-to-end video compression methods based on deep learning have shown significant potential. These methods, by jointly optimizing the entire coding process, are expected to overcome the performance bottlenecks of traditional frameworks.

[0003] In learning-based video encoders, the ability to fully mine and utilize the spatiotemporal correlations (i.e., "contextual" information) between video frames is crucial for improving compression efficiency. Some advanced solutions, such as Deep Contextual Video Compression (DCVC), employ conditional coding architectures to extract single-scale contextual features from previously reconstructed frames. However, extracting information solely from reconstructed frames with a low number of channels easily leads to the loss of rich, detailed features. Furthermore, single-scale contextual modeling struggles to effectively capture the spatiotemporal non-uniformity prevalent in video content, such as complex motion patterns and fine-grained texture variations.

[0004] Furthermore, existing learning codecs often suffer from detail loss and compression artifacts in the reconstructed frames generated at the decoding end due to operations such as quantization. Although post-processing modules can enhance these effects, most post-processing methods fail to fully utilize the rich complementary information between adjacent frames in the video sequence, thus limiting the enhancement results. Summary of the Invention

[0005] To address the aforementioned issues, this invention provides an end-to-end video compression method based on temporal context mining and reconstructed frame enhancement. This method constructs an end-to-end deep learning model, including a conditional encoder, a conditional decoder, a reconstructed frame enhancement module, a motion vector encoder, a motion vector decoder, a motion estimation module, an enhancement context mining module, a feature extraction module, and a decoded image buffer.

[0006] Process the current video frame to be encoded using a deep learning model. Obtain the decoded frame This includes the following steps:

[0007] S1. Extract the decoded frame of the previous video frame from the decoded image buffer. , will the current video frame With decoded frames The input motion estimation module estimates the current video frame. Compared to the decoded frame The motion relationship is determined, and the motion vector is output. ;

[0008] S2. Motion vectors are processed in a lossy manner using a motion vector encoder and a motion vector decoder. Compression and reconstruction are performed to obtain the reconstructed motion vectors. ;

[0009] S3. Reconstruct motion vectors Features of the previous video frame Input the enhanced context mining module to obtain the context. The enhanced context mining module includes a deformable convolutional fine alignment unit and a multi-scale feature extraction unit.

[0010] S4. Move the current video frame and its context Input conditional encoder and conditional decoder to obtain reconstructed frames. ;

[0011] S5. Reconstruct motion vectors ,feature Reconstructed frames Input the reconstructed frame enhancement module to obtain the decoded frame. The image is then fed into the decoded image buffer; the reconstructed frame enhancement module includes an alignment unit and a reconstruction unit.

[0012] The beneficial effects of this invention are:

[0013] This invention provides a richer and more accurate spatiotemporal context by mining multi-scale contextual information of propagation features and using deformable convolution to achieve accurate feature alignment, thereby significantly improving the efficiency of conditional coding.

[0014] This invention designs an independent reconstructed frame enhancement module. This module adopts an optical flow-guided deformable alignment method to make full use of the information of adjacent frames to repair compression artifacts and enhance image details, thereby significantly improving the overall quality of the final output frame.

[0015] The entire solution achieves compression performance exceeding that of DCVC without using the autoregressive entropy model, which is unfriendly to parallelization, and combines high compression efficiency with high practical value. Attached Figure Description

[0016] Figure 1This is a flowchart of the end-to-end video compression method based on temporal context mining and reconstructed frame enhancement according to the present invention;

[0017] Figure 2 This is a schematic diagram of the end-to-end deep learning model of the present invention;

[0018] Figure 3 This is a schematic diagram of the enhanced context mining module of the present invention;

[0019] Figure 4 This is a schematic diagram of the lightweight network of the present invention;

[0020] Figure 5 This is a schematic diagram of the reconstructed frame enhancement module of the present invention;

[0021] Figure 6 This is a schematic diagram of the residual block of the present invention;

[0022] Figure 7 The figure shows the simulation results of an embodiment of the present invention. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] Please see Figures 1-6 This invention provides an end-to-end video compression method based on temporal context mining and reconstructed frame enhancement, achieving efficient compression. Its core lies in an end-to-end deep learning model, which is trained end-to-end with rate-distortion optimization as the objective. At the encoding end, the enhanced context mining module effectively extracts and utilizes the temporal correlation of video sequences; at the decoding end, an optical flow-guided reconstructed frame enhancement module is used to further improve the quality of the reconstructed frames.

[0025] The deep learning model constructed in this embodiment of the invention is a complex closed-loop system, such as... Figure 2 As shown, it includes a conditional encoder, a conditional decoder, a reconstructed frame enhancement module, a motion vector encoder, a motion vector decoder, a motion estimation module, an enhanced context mining module, a feature extraction module, and a decoded image buffer.

[0026] In this embodiment of the invention, the processing object is a series of video frames. This method can be deployed at both the video encoding and decoding ends, and is suitable for end-to-end learning-based video compression systems.

[0027] In this embodiment of the invention, the current video frame to be encoded is processed using a deep learning model. Obtain the decoded frame This includes the following steps:

[0028] S1. Extract the decoded frame of the previous video frame from the decoded image buffer. , will the current video frame With decoded frames The input motion estimation module estimates the current video frame. Compared to the decoded frame The motion relationship is determined, and the motion vector is output. .

[0029] In some embodiments, the motion estimation module may employ a pre-trained optical flow estimation network, Spynet.

[0030] S2. Motion vectors are processed in a lossy manner using a motion vector encoder and a motion vector decoder. Compression and reconstruction are performed to obtain the reconstructed motion vectors. .

[0031] Specifically, the encoding end will use motion vectors The input is fed into a motion vector encoder for lossy compression, generating a corresponding motion bitstream; the decoder then decodes the motion bitstream using a corresponding motion vector decoder to obtain the reconstructed motion vectors. .

[0032] In particular, the structure, entropy modeling method, and quantization strategy of the motion vector encoder and motion vector decoder can all adopt the design scheme in the existing DCVC framework.

[0033] S3. Reconstruct motion vectors Features of the previous video frame Input the enhanced context mining module to obtain the context. The enhanced context mining module includes a deformable convolutional fine alignment unit and a multi-scale feature extraction unit.

[0034] This step addresses the problem that existing methods typically generate only single-scale context features in spatiotemporal context mining, making it difficult to effectively model complex non-uniform spatiotemporal characteristics in video sequences (such as different objects moving at different speeds and directions), thus limiting the ability of conditional encoders to eliminate spatiotemporal redundancy. An enhanced context mining (ECM) module is proposed.

[0035] Specifically, the features of the previous video frame The feature extraction module is used to obtain the feature data, and this module can adopt the design scheme of the existing DCVC framework.

[0036] In some embodiments, such as Figure 3 As shown, the enhanced context mining module adopts a cascaded architecture of "motion-compensated coarse alignment + deformable convolutional fine alignment + multi-scale feature fusion," and its specific processing steps include:

[0037] S31. Based on reconstructed motion vectors Features Perform coarse alignment to obtain coarse alignment features. ; can be represented as:

[0038]

[0039] In the formula, warp() represents motion compensation.

[0040] S32. Reconstruct the motion vectors Coarse alignment feature Refined features are obtained through deformable convolutional fine alignment units. .

[0041] Due to the complexity of object motion, optical flow estimation often suffers from significant errors in occlusion, boundary conditions, and high-speed motion regions, leading to coarsely aligned features obtained through simple motion compensation (warping) operations. Precise alignment is difficult to achieve. To address this issue, this invention introduces deformable convolutional fine alignment units to perform coarse alignment of features. Further refinement.

[0042] Preferably, step S32 specifically includes:

[0043] S321. Reconstruct motion vectors Coarse alignment feature After splicing, the data is input into a lightweight network to obtain the first spatial offset and the first modulation mask.

[0044] In some embodiments, such as Figure 4 As shown, the lightweight network includes three convolutional layers, and a ReLU activation function layer is cascaded between every two convolutional layers; the output of the last convolutional layer can be split to obtain the spatial offset offset1 and the modulation mask mask1.

[0045] In some embodiments, the three convolutional layers of the lightweight network all use 3×3 convolutional kernels and are uniformly set to a stride of 1. The number of input channels for the three convolutional layers are 66, 64, and 64, respectively, and the number of output channels are 64, 64, and 81, respectively.

[0046] S322. Coarse Alignment Feature Spatial offset1 and modulation mask1 are passed through a deformable convolutional layer and a ReLU activation function layer to obtain refined features. And use it as the input of the multi-scale feature extraction unit.

[0047] The spatial offset allows for flexible adjustment of the sampling position to accurately compensate for residual local errors in optical flow estimation, thereby explicitly correcting geometric deformations in local regions. Simultaneously, a modulation mask assigns learnable weights to each sampling position, enabling spatially adaptive adjustment of feature intensity. This allows for automatic enhancement of responses in reliable regions during feature fusion and suppression of invalid responses caused by noise or severe misalignment. When handling complex scenes such as non-rigid motion, object edges, and occluded regions, this method achieves finer feature alignment and provides more robust temporal contextual features for subsequent encoding and modeling.

[0048] In some embodiments, the Deformable Convolution layer uses a 3×3 convolution kernel with a stride of 1, and both the number of input channels and the number of output channels are set to 64.

[0049] S33. Refinement features Input the multi-scale feature extraction unit to obtain the concatenated features. .

[0050] While deformable convolutional fine-alignment units can obtain relatively accurate and refined features, However, this feature is only constructed based on a single scale. Single-scale features have inherent limitations in characterizing non-uniform motion and complex textures, while multi-scale features can cover a wider receptive field, capturing both local details and global scene information, and establishing more robust temporal correlations. This helps to more comprehensively describe the spatiotemporal non-uniformity that is prevalent in videos. Dilated convolution, as a classic multi-scale modeling method, can effectively expand the receptive field while maintaining the spatial localization accuracy of features with a low number of parameters. In addition, this structure avoids the information loss that may be caused by traditional multi-layer downsampling operations, enabling the network to model long-distance contextual dependencies while maintaining high spatial resolution. To this end, this invention designs a multi-scale feature extraction unit based on dilated convolution, which refines features under multiple receptive fields. Parallel extraction is performed to enhance the model's ability to perceive multi-scale contextual information.

[0051] Specifically, the multi-scale feature extraction unit consists of three parallel branches, each including a dilated convolutional layer and a ReLU activation function layer, used to capture local high-frequency motion, medium-range structural deformation, and large-scale background consistency, respectively, to obtain features at three scales; then the outputs of the three branches are concatenated along the channel dimension to obtain the concatenated features. The resulting temporal context features not only contain precisely aligned fine-grained motion information but also integrate cross-scale structured temporal cues, making them a more complete, robust, and expressive temporal prior.

[0052] In some embodiments, the three dilated convolutional layers all use 3×3 convolutional kernels with dilation coefficients of 1, 2, and 4, respectively. The number of input / output channels for each of the three dilated convolutional layers is 64.

[0053] S34. Features of splicing After passing through convolutional layers and ReLU activation function layers, the context is obtained. .

[0054] The enhanced context mining module, guided by motion vector reconstruction, adaptively aligns features from the previous video frame and extracts and fuses temporal contextual information through multi-scale feature modeling, thereby obtaining a more accurate and robust contextual representation. This contextual feature effectively mitigates motion compensation errors and local misalignment issues, providing more reliable prior information for subsequent conditional encoding and decoding processes.

[0055] S4. Move the current video frame and its context Input conditional encoder and conditional decoder to obtain reconstructed frames. .

[0056] Specifically, the current video frame With context These features are input together to the conditional encoder for feature encoding, quantization, and entropy encoding. At the decoding end, the corresponding conditional decoder uses the decoded features to reconstruct the current video frame. .

[0057] In particular, the structure and entropy modeling method of the conditional encoder and conditional decoder can adopt the design scheme in the existing DCVC framework.

[0058] S5. Reconstruct motion vectors ,feature Reconstructed frames Input the reconstructed frame enhancement module to obtain the decoded frame. The image is then fed into the decoded image buffer; the reconstructed frame enhancement module includes an alignment unit and a reconstruction unit.

[0059] In some embodiments, such as Figure 5 As shown, step S5 specifically includes:

[0060] S51. Reconstruction Frame Feature extraction is performed to obtain features By reconstructing motion vectors Features Preliminary geometric alignment is performed to obtain coarse alignment features. .

[0061] S52. Features Coarse alignment feature Input alignment unit to obtain reference feature .

[0062] In some embodiments, the optical flow estimation itself is noisy, especially in occlusion, boundary, and high-motion regions, making it difficult for bilinear warps to achieve sufficiently accurate spatial alignment. Therefore, an alignment unit is proposed, the process of which includes:

[0063] Features Coarse alignment feature After concatenation, the data is input into a shallow convolutional network to obtain shallow features. The shallow convolutional network includes three simple convolutional layers, and each simple convolutional layer is followed by a leaky ReLU layer. All three simple convolutional layers use 3×3 convolutional kernels.

[0064] The shallow features are passed through the first branch, and the output of the first branch is compared with the reconstructed motion vector. The sum is used to obtain the spatial offset offset2; the first branch includes a 3×3 convolutional layer;

[0065] The shallow features are passed through the second branch to obtain the modulation mask mask2; the second branch includes a 3×3 convolutional layer and a sigmoid activation function layer;

[0066] Spatial offset2, modulation mask, and features Input a deformable convolutional layer with a 3×3 kernel and a stride of 4 to obtain reference features. .

[0067] This deformable alignment process can automatically correct the sampling position in regions with local structural misalignment, motion deformation, and unstable optical flow estimation, thereby obtaining more reliable alignment results and significantly reducing reconstruction artifacts caused by temporal error propagation.

[0068] S53. Features and reference features After being stitched together, the data is input into the reconstruction unit to obtain the decoded frame. .

[0069] The reconstruction unit consists of a cascaded first convolutional layer, three residual blocks, and a second convolutional layer.

[0070] The reconstruction unit effectively integrates texture information from the previous frame with high-frequency cues from the current frame while maintaining the global structure, enabling the model to compensate for details in areas with significant compression impairment. Furthermore, this structure helps enhance local consistency and global perceptual quality, resulting in more stable and clearer enhanced reconstructed frames.

[0071] In some embodiments, such as Figure 6 As shown, the residual block includes two convolutional layers, each followed by a ReLU activation function layer; an identity shortcut connection is set between the input of the first convolutional layer and the output of the second convolutional layer. Each convolutional layer uses a 3×3 convolutional kernel with a stride of 1, and both the input and output channels are set to 64.

[0072] The reconstructed frame enhancement module introduces a motion-guided feature alignment and fusion mechanism to finely align the information of the previous video frame and jointly model it with the features of the current frame, thereby further improving the detail quality and temporal consistency of the reconstructed frame and reducing blur and artifacts in the reconstruction process.

[0073] Finally, the decoded frame Stored in the decoded image buffer as a reference frame for subsequent frame motion estimation and context modeling.

[0074] In some embodiments, the training phase utilizes the training set from the Vimeo-90kseptuplet dataset, widely used in the field of video compression. This training set contains 64,612 sequences, each consisting of 7 consecutive high-resolution frames (448×256 pixels). The video content covers diverse scenes including natural landscapes, human activities, animals, and cityscapes, featuring rich texture details, complex motion patterns, and significant lighting variations. During training, the input video frames are randomly cropped into 256×256 image patches.

[0075] Preferably, in video compression tasks, the core objective is typically to maximize the quality of the reconstructed video under a given bitrate constraint. Accordingly, the loss function L used in the training process aims to jointly optimize both bitrate and distortion metrics:

[0076]

[0077] In the formula, Indicates input frame With the reconstructed decoded frames The distortion between them is usually measured by mean squared error (MSE) or multi-scale structural similarity (MS-SSIM). For total bitrate estimation, This represents the number of bits consumed in encoding and quantizing the latent variables of the motion vector and their corresponding prior distributions. This represents the number of bits consumed in encoding and quantizing the latent variables of the current frame and their corresponding prior distributions. Expected bitrate for each term. It can be approximated by calculating the cross-entropy between the actual marginal distribution and the estimated distribution of the latent variables. The parameter λ is used to balance the bit rate R and the distortion D.

[0078] To evaluate the performance of this invention, we selected MCL-JCV as the test dataset. For compression efficiency, bits per pixel (bpp) was used to measure the coding overhead per pixel. For reconstruction quality, Peak Signal-to-Noise Ratio (PSNR) and Multi-Scale Structural Similarity (MS-SSIM) were used as quantification metrics to evaluate the differences between the decoded frame and the original frame in terms of global fidelity and structural information preservation.

[0079] We trained four models using four different sets of λ parameters to cover a wider range of coding bitrates. In peak signal-to-noise ratio (PSNR)-guided training, λ values ​​were set to 256, 512, 1024, and 2048, respectively; while in multi-scale structural similarity (MS-SSIM)-guided training, λ values ​​were set to 8, 16, 32, and 64, respectively. The optimizer used was AdamW, with an initial learning rate of 1×10⁻⁶. -4 And after the model converges, the learning rate is decayed to 1×10. -5 The training batch size is fixed at 18.

[0080] To evaluate the effectiveness of the proposed method, we conducted comparative experiments with several state-of-the-art solutions, including learning-based models DCVC and RLVC, as well as traditional compression standards x264 (very-slow) and x265 (very-slow). Figure 7 As shown, our method significantly outperforms all comparable methods in both PSNR and MS-SSIM metrics. Regarding PSNR, compared to x265 (veryslow), our method achieves a 35.76% bitrate saving, and further improves upon DCVC, the current leading method in video encoding and decoding, by 3.3%. In terms of MS-SSIM, our method achieves a 53.08% bitrate saving compared to x265 (veryslow), and also leads DCVC by 3.83%.

[0081] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An end-to-end video compression method based on temporal context mining and reconstructed frame enhancement, characterized in that, Construct an end-to-end deep learning model, which includes a conditional encoder, a conditional decoder, a reconstructed frame enhancement module, a motion vector encoder, a motion vector decoder, a motion estimation module, an enhanced context mining module, a feature extraction module, and a decoded image buffer; Process the current video frame to be encoded using a deep learning model. Obtain the decoded frame This includes the following steps: S1. Extract the decoded frame of the previous video frame from the decoded image buffer. , will the current video frame With decoded frames The input motion estimation module estimates the current video frame. Compared to the decoded frame The motion relationship is determined, and the motion vector is output. ; S2. Motion vectors are processed in a lossy manner using a motion vector encoder and a motion vector decoder. Compression and reconstruction are performed to obtain the reconstructed motion vectors. ; S3. Reconstruct motion vectors Features of the previous video frame Input the enhanced context mining module to obtain the context. The enhanced context mining module includes a deformable convolutional fine alignment unit and a multi-scale feature extraction unit. S4. Move the current video frame and its context Input conditional encoder and conditional decoder to obtain reconstructed frames. ; S5. Reconstruct motion vectors ,feature Reconstructed frames Input the reconstructed frame enhancement module to obtain the decoded frame. The image is then fed into the decoded image buffer; the reconstructed frame enhancement module includes an alignment unit and a reconstruction unit.

2. The end-to-end video compression method based on temporal context mining and reconstructed frame enhancement according to claim 1, characterized in that, Step S3 specifically includes: S31. Based on reconstructed motion vectors Features Perform coarse alignment to obtain coarse alignment features. ; S32. Reconstruct the motion vectors Coarse alignment feature Refined features are obtained through deformable convolutional fine alignment units. ; S33. Refinement features Input the multi-scale feature extraction unit to obtain the concatenated features. ;in: The multi-scale feature extraction unit consists of three parallel branches, each including a dilated convolutional layer and a ReLU activation function layer; the outputs of the three branches are concatenated along the channel dimension to obtain the concatenated features. ; S34. Features of splicing After passing through convolutional layers and ReLU activation function layers, the context is obtained. .

3. The end-to-end video compression method based on temporal context mining and reconstructed frame enhancement according to claim 2, characterized in that, Step S32 specifically includes: S321. Reconstruct motion vectors Coarse alignment feature After concatenation, the data is input into a lightweight network to obtain the spatial offset offset1 and the modulation mask mask1. The lightweight network includes three convolutional layers, and a ReLU activation function layer is cascaded between every two convolutional layers. S322. Coarse Alignment Feature Spatial offset1 and modulation mask1 are passed through a deformable convolutional layer and a ReLU activation function layer to obtain refined features. And use it as the input of the multi-scale feature extraction unit.

4. The end-to-end video compression method based on temporal context mining and reconstructed frame enhancement according to claim 3, characterized in that, The lightweight network uses 3×3 convolutional kernels for all three convolutional layers and the deformable convolutional layer, and the stride is uniformly set to 1.

5. The end-to-end video compression method based on temporal context mining and reconstructed frame enhancement according to claim 2, characterized in that, All three dilated convolutional layers use 3×3 convolutional kernels, with dilation coefficients of 1, 2, and 4, respectively.

6. The end-to-end video compression method based on temporal context mining and reconstructed frame enhancement according to claim 1, characterized in that, Step S5 specifically includes: S51. Reconstruction Frame Feature extraction is performed to obtain features By reconstructing motion vectors Features Preliminary geometric alignment is performed to obtain coarse alignment features. ; S52. Features Coarse alignment feature Input alignment unit to obtain reference feature ; S53. Features and reference features After being stitched together, the data is input into the reconstruction unit to obtain the decoded frame. ;in: The reconstruction unit consists of a cascaded first convolutional layer, three residual blocks, and a second convolutional layer.

7. The end-to-end video compression method based on temporal context mining and reconstructed frame enhancement according to claim 6, characterized in that, The processing steps of shallow convolutional networks include: Features Coarse alignment feature After concatenation, the data is input into a shallow convolutional network to obtain shallow features. The shallow convolutional network includes three simple convolutional layers, and each simple convolutional layer is followed by a leaky ReLU layer. All three simple convolutional layers use 3×3 convolutional kernels. The shallow features are passed through the first branch, and the output of the first branch is compared with the reconstructed motion vector. The sum is used to obtain the spatial offset offset2; the first branch includes a 3×3 convolutional layer; The shallow features are passed through the second branch to obtain the modulation mask mask2; the second branch includes a 3×3 convolutional layer and a sigmoid activation function layer; Use spatial offset2, modulation mask mask2, and features Input a deformable convolutional layer with a 3×3 kernel and a stride of 4 to obtain reference features. .

8. The end-to-end video compression method based on temporal context mining and reconstructed frame enhancement according to claim 6, characterized in that, The residual block includes two convolutional layers, each followed by a ReLU activation function layer; an identity shortcut connection is provided between the input of the first convolutional layer and the output of the second convolutional layer.

Images