End-to-end video compression method based on enhanced spatio-temporal context mining and enhanced reconstructed frames

Abstract

The present application belongs to the technical field of scalable video coding, and particularly relates to an end-to-end video compression method based on enhanced spatio-temporal context mining and enhanced reconstructed frame, which is realized through an end-to-end deep learning model, wherein the model efficiently utilizes motion vector time series information through an improved spatio-temporal context mining module at the encoding end, and utilizes the motion vector generated by an optical flow network and the motion vector compensated to the current time to jointly generate the final required motion vector at the current time, so as to significantly improve the accuracy of the motion vector. The scheme can effectively utilize the redundant information between different levels, reduce unnecessary data transmission and improve the video compression efficiency through inter-layer prediction, fusion of motion vector information of different layers, fusion of texture information of different layers and the like.

Description

Technical Field

[0001] This invention belongs to the field of quality scalable video coding technology, specifically relating to an end-to-end video compression method based on enhanced spatiotemporal context mining and enhanced reconstructed frames. Background Technology

[0002] As video data accounts for an increasingly larger proportion of internet traffic, effectively utilizing limited bandwidth and coping with complex network conditions to transmit videos with varying quality levels has become crucial. To address this, several scalable video coding standards have emerged in recent years, such as H.264 / SVC, H.265 / SHVC, and H.266 / VVC. These standards significantly reduce compression costs when transmitting videos at different resolutions and frame rates. Traditional Scalable Video Coding (SVC) achieves scalability in spatial, quality, and temporal dimensions through a block-based coding approach.

[0003] However, traditional block-based scalable video coding methods have some limitations, especially in terms of flexibility and performance improvement. First, these methods are based on block-level prediction and do not utilize all the information in the entire video frame. Second, the modules of these encoders are designed independently; encoders composed of multiple individual modules cannot achieve end-to-end optimization, thus failing to maximize video compression efficiency. Furthermore, after decades of development, the performance improvements of these traditional encoders have reached a plateau, and the rate of improvement is slow.

[0004] With the development of deep neural network (DNN) technology, researchers have begun to explore its application in video coding, thereby driving new progress in video coding technology. However, current research focuses mostly on single-layer video coding frameworks, with relatively little research on quality-scalable video coding frameworks. For such single-layer video coding frameworks, firstly, the framework structure is relatively simple, and video transmission can only be achieved under good network conditions. Secondly, it cannot effectively utilize bandwidth and cannot dynamically adjust according to network conditions or the capabilities of terminal devices. Thirdly, because single-layer coding compresses under a fixed setting, it may not be able to fully utilize redundant information between different video layers. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides an end-to-end video compression method based on enhanced spatiotemporal context mining and enhanced reconstructed frames, comprising:

[0006] S1: Encode the base layer video to obtain the motion vector of the base layer at the current time step. and the motion vector of the previous moment And calculate the motion difference Where B represents the basic layer and t represents the current time.

[0007] S2: Utilizing the motion difference The motion vector of the enhancement layer at the previous moment Perform time-domain compensation to obtain the compensated motion vector. ;

[0008] S3: Compensate motion vector The original motion vector estimated by the optical flow network The final motion vector is obtained by fusion. ;

[0009] S4: Based on the final motion vector Complete the encoding and reconstruction of the enhancement layer video frames.

[0010] The beneficial effects of this invention are:

[0011] This invention compensates for the motion vector of the enhancement layer at the previous moment by introducing the motion vector difference of the base layer. This allows the motion information of the enhancement layer to be dynamically corrected according to the time change trend of the base layer. The compensated motion vector can more accurately reflect the true motion state at the current moment, effectively reducing the accumulated error caused by the independent estimation of the motion vector of the enhancement layer in traditional methods, thereby improving the accuracy of motion vector prediction and motion estimation of the enhancement layer.

[0012] This invention fuses the compensated motion vector with the motion vector directly estimated by the optical flow network. It achieves deep fusion of dual-source motion information through convolutional networks and self-attention mechanisms. The self-attention module can automatically weight the motion region based on global correlation, making the system pay more attention to the real motion region, suppressing noise and optical flow estimation failure regions, and greatly improving the stability of motion vectors and robustness in complex scenes. Attached Figure Description

[0013] Figure 1 This is a schematic diagram of the encoding framework of the basic layer of the present invention;

[0014] Figure 2 This is a schematic diagram of the quality-scalable video enhancement layer coding framework of the present invention;

[0015] Figure 3 This is a schematic diagram illustrating the relationship between motion vectors at different times in the basic layer and the enhancement layer of the present invention;

[0016] Figure 4 This is a schematic diagram of the motion vector prediction structure based on temporal and inter-layer relationships of the present invention;

[0017] Figure 5 This is a flowchart of the multiple motion vector fusion process of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] An end-to-end video compression method based on enhanced spatiotemporal context mining and enhanced reconstructed frames is proposed, the core of which lies in the implementation of an end-to-end deep learning model. At the encoding end, this model efficiently utilizes the temporal information of motion vectors through an improved spatiotemporal context mining module. We use motion vectors generated by an optical flow network and motion vectors compensated to the current time step to jointly generate the final required motion vector for the current time step, thereby significantly improving the accuracy of the motion vectors. The overall training of the model aims at rate-distortion optimization.

[0020] The detailed implementation steps of this embodiment are as follows:

[0021] S1. Interlayer motion vector information Extraction:

[0022] In this embodiment, the motion vectors of the base layer and the enhancement layer at corresponding moments have a strong similarity, so we can utilize the motion difference between adjacent motion vectors of the base layer (BL). The motion vector of the enhancement layer (EL) at the previous moment Perform time alignment and compensation to obtain the motion vector of the enhancement layer at the current time step, obtained by compensation to the current time step. The entire process is as follows.

[0023] The basic layer adopts the DCVC-TCM end-to-end video coding framework, including context encoding / decoding, motion vector encoding / decoding, motion estimation, temporal context mining, and entropy model modules. The overall process is as follows: Figure 1 As shown, the specific process is as follows:

[0024] First, we need to use the previous video frame. and the current video frame The motion vector is calculated in the motion estimation module. Then, it needs to be encoded and decoded using motion vector encoding and decoding to obtain motion vectors at the decoding end. Then, at the decoding end, this decoded motion vector is used to analyze the feature map of the previous frame. Motion compensation is performed by the temporal context mining module. This yields conditional features representing the current moment, which are then input into the context codec. The codec uses these features to encode and decode the current video frame, resulting in a reconstructed video frame at the decoding end. To optimize the context codec's encoding and decoding process, the conditional features are also input into the entropy model. The entropy model extracts the data distribution of the conditional feature map, which is beneficial for encoding and decoding the current video frame. Note that all the information mentioned above can be obtained directly at the encoding end, without considering data issues at the encoding end; the same applies to the enhancement layer below.

[0025] The enhancement layer includes: context encoding / decoding, motion vector encoding / decoding, motion estimation, temporal context mining, and an entropy model module, and its data flow is the same as the base layer. However, the enhancement layer's prediction and utilization of motion vectors is too simplistic and does not take advantage of the multi-layer video coding framework, resulting in insufficient quality of the reconstructed video frames. To address this issue, we propose a method to enhance the motion vectors of the enhancement layer by utilizing the relationships between motion vectors at different times in the base layer. The specific framework of the enhancement layer is as follows: Figure 2 As shown, the functions and locations of the added modules are explained below:

[0026] like Figure 2 As shown, after simple motion estimation and motion vector encoding / decoding, the process is the same as the basic layer, so it will not be repeated here. We obtain the simple motion vector at the current moment. Then we utilize the motion vectors from the base layer. and Calculate the relationship between the two, and then apply this relationship to the motion vector of the previous moment. Thus, the motion vector of the previous moment This compensates for the current moment, and then the compensated motion vector is summed with the simple motion vector obtained at the current moment. The motion vectors are fused to obtain higher quality motion vectors, and then these higher quality motion vectors are applied to the motion compensation of subsequent conditional features to obtain higher quality conditional features.

[0027] First, the system determines the motion vector of the enhancement layer at the previous moment. Does it exist? Also check the motion difference of the base layer. Has the calculation been successful? Generally, neither of these two parameters exists in the first frame, but they will exist in subsequent video frames. This invention applies to video frames from the first frame onwards. When both are available, proceed with the motion compensation process of this embodiment; if either is absent, it is considered that cross-time inference cannot be performed, and the original motion prediction value obtained from the current enhancement layer model is used directly. As the final output.

[0028] When cross-time inference is possible, the system first performs... and Enable learnable gradients so they can participate in parameter updates of subsequent network modules. Then, through a motion warping module, such as... Figure 4 As shown, the motion difference of the basic layer is utilized. The motion field of the enhancement layer at the previous moment Temporal compensation is performed to obtain the augmentation layer motion estimate aligned to the current time. This process enables time prediction of the motion field of the enhanced layer based on the motion trend of the basic layer.

[0029] S2. Motion vector correction from the previous moment:

[0030] As shown in step one, we can utilize To the motion vector of the enhancement layer at the previous moment Correction is performed to obtain the compensated motion vector at the current moment. However, in quality-scalable video coding, the motion vectors of the preceding and following frames of the base layer and the corresponding motion vectors of the enhancement layer may differ. The specific process is as follows: Figure 4 As shown, we first use 2D convolution and activation functions to extract features from the motion vector differences of the basic layers. Then, we employ a deformable convolutional network to perform multi-level correction on the features, thereby achieving more refined repair. After repair, the features are uniformly input into the self-attention module to achieve more weighted fusion in the spatial domain.

[0031] Due to the diversity and complex motion patterns of video sequences, the corrected motion vectors may exhibit spatial instability, and differences also exist between different motion vector channels. To improve the motion accuracy of motion vectors, we first employ a self-attention mechanism to enhance the long-range dependencies of motion vectors in the spatial domain. Subsequently, we input this into a channel attention module to perform weighted modeling of different channels, highlighting motion vector channels that contribute more to motion estimation, thereby improving the stability and robustness of the motion vectors.

[0032] Since there may be some differences in motion vectors between the enhancement layer and the base layer, a single network cannot simultaneously model the overall motion trend and correct local detail errors. Therefore, this paper employs a multi-level network to progressively repair the motion vectors and fully utilizes the continuity of motion in the time dimension to improve the overall accuracy and stability of the motion vectors.

[0033] S3. Fuse multiple motion vectors:

[0034] like Figure 5 As shown, we fuse the original motion vectors with the decoded motion vectors. and the corrected motion vector To generate the final motion vector To improve the accuracy and stability of motion vector prediction, we employ residual networks and attention mechanisms to further refine and improve the motion vectors. The specific steps are as follows:

[0035] First, we concatenate the two motion vector features along the channel dimension. Then, to better correct the errors and prediction biases of the motion vectors step by step, we subsequently use a residual network to process them.

[0036]

[0037] in, This represents the motion vector after being repaired by the neural network; This represents a neural convolutional network with residual connections. This indicates that two feature data are joined along the channel dimension. This represents the motion vector after it has been repaired by the time-motion vector fusion module. This represents the motion vector simply generated by the motion estimation process.

[0038] To enable better motion vector compensation in subsequent steps, we employ a self-attention mechanism to repair the motion vectors. Our model allows the motion vectors to capture long-range spatial dependencies and progressively refine features.

[0039]

[0040] in, Represents the final motion vector. This represents the motion vector after being repaired by the neural network; This indicates a self-attention mechanism repair module. Dot product, This represents the proportion of self-attention-generated features within the overall features. The proportion of attention-generated features increases when there are more distant pixel correlations in the motion vector, and vice versa. Therefore, this approach can automatically adapt to the magnitude of the motion trend. In one implementation, It can be generated by a learnable gating module and constrained to the [0,1] interval using a sigmoid function to ensure stable and controllable fusion weights; furthermore, It can also be generated adaptively based on spatial location.

[0041] Because a multi-time motion vector fusion mechanism is adopted, the quality of motion vector reconstruction can be significantly improved, avoiding the low quality of reconstructed frames caused by motion vector instability, thereby further improving the quality of reconstructed frames.

[0042] Based on the final motion vector The enhancement layer video frame encoding and reconstruction are completed. The generation process mainly includes motion vector prediction and motion vector fusion. Its network architecture is as follows: Figure 4 , Figure 5 As shown, the specific process is explained below:

[0043] exist Figure 4 In this process, we use convolutional neural networks and activation functions to extract the motion vectors of the basic layers. and the motion vector of the previous moment We then examine the relationship between the two and apply this relationship to a convolutional network, using deformable convolutional modules to add to the motion vector of the enhancement layer in the previous time step. This allows us to obtain a repaired motion vector. Then we compare this repaired motion vector with the motion vector simply obtained through the motion compensation process. Enter together Figure 4 The motion vector shown is processed through a channel attention mechanism, a convolutional neural network, and a self-attention mechanism to obtain the final motion vector used for motion compensation. .

[0044] Model training and optimization:

[0045] The entire model is trained in an end-to-end manner. The training dataset uses publicly available large-scale video datasets, such as the Vimeo-90k Septuplet dataset, with videos randomly cropped into 256×256 segments for training.

[0046] First, we need to train the base layer. We use the DCVC-TCM model as our base layer, and its training loss function is:

[0047]

[0048] Where D is the distortion function, and in this embodiment, mean squared error (MSE) is used. These are hyperparameters related to the control rate-distortion tradeoff, among which This represents the number of bits required for encoding. To cover different code rate points, this embodiment trains four models, which... The values ​​were set to 180, 512, 1400, and 5000 respectively. The value can be set very flexibly and can be customized as needed. This is just one example.

[0049] The optimizer used is AdamW, with a batch size of 26. Other optimizers commonly used in deep learning can also be used. The batch size can also be adjusted as needed.

[0050] After training the basic layer model, we use the LSSVC enhancement layer as the basic framework for our enhancement layer, and its training function is:

[0051]

[0052] Where D is the distortion function, and in this embodiment, mean squared error (MSE) is used. These are hyperparameters related to the control rate-distortion tradeoff, among which This represents the number of bits required for encoding. To cover different code rate points, this embodiment trains four models, which... The values ​​were set to 220, 480, 1000, and 2200 respectively. The optimizer used was AdamW, and the batch size was set to 26.

[0053] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An end-to-end video compression method based on enhanced spatiotemporal context mining and enhanced reconstructed frames, characterized in that, include: S1: Encode the base layer video to obtain the motion vector of the base layer at the current time step. and the motion vector of the previous moment And calculate the motion difference Where B represents the basic layer and t represents the current time. S2: Utilizing the motion difference The motion vector of the enhancement layer at the previous moment Perform time-domain compensation to obtain the compensated motion vector. ; S3: Compensate motion vector The original motion vector estimated by the optical flow network The final motion vector is obtained by fusion. ; S4: Based on the final motion vector Complete the encoding and reconstruction of the enhancement layer video frames.

2. The end-to-end video compression method based on enhanced spatiotemporal context mining and enhanced reconstructed frames according to claim 1, characterized in that, The basic layer adopts the DCVC-TCM end-to-end video coding framework, including context encoding / decoding, motion vector encoding / decoding, motion estimation, temporal context mining, and entropy model modules; Using the previous video frame and the current video frame The motion vector is calculated in the motion estimation module. The motion vectors are encoded and decoded using a motion vector encoding / decoding module, and then used at the decoding end to obtain motion vectors. Finally, the decoded motion vectors are used to analyze the feature map of the previous frame. Motion compensation is performed by the temporal context mining module, which obtains conditional features representing the current moment. These features are then input into the context codec, which uses them to encode and decode the current video frame, thus reconstructing the video frame at the decoding end. Simultaneously, to optimize the encoding and decoding process of the context codec, the conditional features are input into the entropy model. The entropy model extracts the data distribution of the conditional feature map, and the extracted distribution is beneficial for the encoding and decoding of the current video frame.

3. The end-to-end video compression method based on enhanced spatiotemporal context mining and enhanced reconstructed frames according to claim 1, characterized in that, The enhancement layer has the same structure and data flow as the basic layer. After motion estimation and motion vector encoding / decoding, it obtains the simple motion vector at the current moment. Then, using motion vectors from the base layer and The relationship between the two is calculated and applied to the motion vector of the previous moment. The motion vector of the previous moment Compensate to the current moment, then sum the compensated motion vector with the current motion vector. By fusing the motion vectors, a higher quality motion vector is obtained. This higher quality motion vector is then used in the motion compensation of the subsequent conditional features, thereby obtaining a higher quality conditional feature.

4. The end-to-end video compression method based on enhanced spatiotemporal context mining and enhanced reconstructed frames according to claim 1, characterized in that, Compensation motion vector The original motion vector estimated by the optical flow network The final motion vector is obtained by fusion. ,include: First, consider the two motion vectors... and The channels are connected in the same dimension, and a residual network is used to progressively repair the errors and prediction biases of the spliced ​​motion vectors. A self-attention mechanism is adopted to enable the motion vectors to capture long-distance spatial dependencies, and the features are gradually refined to obtain the final motion vector. .

5. The end-to-end video compression method based on enhanced spatiotemporal context mining and enhanced reconstructed frames according to claim 4, characterized in that, A residual network is used to progressively correct the errors and prediction biases in the spliced ​​motion vectors, including: ； in, This represents the motion vector after being repaired by the neural network; This represents a neural convolutional network with residual connections. This indicates that two feature data are joined along the channel dimension. This represents the motion vector after it has been repaired by the time-motion vector fusion module. This represents the motion vector simply generated by the motion estimation process.

6. The end-to-end video compression method based on enhanced spatiotemporal context mining and enhanced reconstructed frames according to claim 4, characterized in that, A self-attention mechanism is employed to capture long-range spatial dependencies in motion vectors, and features are progressively refined, including: ； in, Represents the final motion vector. This represents the motion vector after being repaired by the neural network; This indicates a self-attention mechanism repair module. This represents the learnable weight parameters. This indicates dot product.

7. The end-to-end video compression method based on enhanced spatiotemporal context mining and enhanced reconstructed frames according to claim 1, characterized in that, Based on the final motion vector Complete the encoding and reconstruction of the enhancement layer video frames, including: Extracting motion vectors of the base layer using convolutional neural networks and activation functions and the motion vector of the previous moment The relationship between the two is then applied to a convolutional network, using deformable convolutional modules to add to the motion vector of the enhancement layer in the previous time step. This allows us to obtain a repaired motion vector. Then, the repaired motion vector and the motion vector simply obtained through the motion compensation process are compared. After processing through channel attention, convolutional neural networks, and self-attention mechanisms, the final motion vectors used for motion compensation are obtained. .

Images