An optical flow guided feature space context video compression method and system

Abstract

The application discloses a feature space context video compression method and system based on optical flow guidance, and the method comprises the following steps: selecting a current frame and a reference frame in a given video sequence and sequentially performing preprocessing, optical flow guidance processing, context compression processing and frame reconstruction processing to obtain a reconstructed frame of the current frame; a loss function is constructed, and under PSNR and MS-SSIM measurement standards, an evaluation index model is trained respectively to obtain a trained model; the above operation processing is performed on a current frame and a reference frame to be compressed, and the trained model is input for P frame compression to output a compressed and reconstructed current frame. The feature space context video compression method solves the problem of overflow of offset maps in the feature space, improves the image quality of the reconstructed frame, and further improves the compression ratio of the video frame by using a context compression module to extract time priori and hierarchical priori.

Description

Technical Field

[0001] This invention belongs to the field of video encoding and decoding technology, and specifically relates to a feature space context video compression method and system based on optical flow guidance. Background Technology

[0002] Today, video accounts for a significant portion of internet traffic. There is a need to establish an efficient video compression system to reduce transmission and storage costs for various practical applications. Over the past two decades, traditional video compression standards such as H.264, H.265, and more recently H.266 have been developed to improve compression ratios, achieving encouraging results based on hand-crafted techniques. However, higher compression performance is often accompanied by a significant increase in coding complexity, and these hand-designed modules cannot be optimized end-to-end for large-scale video sets.

[0003] An I-frame, also known as an intra-coded frame, is an independent frame containing all its own information. It can be decoded independently without referencing other images; it can be simply understood as a still image. The first frame in a video sequence is always an I-frame because it is the keyframe. A P-frame, also known as an inter-predictive coded frame, requires reference to the preceding I-frame for encoding.

[0004] With the rapid development of deep learning, many learning-based video compression works have explored a new direction. Most of these follow a hybrid predictive coding framework. Currently, most works are based on pixel space; however, in videos with complex, non-rigid motion patterns, video compression often fails and introduces artifacts. To address these issues, existing Feature Space Video Compression (FVC) networks perform operations in the feature space, but lack explicit guidance, often leading to overflow of the offset feature map in the feature space, resulting in degraded reconstructed image quality. Furthermore, according to information theory, the entropy of residual coding-based schemes is greater than or equal to that of conditional coding-based schemes. Summary of the Invention

[0005] The purpose of this invention is to provide a feature space context video compression method and system based on optical flow guidance, which solves the problems of offset map overflow and reduced quality of reconstructed images in the feature space.

[0006] To achieve the above objectives, the solution of the present invention is:

[0007] A feature space context video compression method based on optical flow guidance includes the following steps:

[0008] Step 1: Select the current frame and reference frame in the given video sequence and perform preprocessing to obtain the current frame features, reference frame features, and the original optical flow between the current frame and the reference frame;

[0009] Step 2: Process the current frame features, reference frame features, and the original optical flow between the current frame and the reference frame from Step 1 to obtain the predicted frame features;

[0010] Step 3: Process the current frame features from Step 1 and the predicted frame features from Step 2 to obtain the reconstructed features of the current frame;

[0011] Step 4: Process the reconstruction features of the current frame from Step 3 to obtain the reconstructed frame of the current frame;

[0012] Step 5: Construct a loss function, calculate the error between the reconstructed frame of the current frame and the current frame, and the number of bits required to reconstruct the current frame; using the constructed loss function, train the evaluation index model under the PSNR metric and the MS-SSIM metric respectively to obtain the trained model.

[0013] Step 6: Perform the operations from Step 1 to Step 4 on the current frame and the reference frame to be compressed, and input the trained model from Step 5 to perform P-frame compression, outputting the compressed and reconstructed current frame.

[0014] The specific content of step 1 above is as follows: extract features from the current frame and reference frame in the given video sequence to obtain the current frame features and reference frame features. The reference frame is obtained by compressing and reconstructing the I frame and serves as the reference frame for the P frame. Extract the original optical flow between the current frame and the reference frame using the optical flow method, and extract the features of the current frame and the reference frame respectively.

[0015] The specific content of step 2 above is as follows:

[0016] Optical flow compression is performed on the original optical flow to obtain the reconstructed optical flow;

[0017] Spatial transformation is performed on the reconstructed optical flow and reference frame features to obtain intermediate features;

[0018] Motion estimation processing is performed on intermediate features and current frame features to obtain an estimated offset map;

[0019] The estimated offset map is subjected to feature compression to obtain the reconstructed offset map;

[0020] Convolution calculations are performed on the reconstructed offset map and reconstructed optical flow to obtain the offset and modulation mask of the variable convolution. Variable convolution calculations are performed on the reference frame features, the offset of the variable convolution, and the modulation mask to obtain the variable convolution features. Motion compensation processing is performed on the variable convolution features and the reference frame features to obtain the coarse features of the predicted frame. The coarse features of the predicted frame are added to the variable convolution features to obtain the predicted frame features.

[0021] The specific content of step 3 above is as follows:

[0022] Temporal prior encoding is performed on the predicted frame features to obtain the temporal prior information of the predicted frame features;

[0023] Context encoding is performed on the features of the predicted frame and the features of the current frame to obtain hierarchical prior information;

[0024] Prior fusion processing and context decoding processing are performed on temporal prior information and hierarchical prior information to obtain the reconstructed features of the current frame.

[0025] The specific content of step 4 above is as follows:

[0026] The reconstructed features of the current frame are processed by residual processing and deconvolution to obtain the reconstructed frame of the current frame.

[0027] In step 5 above, based on the rate-distortion optimization function, different evaluation index models are trained using the loss function under the PSNR and MS-SSIM metrics, as shown in the following formula:

[0028]

[0029] Where λ is a Lagrange multiplier, representing the coordination between distortion D and rate R, and d(·) represents the current frame x. t and the current reconstructed frame The distortion, under the PSNR standard, d(·) represents the mean square error; under the MS-SSIM standard, d(·) represents 1-MSSSIM, R s R f and R c These represent the spatial distribution of the encoded optical flow, offset map, and latent context, respectively.

[0030] In step 6 above, the methods for compressing I-frames include, but are not limited to, traditional image compression methods and deep learning-based image compression methods.

[0031] A feature space context video compression system guided by optical flow includes:

[0032] The preprocessing module is used to select the current frame and the reference frame in a given video sequence, and obtain the features of the current frame, the features of the reference frame, and the raw optical flow between the current frame and the reference frame; wherein the reference frame is the frame before the current frame.

[0033] The optical flow guidance module is used to obtain the predicted frame features based on the current frame features, reference frame features, and the original optical flow between the current frame and the reference frame output by the preprocessing module.

[0034] The context compression module is used to obtain the reconstructed features of the current frame based on the current frame features output by the preprocessing module and the predicted frame features output by the optical flow guidance module.

[0035] The frame reconstruction module is used to obtain the reconstructed frame of the current frame based on the reconstruction features of the current frame mentioned above.

[0036] The model training module is used to construct a loss function, calculate the error between the reconstructed frame and the current frame, and the bits required to reconstruct the current frame. Using the constructed loss function, the model is trained on the evaluation metrics under both PSNR and MS-SSIM standards to obtain the trained model.

[0037] The compression and reconstruction module is used to process the current frame to be compressed and the reference frame after they have been processed by the preprocessing module, optical flow guidance module, context compression module and frame reconstruction module, and then input them into the trained model for P-frame compression, and output the compressed and reconstructed current frame.

[0038] The present invention proposes a feature space context video compression method and system based on optical flow guidance. On the one hand, it solves the problem of offset map overflow in the feature space and improves the image quality of the reconstructed frames. On the other hand, it uses the context compression module to extract temporal priors and hierarchical priors, which further improves the compression ratio of video frames. Attached Figure Description

[0039] Figure 1 This is a flowchart of the feature space context video compression method based on optical flow guidance according to the present invention.

[0040] Figure 2 This is a flowchart of the optical flow guiding module of the present invention.

[0041] Figure 3 This is a flowchart of the context compression module of the present invention.

[0042] Figure 4 This is a structural diagram of the Leaky Resblock residual network of the present invention.

[0043] Figure 5 This is a structural diagram of the Resblock residual network of the present invention. Detailed Implementation

[0044] The technical solution and beneficial effects of the present invention will be described in detail below with reference to the accompanying drawings.

[0045] In the feature space context video compression method proposed in this invention, the video compression system consists of intra-frame coding (I-frame coding) and inter-frame coding (P-frame coding). The coding structure is IPPPP…IPPPP…, where GOP (Group of Pictures) is the interval between two I-frames, and the GOP can be specifically set according to the characteristics of the encoded video.

[0046] like Figure 1 As shown, this invention proposes a feature space context video compression method based on optical flow guidance, comprising the following steps:

[0047] Step 1, for a given video sequence V = {x1, x2, ..., x...} n-1 ,x n Preprocessing is performed. This invention employs optical flow networks, including but not limited to Spynet and PWCNet, to extract motion vector information between adjacent frames of the current frame and the reference frame. A feature extraction network is then used to process the current frame x... t ,t∈(2,n) and reference frame Transforming t∈(2,n) into the feature space yields the current frame features f. t and reference frame features f t-1 The feature extraction network consists of one Conv3×3 and three Resblocks. The structure of the Resblock is as follows: Figure 5 As shown, it consists of one Conv3×3, one ReLU, and one Conv3×3 connected in series; optical flow is used to extract the current frame x. t and reference frame The raw optical flow S between two adjacent frames t→t-1 .

[0048] Step 2: Construct an optical flow guidance module to process the current frame features, reference frame features, and the original optical flow between two adjacent frames from Step 1 to obtain the predicted frame features; such as... Figure 2 The diagram shows the flowchart of the optical flow guidance module. The optical flow compression network extracts the original optical flow S. t→t-1 Compression and reconstruction are performed to obtain the reconstructed optical flow. The optical flow compression network is an autoencoder-style network. It first passes through a 3×3 convolutional layer, then three residual blocks. The output feature map is then added to the feature map after the convolutional layer and fed into the next block for processing. At the encoder end, there are three such processing structures. Finally, another 3×3 convolutional layer is applied. After processing, the data is quantized, encoded, and then sent to the optical flow compression decoder for decoding. Decoding this decoder involves a 3×3 deconvolutional layer, followed by processing three residual blocks. The processed result is added to the initial input result and then deconvolved again. This process is repeated three times to obtain the reconstructed optical flow.

[0049] Encoding the original optical flow S using a priori network t→t-1 Bit estimation is performed on the potential coding space to obtain the estimated coded optical flow bits R. s ; for reconstructing optical flow and reference frame features Intermediate features are obtained by performing spatial transformation. The formula is shown below:

[0050]

[0051] Where W represents spatial transformation;

[0052] Build as Figure 2 The motion estimation network shown contains two Conv3×3 layers, which incorporate intermediate features. and current frame features f t The offset map is obtained by feeding it into the motion estimation network. t ;

[0053] Build as Figure 2 The feature compression network shown has the same structure as the optical flow compression network, and is used for the offset map o. t Compressed reconstruction yields the reconstructed offset map. and the estimated bit R of the encoded offset map f Reconstructing optical flow and reconstructed offset map The data is fed into the variable convolution module to calculate the DCN offsets. t→t-1 and modulation mask (DCNmasks) m t→t-1 The formula is shown below:

[0054]

[0055]

[0056] Here, c represents the channel concatenation operation, σ represents the activation function, and {C} o C m} represents the corresponding convolution operation, and then the reference frame features f are processed. t-1 o t→t-1 and m t→t-1 The data is fed into a variable convolutional network (DCN) block and combined with the reference frame features f. t-1 The data is fed into the motion compensation network in series, such as... Figure 2 As shown, the module contains two Conv3×3 layers, which are added to the previous DCN elements to obtain the predicted frame features, as shown in the formula below:

[0057]

[0058] Among them, C mc This represents a motion compensation network containing two Conv3×3 modules, where D represents variable convolution.

[0059] Step 3: Construct a context compression module to process the current frame features from Step 1 and the predicted frame features from Step 2 to obtain the reconstructed features of the current frame; for example... Figure 3 The diagram shows a flowchart of the context compression module of this invention, which constructs a temporal prior encoder to extract features of the predicted frame. The time prior encoder is composed of one Conv5×5, one GDN, one Conv5×5, one GDN, one Conv5×5, one GDN, one Conv5×5, one GDN, one Conv5×5 and one Conv5×5 connected in series.

[0060] Then based on the predicted frame features and current frame features f t Hierarchical prior information is extracted through a context encoder. The context encoder consists of one Conv5×5, one GDN, one Leaky Resblock, one Conv5×5, one GDN, one Leaky Resblock, one Conv5×5, one GDN, and one Conv5×5 connected in series. The structure of the Leaky Resblock is as follows: Figure 4 As shown, it consists of one Conv3×3, one Leaky ReLu, one Conv3×3 and one Leaky ReLu connected in series;

[0061] The context features after the context encoder are encoded into hyperprior features by the hyperprior encoder, then quantized (Q) and arithmetic encoder (AE) to obtain the hyperprior probability distribution in the context space. Finally, the hyperprior features are obtained by arithmetic decoder (AD) and hyperprior decoder.

[0062] A prior fusion network is constructed to fuse temporal and hierarchical prior information to estimate the mean μ of the hidden coding space. t and scale σ t It uses an arithmetic encoder (AE) and an arithmetic decoder (AD) to estimate the encoded bits R. c ;

[0063] Finally, the context decoder is used to obtain the final reconstructed features of the current frame. The context decoder consists of one Subpixel_conv3×3, one GDN, one Subpixel_conv3×3, one GDN, one Leaky ReLu, one Subpixel_conv3×3, one GDN, one Leaky ReLu, and one Subpixel_conv3×3.

[0064] Step 4: Construct a frame reconstruction module to reconstruct the features of the current frame from Step 3. Processing is performed to obtain the reconstructed frame of the current frame. Frame reconstruction module such as Figure 1 As shown, it consists of 3 Resblocks and 1 Deconv3×3.

[0065] Step 5: Construct the loss function and calculate the reconstructed frame for the current frame. Error between the original frame and the reconstructed frame of the current frame The number of bits required; under different metric standards, using the constructed loss function, different evaluation index models are trained to obtain the trained model; the loss function is divided into PSNR standard and MS-SSIM standard metric. According to the Rate Distortion Optimization (RDBEM) function, under different metric standards, different evaluation index models are trained using the loss function to achieve the effect under each metric. Under the PSNR metric, a PSNR model is generated with a higher PSNR value; under the MS-SSIM metric, an MS-SSIM model is generated with a higher MS-SSIM value.

[0066]

[0067] Where λ is a Lagrange multiplier, representing the coordination between distortion D and rate R, and d(·) represents the current frame x. t and the current reconstructed frame The distortion, under the PSNR standard, d(·) represents the mean square error; under the MS-SSIM standard, d(·) represents 1-MSSSIM. Through entropy coding, the coded optical flow, offset map features, and bits in the context space are calculated, where R... s R f and R c These represent the spatial distributions of the encoded optical flow, offset map, and latent context, respectively.

[0068] The hyperparameters of the target relative to the network are solved using forward propagation of the network. In this embodiment, gradient descent is used to update the hyperparameters, including but not limited to Adam and SGD optimizers. During network training, the Vimeo-90k dataset is used as the training set, which includes 89,800 video clips, each with 7 consecutive frames. We train for approximately 3,000,000 steps. The learning rate for the first 2,000,000 steps is 5e-5 with a batch size of 4, and the learning rate for the last 1,000,000 steps is 5e-6 with a batch size of 2.

[0069] Step 6: Compress and reconstruct the I-frame to obtain a reference frame. This reference frame, along with the original uncompressed current frame, is fed into the trained network for P-frame compression. The output is the compressed and reconstructed current frame. After the model learns the model parameters, in this embodiment, image compression methods (not limited to deep learning, such as cheng2020-anchor) are first used to compress the I-frame. The compressed and reconstructed I-frame is used as a reference frame. This reference frame, along with the original uncompressed current frame, is fed into the trained network for P-frame compression to obtain the compressed and reconstructed current frame. The reconstructed current frame is then used as the reference frame for the next P-frame compression. The GOP (Group of Pictures) can be specifically set according to the characteristics of the encoded video.

[0070] This invention proposes a feature space context video compression system based on optical flow guidance, comprising:

[0071] The preprocessing module is used to select the current frame and the reference frame in a given video sequence, and obtain the features of the current frame, the features of the reference frame, and the raw optical flow between the current frame and the reference frame; wherein the reference frame is the frame before the current frame.

[0072] The optical flow guidance module is used to obtain the predicted frame features based on the current frame features, reference frame features, and the original optical flow between the current frame and the reference frame output by the preprocessing module.

[0073] The context compression module is used to obtain the reconstructed features of the current frame based on the current frame features output by the preprocessing module and the predicted frame features output by the optical flow guidance module.

[0074] The frame reconstruction module is used to obtain the reconstructed frame of the current frame based on the reconstruction features of the current frame mentioned above.

[0075] The model training module is used to construct a loss function, calculate the error between the reconstructed frame and the current frame, and the bits required to reconstruct the current frame. Using the constructed loss function, the model is trained on the evaluation metrics under both PSNR and MS-SSIM standards to obtain the trained model.

[0076] The compression and reconstruction module is used to process the current frame to be compressed and the reference frame after they have been processed by the preprocessing module, optical flow guidance module, context compression module and frame reconstruction module, and then input them into the trained model for P-frame compression, and output the compressed and reconstructed current frame.

[0077] In summary, this invention proposes a feature space context video compression method and system based on optical flow guidance, addressing the problems of offset map overflow and reduced reconstructed image quality in the feature space. The method includes the following steps: selecting the current frame and reference frame from a given video sequence and preprocessing them to obtain current frame features, reference frame features, and the original optical flow between the current and reference frames; processing the current frame features, reference frame features, and the original optical flow between the current and reference frames to obtain predicted frame features; processing the current frame features and predicted frame features to obtain reconstructed features of the current frame; processing the reconstructed features of the current frame to obtain the reconstructed frame of the current frame; constructing a loss function to calculate the error between the reconstructed frame and the current frame, and the bits required for the reconstructed frame; using the constructed loss function, training the evaluation index model under both the PSNR and MS-SSIM metrics to obtain the trained model; performing the above operations on the current frame and reference frame to be compressed, inputting them into the trained model for P-frame compression, and outputting the compressed and reconstructed current frame. The feature space context video compression method of the present invention solves the problem of offset map overflow in the feature space by utilizing optical flow motion information, thereby improving the image quality of the reconstructed frame; and further improves the compression ratio of the video frame by using the context compression module to extract temporal prior and hierarchical prior.

[0078] The above embodiments are merely illustrative of the technical concept of the present invention and should not be construed as limiting the scope of protection of the present invention. Any modifications made to the technical solutions based on the technical concept proposed in this invention shall fall within the scope of protection of this invention.

Claims

1. A feature space context video compression method based on optical flow guidance, characterized in that, The steps include the following: Step 1: Select the current frame and reference frame in the given video sequence and perform preprocessing to obtain the current frame features, reference frame features, and the original optical flow between the current frame and the reference frame; Step 2: Process the current frame features, reference frame features, and the original optical flow between the current frame and the reference frame from Step 1 to obtain the predicted frame features; The specific content of step 2 is as follows: Optical flow compression is performed on the original optical flow to obtain the reconstructed optical flow; Spatial transformation is performed on the reconstructed optical flow and reference frame features to obtain intermediate features; Motion estimation processing is performed on intermediate features and current frame features to obtain an estimated offset map; The estimated offset map is subjected to feature compression to obtain the reconstructed offset map; Convolution calculations are performed on the reconstructed offset map and reconstructed optical flow to obtain the offset and modulation mask of the variable convolution. The variable convolution is calculated by performing variable convolution on the reference frame features, the offset of the variable convolution, and the modulation mask to obtain the variable convolution features. The variable convolution features and the reference frame features are then subjected to motion compensation processing to obtain the coarse features of the predicted frame. Finally, the coarse features of the predicted frame are added to the variable convolution features to obtain the predicted frame features. Step 3: Process the current frame features from Step 1 and the predicted frame features from Step 2 to obtain the reconstructed features of the current frame; Step 4: Process the reconstruction features of the current frame from Step 3 to obtain the reconstructed frame of the current frame; Step 5: Construct a loss function, calculate the error between the reconstructed frame of the current frame and the current frame, and the number of bits required to reconstruct the current frame; using the constructed loss function, in Under the metric standard and the MS-SSIM metric standard, the evaluation index model is trained to obtain the trained model; Step 6: Perform the operations from Step 1 to Step 4 on the current frame and the reference frame to be compressed, and input the trained model from Step 5 to perform P-frame compression, outputting the compressed and reconstructed current frame.

2. The feature space context video compression method as described in claim 1, characterized in that, The specific content of step 1 is as follows: extract features from the current frame and reference frame in the given video sequence to obtain the current frame features and reference frame features. The reference frame is obtained by compressing and reconstructing the I frame and serves as the reference frame for the P frame. Extract the original optical flow between the current frame and the reference frame using the optical flow method, and extract the features of the current frame and the reference frame respectively.

3. The feature space context video compression method as described in claim 1, characterized in that, The specific content of step 3 is as follows: Temporal prior encoding is performed on the predicted frame features to obtain the temporal prior information of the predicted frame features; Context encoding is performed on the features of the predicted frame and the features of the current frame to obtain hierarchical prior information; Prior fusion processing and context decoding processing are performed on temporal prior information and hierarchical prior information to obtain the reconstructed features of the current frame.

4. The feature space context video compression method as described in claim 1, characterized in that, The specific content of step 4 is as follows: The reconstructed features of the current frame are processed by residual processing and deconvolution to obtain the reconstructed frame of the current frame.

5. The feature space context video compression method as described in claim 1, characterized in that, In step 5, based on the rate-distortion optimization function, Under the metric standard and the MS-SSIM metric standard, different evaluation index models are trained using the loss function, as shown in the following formulas: ， in, It is a Lagrange multiplier, which is distorted. and rate Coordination, Represents the current frame and the current reconstructed frame The distortion, in Under the standard, Represents the mean squared error; under the MS-SSIM standard, Represents 1-MSSSIM, , and These represent the spatial distribution of the encoded optical flow, offset map, and latent context, respectively.

6. The feature space context video compression method as described in claim 1, characterized in that, In step 6, the methods for compressing I-frames include, but are not limited to, traditional image compression methods and deep learning-based image compression methods.

7. A feature spatial context video compression system based on optical flow guidance, characterized in that, include: A preprocessing module is used to select the current frame and a reference frame in a given video sequence, and obtain the features of the current frame, the features of the reference frame, and the raw optical flow between the current frame and the reference frame; wherein the reference frame is the frame preceding the current frame. The optical flow guidance module is used to obtain the predicted frame features based on the current frame features, reference frame features, and the original optical flow between the current frame and the reference frame output by the preprocessing module. The context compression module is used to obtain the reconstructed features of the current frame based on the current frame features output by the preprocessing module and the predicted frame features output by the optical flow guidance module. A frame reconstruction module is used to obtain a reconstructed frame of the current frame based on the reconstruction features of the current frame; The model training module is used to construct the loss function, calculate the error between the reconstructed frame and the current frame, and the bits required to reconstruct the current frame; using the constructed loss function, in Under both the metric standard and the MS-SSIM metric standard, the evaluation index model is trained to obtain the trained model; and, The compression and reconstruction module is used to process the current frame to be compressed and the reference frame after they have been processed by the preprocessing module, optical flow guidance module, context compression module and frame reconstruction module, and then input them into the trained model for P-frame compression, and output the compressed and reconstructed current frame. The specific contents of the optical flow guidance module are as follows: Optical flow compression is performed on the original optical flow to obtain the reconstructed optical flow; Spatial transformation is performed on the reconstructed optical flow and reference frame features to obtain intermediate features; Motion estimation processing is performed on intermediate features and current frame features to obtain an estimated offset map; The estimated offset map is subjected to feature compression to obtain the reconstructed offset map; Convolution calculations are performed on the reconstructed offset map and reconstructed optical flow to obtain the offset and modulation mask of the variable convolution. The variable convolution feature is obtained by performing variable convolution calculation on the reference frame feature, the offset of the variable convolution and the modulation mask. The variable convolution feature and the reference frame feature are then subjected to motion compensation processing to obtain the coarse feature of the prediction frame. Finally, the coarse feature of the prediction frame is added to the variable convolution feature to obtain the prediction frame feature.

Images