A variable bit rate video compression method, system, device and storage medium

Abstract

The present invention provides a variable bit rate video compression method, system, device and storage medium, including: passing a reference frame image through a convolutional neural network to obtain a first feature map of the reference frame, scaling and quantizing to obtain a second feature map of the reference frame, and performing inverse scaling to obtain a reconstructed reference frame. Passing a predicted frame image through a convolutional neural network to obtain a first feature map of the predicted frame, and scaling and quantizing to obtain a second feature map of the predicted frame. Subtracting the second feature map of the predicted frame from the second feature map of the reference frame to obtain a feature map residual; using the second feature map of the reference frame as prior information, performing lossless entropy coding on the feature map residual and transmitting it to a decoding end; adding the feature map residual to the second feature map of the reference frame and inverse scaling, and then obtaining a reconstructed predicted frame through a convolutional neural network. The present invention scales the feature map using a scale factor to obtain feature maps corresponding to different bit rate qualities.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and relates to an end-to-end optimized variable bit rate video compression scheme using temporal and spatial domain context information, and specifically to a variable bit rate video compression method, system, device and storage medium. Background Art

[0002] End-to-end optimization is a technique commonly used in image and video compression. It uses convolutional neural networks to model compression as a rate-distortion bidirectional optimization problem, and can jointly optimize various modules. Variable bitrate schemes can output output images or video sequences of different qualities and bitrates for input images or video sequences through a single model. After searching the literature on the prior art, it was found that Lu et al. proposed the first end-to-end video coding framework in their paper “DVC: ​​An End-to-End DeepVideo Compression Framework” published at the 2019 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), which implemented the transformation, motion estimation, motion compensation, and entropy coding modules in the traditional framework using deep convolutional neural networks. Abdelaziz et al. published their paper “Neural Inter-Frame Compression For Video Coding” at the 2019 IEEE International Conference on Computer Vision (ICCV), which used bidirectional prediction for motion estimation and motion compensation. Yang et al. published "Learning For Video Compression with HierarchicalQuality and Recurrent Enhancement" at the 2020 IEEE Conference on Computer Vision and PatternRecognition (CVPR). They proposed a hierarchical end-to-end optimized video compression model using a weighted recurrent reinforcement network and combining unidirectional and bidirectional predictions. However, since symbols can only be restored sequentially, the context model brings computational complexity overhead. When a three-dimensional context model is used, the required encoding and decoding time will increase significantly.

[0003] However, the above methods usually use optical flow networks and residual networks for motion estimation and residual compensation. The input of the optical flow network at the encoding end is the reconstructed video frame, so the reconstruction quality directly affects the prediction efficiency. Lower reconstruction quality will lead to an increase in the corresponding residual information encoding overhead. When the residual coding overhead is limited, the quality of the reconstructed frame will be significantly reduced, eventually causing serious inter-frame error propagation problems. In addition, the complex network structure and prediction compensation framework make the model design of variable bit rate coding more complicated, so this type of method trains multiple models independently for different bit rate points. Summary of the invention

[0004] The present invention aims at the deficiencies in the current video compression schemes applied to end-to-end optimization, and therefore proposes a variable bit rate video compression method without motion estimation.

[0005] According to a first aspect of the present invention, there is provided a variable bit rate video compression method, comprising:

[0006] S1, variable bit rate intra-frame encoding and decoding: At the encoding end, the reference frame image is passed through a convolutional neural network to obtain the first feature map of the reference frame, and the quantized second feature map of the reference frame is obtained by scaling the feature map and quantizing it. After lossless entropy coding and transmission. At the decoding end, the quantized second feature map of the reference frame is inversely scaled and reconstructed through a convolutional neural network.

[0007] S2, variable bit rate inter-frame coding: At the encoding end, the predicted frame image is passed through a convolutional neural network to obtain a first feature map of the predicted frame, and a quantized second feature map of the predicted frame is obtained by scaling and quantizing the feature map. The second feature map of the predicted frame is subtracted from the second feature map of the reference frame to obtain a feature map residual.

[0008] S3, inter-frame residual entropy coding: using the obtained second feature map of the reference frame as prior information, performing lossless entropy coding on the feature map residual, and transmitting it to the decoding end.

[0009] S4, variable bit rate inter-frame decoding: At the decoding end, the feature map residual is added to the second feature map of the reference frame obtained and inversely scaled, and then a reconstructed prediction frame is obtained through a convolutional neural network.

[0010] Preferably, the variable bit rate intra-frame coding and decoding includes:

[0011] S11. Image feature extraction: The image is passed through a convolutional neural network to extract the first feature map.

[0012] S12, feature map scaling: multiplying the first feature map by a scaling factor to obtain the second feature map. The scaling factor is a scalar greater than 0, and the range is specified in advance.

[0013] S13, feature-derivable quantization: for the second feature map, simulate quantization by adding uniform noise to generate a second feature map of the reference frame;

[0014] S14, intra-frame entropy coding: The quantized second feature map of the reference frame is encoded and transmitted using lossless entropy coding.

[0015] S15, feature map inverse scaling: at the decoding end, the second feature map of the reference frame is multiplied by an inverse scaling factor to obtain the reconstructed first feature map. The inverse scaling factor corresponds to the scaling factor one by one.

[0016] S16, image reconstruction: the reconstructed first feature map is passed through a convolutional neural network to obtain a reconstructed image; by changing the scale factor and the inverse scale factor, the variable bit rate intra-frame encoding and decoding method can obtain reconstructed images with different bit rates and qualities.

[0017] Preferably, the variable bit rate inter-frame coding includes:

[0018] S21. Image feature extraction: The image is passed through a convolutional neural network to extract a first feature map.

[0019] S22, feature map scaling: multiplying the first feature map by a scaling factor to obtain the second feature map. The scaling factor is a scalar greater than 0, and the range is specified in advance.

[0020] S23, feature-derivable quantization: for the second feature map, simulate quantization by adding uniform noise to generate a second feature map of the prediction frame;

[0021] S24, feature residual acquisition: subtract the second feature map of the reference frame from the second feature map of the prediction frame to obtain the feature map residual to be encoded.

[0022] Preferably, the entropy coding of the inter-frame residual includes:

[0023] S31, extracting super prior information: after splicing the second feature map of the reference frame and the second feature map of the prediction frame, obtain super prior information through a convolutional neural network;

[0024] S32, super-prior derivable quantization: for the super-prior information, simulate quantization by adding uniform noise to generate quantized super-prior information;

[0025] S33, super-prior information reconstruction: the quantized super-prior information is passed through a convolutional neural network to obtain reconstructed super-prior information;

[0026] S34, context modeling: modeling is performed according to the second feature map of the encoded reference frame and the partial feature map whose feature map residual has been encoded, to obtain context information for encoding the feature residual;

[0027] S35, construction entropy estimation: according to the construction distribution, estimate the bit rate L required for quantized super-prior information encoding BPP-H ; The entropy coding of the method includes encoding two parts of information, namely, super-prior information and feature map residual.

[0028] S36, conditional entropy estimation: According to the mean and variance of the Gaussian distribution, it means using Gaussian distribution to encode the feature map residual and estimating the bit rate L required for encoding the feature map residual. BPP-F ;

[0029] Preferably, the context modeling is modeled based on the reference frame feature map encoded in the time domain and the partial feature map in which the feature parameter residual has been encoded, to obtain the context for encoding the feature residual, including:

[0030] Temporal context information extraction: Using the second feature map of the reference frame, the context information in the temporal dimension is obtained through a convolutional neural network;

[0031] Spatial context information extraction: Extract the feature residuals between the reference frame and the predicted frame, and obtain the context information in the spatial dimension through a convolutional neural network;

[0032] Entropy coding distribution estimation: The context information and the super prior information are integrated to estimate the mean and variance of the Gaussian distribution used for coding, and output them to arithmetic coding.

[0033] Preferably, the variable bit rate inter-frame decoding includes:

[0034] S41, feature map inverse scaling: At the decoding end, the second feature map of the reference frame is multiplied by an inverse scale factor to obtain the reconstructed first feature map. The inverse scale factor corresponds to the scale factor one by one.

[0035] S42, image reconstruction: passing the reconstructed first feature map through a convolutional neural network to obtain a reconstructed image;

[0036] S43, residual compensation: adding the residual feature map to the decoded second feature map of the reference frame to obtain the second feature map reconstructed by the predicted frame.

[0037] According to a second aspect of the present invention, there is provided a video compression system, comprising:

[0038] Variable bit rate intra-frame encoding and decoding module: used to compress the reference frame image to obtain the feature map of the reference frame image, the reference frame image bit stream and the reconstructed reference frame image;

[0039] Variable bit rate inter-frame coding module: used to compress and transform the predicted frame image to obtain the feature map residual to be encoded.

[0040] Inter-frame residual entropy coding module: used to perform context modeling and compression on the feature map residual to be encoded to obtain the bit stream of the feature map residual.

[0041] Variable bit rate inter-frame decoding module: used to decode the bit stream of the feature map residual, and further obtain the predicted frame reconstructed feature map and the reconstructed predicted frame image.

[0042] According to a third aspect of the present invention, there is provided a variable bit rate video compression device, which compresses video using the above method.

[0043] According to a fourth aspect of the present invention, there is provided a terminal, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor can be used to execute the method described, or, to run the system described, or, to run the device described when executing the program.

[0044] According to a fifth aspect of the present invention, a computer-readable storage medium stores a computer program thereon, characterized in that when the program is executed by a processor, it can be used to execute the method described, or to run the system described, or to run the device described.

[0045] Compared with the prior art, the present invention has the following beneficial effects:

[0046] The present invention uses a scale factor to scale the feature map, thereby obtaining feature maps corresponding to different bit rate qualities;

[0047] The present invention effectively reduces the inter-frame coding overhead through the time domain context prediction module, and realizes variable bit rate coding by controlling the quantization step size through the scale factor. The implicit representation that has been encoded in the previous frame is used to provide additional time domain context prior information for the current frame. Under the premise of using a single model, the video compression performance exhibited by the model optimized for different bit rate points is comparable to that of the current mainstream end-to-end method, and has extremely strong practical application value. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0049] Figure 1 The present invention is a flow chart of a variable bit rate video compression method according to an embodiment of the present invention.

[0050] Figure 2The present invention is a flow chart of a variable bit rate intra-frame encoding and decoding subsystem in a variable bit rate video compression system according to an embodiment of the present invention.

[0051] Figure 3 The present invention is a flow chart of a variable bit rate inter-frame coding subsystem in a variable bit rate video compression system according to an embodiment of the present invention.

[0052] Figure 4 The present invention is a flowchart of an inter-frame residual entropy coding subsystem in a variable bit rate video compression system according to an embodiment of the present invention.

[0053] Figure 5 Flow chart of a variable bit rate inter-frame decoding subsystem in a variable bit rate video compression system according to an embodiment of the present invention

[0054] Figure 6 The present invention is a flowchart of a context modeling module in a variable bit rate video compression system according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0056] like Figure 1 As shown, the present invention provides a flow chart of a variable bit rate video compression method according to an embodiment, including:

[0057] S1, variable bit rate intra-frame encoding and decoding: at the encoding end, the reference frame image is passed through a convolutional neural network to obtain a first feature map of the reference frame, and the second feature map of the reference frame is obtained by scaling and quantizing the feature map; it is then losslessly entropy encoded and transmitted; at the decoding end, the second feature map of the reference frame is inversely scaled and then reconstructed through a convolutional neural network to obtain a reference frame;

[0058] S2, variable bit rate inter-frame coding: At the encoding end, the predicted frame image is passed through a convolutional neural network to obtain the first feature map of the predicted frame, and the second feature map of the predicted frame is obtained by scaling and quantizing the feature map; the second feature map of the predicted frame is subtracted from the second feature map of the reference frame to obtain the feature map residual.

[0059] S3, inter-frame residual entropy coding: using the second feature map of the reference frame as prior information, the feature map residual is losslessly entropy coded and transmitted to the decoding end;

[0060] S4, variable bit rate inter-frame decoding: At the decoding end, the feature map residual is added to the second feature map of the reference frame and inversely scaled, and then the reconstructed prediction frame is obtained through a convolutional neural network.

[0061] Based on the above-mentioned embodiment, the present invention provides a first preferred embodiment. It uses a variable bit rate video compression method provided by the present invention and combines the Vimeo data set to train the video compression system. Specifically, it includes:

[0062] S101, feature map extraction: crop a single frame image in the Vimeo dataset into a 256×256 image, use two adjacent frames as a reference frame and a prediction frame respectively, and input them into a pre-established intra-frame image coding convolutional neural network to obtain a multi-channel feature map as the first feature map of the reference frame and the prediction frame.

[0063] S102, feature map scale scaling: multiply the first feature map of the reference frame and the prediction frame by the scale factor s to obtain the second feature map of the reference frame and the prediction frame, and encode the scale factor s to generate a scale factor binary code stream.

[0064] S103, feature-derivative quantization: for the second feature map, simulate quantization by adding uniform noise to generate second feature maps of the reference frame and the predicted frame.

[0065] S104, feature map inverse scaling: multiplying the second feature maps of the reference frame and the predicted frame by the inverse scale factor to obtain the first feature maps reconstructed from the reference frame and the predicted frame;

[0066] S105, image reconstruction: inputting the first feature map reconstructed from the reference frame and the predicted frame into a pre-established intra-frame image decoding convolutional neural network to obtain a reconstructed reference frame and predicted frame image;

[0067] S106, obtaining feature map residual: subtracting the second feature map of the reference frame and the second feature map of the prediction frame to obtain a feature map residual;

[0068] S107, super prior information extraction: after splicing the second feature maps of the reference frame and the predicted frame, super prior information is obtained through a convolutional neural network; wherein, splicing refers to splicing according to channels, that is, if the second feature map of the reference frame and the second feature map of the predicted frame both have N channels, then feature maps of 2N channels are obtained after splicing.

[0069] S108, super-prior derivable quantization: for the super-prior information, simulate quantization by adding uniform noise to generate quantized super-prior information;

[0070] S109, super prior information reconstruction: the quantized super prior information is passed through a convolutional neural network to obtain reconstructed super prior information;

[0071] S110, Spatial Context Modeling: Using the local information in the feature map residual, the spatial context information is obtained through a convolutional neural network;

[0072] S111, temporal context modeling: using the second feature map of the reference frame, through a convolutional neural network, to obtain temporal context information;

[0073] S112, feature map conditional probability modeling: using the reconstructed hyper-prior information, spatial context information and temporal context information, the mean and variance of the Gaussian distribution used for entropy coding are estimated;

[0074] S113, construction entropy estimation: according to the construction distribution, estimate the bit rate L required for encoding the noisy super-prior information BPP-H ; The distribution is constructed to estimate the bit rate required to encode the super prior information. The bit rate required for encoding the super prior information L BPP-H The neural network obtained through training is adaptively mapped to the code table for estimation.

[0075] S114, conditional entropy estimation: Estimate the bit rate L required for feature map residual coding based on the mean and variance of the Gaussian distribution BPP-F ; The feature map residual encoding is achieved by encoding the residual items one by one. For the i-th residual item y i , according to the estimated Gaussian distribution mean μ i and variance Encoding;

[0076] Using the training data set, the back gradient propagation algorithm is used to learn the deep network model; the loss function of the deep network model is defined as L = L BPP-H +L BPP-F .

[0077] The present invention provides a second preferred embodiment, which uses the video encoder in the video compression system obtained through the above training to compress the UVG test set video sequence to obtain a binary code stream of the compressed video sequence. Specifically, it includes:

[0078] S201, extracting feature maps of reference frame and prediction frame: inputting reference frame and prediction frame through convolutional neural network to generate multi-channel feature maps of reference frame and prediction frame as first feature maps;

[0079] S202, scaling the feature maps of the reference frame and the predicted frame: taking the result of multiplying the first feature map obtained in S201 by the scale factor as the second feature map, and outputting a scale factor binary code stream;

[0080] S203, uniform quantization: quantizing the second feature map obtained in S202 into an integer;

[0081] S204, obtaining an inter-frame feature map residual: subtracting the reference frame obtained in S203 from the quantized second feature map of the prediction frame to obtain a feature map residual;

[0082] S205, super prior information extraction: extracting super prior information from the second feature map obtained in S202 through a convolutional neural network;

[0083] S206, super a priori information encoding: quantizing and arithmetic encoding the super a priori information obtained in S205 to obtain a super a priori information binary code stream;

[0084] S207, super a priori information decoding: recovering the super a priori information from the super a priori information binary code stream outputted from S206 by arithmetic decoding;

[0085] S208, super prior information reconstruction: generating reconstructed super prior information from the super prior information obtained in S207 through a convolutional neural network;

[0086] S209, context reconstruction modeling: using the reconstructed super prior information obtained in S208, the spatial context information of the feature map residual in S204 and the temporal context information of the reference frame feature map in S203 are collected to estimate the mean and variance of the Gaussian distribution used to encode the feature map residual;

[0087] S210, feature map encoding: performing arithmetic encoding on the quantized feature map residual according to the coded Gaussian distribution generated in S209 to obtain a feature map residual binary code stream;

[0088] S211, code stream synthesis: merging the scale factor binary code stream in S202, the super prior information binary code stream in S206 and the feature map residual binary code stream in S209 to obtain a compressed prediction frame image binary code stream.

[0089] The present invention provides a third preferred embodiment, which uses the video decoder in the trained video compression system to decompress the binary code stream generated by the UVG test set video sequence to obtain a reconstructed video sequence, specifically including:

[0090] S301: code stream decomposition: decomposing the compressed prediction frame image binary code stream to obtain a scale factor binary code stream;

[0091] S302: super a priori information binary code stream and feature map residual binary code stream;

[0092] S303: Super a priori information decoding: performing arithmetic decoding on the super a priori information binary code stream obtained in S301.

[0093] S304: Super prior information reconstruction: Restore the super prior information from the arithmetic decoding result generated in S302 through a convolutional neural network.

[0094] S305: Context modeling: extract the spatial context information of the progressively recovered feature map residual and the temporal context information of the second feature map decoded by the reference frame, and estimate the mean and variance of the Gaussian distribution required for encoding the second feature map in combination with the super prior information recovered in S303.

[0095] S306: Feature map decoding: Perform arithmetic decoding on the feature map residual binary code stream generated by S301 according to the Gaussian distribution generated by S304.

[0096] S307: Prediction frame residual compensation: adding the feature map residual obtained in S306 to the decoded second feature map of the reference frame to obtain a second feature map reconstructed by the prediction frame;

[0097] S308: feature map inverse scaling: using the scale factor binary code stream restored in S301, the second feature map of the prediction frame obtained in S307 is inversely scaled to obtain a first feature map reconstructed from the prediction frame;

[0098] S309: Prediction frame image reconstruction: Reconstruct the first feature map obtained by inverse scaling in S308 to generate a prediction frame reconstructed image.

[0099] In a first preferred embodiment, the variable bit rate video compression method provided by the present invention is used in combination with the Vimeo data set to train the video compression system, the convolutional neural network for feature map extraction and image reconstruction adopts pre-set image intra-frame coding model parameters, the scale factor s ranges from 0.1 to 1.5, and the Adam algorithm is used to optimize the model, wherein the batch size is set to 8, the initialization learning rate is 1e-4, and is reduced to 5e-5, 1e-5, 5e-6, and 1e-6 at 1200000, 1600000, 1800000, and 1900000 iterations, respectively. A total of 2500000 steps of training are performed, the model converges, and the training is stopped.

[0100] In the second preferred embodiment, the video encoder in the trained video compression system is used to compress the six video sequences in the UVG test set, and the image group size (GOP) is set to 12. By changing the size of the scale factor s, binary streams with different bit rates can be obtained. When the scale factor is set to 0.1, the scale factor binary stream, the super prior information binary stream and the feature Figure 2 The average value of the sum of the three bit rates is 0.0625bpp. When the scale factor is set to 0.2, the average value of the sum of the three bit rates is 0.0874bpp. When the scale factor is set to 0.3, the average value of the sum of the three bit rates is 0.132bpp.

[0101] In a third preferred embodiment, the video decoder in the trained video compression system is used to decompress the binary code stream generated by compressing the 6 video sequences of the UVG test set to obtain a reconstructed video sequence. When the scale factor is set to 0.1, the average peak signal-to-noise ratio (PSNR) of all images of the reconstructed 6 video sequences is 34.32. When the scale factor is set to 0.2, the average peak signal-to-noise ratio is 35.82. When the scale factor is set to 0.3, the average peak signal-to-noise ratio is 36.97.

[0102] In addition, compared with training multiple codec network models at different bitrates to achieve variable bitrates, the variable bitrate video compression method provided by the present invention for scaling feature maps based on scale factors also greatly reduces the training time and the total number of model parameters, and only a single model is needed to achieve video coding and decoding at any bitrate. Therefore, different scale factors can be set according to actual requirements to balance the binary code stream size and the average peak signal-to-noise ratio.

[0103] According to the same concept of the present invention, in another embodiment of the present invention, a variable bit rate video compression system is provided, comprising:

[0104] S100, variable bit rate intra-frame coding and decoding subsystem: At the encoding end, the reference frame image is passed through a convolutional neural network to obtain the first feature map of the reference frame, and the quantized second feature map of the reference frame is obtained by scaling the feature map and quantizing it. After lossless entropy coding and transmission. At the decoding end, the quantized second feature map of the reference frame is inversely scaled and reconstructed through a convolutional neural network. For details, please refer to Figure 2 ,include:

[0105] S110, image feature extraction module: pass the reference frame image through a convolutional neural network to generate a multi-channel feature map of the reference frame image as a first feature map of the reference frame.

[0106] S120, feature map scaling module: taking the result of multiplying the first feature map of the reference frame generated in S110 by the scale factor as the second feature map of the reference frame, and outputting a scale factor binary code stream;

[0107] S130, uniform quantization module: uniformly quantize the second feature map of the reference frame generated in S120 into integers to obtain the second feature map of the reference frame;

[0108] S140, an intra-frame entropy coding module, performing arithmetic coding on the second feature map of the reference frame generated in S130 to obtain a binary code stream of the feature map of the reference frame;

[0109] S150, code stream synthesis module: merge the scale factor binary code stream generated by S120 and the second feature of the reference frame generated by S140 Figure 2The binary code stream of the compressed reference frame image is obtained.

[0110] S160, a code stream decomposition module: decomposes the compressed reference frame image binary code stream to obtain a scale factor binary code stream and a binary code stream of a second feature map of the reference frame.

[0111] S170, feature map decoding module: features of the reference frame generated by S160 Figure 2 The binary code stream is arithmetically decoded to obtain a second feature map reconstructed from the reference frame;

[0112] S180, feature map inverse scaling module: inversely scale the second feature map reconstructed from the reference frame obtained in S170 using the scale factor binary code stream obtained in S160 to obtain a first feature map reconstructed from the reference frame;

[0113] S190, image reconstruction module: reconstruct the first feature map of the reference frame obtained in S180 to generate a reference frame reconstructed image.

[0114] S200, variable bit rate inter-frame coding subsystem: At the encoding end, the predicted true image is passed through a convolutional neural network to obtain the first feature map of the predicted frame, and the quantized second feature map of the predicted frame is obtained by scaling and quantizing the feature map. The second feature map of the predicted frame is subtracted from the second feature map of the reference frame to obtain the feature map residual. For details, please refer to Figure 3 ,include:

[0115] S210, image feature extraction module: This module is exactly the same as S110, and passes the predicted frame image through a convolutional neural network to generate a multi-channel feature map of the predicted frame image as the first feature map of the predicted frame.

[0116] S220, feature map scaling module: This module is identical to S120, and uses the result of multiplying the first feature map of the prediction frame generated by S210 by the scale factor as the second feature map of the prediction frame, and outputs a scale factor binary code stream;

[0117] S230, uniform quantization module: This module is exactly the same as S130, and uniformly quantizes the second feature map of the prediction frame generated in S220 into integers to obtain the second feature map of the prediction frame;

[0118] S240, feature map residual acquisition module: subtract the quantized second feature map of the reference frame generated in S130 from the second feature map of the prediction frame generated in S230 to obtain the feature map residual to be encoded.

[0119] S300, inter-frame residual entropy coding subsystem: using the obtained second feature map of the reference frame as prior information, the feature map residual is losslessly entropy coded and transmitted to the decoding end. For details, please refer to Figure 4 ,include:

[0120] S310, super prior information extraction module: extracting super prior information from the second feature map of the prediction frame generated in S230 through a convolutional neural network;

[0121] S320, a super a priori information encoding module: quantizing and arithmetic encoding the super a priori information extracted in S310 to obtain a super a priori information binary code stream;

[0122] S330, a super a priori information decoding module: performing arithmetic decoding on the super a priori information binary code stream generated in S320;

[0123] S340, super prior information reconstruction module: restore the super prior information from the arithmetic decoding result generated by S330 through a convolutional neural network;

[0124] S350, context modeling module: performs context reorganization modeling on the feature map residual generated by S240, see Figure 6 ,include:

[0125] S351, spatial context information extraction submodule: extracting spatial context information provided by the feature map residual generated in S240;

[0126] S352, temporal context information extraction submodule: extracting temporal context information provided by the second feature map of the reference frame generated in S130;

[0127] S353, entropy coding distribution estimation submodule: using the super prior information obtained in S340 and the context information obtained in S351 and S352 to estimate the mean and variance of the Gaussian distribution required for encoding the feature map residual;

[0128] S360, feature map encoding module: performing arithmetic encoding on the feature map residual generated in S240 according to the coded Gaussian distribution generated in S353 to obtain a feature map residual binary code stream;

[0129] S370, code stream synthesis module: The super prior information binary code stream generated by S320 and the feature map residual binary code stream generated by S360 are used to obtain a compressed prediction frame image binary code stream.

[0130] S400, variable bit rate inter-frame decoding subsystem: At the decoding end, the feature map residual is added to the second feature map of the reference frame and inversely scaled, and then the reconstructed prediction frame is obtained through the convolutional neural network. For details, please refer to Figure 5 ,include:

[0131] S410, a code stream decomposition module: decomposing the compressed prediction frame image binary code stream to obtain a super prior information binary code stream and a feature map residual binary code stream;

[0132] S420, super a priori information decoding module: This module is identical to S330, and performs arithmetic decoding on the super a priori information binary code stream generated by S410;

[0133] S430, super prior information reconstruction module: This module is exactly the same as S340, and recovers the super prior information from the arithmetic decoding result generated in S420 through a convolutional neural network;

[0134] S440, context modeling module: This module is exactly the same as S350, and performs context modeling on the gradually recovered feature map residual. Figure 6 ,include:

[0135] S441, spatial context information extraction submodule: This module is exactly the same as S351, extracting the spatial context information provided by the progressively recovered feature map residual;

[0136] S442, temporal context information extraction submodule: This module is identical to S352, and extracts temporal context information provided by the second feature map of the reference frame;

[0137] S443, entropy coding distribution estimation submodule: This module is exactly the same as S353, and uses the super prior information obtained in S430 and the context information obtained in S441 and S442 to estimate the mean and variance of the Gaussian distribution required for encoding the feature map residual;

[0138] S450, feature map decoding module: according to the Gaussian distribution generated by S443, the feature map generated by S410 is decoded Figure 2 Perform arithmetic decoding on the binary code stream;

[0139] S460, feature map residual compensation module: add the feature map residual obtained by decoding in S450 to the second feature map of the reference frame decoded in S170 to obtain a second feature map reconstructed by the predicted frame;

[0140] S470, a feature map inverse scaling module: inversely scaling the second feature map reconstructed from the prediction frame obtained in S460 using the scale factor binary code stream obtained in S160 to obtain a first feature map reconstructed from the prediction frame;

[0141] S480, prediction frame image reconstruction module: reconstruct the first feature map obtained in S470 to generate a prediction frame reconstructed image.

[0142] Based on the same inventive concept, in other embodiments of the present invention, a video compression device is also provided, which uses the variable bit rate video compression method in the above embodiment to perform video compression.

[0143] Based on the same inventive concept, in other embodiments of the present invention, a terminal is also provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, it can be used to execute the method in the above-mentioned embodiment, or, to run the system in the above-mentioned embodiment, or, to run the device in the above-mentioned embodiment.

[0144] Based on the same inventive concept, in other embodiments of the present invention, a computer-readable storage medium is also provided, on which a computer program is stored. When the program is executed by a processor, it can be used to execute the method in the above embodiment, or to run the system in the above embodiment, or to run the device in the above embodiment.

[0145] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A variable bit rate video compression method, characterized in that: include: Variable bit rate intra-frame coding and decoding: At the encoding end, the reference frame image is passed through a convolutional neural network to obtain a first feature map of the reference frame, and the second feature map of the reference frame is obtained by scaling and quantizing the feature map; it is then losslessly entropy encoded and transmitted; at the decoding end, the second feature map of the reference frame is inversely scaled and then reconstructed through a convolutional neural network to obtain a reference frame; Variable bit rate inter-frame coding: At the encoding end, the predicted frame image is passed through a convolutional neural network to obtain a first feature map of the predicted frame, and the second feature map of the predicted frame is obtained by scaling and quantizing the feature map; the second feature map of the predicted frame is subtracted from the second feature map of the reference frame to obtain a feature map residual; Inter-frame residual entropy coding: Based on the second feature map of the reference frame, the feature map residual is losslessly entropy coded and transmitted to the decoding end; Variable bit rate inter-frame decoding: At the decoding end, the feature map residual is added to the second feature map of the reference frame and inversely scaled, and then a reconstructed prediction frame is obtained through a convolutional neural network; The inter-frame residual entropy coding comprises: After splicing the second feature map of the reference frame and the second feature map of the prediction frame, super prior information is obtained through a convolutional neural network; For the super prior information, quantization is simulated by adding uniform noise to generate quantized super prior information; The quantized super-prior information is passed through a convolutional neural network to obtain reconstructed super-prior information; Modeling is performed according to the second feature map of the reference frame and the feature map residual to obtain context information for encoding the feature residual; According to the constructed distribution, the code rate L required for quantized super-prior information encoding is estimated BPP-H ; Construct a distribution to estimate the bit rate required to encode the super prior information. The bit rate required for encoding the super prior information is L BPP-H The neural network obtained through training is adaptively mapped to the code table for estimation; According to the mean and variance of the Gaussian distribution, the bit rate L required for the residual coding of the feature map is estimated. BPP-F ; The feature map residual encoding is achieved by encoding the residual items one by one. For the i-th residual item y i , according to the estimated Gaussian distribution mean μ i and variance σ i 2 Encoding; The modeling based on the second feature map of the reference frame and the feature map residual to obtain context information for encoding the feature residual includes: Spatial context information extraction: Extract the context information of the feature residual between the reference frame and the predicted frame in the spatial dimension; Temporal context information extraction: Using the reference frame feature map, extract context information in the temporal dimension; Entropy coding distribution estimation: The context information and the super prior information are integrated to estimate the mean and variance of the Gaussian distribution used for coding, and output them to arithmetic coding.

2. A variable bit rate video compression method according to claim 1, characterized in that: The variable bit rate intra-frame encoding and decoding includes: The reference frame image is passed through a convolutional neural network to extract the first feature map of the reference frame; Multiplying the first feature map of the reference frame by a scale factor to obtain a second feature map of the reference frame; For the second feature map of the reference frame, simulate quantization by adding uniform noise to generate a quantized second feature map of the reference frame; quantizing the second feature map of the reference frame, and encoding and transmitting the same using lossless entropy coding; At the decoding end, multiplying the reference frame quantized second feature map by the inverse scale factor to obtain a reconstructed reference frame first feature map; The reconstructed reference frame first feature map is passed through a convolutional neural network to obtain a reconstructed reference frame image.

3. A variable bit rate video compression method according to claim 1, characterized in that: The variable bit rate inter-frame coding comprises: The predicted frame image is passed through a convolutional neural network to extract the first feature map of the predicted frame; Multiplying the first feature map of the prediction frame by a scale factor to obtain a second feature map of the prediction frame; For the second feature map of the prediction frame, quantization is simulated by adding uniform noise to generate a quantized second feature map of the prediction frame; The second feature map of the reference frame is subtracted from the second feature map of the prediction frame to obtain a feature map residual to be encoded.

4. The variable bit rate video compression method according to claim 1, characterized in that: The variable bit rate inter-frame decoding includes: At the decoding end, the feature map residual is added to the decoded second feature map of the reference frame to obtain a second feature map reconstructed by the predicted frame; Multiplying the second feature map reconstructed from the prediction frame by the inverse scale factor to obtain a reconstructed first feature map of the prediction frame; The reconstructed prediction frame first feature map is passed through a convolutional neural network to obtain a reconstructed prediction frame image.

5. A variable bit rate video compression system, used to execute the variable bit rate video compression method according to any one of claims 1 to 4, characterized in that: include: Variable bit rate intra-frame encoding and decoding module: At the encoding end, the reference frame image is passed through a convolutional neural network to obtain the first feature map of the reference frame, and the quantized second feature map of the reference frame is obtained by scaling and quantizing the feature map; after lossless entropy coding and transmission; At the decoding end, after inverse scaling of the quantized second feature map of the reference frame, a reconstructed reference frame is obtained through a convolutional neural network; Variable bit rate inter-frame coding module: At the encoding end, the predicted frame image is passed through a convolutional neural network to obtain the first feature map of the predicted frame, and the quantized second feature map of the predicted frame is obtained by scaling and quantizing the feature map. Subtracting the quantized second feature map of the prediction frame from the quantized second feature map of the reference frame to obtain a feature map residual; Inter-frame residual entropy coding module: using the obtained second feature map of the reference frame as prior information, performing lossless entropy coding on the feature map residual and transmitting it to the decoding end; Variable bit rate inter-frame decoding module: At the decoding end, the feature map residual is added to the second feature map of the reference frame obtained and inversely scaled, and then a reconstructed prediction frame is obtained through a convolutional neural network.

6. A video compression device, characterized in that: Video compression is performed using a variable bit rate video compression method as described in any one of claims 1 to 4.

7. A terminal comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, it can be used to perform the method according to any one of claims 1 to 4.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it can be used to execute the method according to any one of claims 1 to 4.

Images