Binocular video compression method based on parallax compensation and deep learning

Abstract

The invention provides a binocular video compression method based on parallax compensation and deep learning, and the method comprises the steps: firstly obtaining training data, carrying out the preprocessing, and then constructing a binocular video compression network; the binocular video compression network comprises a motion estimation network, a motion information compression network, a motion compensation network, a binocular video frame information compression network and a frame reconstruction network, then carrying out supervised training on the binocular video compression network, and carrying out video compression processing by using the trained binocular video compression network; according to the method, a binocular feature interaction model and a binocular entropy model are arranged in a motion information compression network and a binocular frame information compression network, the binocular feature interaction model aligns binocular visual angle features by using a mixed parallax compensation mechanism in the encoding and decoding process, and the binocular entropy model performs accurate probability distribution modeling through autoregression probability estimation. According to the method, the compression performance can be remarkably improved while the reconstruction quality is guaranteed, and the method is suitable for scenes such as virtual reality and automatic driving needing efficient binocular video processing.

Description

Technical Field

[0001] The present invention relates to the technical field of binocular video compression, and in particular to a binocular video compression method based on parallax compensation and deep learning. Background Art

[0002] Binocular video compression is an important emerging research direction in the field of video signal processing. In recent years, the rapid popularity of intelligent devices such as virtual reality devices and autonomous driving has generated a large demand for the transmission and storage of binocular video data. This requires binocular video compression technology to not only compress binocular videos at a higher rate to meet the storage and transmission needs of high-resolution images, but also to ensure the fidelity of reconstructed binocular videos to provide users with high-quality reconstructed videos.

[0003] A major challenge in binocular video compression technology is to reduce cross-view redundancy between the left and right views in binocular video through parallax compensation. By estimating the parallax between the two views, the two binocular views are aligned with each other. Parallax compensation can use information shared across views to complete the details in both views. Through information reuse, it can enhance the quality of binocular video reconstruction and improve compression efficiency. Traditional binocular video compression technology uses hand-designed parallax compensation mechanisms to enhance binocular video compression performance, but the compression effect is limited by the hand-designed algorithm and structure.

[0004] In order to achieve better disparity compensation, disparity compensation methods based on deep neural networks have received extensive research and attention. Existing disparity compensation methods based on deep neural networks are mainly divided into two categories. The first category uses explicit disparity information to align cross-perspective features, such as horizontally translating features under different disparity values, extracting cross-perspective information from the features obtained after translation, and using the predicted disparity value between two perspectives to align features of the two perspectives, as well as using transformation matrices to align features of the two perspectives. Although this type of method reduces the search space by explicitly using disparity information, its feature matching-based characteristics make it difficult to handle occlusion and mismatching, resulting in insufficient robustness of the model results. The second type of method uses a cross-attention mechanism to generate a similarity map to implicitly utilize disparity information. This type of method can use the attention map to dynamically perform global matching on the information of the two perspectives, thereby better handling noise and occlusion. Although this type of method is more flexible, this implicit global matching requires a global similarity map to perform global information matching. Compared with explicitly using disparity information for alignment, this method is usually more difficult to optimize and will incur greater computational overhead, resulting in slower convergence and greater training overhead. Summary of the Invention

[0005] To address the above problems, the present invention proposes a binocular video compression method based on disparity compensation and deep learning. This method combines the advantages of explicit and implicit disparity compensation methods, and uses attention maps to align cross-perspective information under different disparity value translations to achieve more robust cross-perspective information alignment.

[0006] A binocular video compression method based on parallax compensation and deep learning, the method comprising:

[0007] Step S1, obtain training data and crop the artifact parts at the top and bottom of the image;

[0008] Step S2, constructing an end-to-end deep binocular video compression network, the binocular video compression network includes a motion estimation network, a motion information compression network, a motion compensation network, a binocular video frame information compression network and a frame reconstruction network, the motion information compression network includes a motion information encoder, a motion information decoder and an entropy model, the motion information encoder includes four downsampling modules and three binocular feature interaction modules, the motion information decoder includes four upsampling modules and three binocular feature interaction modules, the entropy model includes a binocular entropy model, a binocular feature interaction module and a super priori module, the binocular video frame information compression network includes a binocular frame information encoder, a binocular frame information decoder and an entropy model, the binocular frame information encoder includes four downsampling modules and three binocular feature interaction modules, the binocular frame information decoder includes four upsampling modules and three binocular feature interaction modules; the binocular video frames to be compressed and their respective reference frames estimate motion information through a motion estimation network, the reference frame is a decoded frame at the previous moment, and the motion information is estimated through motion estimation. The motion information compression network reconstructs the motion information after compression and decompression, and the reference frame information is aligned using the reconstructed motion information. The aligned features are passed through the motion compensation network to obtain multi-scale conditional context information. The binocular video frame to be compressed is compressed and reconstructed by the binocular video frame information compression network using the multi-scale conditional context information as a condition, and its output is reconstructed by the frame reconstruction network to restore the binocular video frame. In the codec of the motion information compression network and the binocular video frame information compression network, the intermediate features of the motion information / video frame information of the left and right perspectives in the codec processing process are interactively aligned through the binocular feature interaction model to utilize the aligned cross-perspective information to enhance the features of the two perspectives and reduce cross-perspective redundancy. The potential representations of the motion information / video frame information of the two perspectives extracted by the encoder are interacted through the binocular entropy model, and the cross-perspective information, the multi-scale conditional context information in the time domain and the super-prior information in the spatial domain are combined to perform channel dimension autoregressive entropy modeling to enhance the accuracy of probability distribution estimation;

[0009] Step S3: Perform supervised training on the binocular video compression network to obtain a trained binocular video compression network, and then use the trained binocular video compression network to perform compression processing on the binocular video to be compressed.

[0010] Furthermore, the binocular feature interaction model includes a downsampling module, a disparity compensation module, a feature enhancement module and an upsampling module. The feature enhancement module consists of ordinary 3D convolution, nonlinear activation function and ordinary 2D convolution, and the disparity compensation module includes Vandermonde product and normalization operations.

[0011] Furthermore, the binocular feature interaction model aligns the intermediate features of the two perspectives, including:

[0012] First, the binocular feature interaction model uses the intermediate features K of the two perspectives in the encoding and decoding process of frame information or motion information L With K R As input, the feature is obtained after the resolution is reduced by the downsampling module. and and Under parallax values of 1 to D, the feature volume V is formed by two viewing angle features under different parallax values. L and V R :

[0013]

[0014] Where c represents the number of channels, h represents the height of the feature, w represents the width of the feature, and the disparity d∈(1,…,D);

[0015] Then the two view feature bodies are normalized by the Vandermonde product to obtain the attention score F under different disparities of the two views * , expressed as: F * =Softplus(Tanh(V L ⊙V R )), where ⊙ represents the Vandermonde product, and Tanh and Softplus are both nonlinear activation functions;

[0016] Then the attention score map F * Then, through the Vandermonde product, we can get the characteristic volume V L and V R Filter cross-view related features, F * Places with larger values, that is, places with stronger cross-view correlation, will be retained while weaker ones will be suppressed, and the feature enhancement module f refine Enhance the filtered features to obtain the final aligned cross-view features and Expressed as:

[0017]

[0018] at last, and After being upsampled by the upsampling module, it is restored to the original input size and added back to the input K L With K R Middle pair of Kings L With K R Enhance and get the enhanced K L With K R ,Right now and and Continue to complete the encoding and decoding process in the compression network.

[0019] Furthermore, the downsampling module is composed of downsampling convolution, nonlinear activation function and ordinary convolution, and the upsampling module is composed of 2D upsampling convolution, nonlinear activation function and ordinary 2D convolution.

[0020] Furthermore, the binocular entropy model includes a spatiotemporal prior information fusion network and a probability distribution estimation network. M∈[L,R], the spatiotemporal prior information fusion network includes a conditional context fusion module, a super prior module and a fusion module, the probability distribution estimation network includes a disparity alignment module, a splicing module and a prior fusion module, the disparity alignment module includes feature translation, Vandermonde product, normalization and convolution modules, the probability distribution estimation network fuses the spatiotemporal prior fusion information φ M , decoded information of the current view and cross-view prior information The three are fused through the prior fusion module, alternating the potential representations y of the two perspectives. L and y R Probability distribution estimation is performed on slices along the channel dimension.

[0021] Furthermore, the spatiotemporal prior fusion information φ M Multi-scale conditional context information And the spatial information extracted by the super prior module Obtained through the fusion of spatiotemporal prior information network, the current perspective has decoded information and cross-view prior information It is generated by quantizing the estimated probability distribution parameters (μ, σ) during the autoregressive probability distribution estimation process, where the conditional context information The scale relationship is

[0022] Furthermore, the conditional context fusion module includes four downsampling modules, namely the first downsampling module, the second downsampling module, the third downsampling module and the fourth downsampling module. The conditional context information of the largest scale As input, after the first downsampling module and the smaller scale conditional context information, After splicing, it enters the next downsampling module until all conditional context information is integrated; the super prior module consists of an encoder and a decoder, starting from the potential representation y M The encoder extracts the edge information, and then the decoder recovers the spatial domain information. The encoder consists of two downsampling modules, namely the fifth downsampling module and the sixth downsampling module, and the decoder consists of two upsampling modules, namely the first upsampling module and the second upsampling module. The fusion module takes the splicing of the outputs of the conditional context fusion module and the super prior module as input, and outputs the fused spatiotemporal prior fusion information φ M The first upsampling module and the second upsampling module are both composed of upsampling convolutional networks, the first downsampling module, the second downsampling module, the third downsampling module, the fourth downsampling module, the fifth downsampling module and the sixth downsampling module are all composed of downsampling convolutions, and the fusion module is composed of an ordinary 2D convolutional network.

[0023] Furthermore, the potential representation y L and y R First, it is divided into N slices along the channel, and then the probability estimation and encoding of the two perspectives (2N slices) are performed alternately to obtain the encoded sequence and And generate the encoded feature sequence in the following order: Specifically:

[0024] For the nth slice of the M perspective The probability distribution estimation network combines three different priors to estimate the probability distribution, namely the decoded information of the current perspective Cross-view prior information Spatiotemporal prior fusion information of the current perspective Among them, when M=L, q=n-1, when M=R, q=n, the cross-view prior information is first aligned to the current view through the disparity alignment module to obtain the aligned cross-view features, and then the spatiotemporal prior fusion information of the current view is Aligned cross-view features and decoded information of the current view The information is spliced through the splicing module, and the spliced information is quantized by the prior fusion module. The probability distribution parameters of in, Used to Quantized to Expressed as:

[0025]

[0026] represents rounding quantization, For arithmetic coding, Encoded as a bitstream.

[0027] The prior fusion module is a neural network. After the three prior information are input into the neural network, the probability distribution parameters are output.

[0028] Furthermore, the parallax alignment module aligns the cross-view prior information, including: By fusing the spatiotemporal prior information with the current view φ M First translate along the width dimension at different parallax values to obtain the feature volume and and The Vandermonde product is calculated and normalized by Tanh and Softplus nonlinear activation functions to obtain the attention score, and then The Vandermonde product is performed with the attention score to weight the features under different disparities, and then the disparity dimension is removed by the 3D convolutional network. Finally, the final fused aligned cross-view feature s is obtained by the 2D ordinary convolutional network. M .

[0029] Furthermore, during the training process of the binocular video compression network, the rate-distortion loss function is used as the loss function of the deep binocular video compression method:

[0030]

[0031] Where t represents the frame number, and Represent the potential representation of the encoding of the view M respectively and the spatial information generated by the super prior module The size of the bit stream, Represents the input video frame and the reconstructed video frame λ is a hyperparameter used to control the trade-off between the encoded bitstream size and the distortion of the reconstructed frame.

[0032] The present invention proposes a binocular video compression method based on disparity compensation and deep learning. The two key modules of the binocular feature interaction model and the binocular entropy model in the binocular video compression network combine the advantages of explicit and implicit disparity compensation methods, and use the attention map to align the cross-view information under different disparity value translations to achieve more robust cross-view information alignment, and effectively remove cross-view redundancy and improve the compression rate. The binocular feature interaction model is used in the codec of the motion information compression network and the binocular frame information compression network to process the intermediate features of the two perspectives in feature extraction and reconstruction, realize the multiplexing of the two perspective information, enhance the information multiplexing capability, and thus improve the reconstruction quality. The intermediate features of the two perspectives are aligned to each other's perspective using the proposed parallax compensation mechanism based on explicit feature translation and attention map weighting. The original input features of the two perspectives are enhanced after being processed by a deep neural network to fuse the aligned cross-perspective features. In the binocular entropy model in the temporal motion information compression network and the binocular frame information compression entropy network of the binocular video compression network, the potential representation obtained by the encoder of the input of the two perspectives is divided into multiple sub-slices along the channel dimension, and then the sub-slices of the two perspectives are alternately autoregressively estimated to optimize the entropy coding efficiency. In the process of estimating the probability distribution of a slice of perspective one, the temporal and spatial characteristics of perspective one are also analyzed. The domain prior information and the decoded information of view one and view two are used as prior to estimate the probability distribution of the current slice. The cross-view features of view two are aligned to the current view through a parallax compensation mechanism based on explicit feature translation and attention map weighting to make a more accurate probability distribution estimate. The probability distribution of the slice of view two is estimated in the same way, that is, the binocular entropy model divides the potential representation of the two views into multiple slices along the channel dimension, and alternately performs autoregressive probability distribution estimation on the slices of the two views. The probability distribution estimation of each feature uses the decoded current feature, cross-view feature and the time and space information of the current view as prior, among which the decoded features across views are passed through The disparity compensation mechanism based on explicit feature translation and attention map weighting is aligned to the current perspective to achieve more accurate probability estimation. At the same time, the scheme of the present invention uses a rate-distortion loss function for optimization. By using the module to eliminate cross-perspective redundant information in the process of feature extraction, reconstruction and entropy coding, the present invention constructs a binocular video compression method based on disparity compensation and deep learning, which can more fully remove the cross-perspective redundancy between the two perspectives in the encoding and decoding process and entropy modeling process of binocular video compression, and can achieve a higher compression rate with better reconstruction effect. It has the characteristics of high efficiency and high performance and is suitable for scenarios such as virtual reality and autonomous driving that require efficient binocular video processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0034] Figure 1 This is a general framework diagram of a binocular video compression method based on parallax compensation and deep learning provided by an embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of the structure of a motion information compression network provided by an embodiment of the present invention;

[0036] Figure 3 Schematic diagram of the structure of a binocular video frame information compression network provided by an embodiment of the present invention;

[0037] Figure 4 Schematic diagram of the structure of the binocular feature interaction model provided by an embodiment of the present invention;

[0038] Figure 5 Schematic diagram of the structure of the binocular entropy model provided by an embodiment of the present invention;

[0039] Figure 6 The probability distribution estimation network in the binocular entropy model provided by the embodiment of the present invention is Schematic diagram of the structure;

[0040] Figure 7 3. It is a structural diagram of the spatiotemporal prior information fusion network in the binocular entropy model provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0042] Figure 1 This is a general framework diagram of a binocular video compression method based on parallax compensation and deep learning provided by an embodiment of the present invention, showing the network processing flow for a single frame. The specific steps of the method are as follows:

[0043] Step S1: Acquire training data and crop artifacts from the top and bottom of the image. In this embodiment, the training dataset for deep binocular video compression uses the monocular video dataset Vimeo90K for pre-training and the binocular autonomous driving dataset Cityscape Raw Sequences for fine-tuning. The Vimeo90K dataset contains 90,000 monocular video sequences of 7 frames in length, which are randomly cropped to a resolution of 256×256 during network pre-training. The Cityscape Raw Sequences dataset contains 5,000 binocular video pairs of 30 frames in length, which are randomly cropped to a resolution of 256×384 during network fine-tuning.

[0044] Step S2, constructing an end-to-end deep binocular video compression network, the binocular video compression network includes a motion estimation network, a motion information compression network, a motion compensation network, a binocular video frame information compression network and a frame reconstruction network, the motion information compression network includes a motion information encoder, a motion information decoder and an entropy model, such as Figure 2 As shown in Figure 1, the motion information encoder includes four downsampling modules and three binocular feature interaction modules, the motion information decoder includes four upsampling modules and three binocular feature interaction modules, the entropy model includes a binocular entropy model, a binocular feature interaction module and a super priori module, and the binocular video frame information compression network includes a binocular frame information encoder, a binocular frame information decoder and an entropy model, as shown in Figure 1. Figure 3 As shown, the binocular frame information encoder includes four downsampling modules and three binocular feature interaction modules, and the binocular frame information decoder includes four upsampling modules and three binocular feature interaction modules; the binocular video frame to be compressed and the respective reference frames estimate motion information through a motion estimation network, the reference frame is a decoded frame at the previous moment, the motion information is compressed and decompressed by the motion information compression network to reconstruct the motion information, the reference frame information is aligned using the reconstructed motion information, the aligned features are passed through a motion compensation network to obtain multi-scale conditional context information, the binocular video frame to be compressed is compressed and reconstructed by the binocular video frame information compression network with the multi-scale conditional context information as a condition, and its output is The binocular video frames are reconstructed and restored through the frame reconstruction network. In the codec of the motion information compression network and the binocular video frame information compression network, the intermediate features of the motion information / video frame information of the left and right views during the codec processing are interactively aligned through the binocular feature interaction model, so as to enhance the features of the two views by using the aligned cross-view information and reduce cross-view redundancy. The potential representations of the motion information / video frame information of the two views extracted by the encoder interact through the binocular entropy model, combining the cross-view information, multi-scale conditional context information in the time domain and the super-prior information in the spatial domain to perform channel dimension autoregressive entropy modeling to enhance the accuracy of probability distribution estimation;

[0045] Given the binocular video frame to be compressed at the current time t Motion estimation network combined with reference binocular video frames The information is first estimated and and the motion information between their respective reference frames The motion information compression network then compresses the motion information and reconstructs it into The reconstructed motion information is used in the motion compensation network to align the reference information at time t-1 And generate multi-scale context information After that, the binocular frame information compression network is used to As a condition for the binocular video frame Perform compression reconstruction and finally obtain the reconstructed binocular video frame And generate the reference information features of the next frame Generated during rebuild The high-dimensional features contain rich information about the current binocular video frame.

[0046] Step S3: Perform supervised training on the binocular video compression network to obtain a trained binocular video compression network, and then use the trained binocular video compression network to perform compression processing on the binocular video to be compressed.

[0047] This method mainly enhances the binocular video compression effect by improving the parallax compensation of the two perspectives in the two compression networks of motion information compression network and binocular frame information compression network. Both compression networks contain two parts: codec and entropy model. The codec is responsible for downsampling the input information (frame information or motion information) of the two perspectives to obtain the potential representation. The entropy model is responsible for quantizing and entropy encoding the obtained latent representation. To leverage cross-view information to assist in the compression and reconstruction of information from both views during encoding and decoding, a binocular feature interaction model is proposed. This model aligns features across views during downsampling or upsampling, and uses the aligned cross-view features to enhance features from both views. To efficiently combine cross-view information with spatiotemporal information during entropy modeling, a binocular entropy model is proposed for more accurate entropy modeling to improve compression. The binocular entropy model first aligns cross-view prior information. This aligned cross-view information is then combined with multi-scale conditional contextual information in the temporal domain and super-prior information in the spatial domain. These three types of information are then used in a probability distribution estimation network to achieve more accurate probability distribution estimation. The encoder of the binocular frame information compression network and the decoder of the motion information compression network both consist of four downsampling modules, and the decoder consists of four upsampling modules. A binocular feature interaction model is implemented after each downsampling module and before each upsampling module to enable information interaction between the two views, eliminating cross-view redundancy during encoding and decoding. The entropy model of the motion information compression network consists of a binocular feature interaction module, a super-prior module and a binocular entropy model. On this basis, the binocular entropy model of the binocular frame information compression network also introduces super-prior information and fuses it with the multi-scale conditional information output by the motion compensation network. The binocular entropy model can combine cross-view information to estimate the final probability distribution.

[0048] like Figure 4 As shown in the figure, the binocular feature interaction model includes a downsampling module, a disparity compensation module, a feature enhancement module and an upsampling module. The feature enhancement module consists of ordinary 3D convolution, nonlinear activation function and ordinary 2D convolution. The disparity compensation module includes Vandermonde product and normalization operation. The binocular feature interaction model enhances the feature representation in the feature extraction and feature reconstruction process by aligning cross-view features. The feature pairs output by the downsampling module / upsampling module in the encoding and decoding process of the two compression networks are {K L ,K R}, its shape is C×H×W, H is the height of the feature, W is the width, and C is the number of channels. In the binocular feature interaction model, K is first L With K R Downsampling to reduce the resolution and To reduce the computational complexity of subsequent operations in this module, and Translate along the width dimension (W), from the disparity value of 1 to the maximum disparity value D, and obtain the feature volume V of two perspectives under D different disparity values L and V R , the shape is D×C×H×W:

[0049]

[0050] d∈(1,…,D) represents the translation when the parallax is d. The maximum parallax value D is determined according to the actual scene. Then, a similarity graph F is generated through V L and V R The dot product between them gives:

[0051] F=V L ⊙V R ;

[0052] The similarity graph is normalized by Softplus and Tanh functions, which can suppress irrelevant features and enhance the relevant features of the two perspectives. The normalized similarity graph F * It is considered as a similarity measure of features between two views at different parallaxes, and can find the most relevant features of two views at different parallaxes:

[0053] F * =Softplus(Tanh(V L ⊙V R ))

[0054] Afterwards, the similarity graph is used as an attention score to find V through dot product and feature enhancement module. L and V R The most effective cross-view features and

[0055]

[0056] and Finally, after the upsampling module restores the resolution to the input size, it is added back to K L With K 1 , used to enhance K L With K R , and get the enhanced output and

[0057] The downsampling module consists of downsampling convolution, nonlinear activation function and ordinary convolution, and the upsampling module consists of 2D upsampling convolution, nonlinear activation function and ordinary 2D convolution.

[0058] like Figure 5 As shown, the binocular entropy model includes a spatiotemporal prior information fusion network and a probability distribution estimation network. like Figure 6 As shown in , the probability distribution estimation network includes a disparity alignment module, a splicing module, and a priori fusion module. The disparity alignment module includes feature translation, Vandermonde product, normalization, and convolution modules, as shown in Figure 7As shown, the spatiotemporal prior information fusion network includes a conditional context fusion module, a super prior module and a fusion module. The conditional context fusion module contains four downsampling modules, namely the first downsampling module, the second downsampling module, the third downsampling module and the fourth downsampling module. The maximum scale context information As input, it passes through the first downsampling module and is spliced with the smaller scale context information. After splicing, it enters the next downsampling module until all conditional context information is fused; the super prior module consists of an encoder and a decoder, first from the potential representation y M The encoder extracts the edge information, and then the decoder recovers the spatial domain information. The encoder consists of two downsampling modules, namely the fifth downsampling module and the sixth downsampling module, and the decoder consists of two upsampling modules, namely the first upsampling module and the second upsampling module. The fusion module takes the splicing of the outputs of the conditional context fusion module and the super prior module as input, and outputs the fused spatiotemporal prior fusion information φ M The first upsampling module and the second upsampling module are both composed of upsampling convolutional networks, the first downsampling module, the second downsampling module, the third downsampling module, the fourth downsampling module, the fifth downsampling module and the sixth downsampling module are all composed of downsampling convolutions, and the fusion module is composed of an ordinary 2D convolutional network.

[0059] The goal of the binocular entropy model is to represent the potential representation y of the two-view video frames obtained by the encoder L and y R Carry out entropy modeling and estimate its probability distribution to perform arithmetic coding on it. A good entropy model can accurately estimate the probability distribution of the potential representation, thereby reducing the length of the encoded bit stream. In order to achieve accurate probability distribution estimation, the present invention proposes a cross-view autoregressive entropy model, which autoregressively L and y R The slices are used for probability distribution estimation and quantization coding. First, the potential representation y L and y R Divide it into N slices along the channel dimension, and then alternately perform probability estimation and encoding on the two perspectives, a total of 2N slices, to obtain the encoded sequence and Generate encoded features in the following order:

[0060] In this process, the probability distribution estimation of each slice uses the previously encoded current view information and cross-view prior information as well as the spatiotemporal prior fusion information of the current view as the prior. For the nth slice of M views, The probability distribution estimation network combines three different priors to estimate the probability distribution, namely the decoded information of the current perspective Cross-view prior information Spatiotemporal prior fusion information of the current perspective Here, when M=L, q=n-1, and when M=R, q=n.

[0061] In the probability distribution estimation network In the above example, due to the cross-view prior information There is a parallax with the current perspective, so the spatiotemporal prior fusion information of the current perspective is used first As the anchor point of the current perspective, cross-perspective prior information By fusing the spatiotemporal prior information with the current view φ M First translate along the width dimension at different parallax values to obtain the feature volume and and The Vandermonde product is calculated and normalized by Tanh and Softplus nonlinear activation functions to obtain the attention score, and then The Vandermonde product is performed with the attention score to weight the features under different parallaxes, and then the parallax dimension is removed by the 3D convolutional network. Finally, the final fused aligned cross-view features are obtained by the 2D ordinary convolutional network.

[0062] Then, the spatiotemporal prior information of the current perspective is fused through the splicing module Aligned cross-view information Decoded information about the current view After splicing, the quantized features are generated through the prior fusion module The probability distribution parameters of in, Used to Quantized to Expressed as:

[0063]

[0064] represents rounding quantization, For arithmetic coding, Encoded as a bitstream.

[0065] The prior fusion module is a neural network. After the three prior information are input into the neural network, the probability distribution parameters are output.

[0066] The training strategy of the binocular video compression network is as follows: Since the binocular video compression method requires considering both the compression rate and the reconstruction quality of the binocular video, in order to achieve multi-objective optimization, the rate-distortion loss function is used as the loss function of the binocular video compression network:

[0067]

[0068] Where t represents the frame number, and Represent the potential representation of the encoding of the view M respectively and the spatial information generated by the super prior module The size of the bit stream, Represents the input video frame and the reconstructed video frame λ is a hyperparameter used to control the trade-off between the encoded bitstream size and the distortion of the reconstructed frame.

[0069] In this embodiment, the network is trained using the Adam optimizer for a total of 2 million iterations, with 1.5 million iterations pre-trained on the monocular dataset and 500,000 iterations fine-tuned on the binocular dataset.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the above embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A binocular video compression method based on parallax compensation and deep learning, characterized in that: The method comprises: Step S1, obtain training data and crop the artifact parts at the top and bottom of the image; Step S2, constructing an end-to-end deep binocular video compression network, the binocular video compression network includes a motion estimation network, a motion information compression network, a motion compensation network, a binocular video frame information compression network and a frame reconstruction network, the motion information compression network includes a motion information encoder, a motion information decoder and an entropy model, the motion information encoder includes four downsampling modules and three binocular feature interaction modules, the motion information decoder includes four upsampling modules and three binocular feature interaction modules, the entropy model includes a binocular entropy model, a binocular feature interaction module and a super priori module, the binocular video frame information compression network includes a binocular frame information encoder, a binocular frame information decoder and an entropy model, the binocular frame information encoder includes four downsampling modules and three binocular feature interaction modules, the binocular frame information decoder includes four upsampling modules and three binocular feature interaction modules; the binocular video frames to be compressed and their respective reference frames estimate motion information through a motion estimation network, the reference frame is a decoded frame at the previous moment, and the motion information is estimated through motion estimation. The motion information compression network reconstructs the motion information after compression and decompression, and the reference frame information is aligned using the reconstructed motion information. The aligned features are passed through the motion compensation network to obtain multi-scale conditional context information. The binocular video frame to be compressed is compressed and reconstructed by the binocular video frame information compression network using the multi-scale conditional context information as a condition, and its output is reconstructed by the frame reconstruction network to restore the binocular video frame. In the codec of the motion information compression network and the binocular video frame information compression network, the intermediate features of the motion information / video frame information of the left and right perspectives in the codec processing process are interactively aligned through the binocular feature interaction model to utilize the aligned cross-perspective information to enhance the features of the two perspectives and reduce cross-perspective redundancy. The potential representations of the motion information / video frame information of the two perspectives extracted by the encoder are interacted through the binocular entropy model, and the cross-perspective information, the multi-scale conditional context information in the time domain and the super-prior information in the spatial domain are combined to perform channel dimension autoregressive entropy modeling to enhance the accuracy of probability distribution estimation; Step S3: Perform supervised training on the binocular video compression network to obtain a trained binocular video compression network, and then use the trained binocular video compression network to perform compression processing on the binocular video to be compressed.

2. The method according to claim 1, characterized in that The binocular feature interaction model includes a downsampling module, a disparity compensation module, a feature enhancement module and an upsampling module. The feature enhancement module consists of ordinary 3D convolution, nonlinear activation function and ordinary 2D convolution. The disparity compensation module includes Vandermonde product and normalization operations.

3. The method according to claim 2, characterized in that The binocular feature interaction model aligns the intermediate features of the two perspectives, including: First, the binocular feature interaction model uses the intermediate features K of the two perspectives in the encoding and decoding process of frame information or motion information L With K R As input, the feature is obtained after the resolution is reduced by the downsampling module. and and Under parallax values of 1 to D, the feature volume V is formed by two viewing angle features under different parallax values. L and V R : Where c represents the number of channels, h represents the height of the feature, w represents the width of the feature, and the disparity d∈(1,…,D); Then the two view feature bodies are normalized by the Vandermonde product to obtain the attention score F under different disparities of the two views * , expressed as: F * =Softplus(Tanh(V L ⊙V R )), where ⊙ represents the Vandermonde product, and Tanh and Softplus are both nonlinear activation functions; Then the attention score map F * Then, through the Vandermonde product, we can get the characteristic volume V L and V R Filter cross-view related features, F * Places with larger values, that is, places with stronger cross-view correlation, will be retained while weaker ones will be suppressed, and the feature enhancement module f refine Enhance the filtered features to obtain the final aligned cross-view features and Expressed as: at last, and After being upsampled by the upsampling module, it is restored to the original input size and added back to the input K L With K R Middle pair of Kings L With K R Enhance and get the enhanced K L With K R ,Right now and and Continue to complete the encoding and decoding process in the compression network.

4. The method according to claim 2, characterized in that The downsampling module consists of downsampling convolution, nonlinear activation function and ordinary convolution, and the upsampling module consists of 2D upsampling convolution, nonlinear activation function and ordinary 2D convolution.

5. The method according to claim 1, wherein The binocular entropy model includes a spatiotemporal prior information fusion network and a probability distribution estimation network. M∈[L,R], the spatiotemporal prior information fusion network includes a conditional context fusion module, a super prior module and a fusion module, the probability distribution estimation network includes a disparity alignment module, a splicing module and a prior fusion module, the disparity alignment module includes feature translation, Vandermonde product, normalization and convolution modules, the probability distribution estimation network combines the spatiotemporal prior fusion information φ M , decoded information of the current view and cross-view prior information The prior fusion module is used to fuse the potential representations y of the two views alternately. L and y R Probability distribution estimation is performed on slices along the channel dimension.

6. The method according to claim 5, characterized in that The spatiotemporal prior fusion information φ M Multi-scale conditional context information And the spatial information extracted by the super prior module Obtained through the fusion of spatiotemporal prior information network, the current perspective has decoded information and cross-view prior information It is generated by quantifying the estimated probability distribution parameters (μ, σ) during the autoregressive probability distribution estimation process.

7. The method according to claim 6, characterized in that The conditional context fusion module contains four downsampling modules, namely the first downsampling module, the second downsampling module, the third downsampling module and the fourth downsampling module. The conditional context information of the largest scale As input, it passes through the first downsampling module and is spliced with the smaller scale conditional context information. After splicing, it enters the next downsampling module until all the conditional context information is fused. The super prior module consists of an encoder and a decoder. It starts from the potential representation y M The encoder extracts the edge information, and then the decoder recovers the spatial domain information. The encoder consists of two downsampling modules, namely the fifth downsampling module and the sixth downsampling module, and the decoder consists of two upsampling modules, namely the first upsampling module and the second upsampling module. The fusion module takes the splicing of the outputs of the conditional context fusion module and the super prior module as input, and outputs the fused spatiotemporal prior fusion information φ M The first upsampling module and the second upsampling module are both composed of upsampling convolutional networks, the first downsampling module, the second downsampling module, the third downsampling module, the fourth downsampling module, the fifth downsampling module and the sixth downsampling module are all composed of downsampling convolutions, and the fusion module is composed of an ordinary 2D convolutional network.

8. The method according to claim 5, characterized in that Potential representation y L and y R First, it is divided into N slices along the channel, and then the probability estimation and encoding of the two perspectives (2N slices) are performed alternately to obtain the encoded sequence and And generate the encoded feature sequence in the following order: Specifically: For the nth slice of the M perspective The probability distribution estimation network combines three different priors to estimate the probability distribution, namely the decoded information of the current perspective Cross-view prior information Spatiotemporal prior fusion information of the current perspective Among them, when M=L, q=n-1, when M=R, q=n, the cross-view prior information is first aligned to the current view through the disparity alignment module to obtain the aligned cross-view features, and then the spatiotemporal prior fusion information of the current view is Aligned cross-view features and decoded information of the current view The information is spliced through the splicing module, and the spliced information is quantized by the prior fusion module. The probability distribution parameters of in, Used to Quantized to Expressed as: represents rounding quantization, For arithmetic coding, Encoded as a bitstream.

9. The method according to claim 8, characterized in that The parallax alignment module aligns cross-view prior information, including: cross-view prior information By fusing the spatiotemporal prior information with the current view M First translate along the width dimension at different parallax values to obtain the feature volume and and The Vandermonde product is calculated and normalized by Tanh and Softplus nonlinear activation functions to obtain the attention score, and then The Vandermonde product is performed with the attention score to weight the features under different disparities, and then the disparity dimension is removed by the 3D convolutional network. Finally, the final fused aligned cross-view feature s is obtained by the 2D ordinary convolutional network. M .

10. The method according to claim 2, characterized in that During the training process, the rate-distortion loss function is used as the loss function of the deep binocular video compression method: Where t represents the frame number, and Represent the potential representation of the encoding of the view M respectively and the spatial information generated by the super prior module The size of the bit stream, Represents the input video frame and the reconstructed video frame λ is a hyperparameter used to control the trade-off between the encoded bitstream size and the distortion of the reconstructed frame.

Images