Binocular video compression method based on context

Abstract

The invention relates to the technical field of video compression, and discloses a context-based binocular video compression method, which comprises the following steps of: respectively processing frame sequences of a left viewpoint and a right viewpoint frame by frame by adopting two same channels to obtain reconstructed frame sequences of the left viewpoint and the right viewpoint, and combining the reconstructed frame sequences into a compressed binocular video. Each of the two channels comprises a context generation unit and a video reconstruction unit, and the context generation unit receives a current frame, a motion reference frame and a parallax reference frame to obtain final context information; the video reconstruction unit receives the current frame and the final context information of the current frame to obtain a reconstructed frame; reconstructing each frame in the frame sequence through an iteration mode; the motion reference frames of the left and right viewpoints are reconstructed frames of the viewpoints at the previous moment, the parallax reference frame of the left viewpoint is a reconstructed frame of the right viewpoint at the current moment, and the parallax reference frame of the right viewpoint is motion context information of the left viewpoint at the current moment. According to the method, the compression ratio of the binocular video can be improved on the premise of ensuring the video quality.

Description

Technical Field

[0001] The present invention relates to the technical field of video compression, and in particular to a context-based binocular video compression method. Background Art

[0002] With the rapid development of multimedia technology and the widespread popularity of Internet applications, the massive growth of high-definition video data has brought huge challenges to storage and transmission. Video compression technology effectively reduces the size of video files by removing redundant information in video data while trying to maintain the visual quality of the video as much as possible.

[0003] Existing neural video compression methods mostly adopt a predictive residual coding framework. However, due to video motion, new content often appears at object boundaries, making it difficult to find good references in previous frames, thus facing challenges such as large residuals and poor inter-frame coding effects. The context-based video compression framework uses the context in the feature domain as a condition, which can more flexibly learn spatio-temporal correlations and improve the quality of the reconstructed video. However, current research mostly focuses on monocular video compression, and there is less exploration of binocular video and multi-viewpoint stereo video compression. In binocular video, in addition to the redundancy within viewpoints, the correlation between viewpoints also provides room for optimization of the compression algorithm. By effectively using this redundant information, the size of the video file can be further reduced and the compression performance can be improved.

[0004] The Chinese patent application with the publication number CN119232941A discloses a binocular video compression method based on deep learning, which uses deformable convolution to process motion information and inter-view information, and realizes high-quality video compression by using the redundancy between viewpoints and the redundancy within the same viewpoints. However, deformable convolution may have problems such as offset overflow during the training process and is relatively unstable. Summary of the Invention

[0005] Considering the problems such as relatively unstable training in the prior art, in contrast, optical flow, as a relatively mature motion modeling method in computer vision, already has many research results and high-quality pre-trained models. Based on this, this method introduces pyramid optical flow to replace the original deformable convolution scheme, enhancing the stability of the model while ensuring the compression efficiency.

[0006] The technical solution adopted by the present invention to solve its technical problems is: to provide a context-based binocular video compression method, which uses a first channel and a second channel with the same structure to process the frame sequences of the right viewpoint and the left viewpoint frame by frame respectively. The specific process of channel processing is as follows:

[0007] The optical flow motion estimation module uses a spatial pyramid optical flow estimation network to calculate motion information. The feature extraction module extracts the shallow features of the motion reference frame. The motion information is input into the motion compression module for compression and then, together with the shallow features of the motion reference frame, is input into the motion compensation module to obtain motion context information. The optical flow disparity estimation module uses a spatial pyramid optical flow estimation network to calculate disparity information. The feature extraction module extracts the shallow features of the disparity reference frame. The disparity information is input into the disparity compression module for compression and then, together with the shallow features of the disparity reference frame, is input into the disparity compensation module to obtain disparity context information. The feature fusion module and the context refinement module are used to fuse the motion context information and the disparity context information to obtain the final context information, which is sequentially input into the context encoding module, the context compression module, and the context decoding module to obtain the reconstructed frame after compression of the current frame;

[0008] Among them, the first channel includes the first optical flow motion estimation module, the first motion compression module, the first motion compensation module, the first optical flow disparity estimation module, the first disparity compression module, the first disparity compensation module, the first feature extraction module, the first feature fusion module, the first context refinement module, the first context encoding module, the first context compression module, and the first context decoding module; the second channel includes the second optical flow motion estimation module, the second motion compression module, the second motion compensation module, the second optical flow disparity estimation module, the second disparity compression module, the second disparity compensation module, the second feature extraction module, the second feature fusion module, the second context refinement module, the second context encoding module, the second context compression module, and the second context decoding module.

[0009] Preferably, the context-based binocular video compression method includes the following steps:

[0010] Obtain the original frame at the current moment of the right view point as the first current frame, obtain the reconstructed frame at the previous moment of the right view point as the first motion reference frame, input it into the first optical flow motion estimation module to calculate the first motion information, and use the first motion compression module for compression to obtain the first compressed motion information; use the first feature extraction module to extract the shallow features of the first motion reference frame and input them into the first motion compensation module, and perform compensation based on the first compressed motion information to obtain the first motion context information; the first motion reference frame at the initial moment is the right view point I frame compressed from the first frame of the right view point;

[0011] Obtain the original frame of the left view at the current moment as the second current frame, obtain the reconstructed frame of the left view at the previous moment as the second motion reference frame, input it into the second optical flow motion estimation module to calculate the second motion information, and use the second motion compression module for compression to obtain the second compressed motion information; use the second feature extraction module to extract the shallow features of the second motion reference frame and input them into the second motion compensation module, and perform compensation based on the second compressed motion information to obtain the second motion context information; the second motion reference frame at the initial moment is the left view I frame compressed from the first frame of the left view;

[0012] The first optical flow disparity estimation module uses the second motion context information as the first disparity reference frame, combines it with the first current frame to calculate the first disparity information, and uses the first disparity compression module for compression to obtain the first compressed disparity information; uses the first feature extraction module to extract the shallow features of the first disparity reference frame and input them into the first disparity compensation module, and perform compensation based on the first compressed disparity information to obtain the first disparity context information; uses the first feature fusion module to fuse and encode the first motion context information and the first disparity context information to obtain the first rough context information; uses the first context refinement module to refine the first rough context to obtain the first final context information;

[0013] The first context encoding module maps the first current frame to the first latent representation with the first final context information as the condition; uses the first context compression module to compress the first latent representation to obtain the first compressed latent representation; uses the first context decoding module to upsample the first compressed latent representation into features with the original resolution, and then cascade the upsampled features with the first final context information to obtain the first reconstructed frame at the current moment;

[0014] The second optical flow disparity estimation module uses the first reconstructed frame at the current moment as the second disparity reference frame, combines it with the second current frame to calculate the second disparity information, and uses the second disparity compression module for compression to obtain the second compressed disparity information; uses the second feature extraction module to extract the shallow features of the second disparity reference frame and input them into the second disparity compensation module, and perform compensation based on the second compressed disparity information to obtain the second disparity context information; uses the second feature fusion module to fuse and encode the second motion context information and the second disparity context information to obtain the second rough context information; uses the second context refinement module to refine the second rough context to obtain the second final context information;

[0015] The second context encoding module maps the second current frame to a second latent representation conditioned on the second final context information; compresses the second latent representation using the second context compression module to obtain a second compressed latent representation; uses the second context decoding module to upsample the second compressed latent representation into features with the original resolution, and then concatenates the upsampled features with the second final context information to obtain the second reconstructed frame at the current moment;

[0016] Taking the current moment as the previous moment and the next moment as the current moment, repeat the above steps until the reconstructed frames of all frames in the frame sequence are obtained, and combine the reconstructed frame sequences of the right view point and the left view point into a compressed stereoscopic video.

[0017] Preferably, the first motion information is expressed as:

[0018] ;

[0019] ;

[0020] ;

[0021] ;

[0022] ;

[0023] Wherein, represents the first motion information at time t; represents the spatial pyramid optical flow estimation network; represents the downsampling result of the original frame of the right view point at time t for input to the k-th pyramid level, represents the downsampling result of the reconstructed frame of the right view point at time t-1 for input to the k-th pyramid level; represents the motion optical flow field of the k-th pyramid level of the right view point, and the initial motion optical flow field of the right view point ; represents the motion residual optical flow of the k-th pyramid level of the right view point; represents the image warping operation;

[0024] The second motion information is expressed as:

[0025] ;

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] wherein, represents the second motion information at time t; k represents the pyramid level, k = 0, 1, 2, 3, 4; represents the downsampling function; represents the downsampling result of the original left - view frame at time t for input to the k - th pyramid level, represents the downsampling result of the reconstructed left - view frame at time t - 1 for input to the k - th pyramid level; represents the upsampling function; represents the motion optical flow field of the k - th pyramid level of the left - view point, the initial motion optical flow field of the left - view point ; represents the motion residual optical flow of the k - th pyramid level of the left - view point; represents the trained convolutional neural network model.

[0031] Preferably, the first disparity information is expressed as:

[0032] ;

[0033] ;

[0034] ;

[0035] ;

[0036] ;

[0037] wherein, represents the first disparity information at time t; represents the downsampling result of the left - view point motion context information at time t ; represents the view - point optical flow field of the k - th layer of the right - view point pyramid; represents the view - point optical flow field of the (k - 1) - th layer of the right - view point pyramid; represents the view - point residual optical flow field of the k - th layer of the right - view point pyramid;

[0038] The second disparity information is expressed as:

[0039] ;

[0040] ;

[0041] ;

[0042] ;

[0043] ;

[0044] Among them, represents the disparity information of the left view point at time t; represents the downsampling result of the reconstructed frame of the right view point at time t ; represents the view point optical flow field of the k-th layer pyramid of the left view point; represents the view point optical flow field of the (k - 1)-th layer pyramid of the left view point; represents the view point residual optical flow field of the k-th layer pyramid of the left view point.

[0045] Preferably, the calculation process of the first motion compression module includes the following steps:

[0046] Using an encoding network to map the first motion information into a first motion latent representation, expressed as:

[0047] ;

[0048] Among them, represents the first motion latent representation at time t; represents the convolutional layer function with a convolutional kernel of ; represents the normalization layer;

[0049] Quantizing and entropy encoding the first motion latent representation to obtain a first motion entropy encoding result; The first entropy encoding result is obtained through a decoding network to obtain first motion compression information, expressed as:

[0050] ;

[0051] Among them, represents the first motion compression information at time t; represents the first motion entropy encoding result at time t; represents the transposed convolutional layer function with a convolutional kernel of ; represents the inverse normalization layer;

[0052] Using a hyperprior encoding network to capture the hidden information of the first motion latent representation, generating the hyperprior edge information of the first motion latent representation; The hyperprior edge information of the first motion latent representation is quantized and entropy encoded and then fed back to the entropy encoder and entropy decoder of the image compression autoencoder network to assist the entropy model in more accurately modeling the probability distribution of the latent representation;

[0053] The calculation process of the second motion compression module includes the following steps:

[0054] The encoding network is used to map the second motion information into a second motion latent representation, expressed as:

[0055] ;

[0056] wherein, represents the second motion latent representation at time t;

[0057] The second motion latent representation is quantized and entropy encoded to obtain a second motion entropy encoding result, and the second motion entropy encoding result is decoded through a decoding network to obtain second motion compressed information, expressed as:

[0058] ;

[0059] wherein, represents the second motion compressed information at time t; represents the second motion entropy encoding result at time t;

[0060] The hyperprior encoding network is used to capture the hidden information of the second motion latent representation and generate the hyperprior marginal information of the second motion latent representation; the hyperprior marginal information of the second motion latent representation is quantized and entropy encoded and then fed back to the entropy encoder and entropy decoder to assist the entropy model in more accurately modeling the probability distribution of the latent representation;

[0061] The calculation process of the first disparity compression module includes the following steps:

[0062] The encoding network is used to map the first disparity information into a first disparity latent representation, expressed as:

[0063] ;

[0064] wherein, represents the first disparity latent representation at time t;

[0065] The first disparity latent representation is quantized and entropy encoded to obtain a first disparity entropy encoding result, and the first disparity entropy encoding result is decoded through a decoding network to obtain first disparity compressed information, expressed as:

[0066] ;

[0067] wherein, represents the first disparity compressed information at time t; represents the first disparity entropy encoding result at time t;

[0068] The hyperprior encoding network is used to capture the hidden information of the first disparity latent representation and generate the hyperprior edge information of the first disparity latent representation; the hyperprior edge information of the first disparity latent representation is fed back to the entropy encoder and entropy decoder of the image compression autoencoder network after quantization and entropy coding to assist the entropy model in more accurately modeling the probability distribution of the latent representation;

[0069] The calculation process of the second disparity compression module includes the following steps:

[0070] The encoding network is used to map the second disparity information into a second disparity latent representation, expressed as:

[0071] ;

[0072] where, represents the second disparity latent representation at time t;

[0073] The second disparity latent representation is quantized and entropy-coded to obtain the second disparity entropy coding result, and the second disparity entropy coding result is decoded through the decoding network to obtain the second disparity compression information, expressed as:

[0074] ;

[0075] where, represents the second disparity compression information at time t; represents the second disparity entropy coding result at time t;

[0076] The hyperprior encoding network is used to capture the hidden information of the second disparity latent representation and generate the hyperprior edge information of the second disparity latent representation; the hyperprior edge information of the second disparity latent representation is fed back to the entropy encoder and entropy decoder after quantization and entropy coding to assist the entropy model in more accurately modeling the probability distribution of the latent representation.

[0077] Preferably, the first motion context information is expressed as:

[0078] ;

[0079] ;

[0080] ;

[0081] where, represents the first motion context information at time t, represents the shallow feature extracted by the first feature extraction module from the first motion reference frame at time t-1, represents the residual block with a convolutional kernel of ​denotes the RELU activation function, and x denotes the input feature of the residual block. denotes the output feature of the residual block;

[0082] The first parallax context information is expressed as:

[0083] ;

[0084] ;

[0085] Among them, denotes the first parallax context information at time t; denotes the shallow feature extracted by the first feature extraction module from the first parallax reference frame at time t;

[0086] The second motion context information is expressed as:

[0087] ;

[0088] ;

[0089] Among them, denotes the second motion context information at time t; denotes the shallow feature extracted by the second feature extraction module from the second motion reference frame at time t - 1;

[0090] The second parallax context information is expressed as:

[0091] ;

[0092] ;

[0093] Among them, denotes the second parallax context information at time t; denotes the shallow feature extracted by the second feature extraction module from the second parallax reference frame at time t.

[0094] Preferably, the first final context information is expressed as:

[0095] ;

[0096] ;

[0097] Among them, denotes the first final context information at time t; denotes the first rough context information at time t; denotes concatenating the features in the channel dimension;

[0098] The second final context information is expressed as:

[0099] ;

[0100] ;

[0101] Among them, represents the second final context information at time t, represents the second rough context information at time t.

[0102] Preferably, the first latent representation and the second latent representation are respectively represented as:

[0103] ;

[0104] ;

[0105] Among them, represents the first latent representation at time t, represents the second latent representation at time t; represents the normalization function.

[0106] Preferably, the process of the first context compression module compressing the first latent representation includes the following steps:

[0107] Using a hyperprior model to obtain the first hyperprior edge information according to the first latent representation;

[0108] Using a time information extraction network to extract features from the first context information to obtain the first context time prior information, represented as:

[0109] ;

[0110] Among them, represents the first context time prior information at time t;

[0111] Fusing the first hyperprior edge information and the first context time prior information to learn the probability distribution of the quantized first latent representation, and feeding it back to the entropy encoder and entropy decoder of the image compression autoencoder network to assist the entropy model in more accurately modeling the probability distribution of the first latent representation, obtaining the first compressed latent representation;

[0112] The process of the second context compression module compressing the second latent representation includes the following steps:

[0113] Using a hyperprior model to obtain the second hyperprior edge information according to the second latent representation;

[0114] Using a time information extraction network Extract features from the second context information to obtain the second context time prior information, expressed as:

[0115] ;

[0116] where represents the second context time prior information at time t;

[0117] Fuse the second hyperprior edge information and the second context time prior information to learn the probability distribution of the quantized second latent representation, and feedback it to the entropy encoder and entropy decoder of the image compression autoencoder network to assist the entropy model to more accurately model the probability distribution of the second latent representation, and obtain the second compressed latent representation.

[0118] Preferably, the first reconstructed frame is expressed as:

[0119] ;

[0120] ;

[0121] where represents the first reconstructed frame at time t; represents the context decoder; represents the first compressed latent representation at time t;

[0122] The second reconstructed frame is expressed as:

[0123] ;

[0124] ;

[0125] where represents the second reconstructed frame at time t, represents the second compressed latent representation at time t.

[0126] The present invention has the following beneficial effects:

[0127] (1) The present invention designs an optical flow motion estimation module and an optical flow disparity estimation module to estimate motion information and disparity information, and uses pyramid optical flow to model the motion information through a multi-scale structure, calculates the optical flow step by step on images with different resolutions, improves the motion modeling accuracy, and at the same time has a clearer structure and more stable training;

[0128] (2) The present invention introduces context time prior information into the entropy coding model, uses a time prior encoder to mine the time correlation in the video sequence, thereby assisting the entropy model to more accurately estimate the probability distribution of the context latent representation, and improves the video compression ratio;

[0129] (3) The present invention designs a context generation unit and a context encoding and decoding module, and uses high-dimensional context to transmit rich information to the context encoder and the context decoder, and assists video encoding and decoding with the context information in the feature domain, which helps to reconstruct high-frequency content to obtain higher video quality.

[0130] The following further elaborates on the present invention in detail with reference to the accompanying drawings and embodiments, but the present invention is not limited to the embodiments. Description of the Drawings

[0131] Figure 1 It is a flowchart of the steps of the context-based binocular video compression method according to an embodiment of the present invention;

[0132] Figure 2 It is a schematic diagram of the dual-channel model of the context-based binocular video compression method according to an embodiment of the present invention;

[0133] Figure 3 It is a schematic diagram of the context compression module of the context-based binocular video compression method according to an embodiment of the present invention;

[0134] Figure 4 It is a comparison diagram of the compression effects between the embodiment of the present invention and LSVC. Detailed Embodiments

[0135] The present invention provides a context-based binocular video compression method, which is implemented by using a dual-channel model. Specifically, the dual-channel model processes the frame sequence of the right view point and the frame sequence of the left view point frame by frame respectively using a first channel and a second channel; wherein, the first channel includes a first context generation unit and a first video reconstruction unit, and the second channel includes a second context generation unit and a second video reconstruction unit.

[0136] Specifically, the first context generation unit includes a first optical flow motion estimation module, a first motion compression module, a first motion compensation module, a first optical flow disparity estimation module, a first disparity compression module, a first disparity compensation module, a first feature extraction module, a first feature fusion module and a first context refinement module; the first video reconstruction unit includes a first context encoding module, a first context compression module and a first context decoding module.

[0137] Specifically, the second context generation unit includes a second optical flow motion estimation module, a second motion compression module, a second motion compensation module, a second optical flow disparity estimation module, a second disparity compression module, a second disparity compensation module, a second feature extraction module, a second feature fusion module and a second context refinement module; the second video reconstruction unit includes a second context encoding module, a second context compression module and a second context decoding module.

[0138] See Figure 1 AndFigure 2 As shown in the figure, the context-based binocular video compression method according to the embodiment of the present invention includes the following steps:

[0139] S101, Obtain the original frame of the right view point at the current moment as the first current frame, obtain the reconstructed frame of the right view point at the previous moment as the first motion reference frame, input the first optical flow motion estimation module to calculate the first motion information, and use the first motion compression module for compression to obtain the first compressed motion information; Use the first feature extraction module to extract the shallow features of the first motion reference frame and input them into the first motion compensation module, and perform compensation based on the first compressed motion information to obtain the first motion context information;

[0140] S102, Obtain the original frame of the left view point at the current moment as the second current frame, obtain the reconstructed frame of the left view point at the previous moment as the second motion reference frame; Input the second optical flow motion estimation module to calculate the second motion information, and use the second motion compression module for compression to obtain the second compressed motion information; Use the second feature extraction module to extract the shallow features of the second motion reference frame and input them into the second motion compensation module, and perform compensation based on the second compressed motion information to obtain the second motion context information;

[0141] S103, The first optical flow disparity estimation module uses the second motion context information as the first disparity reference frame, combines it with the first current frame to calculate the first disparity information, and uses the first disparity compression module for compression to obtain the first compressed disparity information; Use the first feature extraction module to extract the shallow features of the first disparity reference frame and input them into the first disparity compensation module, and perform compensation based on the first compressed disparity information to obtain the first disparity context information; Use the first feature fusion module to fuse and encode the first motion context information and the first disparity context information to obtain the first rough context information; Use the first context refinement module to refine the first rough context to obtain the first final context information;

[0142] S104, The first context encoding module maps the first current frame to the first latent representation with the first final context information as the condition; Use the first context compression module to compress the first latent representation to obtain the first compressed latent representation; Use the first context decoding module to upsample the first compressed latent representation to the features with the original resolution, and then cascade the upsampled features with the first final context information to obtain the first reconstructed frame at the current moment;

[0143] S105. The second optical flow parallax estimation module uses the first reconstructed frame at the current moment as the second parallax reference frame, combines it with the second current frame to calculate the second parallax information, and compresses it using the second parallax compression module to obtain the second compressed parallax information; uses the second feature extraction module to extract the shallow features of the second parallax reference frame and input them into the second parallax compensation module, and compensates based on the second compressed parallax information to obtain the second parallax context information; uses the second feature fusion module to fuse and encode the second motion context information and the second parallax context information to obtain the second rough context information; uses the second context refinement module to refine the second rough context to obtain the second final context information.

[0144] S106. The second context encoding module maps the second current frame to a second latent representation based on the second final context information; uses the second context compression module to compress the second latent representation to obtain the second compressed latent representation; uses the second context decoding module to upsample the second compressed latent representation into features with the original resolution, and then concatenates the upsampled features with the second final context information to obtain the second reconstructed frame at the current moment.

[0145] S107. Use the current moment as the previous moment and the next moment as the current moment, and repeat the above steps until the reconstructed frames of all frames in the frame sequence are obtained. Combine the reconstructed frame sequences of the right view point and the left view point into a compressed binocular video.

[0146] Among them, the first motion reference frame and the second motion reference frame in the first iteration both use the I frame (i.e., the key frame) of the corresponding view point. Three types of frames are defined in H.264, namely I frame, B frame, and P frame. Among them, the I frame is an independent frame with all information, and only the data of this frame is required to complete decoding during decoding; in the embodiment of the present invention, the first frame of each video is compressed into an I frame.

[0147] The following takes the left view point as an example to detail the specific process of channel processing.

[0148] Specifically, the processing flow of the second context generation unit includes the following steps (the modules involved in the following content all refer to the modules in the second context generation unit).

[0149] S201. Use the optical flow motion estimation module to extract the motion information between the current frame and the motion reference frame. The motion information is expressed as follows:

[0150] ;

[0151] Among them, represents the input image of the left view point, represents the moment of the input frame, denote the reconstructed frame of the left view point at the moment; where denotes the spatial pyramid optical flow estimation network. First, an image sequence with different resolutions is obtained by downsampling by a factor of 2 layer by layer. Each time of downsampling halves the width and height of the image. Then, initial optical flow estimation is performed at the top layer of the pyramid. Assuming the initial motion optical flow field is 0, starting from the top layer, the residual flow at each pyramid level is learned through a coarse-to-fine spatial pyramid structure, and the corresponding motion optical flow field is obtained , where k (k = 0, 1, 2, 3, 4) denotes the k-th pyramid level. The specific representation is as follows:

[0152] ;

[0153] ;

[0154] ;

[0155] ;

[0156] ;

[0157] Among them, denotes the downsampling function; denotes the upsampling function; denotes the left view point image that has been downsampled by a factor of 2 (4 - k) times and input to the k-th pyramid level, denotes the reconstructed frame of the previous moment of the left view point that has been downsampled by a factor of 2 (4 - k) times and input to the k-th pyramid level; denotes the motion optical flow field of the k-th pyramid level; denotes the motion residual optical flow of the k-th pyramid level; denotes the trained convolutional neural network model, which uses the upsampled motion optical flow field of the previous pyramid level and the image frame received by the k-th layer to calculate the motion residual optical flow . denotes the image warping operation. In optical flow motion estimation, each pixel in the input image is mapped to the target position according to the optical flow field, thereby generating the transformed output image.

[0158] S202. Use the spatial optical flow pyramid to extract the disparity features between the current frame and the disparity reference frame. The disparity information is represented as follows:

[0159] ;

[0160] ;

[0161] Among them, represents the disparity feature of the left view at time t; represents the reconstructed frame of the right view after (4 - k) times of 2-fold downsampling at the same time. represents the view optical flow field of the k-th layer pyramid; represents the view optical flow field of the (k - 1)-th layer pyramid; among which, the item represents the view residual optical flow field of the k-th layer pyramid, which can be expressed as .

[0162] S203. The motion compression module is adopted to compress the motion information by using the hyperprior model to obtain the compressed motion information , which is expressed as:

[0163] ;

[0164] represents the entropy coding function, which specifically includes the following two aspects.

[0165] On the one hand, the image is compressed through an autoregressive network composed of a motion coding network and a motion decoding network. The motion information is mapped into a motion latent representation by the motion coding network, and it is quantized and entropy coded to obtain , and the compressed motion information is obtained through the motion decoding network. The specific representation is as follows:

[0166] ;

[0167] ;

[0168] Among them, represents the motion coding network, which maps the motion information into a latent representation , represents the normalization layer; represents the motion decoding network, represents the deconvolution layer function with a convolution kernel of , represents the inverse normalization layer.

[0169] On the other hand, the hyperprior motion edge information is introduced through the hyperprior model. The hyperprior coding network is used to capture the hidden information of the motion latent representation and generate the hyperprior motion edge information of the latent representation , which generates after quantization and entropy coding, and is used as auxiliary information for coding and transmission. The hyperprior model is used to learn The probability distribution of is fed back to the entropy encoder (AE) and entropy decoder (AD) of the image compression autoencoder network, assisting the entropy model to represent the potential at the current moment The probability distribution of can achieve more accurate modeling, improve the compression ratio, and guide it to better complete image compression and restoration.

[0170] S204, using a disparity compression module to compress the disparity information using a super prior model to obtain compressed disparity information The specific expressions are as follows:

[0171] ;

[0172] The process of the disparity compression module is consistent with that of the motion coding network. On the one hand, the image is compressed through an autoregressive network consisting of a disparity coding network and a disparity decoding network, and the disparity information is mapped into a disparity potential representation through the disparity coding network. , and quantize and entropy encode it to get , compressed motion information is obtained through the motion decoding network On the other hand, the super-prior disparity edge information is introduced through the super-prior model. The super-prior encoding network is used to capture the hidden information of the disparity potential representation and generate the disparity potential representation The super prior parallax edge information , quantized and entropy encoded to produce , encoded and transmitted as auxiliary information, and learned through the hyper-prior model The probability distribution of is fed back to the entropy model of the image compression autoencoder network, namely the entropy encoder (AE) and the entropy decoder (AD), to assist the entropy model in the potential representation The probability distribution of can achieve more accurate modeling and guide it to better complete image compression and restoration.

[0173] S205, using a feature extraction module to extract features of the left viewpoint reconstructed frame at the previous moment and the right viewpoint reconstructed frame at the current moment, the features of the left viewpoint reconstructed frame at the previous moment It is expressed as follows:

[0174] ;

[0175] The feature extraction module It is expressed as follows:

[0176] ;

[0177] in, express The features of the left viewpoint reconstructed frame at time instant, Indicates the time of the input frame. Represents a convolutional layer function with a convolutional kernel of ; Represents a residual block with a convolutional kernel of . The residual block Is represented as:

[0178] ;

[0179] Among them, Represents the RELU activation function, x represents the input feature, Represents the output feature, Represents the convolutional kernel. When j = 3, it represents a convolutional kernel of .

[0180] The feature of the right-view reconstruction frame at time t Is represented as follows:

[0181] ;

[0182] Among them, Is the right-view reconstruction frame at time t and serves as the disparity reference frame for the left view.

[0183] S206. The motion compensation module is adopted, and the decoded motion feature Will guide the network to extract context information through a warping operation to obtain high-dimensional motion context information, which is specifically represented as follows:

[0184] ;

[0185] Among them, Represents the motion context information of the left view at time t.

[0186] S207 adopts a disparity compensation module, and the decoded disparity feature Will guide the network to extract context information through a warping operation to obtain high-dimensional view context information, which is represented as follows:

[0187] ;

[0188] Among them, Represents the view context information of the left view at time t.

[0189] S208. The feature fusion module is adopted to fuse the left-view motion context information and the view context information together to obtain the rough context information of the left view at time t , which is represented as follows:

[0190] ;

[0191] Among them, It means to splice features in the channel dimension.

[0192] S209. The adopted context refinement module further refines the rough context information of the left view point obtained in the previous step through residual blocks and convolutional layers to obtain the final context information of the left view point at time t, which is specifically represented as follows: The rough context information of the left view point obtained in the previous step is further refined through residual blocks and convolutional layers to obtain the final context information of the left view point at time t. Specifically, it is as follows:

[0193] ;

[0194] Among them, denotes the context refinement module, and its specific representation is as follows:

[0195] .

[0196] Specifically, the processing flow of the video reconstruction module includes the following steps (the modules involved in the following content all refer to the modules in the second context generation unit).

[0197] S301. The context encoding module is adopted to encode the current frame of the left view point at time t into a latent representation with the context information extracted in the previous step as a condition, which is specifically represented as follows: Specifically, it is as follows:

[0198] ;

[0199] Among them, denotes the context encoder, and its specific representation is as follows:

[0200] ;

[0201] Among them, denotes the normalization function.

[0202] S302. The context compression module is adopted to quantize and compress the latent representation of the current frame of the left view point. Specifically, it is as follows:

[0203] ;

[0204] Among them, denotes the fusion prior entropy model. is the hyperprior model, is the temporal prior model.

[0205] Specifically, as shown in Figure 3 input the latent representation of the current frame of the left view point into the hyperprior model to obtain the edge information, and input the final context information Input the temporal prior model to obtain the contextual temporal prior information. By fusing the edge information of the current frame's latent representation and the contextual temporal prior information, learn the probability distribution of the quantized current frame's latent representation, and feedback it to the entropy models of the image compression autoencoder network, namely the entropy encoder (the second AE) and the entropy decoder (the second AD), to assist the entropy models in achieving more accurate modeling of the probability distribution of the latent representation and guiding them to better complete image compression and restoration. The contextual temporal information feature is expressed as follows:

[0206] .

[0207] S303. Adopt the contextual decoding module. Through the transposed convolution layer, inverse normalization layer, and residual block, upsample the reconstructed latent representation of the previous step to the features with the original resolution, and then, taking the contextual information as the condition, concatenate the contextual information with the upsampled features to generate the final left-viewpoint reconstructed frame , which is specifically expressed as follows:

[0208] ;

[0209] Among them, represents the reconstruction module function, which is specifically expressed as follows:

[0210] ;

[0211] Among them, represents the contextual decoder, which is specifically expressed as follows:

[0212] ;

[0213] Among them, represents the transposed convolution layer with a convolution kernel of , represents the inverse normalization layer.

[0214] ​When this embodiment works, the development environment adopts Pytorch, uses the Python language for programming, and uses the Nvidia RTX A6000 GPU for experiments. The datasets adopted in this embodiment are Vimeo-90K and Cityscapes. Vimeo-90K contains two subsets. The subset "Septuplet dataset" of it is selected in this invention example to train the dual-channel model, with a total of 91,701 groups. Each group of data consists of a sequence of 7 consecutive frame images. Before training, the images are first randomly cropped, and the images with the original resolution of 448×256 are cropped into 256×256. Cityscapes consists of large-scale and diverse stereo video sequences collected on 50 different urban streets. Its training set contains 2,975 video pairs, the validation set contains 500 video pairs, and the test set contains 1,525 video pairs. Each video pair contains two 30-frame video sequences with a resolution of 2048×1034. To improve the adaptability of the model in real scenarios, the images need to be preprocessed before training: crop 64 pixels at the top and 128 pixels on the left of each frame of the image to remove calibration artifacts; at the same time, crop 240 pixels at the bottom to remove the interference of the structure of the acquisition vehicle itself, so that the model can focus more on the learning of real scenario content. Subsequently, the random cropping technique is adopted to crop each training image into 320×256 for data augmentation.

[0215] The dual-purpose context video compression training of this invention example adopts a phased training strategy. First, the optical flow motion / disparity estimation, motion / disparity compression, and motion / disparity compensation parts are pre-trained on the Vimeo-90K dataset. The parameters of these modules are shared between the left view and right view modules. Then, the entire dual-channel compression model is trained end-to-end based on the Cityscapes dataset. During the training process, the initial learning rate is set to 5e -5 , the Adam optimizer is used for optimization, and the batchsize of the training network is set to 4. In terms of performance evaluation, Bpp is used to measure the bit rate, and PSNR and MS-SSIM are used to measure the distortion between the compressed video and the original video.

[0216] The relevant data and comparison results of the PSNR index are shown in Table 1 and Figure 4 As shown, it can be seen from Table 1 that the PSNR values of this invention embodiment at different Bpps are all higher than 35, and the compression effect meets the general video compression requirements; Figure 4This is a schematic diagram comparing the compression effects of the embodiments of the present invention and LSVC (learning-based stereo video compression framework). As can be seen from the figure, within a certain range, the compression effect of the embodiments of the present invention at the same bpp is higher than that of LSVC (learning-based stereo video compression framework, Z. Chen, G. Lu, Z. Hu, S. Liu, W. Jiang and D. Xu, "LSVC: A Learning-based Stereo Video Compression Framework," 2022 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), New Orleans, LA, USA, 2022, pp. 6063-6072, doi: 10.1109 / CVPR52688.2022.00598.).

[0217] Table 1 - PSNR data at different Bpps:

[0218]

[0219] It can be seen that a context-based binocular video compression method proposed by the present invention uses an optical flow disparity estimation module and a disparity compression module to compress the redundancy between viewpoints and improve the video compression performance; the context learning module and the context encoding and decoding module use high-dimensional context to transmit rich information to the context encoder and the context decoder, and assist video encoding and decoding based on the context information in the feature domain, which helps to reconstruct high-frequency content to obtain higher video quality. Optical flow explicitly models the inter-frame displacement through pixel-by-pixel motion vectors. The estimated result is a motion field with physical significance, which can provide clear motion trajectories, is more suitable for accurate motion analysis, and has higher interpretability and stability. Therefore, the present invention example adopts pyramid optical flow to estimate the motion optical flow step by step from low resolution to high resolution by constructing an image pyramid.

[0220] The above are only the preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A context-based binocular video compression method, characterized in that: The first channel and the second channel with the same structure are used to process the frame sequence of the right viewpoint and the frame sequence of the left viewpoint frame by frame respectively. The channel processing flow is as follows: The optical flow motion estimation module uses a spatial pyramid optical flow estimation network to calculate motion information, and the feature extraction module extracts shallow features of the motion reference frame. The motion information is input into the motion compression module for compression and then input into the motion compensation module together with the shallow features of the motion reference frame to obtain motion context information; the optical flow disparity estimation module uses a spatial pyramid optical flow estimation network to calculate disparity information, and the feature extraction module extracts shallow features of the disparity reference frame. The disparity information is input into the disparity compression module for compression and then input into the disparity compensation module together with the shallow features of the disparity reference frame to obtain disparity context information; The motion context information and the disparity context information are fused by the feature fusion module and the context extraction module to obtain the final context information, which is then input into the context encoding module, the context compression module and the context decoding module in sequence to obtain the reconstructed frame after the current frame is compressed; Among them, the first channel includes a first optical flow motion estimation module, a first motion compression module, a first motion compensation module, a first optical flow disparity estimation module, a first disparity compression module, a first disparity compensation module, a first feature extraction module, a first feature fusion module, a first context refinement module, a first context encoding module, a first context compression module and a first context decoding module; the second channel includes a second optical flow motion estimation module, a second motion compression module, a second motion compensation module, a second optical flow disparity estimation module, a second disparity compression module, a second disparity compensation module, a second feature extraction module, a second feature fusion module, a second context refinement module, a second context encoding module, a second context compression module and a second context decoding module.

2. The context-based binocular video compression method according to claim 1, characterized in that: The context-based binocular video compression method comprises the following steps: The original frame of the right viewpoint at the current moment is obtained as the first current frame, and the reconstructed frame of the right viewpoint at the previous moment is obtained as the first motion reference frame, which is input into the first optical flow motion estimation module to calculate the first motion information, and the first motion compression module is used to compress it to obtain the first compressed motion information; the first feature extraction module is used to extract the shallow features of the first motion reference frame and input it into the first motion compensation module, and compensation is performed based on the first compressed motion information to obtain the first motion context information; the first motion reference frame at the initial moment is the right viewpoint I frame obtained by compressing the first frame of the right viewpoint; The original frame of the left viewpoint at the current moment is obtained as the second current frame, and the reconstructed frame of the left viewpoint at the previous moment is obtained as the second motion reference frame, which is input into the second optical flow motion estimation module to calculate the second motion information, and the second motion compression module is used to compress it to obtain the second compressed motion information; the second feature extraction module is used to extract the shallow features of the second motion reference frame and input it into the second motion compensation module, and compensation is performed based on the second compressed motion information to obtain the second motion context information; the second motion reference frame at the initial moment is the left viewpoint I frame obtained by compressing the first frame of the left viewpoint; The first optical flow disparity estimation module uses the second motion context information as the first disparity reference frame, calculates the first disparity information in combination with the first current frame, and compresses it using the first disparity compression module to obtain first compressed disparity information; uses the first feature extraction module to extract shallow features of the first disparity reference frame and inputs it into the first disparity compensation module, and compensates based on the first compressed disparity information to obtain first disparity context information; uses the first feature fusion module to fuse and encode the first motion context information and the first disparity context information to obtain first coarse context information; uses the first context refinement module to refine the first coarse context to obtain first final context information; The first context encoding module maps the first current frame to the first potential representation based on the first final context information; the first context compression module is used to compress the first potential representation to obtain a first compressed potential representation; the first context decoding module is used to upsample the first compressed potential representation to features with original resolution, and then the upsampled features are cascaded with the first final context information to obtain a first reconstructed frame at the current moment; The second optical flow disparity estimation module uses the first reconstructed frame at the current moment as the second disparity reference frame, calculates the second disparity information in combination with the second current frame, and compresses it using the second disparity compression module to obtain second compressed disparity information; uses the second feature extraction module to extract shallow features of the second disparity reference frame and inputs it into the second disparity compensation module, and compensates based on the second compressed disparity information to obtain second disparity context information; uses the second feature fusion module to fuse and encode the second motion context information and the second disparity context information to obtain second coarse context information; uses the second context refinement module to refine the second coarse context to obtain second final context information; The second context encoding module maps the second current frame to the second potential representation based on the second final context information; the second context compression module is used to compress the second potential representation to obtain a second compressed potential representation; the second context decoding module is used to upsample the second compressed potential representation to features with original resolution, and then the upsampled features are cascaded with the second final context information to obtain a second reconstructed frame at the current moment; Take the current moment as the previous moment and the next moment as the current moment, repeat the above steps until the reconstructed frames of all frames in the frame sequence are obtained, and combine the reconstructed frame sequence of the right viewpoint and the reconstructed frame sequence of the left viewpoint into a compressed binocular video.

3. The context-based binocular video compression method according to claim 2, characterized in that: The first motion information is expressed as: ； ； ； ； ； in, Represents the first motion information at time t; Represents the spatial pyramid optical flow estimation network; Represents the original frame of the right viewpoint at time t The downsampling result is used to input the k-th pyramid level, Represents the right view reconstructed frame at time t-1 The downsampling result is used to input the kth pyramid level; Represents the motion optical flow field of the kth pyramid level of the right viewpoint, the initial motion optical flow field of the right viewpoint ; Represents the motion residual optical flow of the k-th pyramid level of the right viewpoint; Represents an image deformation operation; The second motion information is expressed as: ； ； ； ； ； in, represents the second motion information at time t; k represents the pyramid level, k=0, 1, 2, 3, 4; represents the downsampling function; Represents the original frame of the left viewpoint at time t The downsampling result is used to input the k-th pyramid level, Represents the reconstructed frame of the left viewpoint at time t-1 The downsampling result is used to input the kth pyramid level; represents the upsampling function; Represents the motion optical flow field of the kth pyramid level of the left viewpoint, the initial motion optical flow field of the left viewpoint ; Represents the motion residual optical flow of the k-th pyramid level of the left viewpoint; Represents a trained convolutional neural network model.

4. The context-based binocular video compression method according to claim 3, characterized in that: The first disparity information is expressed as: ； ； ； ； ； in, represents the first disparity information at time t; Represents the left viewpoint motion context information at time t The downsampling result of Represents the viewpoint optical flow field of the k-th pyramid of the right viewpoint; Represents the viewpoint optical flow field of the k-1th layer of the pyramid of the right viewpoint; Represents the viewpoint residual optical flow field of the k-th pyramid of the right viewpoint; The second disparity information is expressed as: ； ； ； ； ； in, Represents the disparity information of the left viewpoint at time t; Represents the right view reconstructed frame at time t The downsampling result of Represents the viewpoint optical flow field of the k-th pyramid of the left viewpoint; Represents the viewpoint optical flow field of the k-1th layer of the pyramid of the left viewpoint; Represents the viewpoint residual optical flow field of the k-th pyramid layer of the left viewpoint.

5. The context-based binocular video compression method according to claim 4, characterized in that: The calculation process of the first motion compression module includes the following steps: The encoding network is used to map the first motion information into the first motion potential representation, which is expressed as: ； in, represents the first motion potential representation at time t; Represents the convolution kernel as The convolutional layer function is represents the normalization layer; The first motion potential representation is quantized and entropy encoded to obtain a first motion entropy encoding result; the first entropy encoding result is decoded through a network to obtain first motion compression information, which is expressed as: ； in, Represents the first motion compression information at time t; represents the first motion entropy coding result at time t; Represents the convolution kernel as The deconvolution layer function is: represents the inverse normalization layer; A super-prior coding network is used to capture the hidden information of the first motion potential representation, and generate super-prior edge information of the first motion potential representation; the super-prior edge information of the first motion potential representation is fed back to the entropy encoder and entropy decoder of the image compression autoencoder network after quantization and entropy coding, so as to assist the entropy model in achieving more accurate modeling of the probability distribution of the potential representation; The calculation process of the second motion compression module includes the following steps: The encoding network is used to map the second motion information into the second motion potential representation, which is expressed as: ； in, represents the second motion potential representation at time t; The second motion potential representation is quantized and entropy encoded to obtain a second motion entropy encoding result. The second motion entropy encoding result is decoded through a network to obtain second motion compression information, which is expressed as: ； in, Represents the second motion compression information at time t; represents the second motion entropy coding result at time t; A super-prior coding network is used to capture the hidden information of the second motion potential representation and generate super-prior edge information of the second motion potential representation; the super-prior edge information of the second motion potential representation is fed back to the entropy encoder and entropy decoder after quantization and entropy coding, so as to assist the entropy model in achieving more accurate modeling of the probability distribution of the potential representation; The calculation process of the first parallax compression module includes the following steps: The encoding network is used to map the first disparity information into the first disparity potential representation, which is expressed as: ； in, represents the first disparity potential representation at time t; The first disparity potential representation is quantized and entropy encoded to obtain a first disparity entropy encoding result. The first disparity entropy encoding result is decoded through a network to obtain first disparity compression information, which is expressed as: ； in, represents the first disparity compression information at time t; represents the first disparity entropy coding result at time t; A super-prior coding network is used to capture the hidden information of the first disparity potential representation and generate super-prior edge information of the first disparity potential representation; the super-prior edge information of the first disparity potential representation is fed back to the entropy encoder and entropy decoder of the image compression autoencoder network after quantization and entropy coding, so as to assist the entropy model in achieving more accurate modeling of the probability distribution of the potential representation; The calculation process of the second parallax compression module includes the following steps: The encoding network is used to map the second disparity information into a second disparity potential representation, which is expressed as: ； in, represents the second disparity potential representation at time t; The second disparity potential representation is quantized and entropy encoded to obtain a second disparity entropy encoding result. The second disparity entropy encoding result is decoded through a network to obtain second disparity compression information, which is expressed as: ； in, represents the second parallax compression information at time t; represents the second disparity entropy coding result at time t; A super-prior coding network is used to capture the hidden information of the second disparity latent representation, and generate the super-prior edge information of the second disparity latent representation; the super-prior edge information of the second disparity latent representation is quantized and entropy encoded and then fed back to the entropy encoder and entropy decoder to assist the entropy model in achieving more accurate modeling of the probability distribution of the latent representation.

6. The context-based binocular video compression method according to claim 5, characterized in that: The first motion context information is expressed as: ； ； ； in, represents the first motion context information at time t, represents the shallow features extracted by the first feature extraction module from the first motion reference frame at time t-1, Represents the convolution kernel as The residual block, represents the RELU activation function, x represents the input feature of the residual block, Represents the output features of the residual block; The first disparity context information is expressed as: ； ； in, Represents the first disparity context information at time t; represents the shallow features extracted by the first feature extraction module from the first disparity reference frame at time t; The second motion context information is expressed as: ； ； in, Represents the second motion context information at time t; represents the shallow features extracted by the second feature extraction module from the second motion reference frame at time t-1; The second disparity context information is expressed as: ； ； in, Represents the second disparity context information at time t; Represents the shallow features extracted by the second feature extraction module from the second disparity reference frame at time t.

7. The context-based binocular video compression method according to claim 6, characterized in that: The first final context information is expressed as: ； ； in, Represents the first and final context information at time t; Represents the first rough context information at time t; Indicates that the features are concatenated in the channel dimension; The second final context information is expressed as: ； ； in, represents the second final context information at time t, Represents the second coarse context information at time t.

8. The context-based binocular video compression method according to claim 7, characterized in that: The first potential representation and the second potential representation are respectively expressed as: ； ； in, represents the first potential representation at time t, represents the second potential representation at time t; Represents the normalization function.

9. The context-based binocular video compression method according to claim 8, characterized in that: The process of compressing the first potential representation by the first context compression module comprises the following steps: Acquire first super-prior edge information according to the first latent representation using a super-prior model; Using temporal information extraction network Extract features from the first context information to obtain the first context time prior information, which is expressed as: ； in, Represents the first context time prior information at time t; The first super-prior edge information and the first context time prior information are integrated to learn the probability distribution of the quantized first potential representation, and the information is fed back to the entropy encoder and the entropy decoder of the image compression autoencoder network to assist the entropy model in more accurate modeling of the probability distribution of the first potential representation, thereby obtaining the first compressed potential representation; The process of compressing the second potential representation by the second context compression module comprises the following steps: Acquire second super-prior edge information according to the second latent representation using a super-prior model; Using temporal information extraction network Extract features from the second context information to obtain the second context time prior information, which is expressed as: ； in, Represents the second context time prior information at time t; The second super-prior edge information and the second contextual time prior information are integrated to learn the probability distribution of the quantized second latent representation, and the information is fed back to the entropy encoder and entropy decoder of the image compression autoencoder network to assist the entropy model in more accurate modeling of the probability distribution of the second latent representation, thereby obtaining the second compressed latent representation.

10. The context-based binocular video compression method according to claim 9, characterized in that: The first reconstructed frame is expressed as: ； ； in, represents the first reconstructed frame at time t; represents the context decoder; represents the first compressed latent representation at time t; The second reconstructed frame is expressed as: ； ； in, represents the second reconstructed frame at time t, Represents the second compressed latent representation at time t.

Images