A stereo video compression method with dual-branch attention

Abstract

The application provides a stereo video compression method with double-branch attention, comprising the following steps: dividing a stereo video into a left video frame sequence and a right video frame sequence; inputting a video frame of the stereo video at time t, a time-adjacent reconstructed frame and a view-adjacent reconstructed frame of the video frame into a DAN stereo video compressor to obtain a video reconstructed frame at time t; wherein the DAN stereo video compressor comprises the following modules: a feature extraction module, a motion estimation module, a disparity estimation module, an LGEDB-based coding and decoding module, a motion compensation module, a disparity compensation module, a double-branch high-frequency information fusion module and an image reconstruction module; and the LGEDB and the DHFFM can realize higher-quality image reconstruction with the same or lower Bits Per Pixel (BPP).

Description

Technical Field

[0001] This invention belongs to the technical fields of computer vision, deep learning, and video compression, and in particular relates to a stereo video compression method with dual-branch attention. Background Technology

[0002] Stereoscopic dual-view video is a video technology that simulates the parallax of the human eye by providing two different viewpoints (left and right views), thereby creating a three-dimensional effect in the viewer's perception. With the widespread adoption of autonomous vehicles (AVs) equipped with stereo cameras and the development and application of virtual reality (VR) devices, the importance of stereoscopic video data (dual-view) has significantly increased. In AV and VR applications, codecs not only need to maintain low latency but also need to focus on the quality of the reconstructed image after decoding. Therefore, how to efficiently compress stereoscopic video data has become a hot research topic.

[0003] Traditional stereoscopic video compression methods place each view into a separate single-view codec (such as H.264 / AVC or HEVC), but they ignore view redundancy caused by the similarity between two views after compression. To address this deficiency, extensions have been made to the single-view codec. For example, H.264 / MVC, an extension of H.264 / AVC, uses disparity compensation to eliminate disparity redundancy after compression, or MV-HEVC, an extension of HEVC, uses techniques such as Coding Tree Units (CTUs) to eliminate disparity redundancy. Both methods improve compression performance by combining inter-frame and inter-view prediction methods.

[0004] In recent years, due to the powerful learning capabilities of Deep Neural Networks (DNNs), DNN-based stereo video compression coding methods have seen tremendous development and become a research hotspot (all relevant literature is provided). Luo et al. first proposed a DNN-based Stereo Video Compression (LSVC) framework. They reduced time and binocular redundancy by performing motion compression and disparity compression in the feature space, thus achieving efficient compression coding of stereo video. Hou et al. proposed a DNN-based Low-Latency Neural Stereo Streaming (LLSS) method. By introducing a bidirectional feature displacement module and sharing motion and disparity information, they achieved parallel processing of the left and right views, significantly reducing latency and improving rate-distortion (RD) performance. Although existing DNN-based stereo video compression methods achieve better rate-distortion performance compared to traditional stereo video (dual-view) compression coding methods, they almost all only use convolution operations for feature extraction and fusion. Although convolution operations can effectively extract local information, they are poor at capturing non-repetitive and complex texture information. Furthermore, convolution operations cannot capture long-range global information, resulting in defects such as the inability to effectively capture non-repetitive texture details in a local area and the neglect of global features, which seriously affect the quality of image reconstruction during the decoding process. Summary of the Invention

[0005] To address the problems existing in the background technology, the present invention provides a stereoscopic video compression method with dual-branch attention, comprising:

[0006] S1: Divide the stereoscopic video into a left video frame sequence and a right video frame sequence;

[0007] S2: Input the video frame of the stereoscopic video at time t, its temporally adjacent reconstructed frames, and its view-adjacent reconstructed frames into the DAN stereoscopic video compressor for compression to obtain the video reconstructed frame at time t; wherein, the video frame of the stereoscopic video at time t includes: the left video frame at time t and the right video frame at time t; the temporally adjacent reconstructed frame of the left video frame at time t is the left video reconstructed frame at time t-1, and its view-adjacent reconstructed frame is the right video reconstructed frame at time t-1; the temporally adjacent reconstructed frame of the right video frame at time t is the right video reconstructed frame at time t-1, and its view-adjacent reconstructed frame is the left video reconstructed frame at time t-1.

[0008] The DAN stereo video compressor includes: a feature extraction module, a motion estimation module, a disparity estimation module, an LGEDB-based encoding and decoding module, a motion compensation module, a disparity compensation module, a dual-branch high-frequency information fusion module, and an image reconstruction module.

[0009] S3: Store the left video reconstruction frame at time t and the right video reconstruction frame at time t to obtain the compressed stereoscopic video.

[0010] The present invention has at least the following beneficial effects

[0011] The LGEDB and DHFFM proposed in this invention can achieve higher quality image reconstruction with the same or lower bits per pixel (BPP). This invention proposes a Local and Global Dual-Branch Encoder-Decoder Block (LGEDB) based on Transformer and channel attention mechanisms. By fusing self-attention of pixels within a local range and global channel attention, it achieves accurate capture of local non-repeating texture details and global structural information. Simultaneously, this invention also proposes a Dual-Branch High-Frequency Information Fusion Module (DHFFM) based on reversible neural networks and gating mechanisms. This module can effectively extract high-frequency information from motion compensation features and disparity compensation features, and achieve efficient fusion of the two types of features through pixel-by-pixel feature filtering. The LGEDB proposed in this invention can achieve higher quality image reconstruction while maintaining a lower bit rate. Attached Figure Description

[0012] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0013] Figure 2 This is a schematic diagram of the compression process for each frame of stereoscopic dual-view video in an embodiment of the present invention;

[0014] Figure 3 This is a schematic diagram of the overall framework of DAN in an embodiment of the present invention;

[0015] Figure 4 This is a schematic diagram of the LGEDB structure in this invention;

[0016] Figure 5 This is a schematic diagram of the LWAB structure in this invention;

[0017] Figure 6 This is a schematic diagram of the GCAB structure in this invention;

[0018] Figure 7 This is a schematic diagram of the residual module in this invention;

[0019] Figure 8 This is a schematic diagram of the structure of the dual-branch high-frequency information fusion module in this invention;

[0020] Figure 9 This is a schematic diagram of the feature extraction module and the image reconstruction module in this invention;

[0021] Figure 10 This is a schematic diagram of the motion estimation module and disparity estimation module in this invention;

[0022] Figure 11 This is a schematic diagram of the motion compensation module and the parallax compensation module in this invention;

[0023] Figure 12 This is a schematic diagram of the NAF module in this invention;

[0024] Figure 13 This is a schematic diagram of the INN module in this invention;

[0025] Figure 14 This is a schematic diagram showing a simulation comparison between the present invention and existing technologies on different datasets. Detailed Implementation

[0026] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments are only schematic representations of the basic concept of the present invention. Unless otherwise specified, the following embodiments and features can be combined with each other.

[0027] Please see Figure 1 This invention provides a stereoscopic video compression method based on dual-branch attention, comprising:

[0028] S1: Divide the stereoscopic video into a left video frame sequence and a right video frame sequence;

[0029] S2: Input the video frame of the stereoscopic video at time t, its temporally adjacent reconstructed frames, and its view-adjacent reconstructed frames into the DAN stereoscopic video compressor for compression to obtain the video reconstructed frame at time t; wherein, the video frame of the stereoscopic video at time t includes: the left video frame at time t and the right video frame at time t; the temporally adjacent reconstructed frame of the left video frame at time t is the left video reconstructed frame at time t-1, and its view-adjacent reconstructed frame is the right video reconstructed frame at time t-1; the temporally adjacent reconstructed frame of the right video frame at time t is the right video reconstructed frame at time t-1, and its view-adjacent reconstructed frame is the left video reconstructed frame at time t-1.

[0030] The DAN stereo video compressor includes: a feature extraction module, a motion estimation module, a disparity estimation module, an LGEDB-based encoding and decoding module, a motion compensation module, a disparity compensation module, a dual-branch high-frequency information fusion module, and an image reconstruction module.

[0031] S3: Store the left video reconstruction frame at time t and the right video reconstruction frame at time t to obtain the compressed stereoscopic video.

[0032] Preferably, the corresponding left and right video reconstruction frames are obtained by using image compression for the first video frame in the left and right video frame sequences.

[0033] In this embodiment, the first video frame can employ the intra-frame prediction method of H.265. Since there are no previous frames to refer to, compression can only rely on the pixel information within the image. In H.265, the image is divided into multiple small blocks. Each small block can be compressed using various prediction modes to improve the compression ratio and reconstruction quality.

[0034] In this embodiment, the compression of the first video frame can also use the JPEG image compression algorithm. The JPEG image compression algorithm transforms the image from the time domain to the frequency domain, removes redundant information using Discrete Cosine Transform (DCT), and achieves efficient lossy compression through quantization and entropy coding. The resulting left and right reconstructed video frames are used as reference frames for subsequent frames and input into the DAN network.

[0035] In this embodiment, in addition to the two image compression algorithms mentioned above, other existing methods can also be used to compress the first video frame, which will not be elaborated on in this embodiment.

[0036] Preferably, the reconstruction process of the DAN stereoscopic video compressor includes:

[0037] S21: Take the left video frame at time t Right video frame at time t Right video reconstruction frame at time t-1 and the left video reconstruction frame at time t-1 The input feature extraction module extracts the corresponding features F. t L Feature F t R ,feature and characteristics

[0038] S22: Feature F t R and characteristics The input motion estimation module calculates motion information. and feature F t L and characteristics The input motion estimation module calculates motion information.

[0039] S23: Transfer motion information and The motion reconstruction information is obtained by extracting local non-repetitive complex region texture features and global information from the LGEDB-based encoding and decoding modules respectively. and

[0040] S24: Based on motion reconstruction information Using the motion compensation module to analyze feature F t R Motion compensation is performed to obtain feature F MR and based on motion reconstruction information Using the motion compensation module to analyze feature F t L Motion compensation is performed to obtain feature F ML ;

[0041] S25: Based on feature F t R and characteristics Calculate disparity information using the disparity estimation module And based on feature F t L and characteristics Calculate disparity information using the disparity estimation module

[0042] S26: Parallax information and The LGEDB-based encoding and decoding modules are input separately to extract local non-repeating complex region texture features and global information to obtain disparity reconstruction information. and

[0043] S27: Reconstruct information based on parallax Using the disparity compensation module to analyze feature F t R Parallax compensation is performed to obtain feature F DR Based on parallax reconstruction information Using the disparity compensation module to analyze feature F t L Parallax compensation is performed to obtain feature F DL ;

[0044] S28: Feature F MR and feature F DRThe predicted features are obtained by inputting the high-frequency information fusion module into the dual-branch module. Feature F ML and feature F DL The predicted features are obtained by inputting the high-frequency information fusion module into the dual-branch module.

[0045] S29: Feature F t R Subtract features Obtain residual information Feature F t L Subtract features Obtain residual information residual information and residual information The residual reconstruction information is obtained by extracting local non-repetitive complex region texture features and global information from the LGEDB-based encoding and decoding modules respectively. and Reconstruct information from residuals and predictive features The reconstructed features are obtained by adding the features together. Reconstruct information from residuals and predictive features The reconstructed features are obtained by adding the features together. Reconstruct features The input image reconstruction module obtains the right video reconstruction frame at time t. Reconstruct features The input image reconstruction module obtains the left video reconstruction frame at time t.

[0046] In this embodiment, the stereoscopic dual-view video is captured by simultaneously shooting T frames by the left (L) and right (R) cameras, i.e. In this invention, the superscript represents the view, and the subscript 't' represents the time step. The invention employs the same process as LSVC to compress each frame, such as... Figure 2 As shown, the compression will proceed in turn, compressing the right view first and then the left view.

[0047] The left and right views of this invention use the same compression encoding network, such as Figure 3 As shown, we will use the right view as an example to discuss the DAN proposed in this invention in detail. First, the current encoded frame... Temporally adjacent reconstructed frames Reconstructing frames adjacent to the view Input the feature extraction module to obtain the corresponding feature map F t R , and Then take the features F of the current encoded framet R Features of reconstructed frames adjacent to time The data is fed into the motion estimation module to obtain motion information. Then the motion information Motion reconstruction information is obtained by feeding the codec module, which consists of a local and a global dual-branch codec block (LGEDB). Then reconstruct the motion information and F t R Motion compensation is performed to obtain the motion compensation feature F. ML Similarly, the features F of the current encoded frame are... t R Features of reconstructed frames adjacent to the view The data is fed into the disparity estimation module to obtain disparity information. parallax information The parallax reconstruction information is obtained by feeding it into the LGEDB-based encoding and decoding module. Then reconstruct the parallax information and F t R By performing disparity compensation, the disparity compensation feature F can be obtained. DL Next, F ML and F DL The features are fed into the fusion module to obtain the predicted features. And and F t R Subtraction yields residual information. Right now Then the residual information The residual reconstruction information is obtained by feeding it into the LGEDB-based encoding and decoding module. Finally, the residual reconstruction information is... With predictive features Adding them together yields the reconstructed features. Right now The reconstructed image is obtained by reconstructing the features. The reconstructed image is then placed in a buffer. The feature extraction module, motion estimation module, motion compensation module, disparity estimation module, disparity compensation module, and image reconstruction module all use the same setup as LSVC; for details, please refer to [link to relevant documentation]. Figures 9 to 11 As shown, the LGEDB-based encoding and decoding module is the main contribution and innovation of this invention.

[0048] Preferably, the LGEDB-based encoding / decoding module includes: an encoder, a bottleneck layer, and a decoder; the encoder includes N convolutional modules and an LGEDB module connected in series between any two adjacent convolutional modules; the decoder includes N deconvolutional modules and an LGEDB module connected in series between any two adjacent deconvolutional modules.

[0049] The encoder is used to encode the input features of the LGEDB-based encoding and decoding module, the bottleneck layer is used to map the encoded features of the encoder, and the decoder is used to decode the mapped features of the bottleneck layer to obtain the output features of the LGEDB-based encoding and decoding module.

[0050] Preferably, the LGEDB module includes: an LWAB module, a GCAB module, a convolutional layer, and M cascaded residual modules; the feature extraction process of the LGEDB module includes:

[0051] S211: Input the input feature F of the LGEDB module into the LWAB module and the GCAB module respectively for feature extraction to obtain feature f. w and feature f c ;

[0052] S212: Feature f w and feature f c After concatenation, the data are sequentially input into a convolutional layer and M cascaded residual modules for feature processing to obtain feature f;

[0053] S213: Add the feature f and the input feature F to obtain the output feature F of the LGEDB module. LG-out .

[0054] Preferably, the feature extraction process of the LWAB module is as follows:

[0055] Given input features F∈R H×W×C H represents the height of the input feature F, W represents the width of the input feature F, and C represents the number of channels in the input feature F. The input feature is first divided into p×p non-overlapping blocks, i.e. k = 1, 2, ..., L, Indicates the number of blocks; X is divided through a linear layer. k Divide the channel into multiple heads equally along the channel direction to generate a query. key Sum V s k matrix, The number of channels assigned to each head, s = 1, 2, ..., S, where S is the number of heads, is generated using the following formula:

[0056]

[0057] in, Let each represent a linear projection matrix of the s-th head in the k-th block; calculate the feature similarity of the s-th head in the k-th block.

[0058]

[0059] Here, E is the relative position code; since there are S heads, S self-attention calculations are performed in parallel, and then the attention features calculated for each head are... Feature A is obtained by concatenating along the channel dimension. k ∈R p×p×C Then feature A k With X k The self-attention features of the k-th block are obtained by summing them.

[0060]

[0061] Finally, the self-attention features of each block are concatenated back to their original positions to obtain the output features f of LWAB. w ∈R H ×W×C :

[0062]

[0063] Here, concat(·) represents position concatenation.

[0064] Preferably, the feature extraction process of the GCAB module is as follows:

[0065] Given input features F∈R H×W×C H represents the height of the input feature F, W represents the width of the input feature F, and C represents the number of channels of the input feature F. The calculation process is as follows:

[0066] f' = Conv3(ReLU(Conv3(F)))

[0067] f”=ReLU(Conv1(AvgPool(f’)))

[0068] f c =Sigmoid(Conv1(f”))+f'

[0069] Where Conv3(·) represents a 3×3 standard convolution operation, Conv1(·) represents a 1×1 standard convolution operation, ReLU(·) represents the ReLU activation function, Sigmoid(·) represents the Sigmoid activation function, and AvgPool(.) represents the average pooling operation.

[0070] In this embodiment, most existing stereo dual-view video compression coding models use only pure convolution to construct the codec. However, pure convolution operations not only fail to effectively extract features from complex regions with non-repeating textures, but also fail to capture long-range global information, resulting in poor compression performance. The LGEDB proposed in this invention can effectively extract both non-repeating complex region textures and global information.

[0071] like Figure 5 As shown, the LGEDB proposed in this invention consists of a Local Window Attention Branch (LWAB) and a Global Channel Attention Branch (GCAB), which effectively extract local non-repetitive complex region texture features and global information through LWAB and GCAB respectively.

[0072] For LWAB, this invention uses window-based multi-head attention to acquire local features, such as... Figure 3 As shown. Given input features F∈R H×W×C First, the input features are divided into non-overlapping windows of size p×p, i.e. k = 1, 2, ..., L( That is, the number of blocks (i.e., the number of blocks). Then, X is passed through a linear layer. k Divide the channel into multiple headers to generate a query, key, and value matrix: The number of channels assigned to each head, s = 1, 2, ..., S, where S is the number of heads, is generated using the following formula:

[0073]

[0074] in, These represent the linear projection matrices of the s-th head in the k-th block. Simultaneously, the relationships between different blocks... Parameter sharing. Next, calculate the feature similarity of the s-th head in the k-th block.

[0075]

[0076] Where E is the relative position code. Since there are S heads, the self-attention calculation of formula (4) is performed in parallel S times, and then the attention features calculated for each head are... Feature A is obtained by concatenating along the channel dimension. k ∈R p×p×C Then feature A k With X k The self-attention features of the k-th block are obtained by summing them.

[0077]

[0078] Finally, the self-attention features of each block are concatenated back to their original positions to obtain the LWAB output features f of the entire image. w ∈R H×W×C :

[0079]

[0080] Here, concat(·) represents position concatenation.

[0081] From formula (1)-(5) and Figure 5 As can be seen, LWAB constructs the features of a local window by calculating the correlation between each pixel within that window and other pixels within the same window. Therefore, if a local window contains repeating textures, its similarity is high, and the response after LWAB will be large; conversely, if a local window contains non-repeating textures, its response after LWAB will be small. Thus, LWAB can accurately represent the non-repeating texture features within a local region.

[0082] For GCAB, this invention employs the concept of channel attention to achieve effective extraction of global information. For example... Figure 6 As shown, given input features F∈R H×W×C The calculation process is as follows:

[0083] f'=Conv3(ReLU(Conv3(F))) (7)

[0084] f”=ReLU(Conv1(AvgPool(f’))) (8)

[0085] f c =Sigmoid(Conv1(f”))+f’ (9)

[0086] Where Conv3(·) represents a 3×3 standard convolution operation, Conv1(·) represents a 1×1 standard convolution operation, ReLU(·) represents the ReLU activation function, and Sigmoid(·) represents the Sigmoid activation function.

[0087] After LWAB and GCAB respectively, this invention uses concatenation, convolution, and residual blocks to fuse the extracted local features and global information:

[0088] f = RB(RB(RB(Conv3(cat(f) c ,f w ))))) (10)

[0089] Where cat(·) represents channel splicing, and RB(·) represents a residual block, such as Figure 7 As shown. The final output of LGEDB is:

[0090] F LG-out =f+F

[0091] Please see Figure 7 In this embodiment, the residual module RB(·) is composed of a first 3×3 convolution, a ReLU activation function, and a second 3×3 convolution. The input features of the residual module RB(·) and the output features of the second 3×3 convolution are added together to obtain the output features of the residual module RB(·).

[0092] Preferably, the bottleneck layer maps the encoder's encoded features by:

[0093] (r,(μ,σ))=CWAEM(y)

[0094]

[0095] Where y represents the encoded features after encoder encoding, μ, σ and r represent the mean, variance and latent space residual, respectively; Q(·) is the quantization process; CWAEM(·) represents the channel autoregressive entropy model; AE(·) represents the arithmetic encoding process, and AD(·) represents the arithmetic decoding process.

[0096] After achieving effective extraction of texture features and global information from non-repetitive complex regions, this invention uses the proposed LGEDB as a base block and combines it with Channel-Wise Autoregressive Entropy Models (CWAEM) to construct the encoding and decoding module, such as... Figure 4 As shown.

[0097] The constructed encoding and decoding modules will be applied to motion information respectively. Parallax information and residual information To generate the corresponding reconstruction information. Based on motion information. For example, input First, an encoder composed of LGEDBs extracts the latent representation y. Then, y undergoes quantization, CWAEM, arithmetic encoding (AE), arithmetic decoding (AD), and a decoder composed of LGEDBs to obtain motion reconstruction information. This process can be represented as:

[0098]

[0099] (r,(μ,σ))=CWAEM(y) (12)

[0100]

[0101]

[0102] Where E(·) and D(·) represent the encoder and decoder composed of LGEDB, respectively. Q(·) is the quantization process, CWAEM(·) represents the channel autoregressive entropy model, μ, σ and r represent the mean, variance and latent space residual, respectively; AE(·) represents the arithmetic encoding process, and AD(·) represents the arithmetic decoding process.

[0103]

[0104] Among them, Q step The quantization step size is `round`, which represents the rounding function.

[0105] In motion information and parallax information The motion reconstruction information obtained after passing through LGEDB-based encoding and decoding modules respectively. and parallax reconstruction The information is then processed by the motion compensation module and the disparity compensation module to obtain the motion compensation feature F. M and parallax compensation feature F D Subsequently, DHFFM was used to fuse F M and F D .

[0106] Preferably, the feature fusion process of the dual-branch high-frequency information fusion module includes:

[0107]

[0108] θ=INN(INN(F M )+INN(F D ))

[0109]

[0110] Among them, F M ∈{F MR F ML}, F D ∈{F DR F DL}, NAF(·) represents the NAF module, INN(·) represents the invertible neural network; RB() represents the residual module.

[0111] Besides effectively extracting local non-repeating texture information and global information in the codec, the effective fusion of motion compensation information and disparity compensation information is another key factor affecting the compression coding quality of stereo dual-view video. Based on this, this invention proposes a novel dual-branch high-frequency information fusion module (DHFFM) to achieve effective extraction of high-frequency information and accurate feature selection from motion compensation features and disparity compensation features, thereby realizing efficient fusion of motion compensation features and disparity compensation features.

[0112] Gomez et al. have demonstrated that the inverse transformation property of invertible neural networks (INNs) ensures that high-frequency information is not lost during forward propagation and inverse reconstruction. The INN block proposed by Zhou et al. can effectively extract high-frequency information. Furthermore, the gating mechanism in the NAF (Nonlinear Activation Free) block proposed by Chen et al. enables accurate feature selection. Therefore, as... Figure 8 As shown, the DHFFM proposed in this invention first uses INN and NAF to process the motion compensation information F. M and parallax compensation information F D High-frequency information extraction and feature filtering are performed. Then, F... M The high-frequency information extracted and the filtered features are related to F D The extracted high-frequency information and filtered features are added element-wise. Finally, the added high-frequency information and the added filtered features are concatenated, convolved, and subjected to residual blocks to achieve efficient fusion of motion compensation features and disparity compensation features, resulting in predicted features. This process can be represented as:

[0113]

[0114] θ=INN(INN(F M )+INN(F D ))

[0115]

[0116] Among them, F M ∈{F MR F ML}, F D ∈{F DR F DL}, NAF(·) represents the NAF module, INN(·) represents the invertible neural network; RB() represents the residual module.

[0117] The NAF block and INN block are existing network structures; their network structures can be found in [link to network structure]. Figure 12 and Figure 13As shown, no specific description is given in this embodiment.

[0118] As can be seen from the above analysis, due to the introduction of INN and NAF, the DHFFM proposed in this invention can effectively extract high-frequency information from motion compensation features and disparity compensation features, as well as perform pixel-by-pixel feature filtering, thereby achieving efficient fusion of motion compensation features and disparity compensation features and ensuring further improvement in the performance of dual-view stereoscopic video compression coding.

[0119] Predicted features are obtained through DHFFM. After that, With original feature F t R Subtraction yields residual information. Right now Then the residual information The residual reconstruction information is obtained by feeding it into the encoding and decoding module composed of LGEDB. Subsequently, the residual reconstruction information With predictive features Add them together to obtain the reconstructed features. Right now The reconstructed image is obtained through the image reconstruction module. The reconstructed image It will be stored in the buffer for use in the compression encoding of the next frame.

[0120] exist Figure 8 The residual module RB(·) is composed of a first 3×3 convolution, a ReLU activation function, and a second 3×3 convolution. The input features of the residual module RB(·) and the output features of the second 3×3 convolution are added together to obtain the output features of the residual module RB(·).

[0121] Preferably, the loss function used during the training of the DAN stereo video compressor includes:

[0122]

[0123] Where L represents the loss function, and D(·) is the distortion loss. This represents the video frame of view v at time t; R(y) represents the video reconstructed frame of view v at time t, and R(y) represents the bit rate of the encoder's coding features during compression.

[0124] Following the principles of end-to-end video compression, all networks in this invention use the same loss function, which is used to update all parameters simultaneously. The loss function is set as follows:

[0125]

[0126] Where D(·) represents the distortion loss, indicating the distortion between the reconstructed frame and the original frame. This invention uses Mean Square Errors (MSE) loss and Multi Scale Structural Similarity Index Measure (MS-SSIM) loss to calculate the distortion loss. R(·) represents the bit rate during compression, the superscript v indicates whether the left or right view is used, and the subscript t represents the time step, t∈{1,2,3,4}. L represents the loss function, and D(·) represents the distortion loss. This represents the video frame of view v at time t; Let R(y) represent the video reconstructed frame of view v at time t, and let R(y) represent the bit rate of the encoder's coding features during compression. λ is the control rate-distortion hyperparameter, and in this invention, four different λ values ​​are used (λ = 512, 1024, 2048, 4096).

[0127] Experimental setup

[0128] In the experiments of this invention, the device was first pre-trained using the single-view video dataset Vimeo-90K, and then trained using the stereo dual-view video dataset Cityscape. This invention was evaluated using the stereo dual-view video datasets Cityscape, KITTI2012, and KITTI 2015 as test datasets.

[0129] The Cityscape stereo video dataset contains 2975 and 1525 stereo video sequence pairs for training and testing, respectively. Each sequence pair contains 30 frames, and each image has a resolution of 2048×1024. KITTI 2012 and KITTI 2015 contain 195 and 200 stereo video sequence pairs, respectively. Each sequence pair contains 21 frames, and each image has a resolution of 1241×376. This invention follows the LSVC image preprocessing procedure, preprocessing the Cityscape test set and KITTI separately. After preprocessing, the resolutions of all frames are 1920×768 and 1152×256, respectively.

[0130] This invention follows the LSVC training strategy, training the model in three stages. First, in the first stage, the feature extraction module, motion estimation module, LGEDB-based encoding / decoding module, motion compensation module, and image reconstruction module are pre-trained using the Vimeo-90 K dataset. (The last sentence appears to be incomplete and possibly refers to a specific dataset or training method.) -5 The learning rate is used for two million iterations. Then, the DAN is initialized using one-stage pre-trained weights, using the Cityscape training set and 1×10⁻⁶ weights. -5The learning rate was used to iterate 400,000 times over the LGEDB-based codec module and DHFFM, while the parameters of other modules were frozen. Finally, we used the Cityscape training set and 1×10 -5 The learning process first iterates the entire DAN 300,000 times, then adjusts the learning rate to 5×10. -6 Perform 200,000 iterations.

[0131] Furthermore, this invention employs random rotations of 90, 180, and 270 degrees, along with horizontal flipping, to augment the training dataset. The Adam optimizer is used. During the pre-training phase, the batch size is 8, and the input Vimeo-90K dataset is set to 256×256. In subsequent phases, the batch size is 4, and the input Cityscape training set size is 512×256. During testing, the first frame is compressed using the image compression method proposed by Cheng et al., and fine-tuned on the Cityscape training set according to the training method described in the literature. All experiments in this invention were conducted on an NVIDIA 4090 GPU and the PyTorch deep learning framework for training and testing.

[0132] To demonstrate the superiority of the proposed method, we compared the proposed network with existing, highly representative video compression methods. The comparison literature is shown in Table 1.

[0133]

[0134]

[0135] To verify the superiority of the proposed method, we compared the proposed DAN with six state-of-the-art methods in recent years: LSVC, LLSS, FVC, DCVC, H.265, and MV-HEVC. LSVC, LLSS, FVC, and DCVC are DNN-based methods, while H.265 and MV-HEVC are traditional methods. For H.265, we used HM-16.20 with the configuration "lowdelay_P_main"; for MV-HEVC, we used HTM-16.3 with the configuration "baseCfg_2_view". For the DNN-based methods, we adopted the settings from their respective literature. Figure 14 The figures show the rate-distortion curves of LSVC, LLSS, FVC, DCVC, H.265, MV-HEVC, and the DAN proposed in this invention on the Cityscape, KITTI 2012, and KITTI 2015 test sets. Figure 14 As shown, the rate-distortion curve of the DAN method proposed in this invention is located at the top, which is significantly better than other methods.

[0136] Table 2 shows the BD-BR (baseline selection for BD-BR) of LSVC, LLSS, FVC, DCVC, H.265, and the DAN proposed in this invention on the Cityscape, KITTI 2012, and KITTI 2015 test sets, respectively (the baseline for BD-BR is MV-HEVC).

[0137] Table 2 shows the BD-BR results for all methods on the Cityscape, KITTI 2012, and KITTI 2015 datasets (with MV-HEVC as the baseline).

[0138]

[0139] As shown in Table 2, the DAN proposed in this invention outperforms the most representative state-of-the-art methods in recent years on all test sets: compared with the second best method LLSS, the DAN proposed in this invention improves BD-DR by 3.6%, 7.3% and 7.5% on the Cityscape, KITTI 2012 and KITTI 2015 test sets, respectively.

[0140] In summary, the LGEDB and DHFFM proposed in this invention can achieve higher quality image reconstruction with the same or lower bits per pixel (BPP). This invention proposes a Local and Global Dual-Branch Encoder-Decoder Block (LGEDB) based on Transformer and channel attention mechanisms. By fusing self-attention of pixels within a local range and global channel attention, it achieves accurate capture of local non-repeating texture details and global structural information. Simultaneously, this invention also proposes a Dual-Branch High-Frequency Information Fusion Module (DHFFM) based on reversible neural networks and gating mechanisms. This module can effectively extract high-frequency information from motion compensation features and disparity compensation features, and achieves efficient fusion of the two types of features through pixel-by-pixel feature filtering. The LGEDB proposed in this invention can achieve higher quality image reconstruction while maintaining a lower bit rate.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit it. Although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of the present invention without departing from the spirit and scope of the present invention, and all such modifications or substitutions should be covered within the scope of the claims of the present invention.

Claims

1. A stereoscopic video compression method based on dual-branch attention, characterized in that, include: S1: Divide the stereoscopic video into a left video frame sequence and a right video frame sequence; S2: Input the video frame of the stereoscopic video at time t, its temporally adjacent reconstructed frames, and its view-adjacent reconstructed frames into the DAN stereoscopic video compressor for compression to obtain the video reconstructed frame at time t; wherein, the video frame of the stereoscopic video at time t includes: the left video frame at time t and the right video frame at time t; the temporally adjacent reconstructed frame of the left video frame at time t is the left video reconstructed frame at time t-1, and its view-adjacent reconstructed frame is the right video reconstructed frame at time t-1; the temporally adjacent reconstructed frame of the right video frame at time t is the right video reconstructed frame at time t-1, and its view-adjacent reconstructed frame is the left video reconstructed frame at time t-1. The reconstruction process of the DAN stereoscopic video compressor includes: S21: Take the left video frame at time t The right video frame at time t Right video reconstruction frame at time t-1 and the left video reconstruction frame at time t-1 The input feature extraction module extracts the corresponding features. ,feature ,feature and characteristics ; S22: Features and characteristics The input motion estimation module calculates motion information. and features and characteristics The input motion estimation module calculates motion information. ; S23: Transfer motion information and The motion reconstruction information is obtained by extracting local non-repetitive complex region texture features and global information from the LGEDB-based encoding and decoding modules respectively. and ; S24: Based on motion reconstruction information Using motion compensation module for features Features obtained through motion compensation and based on motion reconstruction information Using motion compensation module for features Features obtained through motion compensation ; S25: Based on features and characteristics Calculate disparity information using the disparity estimation module and based on characteristics and characteristics Calculate disparity information using the disparity estimation module ; S26: Parallax information and The LGEDB-based encoding and decoding modules are input separately to extract local non-repeating complex region texture features and global information to obtain disparity reconstruction information. and ; S27: Reconstruct information based on parallax Using the parallax compensation module for features Features are obtained by performing parallax compensation. Based on parallax reconstruction information Using the parallax compensation module for features Features are obtained by performing parallax compensation. ; S28: Features and characteristics The predicted features are obtained by inputting the high-frequency information fusion module into the dual-branch module. , will feature and characteristics The predicted features are obtained by inputting the high-frequency information fusion module into the dual-branch module. ; S29: Features Subtract features Obtain residual information , will feature Subtract features Obtain residual information ; residual information and residual information The residual reconstruction information is obtained by extracting local non-repetitive complex region texture features and global information from the LGEDB-based encoding and decoding modules respectively. and Reconstruct information from residuals and predictive features The reconstructed features are obtained by adding the features together. Reconstruct information from residuals and predictive features The reconstructed features are obtained by adding the features together. ; Reconstruct features The input image reconstruction module obtains the right video reconstruction frame at time t. ; Reconstruct features The input image reconstruction module obtains the left video reconstruction frame at time t. ; The DAN stereo video compressor includes: a feature extraction module, a motion estimation module, a disparity estimation module, an LGEDB-based encoding and decoding module, a motion compensation module, a disparity compensation module, a dual-branch high-frequency information fusion module, and an image reconstruction module. The LGEDB-based encoding / decoding module includes an encoder, a bottleneck layer, and a decoder; the encoder includes N convolutional modules and an LGEDB module connected in series between any two adjacent convolutional modules; the decoder includes N deconvolutional modules and an LGEDB module connected in series between any two adjacent deconvolutional modules. The encoder is used to encode the input features of the LGEDB-based encoding and decoding module, the bottleneck layer is used to map the encoded features of the encoder, and the decoder is used to decode the mapped features of the bottleneck layer to obtain the output features of the LGEDB-based encoding and decoding module. The LGEDB module includes: an LWAB module, a GCAB module, convolutional layers, and M cascaded residual modules; the feature extraction process of the LGEDB module includes: S211: Input the input features F of the LGEDB module into the LWAB module and the GCAB module respectively for feature extraction to obtain the features. and characteristics ; S212: Features and characteristics After concatenation, the data are sequentially input into a convolutional layer and M cascaded residual modules for feature processing to obtain feature f; S213: Add the feature f and the input feature F to obtain the output feature of the LGEDB module. ; The feature fusion process of the dual-branch high-frequency information fusion module includes: in, , , Represents the NAF module. Represents a reversible neural network; Represents the residual module; S3: Store the left video reconstruction frame at time t and the right video reconstruction frame at time t to obtain the compressed stereoscopic video.

2. The stereoscopic video compression method with dual-branch attention according to claim 1, characterized in that, The corresponding left and right video reconstructed frames are obtained by using image compression on the first video frame in the left and right video frame sequences.

3. The stereoscopic video compression method with dual-branch attention according to claim 1, characterized in that, The feature extraction process of the LWAB module is as follows: Given input features H represents the height of the input feature F, W represents the width of the input feature F, and C represents the number of channels in the input feature F. The input feature is first divided into... Non-overlapping blocks of different sizes, i.e. , , Indicates the number of blocks; through a linear layer Divide the channel into multiple heads equally along the channel direction to generate a query. ,key Sum matrix, , The number of channels allocated to each head. , The number of heads is given by the following formula: in, These represent the linear projection matrices of the s-th head in the k-th block, respectively; calculate the... Feature similarity of the s-th head in each block : Where E is the relative position code; because there is The head, therefore, executes in parallel. The self-attention calculation is performed once, and then the attention features are calculated for each head. Features are obtained by splicing along the channel dimension. Then the features and The self-attention features of the k-th block are obtained by summing them. : Finally, the self-attention features of each block are concatenated back to their original positions to obtain the output features of LWAB. : in, This indicates positional splicing.

4. The stereoscopic video compression method with dual-branch attention according to claim 1, characterized in that, The feature extraction process of the GCAB module is as follows: Given input features H represents the height of the input feature F, W represents the width of the input feature F, and C represents the number of channels of the input feature F. The calculation process is as follows: in, represent Standard convolution operation, represent Standard convolution operation, represent Activation function represent Activation function; This indicates the average pooling operation.

5. The stereoscopic video compression method with dual-branch attention according to claim 1, characterized in that, The bottleneck layer maps the encoder's encoded features in the following ways: in, This represents the encoded features after encoding by the encoder. , and These represent the mean, variance, and latent space residuals, respectively. For the quantification process; This represents the channel autoregressive entropy model; Represents the arithmetic encoding process. This represents the arithmetic decoding process.

6. The stereoscopic video compression method with dual-branch attention according to claim 1, characterized in that, The loss function used during the training of the DAN stereo video compressor includes: in, Represents the loss function. For distortion loss, Represents a view In time Video frames at that time; Represents a view In time Video reconstruction frames at that time This represents the bit rate of the encoder's coding features during the compression process.

Images