Lightweight stereo image super-resolution method based on multi-attention mechanism fusion

Abstract

The present invention discloses a lightweight binocular image super-resolution method based on multi-attention mechanism fusion, which mainly solves the problem that it is difficult to balance model performance and computational efficiency in binocular image super-resolution tasks. First, a modified binary feature fusion framework is introduced to fuse multi-level image features extracted under channel attention and spatial attention mechanisms; secondly, the global disparity information of the binocular image is extracted through the dual-channel attention mechanism, and a pyramid sampling mechanism is introduced to reduce the amount of module calculation. Experiments have shown that the present invention achieves a significant improvement in super-resolution performance with fewer parameters, confirming the portability of lightweight networks in binocular image super-resolution tasks.

Description

Technical Field

[0001] The present invention relates to a lightweight binocular image super-resolution method based on multi-attention mechanism fusion, belonging to the technical field of image processing. Background Art

[0002] Binocular vision is inspired by bionics. The left and right eyes of human beings form a three-dimensional spatial perception of scenes due to their different positions and the differences in the scenes they see. Binocular vision uses binocular cameras to imitate this visual perception mechanism and construct a universal binocular stereoscopic vision perception system similar to the human eye.

[0003] Unlike single image super-resolution, stereo super-resolution is essentially a multi-input and multi-output process. Low-resolution left and right views are input, and the corresponding high-resolution stereo images need to be reconstructed. If the stereo image is regarded as the front and back frames in the video, the task is simplified to a video super-resolution task with two frames of images. However, the interaction between the stereo images is represented by disparity, which is different from the small motion offset between video frames, and the research methods are quite different. At the same time, since the image of the second perspective can provide additional information for a single image, if it is regarded as a single image super-resolution task with reference, the information that can be provided is limited to the information in the low-resolution scene, and the reconstruction of high-level features is extremely useful. Therefore, stereo image super-resolution should not only use the relevant information between images on the basis of single image super-resolution, but also add a disparity compensation mechanism to the traditional multi-image super-resolution task.

[0004] Most lightweight network designs based on computer vision tasks such as image classification and semantic representation have been gradually proposed in recent years, including classic MobileNet, XCeption and other structures, and most of them have also been introduced into the field of image super-resolution, achieving good results. With the addition of attention mechanisms, network performance is further improved. How to reduce the number of parameters of efficient attention mechanisms is also one of the important research topics in this direction. For binocular image super-resolution tasks, the lightweight model needs to take into account the efficiency of the feature extraction backbone network, and also be able to reduce parameter overhead while ensuring performance as much as possible in the binocular feature matching stage. Summary of the invention

[0005] Technical problem: The purpose of this invention is to provide a lightweight binocular image super-resolution method based on the fusion of multiple attention mechanisms. Aiming at the problem that it is difficult to balance model performance and operating efficiency in binocular super-resolution tasks, this paper explores the existing lightweight models in single image super-resolution methods and seeks a lightweight design idea that is suitable for binocular image super-resolution networks.

[0006] Technical solution: In order to achieve a balance between model performance and computational efficiency in binocular super-resolution tasks, this paper proposes a lightweight binocular image super-resolution method based on multi-attention mechanism fusion based on the study of single image super-resolution lightweight network. The specific steps are as follows:

[0007] Step 1: Build a network model

[0008] A low-resolution left view image and a right view image are used as network inputs, and a high-resolution left view image is obtained by super-resolution processing of the left view; the constructed network model includes three submodules, namely, a feature extraction module, a disparity attention extraction module and a feature reconstruction module;

[0009] First, input a low-resolution binocular image pair and After a 3×3 convolutional layer, shallow features of the left view image and the right view image are extracted:

[0010]

[0011]

[0012] Among them, H sfe represents a 3×3 convolutional layer with shared weights, and Represents the shallow features of the left view and the right view extracted from the low-resolution binocular image pair; input m shared weighted feature fusion groups to further extract deeper features:

[0013]

[0014]

[0015] In the above formula, Represents the mth feature fusion group. Similarly, and Represent the m-1th and 1st feature fusion groups respectively, and Refers to the deeper feature tensor output after the shallow features pass through m feature fusion groups;

[0016] After extracting the independent features of the low-resolution image pair, the binocular features are matched through the disparity attention module based on the multi-scale pyramid sampling mechanism, and the disparity fusion feature tensor of the left view image is output:

[0017]

[0018] Among them, H DCPAM Characterize the dual-channel disparity attention module, represents the feature tensor obtained by the disparity attention module; then, n feature fusion groups are used again to further extract and fuse the previous level features, which is represented as:

[0019]

[0020] Similar to the feature extraction stage, represents the nth feature fusion group in the feature reconstruction stage, and Represent the n-1th and 1st feature fusion groups respectively, Represents the fused feature tensor of the left view obtained after n feature fusion groups;

[0021] Finally, the left view image after bicubic upsampling is added pixel by pixel to the upper-level output feature to obtain the final left view super-resolution reconstruction result:

[0022]

[0023] Among them, H 5 and H 3 Refers to 5×5 and 3×3 convolution respectively, H ps represents the pixel reorganization layer, H up is the bicubic upsampling operation, λ 1 and λ 2 represents a trainable scalar parameter, This is the high-resolution left image after the final super-resolution;

[0024] Step 2: Build a binocular image dataset and set training parameters for network training.

[0025] The dataset images are divided into training set, validation set and test set, the training parameters are set to train the network model, and the network training is performed on the training set to obtain the trained network model;

[0026] Step 3: Input the binocular image to be processed into the trained network model to perform binocular image super-resolution reconstruction.

[0027] in,

[0028] The feature fusion block described in step 1 takes the multi-attention fusion module as the basic module, integrates the multi-level features extracted by channel attention, spatial attention and dilated convolution, and is built on the basic architecture of the modified binary feature fusion structure.

[0029] The disparity attention module described in step 1 is a dual-channel attention module, which aims to extract local epipolar features and global disparity information.

[0030] The multi-scale pyramid sampling mechanism described in step 1 has a convolution kernel size of [12, 15, 18, 21] at each level.

[0031] In step 1, m=2, n=2, and the number of feature fusion blocks of the left and right view branches in the feature extraction stage is the same.

[0032] The low-resolution left view image and the right view image in step 1 can be used to reconstruct a high-resolution right view image in the same way by swapping the left and right view low-resolution images.

[0033] In step 2, during the training of the network model, super-resolution loss is used as the loss function.

[0034] In step 2, during the training of the network model, Adam is used as the training optimizer. The learning rate is initialized to 0.00002. The learning rate is optimized during the training process and reduced to half of the original rate every 30 iterations. The batch size is set to 4 and convergence is achieved after 120 training iterations.

[0035] In step 2, during the training of the network model, the network environment was built based on pytorch1.8 using Nvidia RTX3090Ti GPU.

[0036] Beneficial effects: Due to the adoption of the above technical solution, the present invention has the following advantages and beneficial effects compared with the prior art:

[0037] 1. The present invention uses a modified binary feature fusion framework to fuse the features of each level extracted by the multi-attention mechanism, thereby improving the performance of super-resolution reconstruction while reducing model parameters and computational complexity.

[0038] 2. The dual-channel disparity attention mechanism is introduced to realize the feature extraction along the epipolar line and global disparity, and provide effective compensation for the interactive information between the left and right views. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 It is the overall network structure diagram.

[0040] Figure 2(a) is a schematic diagram of the feature fusion group structure, Figure 2(b) is a schematic diagram of the multi-attention fusion module, and Figure 2(c) is a schematic diagram of the dual-channel attention module.

[0041] Figure 3 This is a comparison chart of the results of the present invention and the prior art on low-resolution binocular images.

[0042] Figure 4 It is a comparison chart of the operating efficiency of the present invention and the prior art. DETAILED DESCRIPTION

[0043] The present invention is described in detail below with reference to the accompanying drawings and embodiments. The binocular image super-resolution method based on multi-attention mechanism fusion proposed in the present invention specifically includes the following steps:

[0044] Step 1: Build a network model

[0045] like Figure 1 As shown, the network design of the present invention includes three sub-steps, namely feature extraction, parallax attention extraction and feature reconstruction. Among them, the feature extraction module stage adopts a parameter-sharing modified binary feature fusion framework to fuse the different levels of features in a single view extracted by the multi-attention mechanism; the dual-channels parallax attention extraction stage introduces a dual-channels parallax attention mechanism (DCPAM) and a pyramid sampling mechanism to fuse the local and global information between the two views; the feature reconstruction stage continues the feature fusion group of the feature extraction module and reconstructs the high-resolution left view image through the automatic calculation parameter unit.

[0046] Step 1.1: Feature extraction

[0047] First, input a low-resolution binocular image pair and After a 3×3 convolutional layer, shallow features of the left view image and the right view image are extracted:

[0048]

[0049]

[0050] Among them, H sfe represents a 3×3 convolutional layer with shared weights, and Represents the shallow features of the left view and the right view extracted from the low-resolution binocular image pair. After that, m feature fusion groups with shared weights are input to further extract deeper features:

[0051]

[0052]

[0053] In the above formula, represents the mth feature fusion group, and Represent the m-1th and 1st feature fusion groups respectively, and Refers to the deeper feature tensor output after the shallow features pass through m feature fusion groups.

[0054] Specifically, as shown in Figure 2(a), the feature fusion group is built with the multi-attention fusion module as the basic module and the modified binary fusion framework as the feature fusion framework.

[0055] As shown in Figure 2(b), the main branch of the multi-attention fusion module first extracts spatial features. The input features are first extracted by stacking 3×3 convolution layers, and the number of channels is reduced to 1 / 4 of the original by 1×1 convolution. In order to increase the receptive field, convolution kernels with strides of 2 and 7×7 are used to reduce the spatial dimension. Then, after the pooling layer, dilation convolution operations with expansion rates of 1 and 2 are performed respectively. After upsampling to the original input feature dimension, the spatial feature tensors of the left view image and the right view image are output through the 1×1 convolution kernel Sigmoid normalization operation.

[0056] The auxiliary path of the multi-attention fusion module adopts an efficient channel attention mechanism, which consists of 1×1 point-by-point convolution and depth-by-depth convolution to obtain the channel feature tensors of the left view image and the right view image. The outputs of each multi-attention fusion module are cascaded in the form of a modified binary fusion framework.

[0057] The modified binary fusion framework connects the currently generated output feature tensor with the input feature tensor of the next level and builds it in a layer-by-layer recursive manner. At the same time, a channel reorganization module is added to reduce the number of channels by 1×1 convolution to make it the same as the number of channels of the input feature tensor, and add it to the output feature tensor in a pixel-by-pixel addition manner.

[0058] Step 1.2: Disparity Attention Extraction

[0059] After extracting the independent features of the low-resolution image pair, the binocular features are matched through the disparity attention module based on the multi-scale pyramid sampling mechanism, and the disparity fusion feature tensor of the left view image is output:

[0060]

[0061] Among them, H DCPAM Characterize the dual-channel disparity attention module, Represents the feature tensor obtained by the disparity attention module.

[0062] Specifically, as shown in Figure 2(c), the DLPAM module includes a local parallax attention module (LPAM) on the left branch and a global parallax attention module (GPAM) on the right branch. In LPAM, the left view feature tensor extracted in step 1.1 is transformed into and the right view feature tensor The generated sizes are The feature tensor K l and Q l , by merging the channel dimensions to Get k l and q l , the contribution matrix (correlation matrix) of the right view to the left view along the polar line By k l After transposition, k l T and q l After batch matrix multiplication and softmax operation, we get:

[0063]

[0064] LPAM generates the mask of the right view through a single convolution Reshape and Batched matrix multiplication generates a parallax compensation tensor based on the right view

[0065]

[0066] Final local disparity attention feature tensor for the left view The contribution matrix Parallax compensation tensor for the right view And the deeper feature tensor of the left view The three parts are cascaded, and the generation process is as follows:

[0067]

[0068] The global disparity attention feature tensor generation method of the right channel GPAM is similar to that of LPAM. In order to reduce the huge amount of matrix multiplication calculation, a pyramid sampling mechanism is used for data compression. First, refer to the previous feature tensor extraction method in LPAM to obtain Q g , K g and V g , the dimensions are Provide the characteristic information of the right view, and g and K g Stretched The form is q g and k g , through k g After transposition, k g T and q gAfter batch matrix multiplication, softmax operation is performed to generate a similarity matrix of global features.

[0069]

[0070] Based on this, we introduce the pyramid sampling mechanism under different sizes of convolution kernels [12, 15, 18, 21] to convert K g The size is compressed to Corrected to Global feature aggregation consists of three parts: mask G and similarity matrix M g By dot product with V g After pooling and reshaping v g The transpose of v g T Perform matrix multiplication and output the left view global disparity attention feature tensor

[0071]

[0072] The final extracted dual-channel disparity attention tensor is obtained by adding the local and global disparity feature attention tensors pixel by pixel:

[0073]

[0074] Step 1.3: Feature reconstruction

[0075] Use n feature fusion groups to further extract and fuse the previous level features, which can be represented as follows:

[0076]

[0077] Similar to the feature extraction stage, represents the nth feature fusion group in the feature reconstruction stage, and Represent the n-1th and 1st feature fusion groups respectively, Represents the fused feature tensor of the left view image obtained after n feature fusion groups. Finally, the fused feature tensor extracted by n feature fusion groups is sent to the pixel reconstruction module, and a unit for automatically calculating parameters is added to assign different feature weights to the 3×3 and 5×5 convolutional layers respectively. The cascaded automatically weighted feature tensor is added pixel by pixel with the result of the original left view bicubic upsampling to obtain the final high-resolution left view image:

[0078]

[0079] Among them, H 5 and H 3Refers to 5×5 and 3×3 convolution respectively, H ps represents the pixel reorganization layer, H up is the bicubic upsampling operation, λ 1 and λ 2 represents a trainable scalar parameter, This is the high-resolution left view after the final super-resolution.

[0080] Step 2: Build a binocular image dataset, divide the dataset images into training set, validation set and test set, set training parameters to train the network model, and perform network training on the training set to obtain the trained network model;

[0081] Step 2.1: Loss function setting

[0082] To measure the generalized difference between pixels in the super-resolution image and the ground truth image, the mean square error is used to calculate the super-resolution loss. and The loss function refers to the left view result after super-resolution reconstruction of the network and the ground truth high-resolution left view image. The calculation of is as follows:

[0083]

[0084] Step 2.2: Training setup:

[0085] The number of feature fusion groups at each stage in the above network is m=2, n=2. The training optimizer uses Adam, the learning rate is initialized to 0.00002, and the learning rate is optimized during training, reduced to half of the original every 30 iterations. The batch size is set to 4, and gradually converges after 120 training iterations. The network environment is built based on pytorch1.8 using Nvidia RTX3090Ti GPU.

[0086] Step 3: Input the binocular image to be processed into the trained model to perform binocular image super-resolution reconstruction.

[0087] Figure 3 The reconstruction effect of the present invention is compared with the super-resolution reconstruction results of existing binocular image super-resolution technologies (bicubic upsampling, StereoSR and PASSRnet). The present invention can display the shape of characters more clearly, and the background and the main body of the text are clearly distinguished. There is no influence of light spots, and the super-resolution result is good.

[0088] Figure 4The comparison of the operating efficiency between the present invention and the existing super-resolution reconstruction technologies (SRCNN, VDSR, CARN, StereoSR, PASSRnet) is demonstrated. The present invention has a shorter inference time under the same number of test samples, achieving an effective unity of performance and operating efficiency.

Claims

1. A lightweight binocular image super-resolution method based on multi-attention mechanism fusion, characterized in that: The specific steps include: Step 1: Build a network model A low-resolution left view image and a right view image are used as network inputs, and a high-resolution left view image is obtained by super-resolution processing of the left view; the constructed network model includes three submodules, namely, a feature extraction module, a disparity attention extraction module and a feature reconstruction module; First, input a low-resolution binocular image pair and After a 3×3 convolutional layer, shallow features of the left view image and the right view image are extracted: Among them, H sfe represents a 3×3 convolutional layer with shared weights, and Represents the shallow features of the left view and the right view extracted from the low-resolution binocular image pair; input m shared weighted feature fusion groups to further extract deeper features: In the above formula, Represents the mth feature fusion group. Similarly, and Represent the m-1th and 1st feature fusion groups respectively, and Refers to the deeper feature tensor output after the shallow features pass through m: feature fusion groups; After extracting the independent features of the low-resolution image pair, the binocular features are matched through the disparity attention module based on the multi-scale pyramid sampling mechanism, and the disparity fusion feature tensor of the left view image is output: Among them, H DCPAM Characterize the dual-channel disparity attention module, represents the feature tensor obtained by the disparity attention module; then, n feature fusion groups are used again to further extract and fuse the previous level features, which is represented as: Similar to the feature extraction stage, represents the nth feature fusion group in the feature reconstruction stage, and Represent the n-1th and 1st feature fusion groups respectively, Represents the fused feature tensor of the left view obtained after n feature fusion groups; Finally, the left view image after bicubic upsampling is added pixel by pixel to the upper-level output feature to obtain the final left view super-resolution reconstruction result: Among them, H5 and H3 refer to 5×5 and 3×3 convolutions respectively, and H ps represents the pixel reorganization layer, H up is a bicubic upsampling operation, λ1 and λ2 represent trainable scalar parameters, This is the high-resolution left image after the final super-resolution; Step 2: Build a binocular image dataset and set training parameters for network training. The dataset images are divided into training set, validation set and test set, the training parameters are set to train the network model, and the network training is performed on the training set to obtain the trained network model; Step 3: Input the binocular image to be processed into the trained network model to perform binocular image super-resolution reconstruction.

2. The lightweight binocular image super-resolution method based on multi-attention mechanism fusion according to claim 1, characterized in that: The feature fusion group described in step 1 takes the multi-attention fusion module as the basic module, integrates the multi-level features extracted by channel attention, spatial attention and dilated convolution, and is built on the basic architecture of the modified binary feature fusion structure.

3. The lightweight binocular image super-resolution method based on multi-attention mechanism fusion according to claim 1, characterized in that: The disparity attention module described in step 1 is a dual-channel attention module, which aims to extract local epipolar features and global disparity information.

4. The lightweight binocular image super-resolution method based on multi-attention mechanism fusion according to claim 1, characterized in that: The multi-scale pyramid sampling mechanism described in step 1 has a convolution kernel size of [12, 15, 18, 21] at each level.

5. The lightweight binocular image super-resolution method based on multi-attention mechanism fusion according to claim 1, characterized in that: In step 1, m=2, n=2, and the number of feature fusion blocks of the left and right view branches in the feature extraction stage is the same.

6. The lightweight binocular image super-resolution method based on multi-attention mechanism fusion according to claim 1, characterized in that: The low-resolution left view image and the right view image in step 1 can be used to reconstruct a high-resolution right view image in the same way by swapping the left and right view low-resolution images.

7. The lightweight binocular image super-resolution method based on multi-attention mechanism fusion according to claim 1, characterized in that: In step 2, during the training of the network model, super-resolution loss is used as the loss function.

8. The lightweight binocular image super-resolution method based on multi-attention mechanism fusion according to claim 1, characterized in that: In step 2, during the training of the network model, Adam is used as the training optimizer. The learning rate is initialized to 0.00002. The learning rate is optimized during the training process and reduced to half of the original rate every 30 iterations. The batch size is set to 4 and convergence is achieved after 120 training iterations.

9. The lightweight binocular image super-resolution method based on multi-attention mechanism fusion according to claim 1, characterized in that: In step 2, during the training of the network model, the network environment was built based on pytorch1.8 using Nvidia RTX3090Ti GPU.

Images