A multi-scale residual stereo image super-resolution method based on stereo attention mechanism

Abstract

This paper discloses a multi-scale residual binocular image super-resolution method based on a stereo attention mechanism, which belongs to the field of image processing technology. The method comprises the following steps: S1. Establishing a binocular image training set; S2. Designing a network structure; S3. Setting training parameters to train the network; S4. Testing the network performance. The present invention has the following advantages: (1) Compared with the binocular image super-resolution method based on replica translation, the present method is not limited to the super-resolution processing of fixed disparity images, and the stereo attention mechanism is used to extract the interactive information between the left and right views in parallel, thereby improving the performance of the method. (2) Compared with the binocular image super-resolution based on stereo matching, the present method does not need to estimate the disparity between views based on the three-dimensional or four-dimensional matching cost, thereby improving the efficiency of the method.

Description

Technical Field

[0001] The present invention relates to a multi-scale residual binocular image super-resolution method based on a stereo attention mechanism, and belongs to the technical field of image processing. Background Art

[0002] As the visual basis of the human sensory system, images play an important role in capturing, analyzing, expressing and transmitting information. Image processing technology is indispensable to human beings. Since modern times, with the rapid development of automation technology, electronic information technology and computer network technology, the digital information such as shape, color and texture of natural landscapes carried by images has provided a quantitative representation of visual information.

[0003] Compared with analog image processing, digital image processing has the characteristics of good reproducibility, high precision and high flexibility. As a "soft technology", the development of digital image processing technology has laid the foundation for a large number of subsequent studies, such as image reconstruction, image analysis, target detection, semantic understanding, etc. Based on the demand for image quality, high-resolution images are extremely important in many applications. However, due to the limitations of sensor arrangement density and technical constraints, it is obviously not practical to directly improve the image resolution by improving the resolution of the imaging device. Image super-resolution reconstruction provides another more efficient and convenient way, that is, using one or more low-resolution images to generate high-resolution images through computer software, and restore the detailed texture and occluded areas in the original image as much as possible, while reducing costs and obtaining better visual effects.

[0004] Imitating the imaging process of human eyes, stereoscopic vision restores the relative position of the left and right cameras in the image through the visual difference of synchronous imaging. As one of the mainstream visual solutions, it has outstanding advantages compared with other visual solutions. Binocular vision has a wide range of application scenarios. It can provide two-dimensional (image) and three-dimensional (distance) information at the same time. By adjusting the baseline between cameras, the measurement range can be adjusted. At the same time, after being equipped with an image processing chip, it can also have good real-time recognition of the texture, structure, color, etc. of the scene. Stereoscopic vision has important research value in visual inspection, 3D virtual reality technology, computer-aided technology and advanced driver assistance systems due to its flexibility, convenience and low cost.

[0005] The current research focus of stereo vision is to reconstruct the 3D scene by using the disparity information between binocular views. As the underlying image processing process, binocular image super-resolution is essentially a multi-input and multi-output process, that is, inputting low-resolution left and right views, and reconstructing the corresponding high-resolution binocular image by using the interactive information between images. In order to meet the requirements of high-level visual tasks such as depth estimation and stereo matching for input image quality, the research on binocular image super-resolution needs to draw on the modeling ideas of upper-level tasks on the one hand, and overcome the inherent defects in binocular imaging on the other hand (such as improper matching caused by the influence of ambient lighting, single or overly complex textures, etc.).

[0006] The important premise for binocular images to simulate the human visual system is that objects at different depths appear at different positions during the imaging process. At the same time, due to the epipolar constraints, the relative position of the imaged objects only changes horizontally, and rarely changes vertically, which means that the benefit of parallax compensation along the horizontal epipolar line is the highest. Summary of the invention

[0007] Technical problem: The purpose of the present invention is to provide a multi-scale residual binocular image super-resolution method based on a stereo attention mechanism. In view of the shortcomings of the prior art, the stereo disparity attention mechanism is used to improve the resolution of the binocular image by utilizing the complementary information of the left and right views while solving the epipolar disparity problem.

[0008] Technical solution: The present invention discloses a binocular image super-resolution method based on stereo disparity attention mechanism, comprising the following steps:

[0009] Step S1. Establish a binocular image training set: enhance the training samples by randomly performing upside-down and left-right flipping operations on the training samples;

[0010] Step S2. Design the network structure: The overall network consists of three modules: feature extraction, stereo disparity extraction and feature reconstruction. The feature extraction module extracts features from the low-resolution left and right views of the input network by introducing a multi-scale residual structure with shared weights; the stereo disparity extraction module fuses the feature tensors of the left and right views according to the disparity attention map calculated by the stereo disparity attention mechanism; the feature reconstruction module performs super-resolution reconstruction on the fused feature tensors through dense residual blocks and upsampling modules to restore high-resolution left and right view images;

[0011] Step S3. Set training parameters to train the network, send the training samples obtained in step S1 to the network designed in step S2, select a suitable optimizer, set the loss function, learning parameters and maximum number of iterations to train the network until the training is completed and the final network model is obtained;

[0012] Step S4. Test the network performance, input the low-resolution binocular image test pair into the network obtained in step S3, and output a high-resolution binocular image pair.

[0013] in,

[0014] The binocular image training set established in step S1 is specifically to select picture pairs from four common binocular image data sets as the binocular image training set, each pair of images includes two high-resolution images of the left view and the right view of the same scene, and the corresponding low-resolution image pairs are obtained by bicubic downsampling, and the training samples are composed of image blocks cropped from the high-resolution image pair and the low-resolution image pair. Bicubic interpolation is a professional term, also known as cubic convolution interpolation, which uses the grayscale values ​​of 16 points around the sampling point for cubic interpolation to achieve image scaling. Here, "bicubic downsampling" refers to the reduction of the image through the inverse operation of bicubic interpolation. The subsequent "bicubic upsampling" refers to the enlargement of the image through the bicubic interpolation algorithm.

[0015] The step S2 designs the network structure: comprising the following three steps:

[0016] Step S2.1. Feature extraction: Use a multi-scale residual network with shared weights to extract features from the low-resolution left and right views in the input network, and obtain their feature tensors respectively;

[0017] Step S2.2. Stereo disparity extraction: Use the residual unit to perform secondary processing on the feature tensors of the left view and the right view in step S1.1 to obtain the harmonized left and right feature tensors; Taking the processing of the left view as an example, a stereo disparity attention map is generated by batch matrix multiplication of the left view feature tensor and the transposed feature tensor of the right view and Softmax normalization operation, the right view feature tensor is convolved, and batch matrix multiplication is performed with the stereo disparity attention map, the result is concatenated with the left feature tensor, and the fused left view stereo disparity feature tensor is obtained through convolution operation; the left and right view feature tensors in step S1.1 are swapped and the fused right view stereo disparity feature tensor is obtained through similar operation;

[0018] Step S2.3. Feature reconstruction: Use the residual dense block group to process the left and right view feature tensors fused in step S2.2 respectively, use the deconvolution network to operate the high-dimensional feature tensor, fuse the bicubic upsampling results of the input low-resolution image, and restore the high-resolution left and right view images.

[0019] In step S3, the training network uses Nvidia GTX1080Ti GPU to train the network structure, and the training is deployed based on Pytorch1.8; the optimizer is Adam optimizer, and the initial learning rate is set to 2×10 -4 , the maximum number of iterations is 80, and the learning rate is reduced to half every 20 iterations.

[0020] In step S4, the visual effects of the binocular image pair after bicubic upsampling are compared with the binocular image pair after super-resolution using the method.

[0021] Beneficial effects: Due to the adoption of the above technical solution, the present invention has the following advantages compared with the prior art:

[0022] (1) Compared with the binocular image super-resolution method based on replica translation, this method is not limited to the super-resolution processing of fixed disparity images. It uses a convolutional neural network to directly construct the mapping relationship between low-resolution images and high-resolution images, thereby improving the performance of the method.

[0023] (2) Compared with the binocular image super-resolution based on stereo matching, this method does not need to estimate the disparity between views based on the three-dimensional or four-dimensional matching cost. It extracts the interactive information between the left and right views in parallel through the stereo attention mechanism, which has better robustness to the disparity changes in the binocular image and improves the efficiency of the method. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of the overall steps of the present invention;

[0025] Figure 2 A schematic diagram of the overall structure of the network design of the present invention;

[0026] Figure 2 (a) is a schematic diagram of the structure of the residual dilated spatial convolutional pooling pyramid block in feature extraction;

[0027] Figure 2 (b) is a schematic diagram of the structure of the stereo disparity attention module in stereo disparity extraction;

[0028] Figure 2 (c) is a schematic diagram of the structure of the residual dense module in feature reconstruction;

[0029] Figure 3 It is a comparison diagram of the results of the present invention and the prior art under the embodiment. DETAILED DESCRIPTION

[0030] The present invention is described in detail below with reference to the accompanying drawings and embodiments. The image super-resolution method based on the stereo attention mechanism proposed in the present invention specifically comprises the following steps:

[0031] Figure 1 The overall step flow chart of the present invention specifically includes the following steps:

[0032] Step S1. Establish a binocular image training set: 860 pairs of binocular images (each pair of images contains a left view and a right view) from four common binocular image datasets are selected as high-resolution images in the binocular image training set, and the corresponding low-resolution image pairs are obtained by bicubic downsampling. The training samples are composed of high-resolution image pairs and cropped image blocks of low-resolution image pairs.

[0033] Step S2. Design network structure: Figure 2 It is a schematic diagram of the overall structure of the convolutional neural network in the present invention. As can be seen from the figure, the overall network of the present invention consists of three modules: feature extraction, stereo disparity extraction and feature reconstruction. Among them, the feature extraction module extracts features from the low-resolution left and right views of the input network respectively by introducing a multi-scale residual structure with shared weights; the stereo disparity extraction module fuses the feature tensors of the left and right views by using the disparity attention map calculated by the stereo disparity attention mechanism; the feature reconstruction module performs super-resolution reconstruction on the fused feature tensors through dense residual blocks and upsampling modules to restore high-resolution left and right view images. The specific process is as follows:

[0034] Step S2.1. Feature extraction: Use a multi-layer convolutional neural network with shared weights to extract features from the input low-resolution left and right views to obtain feature tensors Specifically:

[0035] Step S2.1.1. Extract shallow features of the input image through a 3×3 convolutional layer and a basic residual block, use LeakyReLU as the activation function, and feed it into a multi-scale residual module to capture multi-scale features;

[0036] Step S2.1.2. The multi-scale residual module is composed of two groups of residual atrous spatial convolutional pyramid blocks (resASPP) and basic residual blocks in cascade. The former is responsible for generating multi-scale information, while the latter is responsible for information fusion. Among them, the single resASPP structure is as follows Figure 2 As shown in (a), it is composed of dilation rates of 1, 2, and 5, and the number of output channels is controlled to 64 through 1×1 convolution.

[0037] Step S2.2. Stereo disparity extraction: By introducing a stereo disparity attention module (such as Figure 2 (b) to establish the similarity between the positions along the epipolar lines. The details are as follows:

[0038] Step S2.2.1. Use a residual block with shared weights to perform the feature tensor obtained in step S2.1 on each Perform two 3×3 convolution operations and obtain the harmonized feature tensor through a 1×1 convolution layer. and They are characterized as:

[0039]

[0040]

[0041] Among them, H rb refers to the residual block, and They represent the 1×1 convolutions for processing the left and right feature maps, respectively.

[0042]

[0043]

[0044] Step S2.2.2. Stereo disparity attention map of the left view As an example, the left view feature tensor in step S2.2.1 First, we need the transposed right view feature tensor Perform batch matrix multiplication and softmax operation. The resulting stereo disparity attention map group under the dual view is expressed as follows: The middle coordinates represent for The contribution of , that is, the correlation between two feature points.

[0045] Step S2.2.3. Figure 3 The disparity attention map obtained by taking the Toy picture pair in the Middlebury dataset as an example is shown. From the analysis of the figure, the calculation of feature similarity by the stereoscopic inspection attention module focuses on the accurate differentiation of features, not

[0046]

[0047] It is mandatory to collect the similarity between all pixel feature points and only focus on the maximum similarity features along the epipolar lines, so the disparity attention map is sparse. The mask mechanism is used to compensate for the stereo disparity attention map in step S2.2.2. Taking the mask calculation of the left view feature tensor as an example, the process is as follows:

[0048] Step S2.2.4. In actual training, the threshold τ of the mask is set to 0.1, and the interaction feature tensor between the disparity attention map and the input image is obtained by batch matrix multiplication:

[0049]

[0050]

[0051] Step S2.2.5. Connect the feature tensor in step S2.2.1, the mask in step S2.3 and the interactive feature tensor in step S2.4, and use 1×1 convolution to perform feature fusion on the cascaded feature tensors to obtain the final stereo disparity feature tensor, which is expressed as follows:

[0052]

[0053]

[0054] in, and They refer to the left and right view stereo disparity feature tensors passing through a 1×1 convolutional layer.

[0055] Step S2.3. Feature reconstruction: Use four residual dense blocks to perform convolution operations on the stereo disparity feature tensor fused in step S2.2.5. The structure of a single residual dense block is as follows: Figure 2 (c) As shown in the embodiment, each residual dense block is provided with 4 3×3 convolutional layers and a ReLU function group, and a deconvolution operation is performed on the feature tensor through a deconvolution layer, the deconvolution layer is composed of 3×3 convolutions, and the feature tensor is mapped to the RGB space by using the convolution layer, and the bicubic upsampling result of the input low-resolution image is fused to obtain the final output high-resolution images of the left and right views.

[0056] Step S3. Set training parameters to train the network: send the training samples generated in step S1 to the designed network for training, set the cost function to the mean square error between the super-resolution image and the real high-resolution image, deploy the training on Nvidia GTX1080Ti GPU based on Pytorch1.8, use Adam optimizer for optimization, and set the initial learning rate to 2×10 -4 , the maximum number of iterations is 80, and the learning rate is reduced to half every 20 iterations.

[0057] Step S4. Test the network performance: input a low-resolution binocular test image into the trained network and output a corresponding high-resolution binocular image pair. Figure 3The visual effect comparison between a binocular image pair after bicubic upsampling and a binocular image pair after super-resolution using this method. From the enlarged details in this figure, it can be seen that this method has a strong reconstruction ability in terms of text details and structure, and can achieve relatively excellent visual effects through parallax compensation between the left and right views.

Claims

1. A multi-scale residual binocular image super-resolution method based on stereo attention mechanism, characterized by: The method comprises the following steps: Step S1. Establishing a binocular image training set: enhancing the training samples by randomly performing upside-down and left-right flipping operations on the training samples; Step S2. Design the network structure: The overall network consists of three modules: feature extraction, stereo disparity extraction and feature reconstruction. The feature extraction module extracts features from the low-resolution left and right views of the input network by introducing a multi-scale residual structure with shared weights; the stereo disparity extraction module fuses the feature tensors of the left and right views according to the disparity attention map calculated by the stereo disparity attention mechanism; the feature reconstruction module performs super-resolution reconstruction on the fused feature tensors through dense residual blocks and upsampling modules to restore high-resolution left and right view images; Step S3. Set training parameters to train the network, send the training samples obtained in step S1 to the network designed in step S2, select a suitable optimizer, set the loss function, learning parameters and maximum number of iterations to train the network until the training is completed and the final network model is obtained; Step S4. Testing the performance of the network, inputting the low-resolution binocular image test pair into the network obtained in step S3, and outputting a high-resolution binocular image pair; The step S2 designs the network structure: comprising the following three steps: Step S2.

1. Feature extraction: Use a multi-scale residual network with shared weights to extract features from the low-resolution left and right views in the input network, and obtain their feature tensors respectively; Step S2.

2. Stereo disparity extraction: Use the residual unit to perform secondary processing on the feature tensors of the left view and the right view in step S1.1 to obtain the harmonized left and right feature tensors; in the left view, a stereo disparity attention map is generated by batch matrix multiplication of the left view feature tensor and the transposed feature tensor of the right view and Softmax normalization operation, and the right view feature tensor is convolved and batch matrix multiplied with the stereo disparity attention map. The result is concatenated with the left feature tensor, and the fused left view stereo disparity feature tensor is obtained through convolution operation; the left and right view feature tensors in step S1.1 are processed by same The operation is performed to obtain the fused right view stereo disparity feature tensor; Step S2.

3. Feature reconstruction: Use the residual dense block group to process the left and right view feature tensors fused in step S2.2 respectively, use the deconvolution network to operate the high-dimensional feature tensor, fuse the bicubic upsampling results of the input low-resolution image, and restore the high-resolution left and right view images.

2. According to claim 1, the multi-scale residual binocular image super-resolution method based on stereo attention mechanism is characterized in that: The binocular image training set established in step S1 is specifically to select picture pairs from four common binocular image data sets as the binocular image training set, each pair of images includes two high-resolution images of the left view and the right view of the same scene, and the corresponding low-resolution image pairs are obtained by bicubic downsampling, and the training samples are composed of image blocks cropped from the high-resolution image pairs and the low-resolution image pairs.

3. The multi-scale residual binocular image super-resolution method based on stereo attention mechanism according to claim 1, characterized in that: In step S3, the training network uses Nvidia GTX1080Ti GPU to train the network structure and is deployed and trained based on Pytorch1.8; the optimizer is Adam optimizer, and the initial learning rate is set to 2×10 -4 , the maximum number of iterations is 80, and the learning rate is reduced to half every 20 iterations.

4. The multi-scale residual binocular image super-resolution method based on stereo attention mechanism according to claim 1, characterized in that: In step S4, the visual effects of the binocular image pair after bicubic upsampling are compared with the binocular image pair after super-resolution using the method.

Images