A binocular stereo matching method based on horizontally deformable attention module

Abstract

The present invention relates to a binocular stereo matching method based on a horizontally deformable attention module, and belongs to the field of image processing. To address the high computational complexity of non-local attention mechanisms, the potential introduction of unnecessary or erroneous contextual information into stereo matching tasks, and the failure to consider the horizontal constraints and disparity continuity constraints present in stereo matching tasks, the present invention processes the features of the left and right views extracted by a ResNet backbone network through a horizontally deformable attention module, performs further feature processing, and forms a matching cost volume through feature concatenation. The final disparity map is then obtained through three-dimensional convolutional disparity regression. The present invention improves the quality of the disparity map.

Description

Technical Field

[0001] The present invention belongs to the field of image processing, and in particular relates to a binocular stereo matching method based on a horizontally deformable attention module. Background Art

[0002] Stereo matching is a fundamental yet challenging task in computer vision, with widespread applications in autonomous driving, dense reconstruction, and other depth-related tasks. The goal of stereo matching is to compute the disparity or depth information corresponding to each pixel in two or more images from different viewpoints. The challenge of stereo matching lies in accurately finding correspondences between images in areas with unfavorable conditions such as texture loss, occlusion, and illumination variations. To address this issue, numerous deep learning-based stereo matching methods have emerged in recent years. These methods typically employ an end-to-end network architecture, comprising modules such as feature extraction, cost calculation, cost aggregation, and disparity regression. The feature extraction module extracts high-level semantic features from the input image; the cost calculation module constructs a cost volume based on feature similarities or differences; the cost aggregation module regularizes and optimizes the cost volume; and the disparity regression module generates the final disparity map from the optimized cost volume. Among these modules, the cost aggregation module is a key factor influencing stereo matching performance, as it fully leverages contextual information to eliminate ambiguity and noise. To capture contextual information, a common approach is to use a non-local attention mechanism, which calculates the correlation between any two positions through a self-attention mechanism and updates the features of each position according to the correlation weight.

[0003] However, the non-local attention mechanism has the following two problems: first, the computational complexity is high because it requires global operations on the entire feature map; second, for the stereo matching task, the non-local attention mechanism may introduce unnecessary or erroneous contextual information, and does not take into account the horizontal constraints and disparity continuity constraints in the stereo matching task. Summary of the Invention

[0004] (1) Technical issues to be solved

[0005] The technical problem to be solved by the present invention is how to provide a binocular stereo matching method based on a horizontal deformable attention module to solve the problems that the non-local attention mechanism has high computational complexity, may introduce unnecessary or erroneous contextual information for the stereo matching task, and does not take into account the horizontal constraints and disparity continuity constraints existing in the stereo matching task.

[0006] (2) Technical solution

[0007] In order to solve the above technical problems, the present invention proposes a binocular stereo matching method based on a horizontally deformable attention module, which includes:

[0008] S1. The left view of the binocular stereo matching method is input with the features of the left view extracted by the first ResNet backbone network, and the right view of the binocular stereo matching method is input with the features of the right view extracted by the second ResNet backbone network;

[0009] S2: The features output by the first ResNet backbone network are input into the first horizontal deformable attention module for further feature processing, and the features output by the second ResNet backbone network are input into the second horizontal deformable attention module for further feature processing;

[0010] S3: The features processed by the first and second level deformable attention modules are concatenated to form a matching cost volume, and then the final disparity map is obtained through 3D convolutional disparity regression.

[0011] Among them, the first horizontal deformable attention module and the second horizontal deformable attention module are the same horizontal deformable attention modules, both including a horizontal attention mechanism and a deformable convolution module.

[0012] (3) Beneficial effects

[0013] The present invention proposes a binocular stereo matching method based on a horizontal deformable attention module. Compared with the existing technology, the method proposed in the present invention constructs a horizontal deformable attention module, in which the horizontal attention mechanism introduces epipolar constraints, better learns the similarity of pixel features in the horizontal direction of the left and right views, and can improve the similarity matching degree of the same pixel in the left and right views. At the same time, the deformable convolution can better utilize contextual feature information and optimize the texture of the disparity map, thereby improving the quality of the disparity map. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 This is an architecture diagram of the binocular stereo matching method based on the horizontal deformable attention module of the present invention (the dotted box is a detailed structural diagram of the horizontal deformable attention module of the present invention);

[0015] Figure 2 Detailed structural diagram of the horizontal deformable attention module of the present invention;

[0016] Figure 3 A comparison chart of the effects of the horizontal attention mechanism. DETAILED DESCRIPTION

[0017] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below with reference to the accompanying drawings and examples.

[0018] In order to better adapt to the characteristics of stereo matching tasks, the purpose of this invention is to propose a stereo matching method based on a horizontally deformable local attention module, which can effectively capture global correspondence clues in the horizontal direction and adjust the distribution of contextual information in discontinuous areas of disparity through deformable convolution, thereby improving the accuracy and robustness of stereo matching.

[0019] Figure 1 This is a neural network architecture diagram for a binocular stereo matching method based on a horizontally deformable attention module. The horizontally deformable attention module primarily processes the features of the left and right views extracted by the ResNet backbone network, performs further feature processing, and forms a matching cost volume through feature concatenation. The final disparity map is then obtained through three-dimensional convolutional disparity regression. The binocular stereo matching method based on a horizontally deformable attention module of the present invention includes the following steps:

[0020] S1. The left view of the binocular stereo matching method is input with the features of the left view extracted by the first ResNet backbone network, and the right view of the binocular stereo matching method is input with the features of the right view extracted by the second ResNet backbone network;

[0021] S2: The features output by the first ResNet backbone network are input into the first horizontal deformable attention module for further feature processing, and the features output by the second ResNet backbone network are input into the second horizontal deformable attention module for further feature processing;

[0022] S3. The features processed by the first level deformable attention module and the second level deformable attention module are concatenated to form a matching cost volume, and then the final disparity map is obtained through three-dimensional convolutional disparity regression.

[0023] Among them, the first horizontal deformable attention module and the second horizontal deformable attention module are the same horizontal deformable attention modules, both including a horizontal attention mechanism and a deformable convolution module.

[0024] The first ResNet backbone network and the second ResNet backbone network are the same ResNet backbone network.

[0025] in, Figure 1 In the neural network architecture, parameters are shared between the first and second ResNet backbone networks that process the left and right views, and between the first and second horizontal deformable attention modules that process the left and right view features. Sharing ensures that the parameters of the two shared neural network modules are consistent during training or inference, and is a strategy used in neural network training.

[0026] The horizontal deformable attention module is the content proposed by this invention. Figure 2 Detailed structural diagram of the horizontal deformable attention module.

[0027] The horizontal attention mechanism focuses only on the local features of pixels in the same row as the central pixel. It can use the epipolar constraint to capture more features that are effective for feature matching. The following describes the process of adaptively aggregating horizontal spatial depth context features.

[0028] Figure 2 The left part of the horizontal attention mechanism represents the input unary feature as X∈R C×H×W Where C, H and W are the number of channels, spatial height and spatial width respectively. First, three convolutional layers f_query, f_key and f_value with 1×1 convolution kernels are used to convolve X to obtain Q∈R C′×H×W , K∈R C′×H×W and V∈R C ×H×W , where C' = C / 2. The convolution kernel parameters of f_query and f_key are shared, and the number of channels of the output features is reduced from 320 to 128, reducing the consumption of calculation and memory. Then the three output features are processed separately, and Q is flattened and transposed to obtain Where N = H × W. Matrix transformation and resampling of K and V are performed to obtain and and After calculating the correlation matrix and processing it with softmax, we get the attention map Y∈R N×W , expressed as:

[0029]

[0030]

[0031] where y J,I express At position I and The correlation of the features at the Jth position in the image, where n = N × W. In stereo matching, the more similar the features are, the more credible the disparity between the two spatial points is. The aggregation of similar depth features can achieve mutual gain, and then the attention map and value features are used to calculate the correlation of the features at the Jth position in the image. Perform correlation matrix operations between them to obtain the result Y'∈R after re-matrix dimension change C×H×W Finally, the context feature information Y' and the input unary feature X are summed to obtain the final output A∈R of the feature processing stage C×H×W , expressed as

[0032] A=sum(αY',X)

[0033] α is a learnable parameter used to adjust the weight of the attention mechanism. Directly adding the attention features to the input features can achieve better performance. The attention mechanism tends to capture global contextual information and is more effective for depth consistency in textureless areas and single objects. Although the horizontal attention mechanism focuses only on horizontal features, the feature X extracted by the ResNet backbone network has a more reliable global receptive field and can provide sufficient global contextual information.

[0034] Although the attention mechanism can improve the boundary processing effect of objects in the disparity map to a certain extent, such as Figure 3 As shown, after using the horizontal attention mechanism ( Figure 3 The third row) relatively does not use the horizontal attention mechanism ( Figure 3 The disparity map of the second row has more obvious boundaries, but the disparity is not accurate enough. Therefore, we consider using deformable convolution to process the horizontal attention mechanism features to improve the object boundary processing effect in the disparity map. Figure 2 The middle right part describes the deformation convolution module processing of feature A after the horizontal attention mechanism. By learning the spatial convolution offset, the deformation convolution module can adaptively sample feature information at a depth position similar to the center pixel. The deformation convolution module in this method consists of a 1×1 two-dimensional convolution, a 3×3 deformable convolution, and a 1×1 two-dimensional convolution, which can be expressed as:

[0035] A'=conv2d(deformconv2d(conv2d(A)))

[0036] Where conv2d represents a 1×1 two-dimensional convolution, deformconv2d represents a 3×3 deformable convolution, and A' is the feature after processing by the horizontal deformable attention module. Figure 3 The fourth line shows the effect of using the horizontal deformable attention module. Compared with the method of using only the horizontal attention mechanism, the accuracy of the object boundary disparity is further improved.

[0037] The horizontal deformable attention module of the combination of the horizontal attention mechanism and the deformable convolution module, the position of the horizontal deformable attention module in the stereo matching network, and the timing of processing all fall within the scope of protection of the present invention.

[0038] Example 1:

[0039] Ablation experiments have verified that both horizontal attention and deformable convolution can improve the quality of generated disparity maps. To verify the effectiveness of this method, a comparative experiment was conducted. While maintaining the remaining components of the deep learning-based binocular stereo matching algorithm network structure, horizontal attention and deformable convolution were added to the feature extraction process. Four methods were used: without horizontal attention and without deformable convolution, with horizontal attention and without deformable convolution, with deformable convolution and without horizontal attention, and with deformable convolution and horizontal attention. The experimental results are shown in Table 1. This experiment used a GeForce RTX2080Ti GPU, an input image resolution of 256×512, and was tested on the SceneFlow dataset.

[0040] Table 1 Experimental comparison results

[0041]

[0042]

[0043] Compared with existing technologies, the method proposed in this invention constructs a horizontal deformable attention module, in which the horizontal attention mechanism introduces epipolar constraints to better learn the similarity of pixel features in the horizontal direction of the left and right views, and can improve the similarity matching degree of the same pixel in the left and right views. At the same time, the deformable convolution can better utilize contextual feature information to optimize the texture of the disparity map, thereby improving the quality of the disparity map.

[0044] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A binocular stereo matching method based on a horizontally deformable attention module, characterized in that: The method includes: S1. The left view of the binocular stereo matching method is input with the features of the left view extracted by the first ResNet backbone network, and the right view of the binocular stereo matching method is input with the features of the right view extracted by the second ResNet backbone network; S2: The features output by the first ResNet backbone network are input into the first horizontal deformable attention module for further feature processing, and the features output by the second ResNet backbone network are input into the second horizontal deformable attention module for further feature processing; S3: The features processed by the first and second level deformable attention modules are concatenated to form a matching cost volume, and then the final disparity map is obtained through 3D convolutional disparity regression. The first horizontal deformable attention module and the second horizontal deformable attention module are the same horizontal deformable attention modules, both including a horizontal attention mechanism and a deformable convolution module; The horizontal attention mechanism specifically includes: The unary feature of the input is represented as X∈R C×H×W Where C, H and W are the number of channels, space height and space width respectively; First, three convolutional layers f_query, f_key, and f_value with 1×1 convolution kernels are used to convolve X to obtain Q∈R C'×H×W , K∈R C'×H×W and V∈R C×H×W , where C' = C / 2; Then, the three output features are processed separately, and Q is flattened and transposed to obtain Where N = H × W; K and V are transformed into matrices and resampled to obtain and and After calculating the correlation matrix and processing it with softmax, we get the attention map Y∈R N×W , expressed as: where y J,I express At position I and The correlation of the features at the J-th position in , where n = N × W. In stereo matching, the more similar the features are, the more credible the disparity between the two spatial points is. The aggregation of similar depth features achieves mutual gain. In Attention Map and Value Features Perform correlation matrix operations between them to obtain the result Y'∈R after re-matrix dimension change C×H×W Finally, the context feature information Y' and the input unary feature X are summed to obtain the final output A∈R of the feature processing stage C×H×W , expressed as: A=sum(αY',X) Among them, α is a learnable parameter used to adjust the operation weight of the attention mechanism.

2. The binocular stereo matching method based on the horizontal deformable attention module according to claim 1, characterized in that: The first ResNet backbone network and the second ResNet backbone network are the same ResNet backbone network.

3. The binocular stereo matching method based on the horizontal deformable attention module according to claim 1, characterized in that: Parameters are shared between the first ResNet backbone network and the second ResNet backbone network.

4. The binocular stereo matching method based on the horizontal deformable attention module according to claim 1, characterized in that Parameters are shared between the first level deformable attention module and the second level deformable attention module.

5. The binocular stereo matching method based on the horizontal deformable attention module according to claim 1, characterized in that: The convolution kernel parameters of f_query and f_key are shared, and the number of channels of the output features is reduced from 320 to 128.

6. The binocular stereo matching method based on the horizontal deformable attention module according to claim 1, characterized in that: The input unary feature X is the feature extracted by the ResNet backbone network, which has a more reliable global receptive field and can provide sufficient global context information.

7. The binocular stereo matching method based on the horizontal deformable attention module according to claim 1, characterized in that: The horizontal attention mechanism focuses on the local features of pixels in the same row as the central pixel and uses epipolar constraints to capture more features that are effective for feature matching.

8. The binocular stereo matching method based on the horizontal deformable attention module according to claim 1, characterized in that: The deformable convolution module consists of a 1×1 two-dimensional convolution, a 3×3 deformable convolution, and a 1×1 two-dimensional convolution, expressed as, A'=conv2d(deformconv2d(conv2d(A))) Where conv2d represents a 1×1 two-dimensional convolution, deformconv2d represents a 3×3 deformable convolution, and A' is the feature after processing by the horizontal deformable attention module.

9. The binocular stereo matching method based on the horizontal deformable attention module according to claim 8, characterized in that: The feature A after the horizontal attention mechanism is processed by the deformation convolution module. By learning the spatial convolution offset, the deformation convolution module can adaptively sample feature information at a depth position similar to the center pixel.

Images