Stereo matching method based on dual attention mechanism and position coding constraint

Abstract

The application provides a stereo matching method based on a double attention mechanism and position coding constraint, relates to the field of computer vision, and comprises the following steps: (1) feature extraction, performing feature extraction on left and right views to obtain a feature map with a channel C e , a spatial resolution consistent with an original input scale (I h , I w ); (2) feature matching, alternately calculating by using self-attention and cross-attention to establish the correlation between left and right views; (3) position coding, establishing the correlation between adjacent points for points in a weak texture region to improve the matching accuracy in the weak texture region; (4) attention mask, covering the regions that do not need to participate in the operation in the matching process to reduce the operation amount; (5) uniqueness constraint, making the points on the two images in the matching process one-to-one corresponding to improve the matching accuracy; and (6) disparity estimation, predicting the disparity of the input image according to the calculation results of the previous steps.

Description

TECHNICAL FIELD

[0001] The application relates to a disparity estimation method for a stereoscopic image pair, in particular to a stereoscopic matching method with a double attention mechanism and position coding constraint, and belongs to the field of computer vision. BACKGROUND

[0002] At present, active sensing technology and passive sensing technology are used in the field of machine vision to obtain image depth information. Active sensing technology includes time of flight, structured light and laser radar, but due to problems such as resolution, sensing distance, light source influence and high cost, there are certain limitations. In contrast, binocular stereo vision in passive sensing technology has the advantages of low cost, easy implementation and easy expansion, and is widely used in the fields of unmanned driving, augmented reality and intelligent robots, and has great application prospects and social significance.

[0003] Binocular stereo vision is an important branch of machine vision, and since the mid-1960s, the technology has been widely researched and applied. With the continuous development of computer hardware and algorithms, binocular stereo vision is widely used in the fields of computer vision, autonomous driving, remote sensing and mapping, defect detection, three-dimensional reconstruction, etc. The basic principle of the technology is similar to the human eye, which obtains image information of an object at different viewing angles through the difference between the left and right eyes, uses the triangulation method to obtain the position coordinates of a point on a two-dimensional image in three-dimensional space, and obtains the three-dimensional information of the reconstructed target through related three-dimensional reconstruction technology.

[0004] A complete binocular stereo vision system is composed of multiple parts, including image acquisition, camera calibration, feature extraction, stereo matching, depth determination and reconstruction, etc. Among them, the core step of binocular stereo vision is stereo matching, which has an important influence on the final result. When stereo matching is performed, the left and right images need to be corrected, and a matching search algorithm is used to find corresponding pixel points and calculate left and right disparity values. The difficulty of this method lies in the matching of left and right camera pictures, and the matching accuracy will directly affect the imaging effect of the final algorithm. In theory, this method can accurately restore the depth information, but in actual application, its accuracy is often restricted by the shooting conditions: due to the complexity of the real scene, the image may be affected by light and produce noise points, or due to the characteristics of object occlusion, non-textured areas and weak textured areas, the matching may be wrong, thereby affecting the disparity accuracy. SUMMARY

[0005] Therefore, the application provides a stereo matching method based on a double attention mechanism and position coding constraint.

[0006] To achieve the above object, the application adopts the following technical scheme:

[0007] A stereo matching method based on a double attention mechanism and position coding constraint comprises the following steps:

[0008] (1) Image feature extraction:

[0009] Given an image pair, which is used as a source image and a target image respectively, a sandglass-shaped encoder-decoder architecture is adopted to extract multi-scale image features, the encoder is composed of five residual blocks, followed by an SPP module, the decoding part uses transpose convolution, dense-blocks and the last convolution output layer, and the obtained feature map is the same size as the original image;

[0010] (2) Feature matching:

[0011] An alternating attention mechanism is adopted: the self-attention calculates the attention between the pixels along the epipolar line in the same image, while the cross-attention calculates the attention of the pixels on the corresponding epipolar line in the left and right images; the self-attention and the cross-attention are alternately calculated in N-1 layers; this alternating scheme constantly updates the feature descriptors according to the image context and the relative position; in the last cross-attention layer, the most focused pixels are used to estimate the disparity;

[0012] (3) Position coding:

[0013] The adaptability to weak texture areas is optimized, and position coding for relative position modeling is introduced;

[0014] (4) Attention mask:

[0015] A binary mask is introduced in the last cross-attention layer, each pixel in the left image only needs to focus on the pixels in the right image on the left side of the same coordinate x_L, which reduces the calculation amount in the matching process;

[0016] (5) Uniqueness constraint:

[0017] A uniqueness constraint is added to the stereo matching process, that is, each pixel in the right image is assigned to at most one pixel in the left image; an optimal transport scheme with entropy regularization is adopted to establish the uniqueness constraint between the left and right views;

[0018] (6) Disparity estimation:

[0019] Predicting the disparity of the input image according to the calculation result of the previous step.

[0020] Preferably, the step (6) uses a modified winner-takes-all method to estimate the disparity, by finding the most likely matching position (denoted as k) from the optimal transport assignment matrix T, and constructing a 3px window N_3(k) around it; the matching probabilities in the 3px window are re-normalized so that their sum is 1; the weighted sum of the candidate disparities is the estimated disparity d(k).

[0021] Preferably, the specific method of the step (1) is: using a sandglass-shaped encoder-decoder architecture to extract multi-scale image features; the encoder consists of five residual blocks, followed by an SPP module to achieve more efficient global context information collection; the decoding part uses transpose convolution, dense-blocks and the last convolution output layer; the extracted features are channel C_e, and the spatial resolution is consistent with the original input scale (I_h, I_w) feature map.

[0022] Preferably, the specific method of the step (2) is: using multi-head attention, grouping operation in the channel dimension of the feature map, enhancing the expression ability of the feature by dividing the channel dimension of the feature descriptor C_e into N_h groups, normalizing the similarity by softmax, obtaining the weighted combination factor a_h, combining the groups divided before, obtaining the output value vector, then adding the output value vector V_o and the original feature descriptor to obtain the enhanced feature descriptor, forming the residual connection.

[0023] Preferably, the specific method of the step (3) is: providing data-related spatial information through position encoding e_p, compared with absolute pixel position, relative pixel distance has translational invariance, so the invention selects relative position encoding.

[0024] Preferably, the specific method of the step (4) is: for a given cost matrix M, the lengths of its two edge distributions a and b are I_w, and the optimal transport is achieved by solving the best coupling matrix T.

[0025] Preferably, the specific method of step (5) is: let x_L and x_R be the projection positions of the same physical point on the left and right views, and take x as positive from left to right; the spatial arrangement of the binocular camera in the stereo matching system ensures that after calibration, all points should satisfy x_R<=x_L; therefore, in the last cross-attention layer, each pixel in the left image only needs to pay attention to the pixel on the left side of the same coordinate x_L in the right image, that is, only to the point x in the right image that satisfies x<=x_L.

[0026] Preferably, the specific method of step (6) is: calculate the disparity by finding the most likely matching position (denoted as k) from the optimal transmission distribution matrix T, and construct a 3px window N_3(k) around it, and re-normalize the matching probability in the 3px window so that the sum is 1, and the weighted sum of the candidate disparity is the estimated disparity d(k).

[0027] Compared with the prior art, the present application has obvious advantages and beneficial effects, specifically, from the above technical solution, it can be known that:

[0028] The present application mainly proposes a stereo matching method based on double attention mechanism and position coding constraint, which can mainly solve the following problems:

[0029] 1) A large number of stereo matching networks establish the correlation between left and right views by establishing a matching cost volume, but are easily affected by image noise, and when the image is in a weak texture or repetitive texture area, the cost value is very likely to fail to accurately reflect the correlation between pixels. The present application proposes a stereo matching method based on double attention mechanism and position coding constraint, which enhances the matching accuracy of the network in the weak texture area;

[0030] 2) Most stereo matching networks almost only use multiple three-dimensional convolution operations to construct a matching cost volume and perform disparity regression operations. In order to ensure the network accuracy, excessive three-dimensional convolution operations make the parameter quantity in the network very large, thereby limiting the operation speed of the network in the stereo matching task. The present application replaces the matching cost construction process with dense pixel sequence matching to establish the correlation between left and right views, reduces a large number of three-dimensional convolution operations, and improves the running speed while ensuring the accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 The flowchart of the binocular stereo matching method of the embodiment of the present application;

[0032] Figure 2 The feature extractor structure of the embodiment of the present application;

[0033] Figure 3 The double attention mechanism model of the embodiment of the present application;

[0034] Figure 4 Attention mask diagram for an embodiment of the present application;

[0035] Figure 5 Input image for an embodiment of the present application;

[0036] Figure 6 Disparity estimation result for an embodiment of the present application. DETAILED DESCRIPTION

[0037] In order to make the objects, technical solutions and advantages of the present application clearer, further detailed description will be made to the present application in combination with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application, and do not constitute limitation to the present application.

[0038] The embodiment of the present application discloses a stereo matching method based on double attention mechanism and position coding constraint, which uses self-attention and cross-attention to construct parallel Vision Transformer, realizes bidirectional calculation, improves the efficiency and flexibility of network learning features, and effectively uses position coding to establish the position constraint relationship between adjacent pixels, thereby improving the network matching ability and running speed. The stereo matching method based on double attention mechanism and position coding constraint provided by the present application will be described in detail through specific embodiments.

[0039] Reference Figure 1 The flowchart of the binocular stereo matching method, the stereo matching method based on double attention mechanism and position coding constraint of the present application includes the following steps:

[0040] Step 1: image feature extraction:

[0041] The present application selects a sandglass-shaped encoder-decoder architecture to extract multi-scale image features. The encoder is composed of five residual blocks, followed by an SPP module to realize more efficient global context information collection. The decoding part uses transpose convolution, dense-blocks and the last convolution output layer. The extracted features are channel C e , spatial resolution consistent with the original input scale (I h ,I w ) feature map.

[0042] Step 2: feature matching:

[0043] The present application adopts multi-head attention in the feature matching process, which groups the feature map in the channel dimension, and divides the channel dimension of the feature descriptor C e into Nh The ability of the group to enhance the expression of features, for the partition of the group is described as: wherein C h is the channel dimension of each head, N h is the number of heads. Thus, each head can have a different representation, and similarity can be computed for each head. For each attention head h, a set of linear projections is used to use the feature descriptor e I as input, to compute the query vector Q h , the key vector K h , and the value vector V h .

[0044]

[0045]

[0046]

[0047] wherein,

[0048] The similarity is then normalized by softmax to get the weighted combination factor a h :

[0049]

[0050] The output value vector can be calculated as follows by combining the previously partitioned groups:

[0051] V o = W o Concat(a1V1,..., a h V h )+bo (5)

[0052] wherein, The output value vector Vo is then added to the original feature descriptor to obtain the enhanced feature descriptor, forming a residual connection:

[0053] e I = e I + V o (6)

[0054] The Q h , K h , V h needed for all operations in the self-attention part of the calculation process are all features generated from the same view.

[0055] The Qh features from the source image generation, K h ,V h features from the target image generation. In the process of calculating cross-attention, the source and target are relative, that is, they are exchanged to realize bidirectional calculation.

[0056] Step 3: Position encoding:

[0057] In order to further improve the accuracy of the stereo matching network in the occluded area and the weak texture area, the application proposes to establish the connection between adjacent points for the points in the weak texture area through position encoding, so as to optimize the adaptability to the weak texture area.

[0058] The multi-layer attention process in step 2 constructs the association between pixels and pixels, but in a large area of weak texture and no texture area, the association between pixels begins to be chaotic and wrong. In order to solve this problem, the application uses relative position information to establish the connection between these points and adjacent points, especially the connection between edge points with significant features, so as to improve the adaptability of the network in these areas. Through position encoding e p provides spatial information related to data. Compared with absolute pixel position, relative pixel distance has translational invariance, so the application selects relative position encoding. Position encoding e p is directly added to the feature descriptor:

[0059] e=e I +e p At this time, the attention between the i-th and j-th pixels in formula (4) can be expanded as:

[0060]

[0061] As shown in formula (7), the fourth term in the formula depends entirely on the position, and the disparity is basically dependent on the image content, so the relative position encoding is used and the fourth term can be omitted:

[0062]

[0063] Wherein, e p ,i-j represents the position encoding (e p ,i-j≠e p ,j-i) between the i-th pixel and the j-th pixel. From formula (8), we can intuitively see that the attention mechanism of the application depends entirely on the similarity of image content and the relative distance between pixels.

[0064] Step 4: Attention mask:

[0065] Let xL and x R is the projection position of the same physical point on the left and right views (taking x as positive from left to right). The spatial arrangement of the binocular cameras in the stereo matching system ensures that after calibration, all points should satisfy x R ≤ x L . Therefore, in the final cross-attention layer, each pixel in the left image only needs to focus on the pixel in the right image at the same coordinate x L left (i.e., only focus on the point x that satisfies x≤x L in the right image), as shown in FIG. 19. Figure 4 To achieve this restriction, the present application introduces a binary mask for attention.

[0066] Step 5: Uniqueness constraint:

[0067] For a given cost matrix M, the lengths of its two edge distributions a and b are Iw. The present application achieves optimal transport with entropy regularization by solving the optimal coupling matrix

[0068]

[0069] where, is the entropy regularization, M is the cost matrix of edge distributions a and b, and its length is I w If the edge distributions a and b are uniform, then is the optimal solution to the matching problem. The soft uniqueness constraint is implemented while reducing ambiguity. Formula (9) is solved by the iterative Sinkhorn algorithm. The value in formula (9) represents the probability of pairwise matching, similar to the softmax attention in formula (4). In the embodiment of the present application, the cost matrix M is set to be the negative value of the attention calculated by the cross-attention module in formula (4), but without softmax, because the optimal transport will normalize the attention values.

[0070] Step 6: Disparity estimation:

[0071] The present application uses an improved winner-takes-all method to regress disparity. The disparity is calculated by finding the most likely matching position (denoted as k) from the optimal transport assignment matrix and constructing a 3px window N3(k) around it. The matching probabilities within the 3px window are re-normalized so that their sum is 1. The weighted sum of the candidate disparities is the estimated disparity The matching probability t in the matching matrix is denoted as t, then we have:

[0072] ​

[0073]

[0074] The method of the application utilizes self-attention and cross-attention to construct a parallel Vision Transformer, realizes bidirectional calculation, improves the efficiency and flexibility of network learning features, and effectively utilizes position encoding to establish the position constraint relationship between adjacent pixels, thereby improving the network matching ability and running speed. Compared with other related stereo matching methods, the method proposed in the application achieves higher precision.

[0075] The technical principles of the application are described above in combination with specific embodiments. These descriptions are only for the purpose of explaining the principles of the application and cannot be interpreted in any way as a limitation on the scope of protection of the application. Based on the explanations herein, those skilled in the art can conceive other specific embodiments of the application without creative labor, and these embodiments will fall within the scope of protection of the application.

Claims

1. A stereo matching method based on a dual attention mechanism and position coding constraint, characterized in that, Comprising the following steps: (1) Image feature extraction: Given a pair of images, respectively as source and target images, a sandglass-shaped encoder-decoder architecture is adopted to extract multi-scale image features, the encoder consists of five residual blocks, followed by an SPP module, the decoding part uses transpose convolution, dense-blocks and the final convolution output layer, the resulting feature map is the same size as the original image; (2) Feature matching: An alternating attention mechanism is adopted: self-attention computes the attention between pixels along the epipolar line in the same image, while cross-attention computes the attention between pixels on the corresponding epipolar line in the left and right images; self-attention and cross-attention are alternately calculated in N-1 layers; this alternating scheme constantly updates the feature descriptors according to the image context and relative position; in the last cross-attention layer, the most attended pixels are used to estimate the disparity; (3) Position encoding: Optimize the adaptability to weak texture areas, introduce position encoding for relative position modeling; (4) Attention mask: In the last cross-attention layer, a binary mask is introduced, each pixel in the left image only needs to focus on the pixel in the right image at the same coordinate x L The left pixel, reducing the computational complexity in the matching process; (5) Uniqueness constraint: Add a uniqueness constraint to the stereo matching process, that is, each pixel in the right image is assigned to at most one pixel in the left image; an optimal transport scheme with entropy regularization is used to establish the uniqueness constraint between left and right views; (6) Disparity estimation: Predict the input image disparity according to the calculation results of the previous steps.

2. The stereo matching method based on dual attention mechanism and position encoding constraint according to claim 1, characterized in that: The step (6) uses a modified winner-takes-all method to estimate the disparity, by finding the most likely matching position from the optimal transmission allocation matrix The position k is denoted as the most likely matching position, and a 3px window N3(k) is constructed around it; the matching probabilities within the 3px window are renormalized so that their sum is 1; The weighted sum of the candidate disparities is the estimated disparity 3. The stereo matching method based on dual attention mechanism and position encoding constraint according to claim 1, characterized in that: The specific method of step (1) is: a sandglass-shaped encoder-decoder architecture is used to extract multi-scale image features; the encoder is composed of five residual blocks, followed by an SPP module to realize more efficient global context information collection; the decoding part uses transpose convolution, dense-blocks and the last convolution output layer; the features extracted through the above two processes are channel C e , the spatial resolution of the feature map is consistent with the original input scale (I h ,I w ).

4. The stereo matching method based on dual attention mechanism and position encoding constraint according to claim 1, characterized in that: The specific method of the step (2) is: using multi-head attention, grouping the feature map in the channel dimension, dividing the channel dimension of the feature descriptor C e into N h groups to enhance the expression ability of the feature, normalizing the similarity through softmax to obtain a weighted combination factor a h , combining the previously divided groups to obtain an output value vector, and then adding the output value vector V o to the original feature descriptor to obtain the enhanced feature descriptor, forming a residual connection.

5. The stereo matching method based on dual attention mechanism and position encoding constraint according to claim 1, characterized in that: The specific method of the step (3) is: through position coding e p The data-related spatial information has translational invariance compared with absolute pixel position, relative pixel distance.

6. The stereo matching method based on dual attention mechanism and position encoding constraint according to claim 1, characterized in that: The specific method of the step (4) is: for a given cost matrix M, the lengths of two edge distributions a and b are I w Optimal transport is achieved by solving the optimal coupling matrix .

7. The stereo matching method based on dual attention mechanism and position encoding constraint according to claim 1, characterized in that: The specific method of step (5) is as follows: Let x L and x R This refers to the projected position of the same physical point on the left and right views, with x taken as positive from left to right; the spatial arrangement of the binocular cameras in the stereo matching system ensures that, after calibration, all points should satisfy x. R ≤x L Therefore, in the final cross-attention layer, each pixel in the left image only needs to focus on the same x-coordinate in the right image. L The pixels on the left are those that satisfy x ≤ x in the right graph. L Point x.

8. The stereo matching method based on dual attention mechanism and position encoding constraint according to claim 1, characterized in that: The specific method of the step (6) is: finding the most possible matching position k from the optimal transmission distribution matrix The disparity is calculated by taking the most possible matching position k from the optimal transmission distribution matrix

Images