A binocular stereo matching method

Abstract

The present invention provides a binocular stereo matching method, comprising the following steps: inputting a left image and a right image, downsampling by adding two weight-sharing stacked twin convolutional neural networks with a concentrated epipolar direction attention mechanism, obtaining a left feature map with a resolution lower than that of the left image and a right feature map with a resolution lower than that of the right image; constructing a cost body using the left feature map and the right feature map, inputting the cost body into a three-dimensional convolution for feature extraction, and obtaining a disparity map of the initial image of the binocular stereo system; guiding and optimizing the disparity map of the initial image according to the constructed guided network of learning weights, upsampling back to the disparity map of the original resolution, and obtaining a depth image of the initial image of the binocular stereo system. By improving GA‑Net, the calculation speed is improved while ensuring a small loss of accuracy; using a low-resolution image to input the convolutional neural network, the calculation cost is saved and the matching speed is improved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a multi-branch binocular stereo matching method focusing on epipolar direction attention. Background Art

[0002] In recent years, with the gradual deepening of research on 3D technology, the perception of 3D fields such as autonomous driving, 3D reconstruction, virtual reality and other fields have gradually become popular. The existing ways to obtain depth or point cloud data include radar and depth camera, but the expensive cost of radar is limited to a certain extent. The human brain can infer the 3D information of surrounding objects based on the horizontal difference between the left and right eye imaging. Binocular cameras came into being with the support of bionics, simulating the human binocular system to measure distance. At the same time, compared with radar, it has many advantages. With the advantages of simple configuration, wide application, reliable results, and larger range, it has become a popular research direction. Among them, binocular stereo matching is the most important link. The accuracy of stereo matching determines the performance and efficiency of the entire binocular system.

[0003] Binocular stereo matching uses two cameras on the same horizontal line to obtain the left and right images at the same time. After accurate epipolar correction, the horizontal position difference between the corresponding pixels between the stereo image pairs can be estimated, that is, the disparity. Since there is a definite corresponding calculation relationship between disparity and depth as shown in the formula, a highly accurate disparity map can be converted into a highly accurate depth map.

[0004]

[0005] Among them, Z is the corresponding depth, B is the optical center distance of the binocular camera, f is the focal length, and d is the parallax.

[0006] Traditional stereo matching algorithms usually include four independent steps: cost calculation, cost aggregation, disparity calculation, and disparity optimization. According to the constraint range of the algorithm, they can be divided into global stereo matching algorithms and local stereo matching algorithms. Among them, classic algorithms such as SGM and AD-Census have been widely used.

[0007] With the development of convolutional neural networks, people used convolutional neural networks for stereo matching for the first time in 2015, and achieved remarkable matching results, taking an important step towards intelligence. Since then, binocular stereo matching methods based on deep learning have made continuous breakthroughs, but there are also many difficult problems to be studied. For example, the classic GA-Net combines deep learning with traditional methods, and uses Semi-Global Aggregation (SGA) and Local Guided Aggregation (LGA) two aggregation branches to learn weights for the aggregation module and optimize the disparity map, achieving good results. However, due to the multi-layer 3D convolutional network structure, the running speed is slow, the computational efficiency is low, and time resources are consumed, which causes certain difficulties in the transplantation of the algorithm. Summary of the invention

[0008] In order to solve the defects in the prior art, the present invention proposes a binocular stereo matching method. The attention module used by different functional networks in the prior art uses convolution to further extract the overall or regional features of the two-dimensional image, usually a fusion extraction of the features of the image globally or within a window. However, for the binocular stereo matching method, the epipolar direction is the priority focus, and the final disparity map is directly related to the disparity d in the epipolar direction. Improving the feature extraction capability in the epipolar direction can further improve the stereo matching effect. Therefore, the method of the present invention improves GA-Net and achieves a substantial increase in speed while ensuring a small loss of accuracy. To achieve the above purpose, the present invention adopts the following specific technical solutions:

[0009] A binocular stereo matching method, comprising:

[0010] S1. Input the left image and the right image, and downsample them through two stacked twin convolutional neural networks with shared weights, each of which adds a concentrated epipolar direction attention mechanism, to obtain a left feature map with a resolution lower than that of the left image and a right feature map with a resolution lower than that of the right image;

[0011] S2. Use the left feature map and the right feature map to construct a cost volume, input the cost volume into a three-dimensional convolution for feature extraction, and obtain a disparity map of the initial image of the binocular stereo system;

[0012] S3. According to the constructed guided network of learning weights, the disparity map of the initial image of the binocular stereo system is guided and optimized, and the disparity map is upsampled back to the original resolution to obtain the depth image of the initial image of the binocular stereo system.

[0013] Preferably, before step S1, step S0 is also included:

[0014] The initial images of the binocular stereo system are respectively subjected to epipolar correction to obtain a left image and a right image.

[0015] Preferably, the epipolar direction attention mechanism includes an attention branch focused on the width direction and a self-reinforced attention branch;

[0016] By focusing on the attention branch in the width direction, the left image is convolved one by one to extract the features of the width of the left image;

[0017] The right image is convolved one by one through its own enhanced attention branch to extract features of the width of the right image;

[0018] The extracted width features of the left image and the right image are added and fused to obtain the depth features of the initial image of the binocular system in the concentrated epipolar direction;

[0019] The deep features are used to guide the weight assignment of stacked Siamese convolutional neural networks that share weights.

[0020] Preferably, the weight-shared stacked twin convolutional neural network is a stacked hourglass-shaped feature extraction network designed based on the feature extraction network of GA-Net.

[0021] Preferably, the guidance network for learning weights includes edge branches, SGA branches and LGA branches.

[0022] Preferably, the cost volume is a four-dimensional cost volume.

[0023] Preferably, step S3 comprises the following steps:

[0024] S301, performing a first disparity aggregation on the disparity map of the initial image of the binocular stereo system by using the SGA branch to obtain a first disparity map;

[0025] S302, performing a second disparity aggregation on the first disparity map by using the LGA branch to obtain a second disparity map;

[0026] S303 : Upsample the second disparity map by using the edge branch to obtain a third disparity map of the original resolution.

[0027] The present invention can achieve the following technical effects:

[0028] 1. By improving GA-Net, the computing speed is improved while ensuring a small loss of accuracy.

[0029] 2. Using low-resolution images to input convolutional neural networks saves computing costs and improves matching speed.

[0030] 3. Use a multi-branch regression network to guide and optimize the disparity results, increase detailed information, and improve matching effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 is a flow chart of a binocular stereo matching method according to an embodiment of the present invention;

[0032] Figure 2 This is a diagram showing the effect of concentrating attention in the epipolar direction according to an embodiment of the present invention;

[0033] Figure 3 is a network diagram of a concentrated epipolar direction attention mechanism according to an embodiment of the present invention;

[0034] Figure 4 This is an aggregation effect diagram of SGA semi-global matching according to an embodiment of the present invention;

[0035] Figure 5 This is a diagram showing the effect of local guide aggregation of an LGA according to an embodiment of the present invention;

[0036] Figure 6 It is a flow chart of a binocular stereo matching method according to another embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0038] The purpose of the present invention is to provide a binocular stereo matching method, which can effectively improve the performance of binocular matching by adding a concentrated epipolar direction attention mechanism and obtain better matching results; at the same time, a low-resolution image is input into a convolutional neural network, which greatly saves the computing cost and significantly improves the matching speed; a multi-branch regression network is used to guide and optimize the disparity results, increase rich detail information, and improve the effect. A binocular stereo matching method provided by the present invention will be described in detail below through a specific embodiment.

[0039] refer to Figure 1 and Figure 6 The flowchart of the binocular stereo matching method shown in FIG. 1 is a binocular stereo matching method of the present invention, which comprises the following steps:

[0040] S0. Perform epipolar correction on the two initial image pairs of the binocular stereo system to obtain an epipolar-corrected image pair, namely a left image and a right image.

[0041] S1. The left image and the right image obtained in the input step S0 are downsampled by two stacked twin convolutional neural networks with shared weights, both of which add a concentrated epipolar direction attention mechanism, to obtain a left feature map with a resolution lower than that of the left image and a right feature map with a resolution lower than that of the right image.

[0042] The two weight-sharing stacked twin convolutional neural networks in the present invention are stacked hourglass-shaped feature extraction networks designed based on the feature extraction network of GA-Net. The two weight-sharing stacked twin convolutional neural networks process the left image and the right image respectively, and downsample the left image and the right image to 1 / 8 resolution of the initial image of the binocular stereo system, so as to reduce the amount of data input to the three-dimensional convolution, thereby greatly improving the speed;

[0043] The binocular stereo matching network is unique compared to other deep learning networks, that is, the vertical epipolar directions of the left and right image pairs are consistent in the initial input, and there is only an offset in the epipolar direction with respect to the disparity d, which is also the desired result of stereo matching.

[0044] The attention module used in other existing networks with different functions uses convolution to further extract the overall features or regional features of the two-dimensional image, which is a fusion extraction of the features of the entire image or within a window. However, for the binocular stereo matching method, the epipolar direction is the priority focus, and the final disparity map is directly related to the disparity d in the epipolar direction. By improving the feature extraction capability in the epipolar direction, the stereo matching effect can be further improved. According to the uniqueness of binocular stereo matching, the target corresponding points of the left and right image pairs should be concentrated in the epipolar direction, so the present invention adds an attention mechanism network that concentrates on the epipolar direction.

[0045] In a preferred embodiment of the present invention, Figure 2 As shown in the diagram of the effect of attention in the concentrated extreme direction, the depth of the color indicates the degree of influence on this point. For example, taking the darkest point in the middle as the center point, the closer it is to it, the higher the weight that should be assigned.

[0046] Therefore, for a feature map, after further feature extraction by focusing attention on the epipolar direction, the features in the epipolar direction are further deeply extracted, and the details in the epipolar direction are better learned, thereby enhancing the effect of binocular matching.

[0047] Figure 3 This is a network diagram of the mechanism for concentrating attention in the direction of the polar line, where C represents the channel, W represents the width, H represents the height, W_attention represents the attention in the direction of the width, and K_attention represents the self-enhanced attention.

[0048] By focusing on the width direction of the attention branch W_attention, the left image is convolved one by one to extract the features of the left image's width; by adding the enhanced attention branch K_attention, the right image is convolved one by one to extract deep features of the right image's width;

[0049] Finally, the deep features extracted from the left image and the deep features extracted from the right image are additively fused in a manner similar to residual addition to obtain the deep features in the epipolar direction. The obtained deep feature information is used to guide the weight allocation of the stacked twin convolutional neural network with shared weights, which can effectively improve the matching effect.

[0050] S2. Use the left feature map and the right feature map to construct a cost volume, input the cost volume into the three-dimensional convolution for feature extraction, and obtain the disparity map of the initial image.

[0051] In a preferred embodiment of the present invention, a four-dimensional cost body is constructed using the left feature map and the right feature map output by a stacked twin network with shared weights, and the four-dimensional cost body is input into a three-dimensional convolutional neural network to extract features to obtain a disparity map of the initial image of the binocular stereo system.

[0052] S3. According to the constructed guided network of learning weights, the disparity map of the initial image of the binocular stereo system is guided and optimized, and the disparity map is upsampled back to the original resolution to obtain the depth image of the initial image of the binocular stereo system.

[0053] In a preferred embodiment of the present invention, a guided network for learning weights is constructed according to the left image, and the guided network for learning weights includes an edge branch, an SGA branch in a GA-Net, and an LGA branch in a GA-Net. Specifically:

[0054] S301, using the SGA branch to perform the first disparity aggregation on the disparity map of the initial image of the binocular stereo system obtained in step S2, by associating the matching cost of each pixel with its pixels in the horizontal and vertical directions to improve the matching effect, and obtain a first disparity map;

[0055] Reference Figure 4 , using the SGA branch network in GA-Ne, based on the traditional SGM semi-global matching method, the feature map is semi-globally aggregated along four directions to further obtain the first disparity map after the first aggregation.

[0056] S302, performing a second disparity aggregation on the first disparity map by using the LGA branch, improving the matching effect by associating the matching cost of each pixel with the pixels in its local window, and obtaining a second disparity map;

[0057] Reference Figure 5, using the LGA branch network in GA-Ne, along the Figure 5 The method shown is used to perform aggregation within a local window, aiming to refine thin structures and target edges in the image, and further obtain a second disparity map after a second aggregation.

[0058] S303: using an edge-guided network, upsampling the second disparity map to obtain a third disparity map of original resolution.

[0059] Using edge branches to guide aggregation, the second disparity map is guided by the edges in the original color image, which can further divide the boundaries of different objects, refine the edges, reduce blur, and improve the matching effect;

[0060] At the same time, the edge-guided network can be used to guide the edge upsampling of the low-resolution second disparity map to obtain the final high-resolution third disparity map. The relationship between vision and depth can be used to obtain the characteristic depth map of the binocular stereo system.

[0061] In another preferred embodiment of the present invention, after multi-branch optimization by the above method, the low-resolution feature depth map is resampled to the original image size to obtain a high-precision depth image corresponding to the initial image, thereby ensuring a certain effect while saving computational costs and improving speed.

[0062] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representations of the above terms do not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine the different embodiments or examples described in this specification and the features of the different embodiments or examples, without contradiction.

[0063] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

[0064] The above specific implementations of the present invention do not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A binocular stereo matching method, characterized in that: The following steps are involved: S1. Input a left image and a right image, and downsample them respectively through two weight-shared stacked twin convolutional neural networks, both of which add a concentrated epipolar direction attention mechanism, to obtain a left feature map with a resolution lower than that of the left image and a right feature map with a resolution lower than that of the right image; The epipolar direction attention mechanism includes an attention branch focused on the width direction and a self-enhanced attention branch; The weight-sharing stacked twin convolutional neural network is a stacked hourglass-shaped feature extraction network designed based on the feature extraction network of GA-Net; In the concentrated epipolar direction attention mechanism, the points closer to the center point are assigned higher weights; Performing convolution operations on the left image one by one through the attention branch focused on the width direction, and extracting features of the width of the left image; Performing convolution operations on the right image one by one through the self-enhanced attention branch, and extracting features of the width of the right image; Additively fusing the extracted width features of the left image and the right image to obtain a depth feature in the epipolar direction of an initial image of a binocular stereo system; The deep features are used to guide the weight allocation of the stacked twin convolutional neural network with shared weights; S2, constructing a cost volume using the left feature map and the right feature map, inputting the cost volume into a three-dimensional convolution for feature extraction, and obtaining a disparity map of an initial image of a binocular stereo system; S3, according to the constructed guided network of learning weights, guiding and optimizing the disparity map of the initial image of the binocular stereo system, upsampling the disparity map back to the original resolution, and obtaining the depth image of the initial image of the binocular stereo system; The guided network of the learning weights includes an edge branch, an SGA branch and an LGA branch.

2. The binocular stereo matching method according to claim 1, characterized in that: Before step S1, step S0 is also included: The initial images of the binocular stereo system are respectively subjected to epipolar correction to obtain the left image and the right image.

3. The binocular stereo matching method according to claim 1, characterized in that: The cost volume is a four-dimensional cost volume.

4. The binocular stereo matching method according to claim 1, characterized in that: Step S3 includes the following steps: S301, performing a first disparity aggregation on the disparity map of the initial image of the binocular stereo system by using the SGA branch to obtain a first disparity map; S302, performing a second disparity aggregation on the first disparity map by using the LGA branch to obtain a second disparity map; S303 . Upsample the second disparity map by using the edge branch to obtain a third disparity map of original resolution.

Images