A binocular stereo matching method integrating gradient cues and multi-scale information

Abstract

The present invention provides a binocular stereo matching method that fuses gradient cues and multi-scale information. The method performs multi-scale feature extraction on the left and right images to obtain multi-scale left and right convolution feature maps, and calculates left and right gradient feature maps. The left and right convolution feature maps of the same resolution are fused with the left and right gradient feature maps to obtain multi-scale left and right fused feature maps. A splicing operation is used to construct a spliced ​​cost volume using the left and right fused feature maps on each disparity plane. A group correlation method is used to construct a group-correlated cost volume using the left and right convolution feature maps on each disparity plane. The spliced ​​cost volume and the group-correlated cost volume of the same resolution are spliced ​​to obtain a multi-scale combined cost volume. The combined cost volumes at different scales are fused to obtain a fused cost volume, which is then upsampled to obtain a final cost volume with the same resolution as the left and right images. A soft argmin operation is used to output the disparity map. The present invention improves the matching accuracy in weak texture areas and fine structure areas.

Description

Technical field

[0002] The present invention relates to the field of computer vision technology, and in particular to a binocular stereo matching method that fuses gradient clues and multi-scale information. [Background Technology]

[0004] Binocular stereo matching draws on the 3D imaging mechanism of human vision. It calculates disparity by finding matching pixel pairs in binocular images, and then obtains scene depth information corresponding to the registered pixels, aiming to reconstruct the scene's 3D structure. Because stereo vision requires no additional imaging equipment and is adaptable to a wide range of environments, it is widely used in advanced driver assistance, mobile robot positioning and navigation, virtual reality, and other fields.

[0005] Binocular stereo matching methods can be divided into traditional stereo matching methods and deep learning-based stereo matching methods. Traditional stereo matching methods rely on manual image feature point matching, making it difficult to obtain high-precision 3D information from dense scenes. Deep learning-based stereo matching methods have powerful human-like data knowledge learning capabilities, improving the ability to express image features and significantly improving stereo matching accuracy. However, the mismatch rate in areas with weak textures and subtle structures is still high.

[0006] Therefore, it is necessary to provide a binocular stereo matching method that integrates gradient cues and multi-scale information to solve the above problems. [Summary of the invention]

[0008] The technical problem to be solved by the present invention is to provide a binocular stereo matching method that fuses gradient cues and multi-scale information. By constructing a multi-scale cost volume containing gradient cues and achieving full fusion of cost volumes between different scales, a cost volume that fuses gradient cues and multi-scale information is obtained, thereby improving the matching accuracy in weak texture areas and fine structure areas.

[0009] To achieve the above object, the technical solution of the present invention is:

[0010] A binocular stereo matching method integrating gradient clues and multi-scale information comprises the following steps:

[0011] S1: Get the left and right images of the left and right cameras;

[0012] S2: Construct a binocular stereo matching network, use the left and right images as inputs of the binocular stereo matching network, perform multi-scale feature extraction on each image, and obtain multi-scale left and right convolution feature maps; pass the left and right convolution feature maps through multiple convolution layers, nonlinear layers, and batch normalization layers in sequence, and then calculate the gradient corresponding to each feature point to obtain multi-scale left and right gradient feature maps; fuse the left and right convolution feature maps with the left and right gradient feature maps of the same resolution to obtain multi-scale left and right fusion feature maps;

[0013] S3: Use the splicing operation to construct a spliced ​​cost volume using the left and right fused feature maps on each disparity plane; use the group correlation method to construct a group-related cost volume using the left and right convolution feature maps on each disparity plane; perform a splicing operation on the spliced ​​cost volume and the group-related cost volume of the same resolution to obtain a multi-scale combined cost volume;

[0014] S4: Fusing the combined cost volumes at different scales to obtain a fused cost volume, and upsampling the fused cost volume to obtain a final cost volume with the same resolution as the left and right images;

[0015] S5: Use soft argmin operation to output disparity map;

[0016] The construction process of the splicing cost body is expressed as follows:

[0017]

[0018] Where, Represents the concatenation operator, Indicates parallax, and Represent the left and right fusion feature maps respectively;

[0019] The construction process of the group-related cost body is expressed as:

[0020]

[0021] Where g represents the group number, Represents the total number of channels of the left and right convolution feature maps, Indicates the total number of groups, 、 Represent the left and right convolution feature maps of the g-th group respectively, represents the inner product operation;

[0022] The construction process of the combined cost body is expressed as follows:

[0023] ;

[0024] The multi-scale combined cost volume is expressed as: 、 、 , the dimensions are: 、 、 ,in, B is the batch size, is the number of channels, is the maximum parallax range;

[0025] The step S4 is specifically as follows:

[0026] S41: Combined cost body 、 、 Perform regularization;

[0027] S42: regularized cost body Downsample to a resolution of , and then with the regularized cost volume Perform splicing operation and merge the two;

[0028] S43: Downsample the cost volume fused in step S42 to a resolution of , and then with the regularized cost volume Perform splicing operation and merge the two;

[0029] S44: Upsample the cost volume fused in step S43 to a resolution of , get the fusion cost body , the dimensions are: ;

[0030] S45: Fusion cost body Upsample to a resolution of Then perform trilinear upsampling to a resolution of , and get the final cost body .

[0031] Preferably, the step S1 is specifically as follows:

[0032] Calibrate the left and right cameras, use the calibrated left and right cameras to capture images, perform distortion correction and epipolar alignment on the images, and obtain a resolution of left and right images.

[0033] Preferably, the step S2 is specifically as follows:

[0034] S21: Construct a binocular stereo matching network, use the left and right images as the input of the binocular stereo matching network, perform multi-scale feature extraction, and obtain 、 、 Left and right convolution feature maps of resolution;

[0035] S22: Pass the left and right convolution feature maps through multiple convolution layers, nonlinear layers, and batch normalization layers in sequence, and then calculate the gradient corresponding to each feature point to obtain 、 、 The left and right gradient feature maps of the resolution are shown in Figure 2, where the gradient calculation process is expressed as:

[0036]

[0037] Indicates that it is located at the feature point The gradient characteristics at represents the gradient direction; Represents the convolution feature map after multiple convolutional layers, nonlinear layers, and batch normalization layers. Indicates the channel, Indicates the total number of channels, Indicates along The direction of the neighboring pixels relative to the feature point The offset, Indicates absolute value;

[0038] S23: The left and right convolution feature maps and the left and right gradient feature maps with the same resolution are fused in turn through multiple convolution layers, nonlinear layers, and batch normalization layers to obtain 、 、 Left and right fused feature maps of different resolutions.

[0039] Preferably, the disparity map output by the soft argmin operation is:

[0040]

[0041] Where, represents the softmax operation along the disparity dimension, Represents the output disparity map.

[0042] Preferably, the method further comprises the following steps:

[0043] Training the binocular stereo matching network. During the training phase, the binocular stereo matching network outputs three levels of disparity maps and uses different training weights for full supervised learning: the first layer is composed of the combined cost volume after regularization in step S41. Output, output disparity map with gradient clues, aiming to retain detail features; the second layer is composed of the fused cost volume in step S44 Output, output disparity map with multi-scale information and gradient clues; the third layer is composed of the final cost volume in step S45 Output, the output learns a disparity map with finer and higher frequency features; the network is trained end-to-end using a smooth L1 loss function, and the loss function of the binocular stereo matching network is expressed as:

[0044]

[0045] Where, Lrepresents the loss function, Indicates the i The weight coefficient of the disparity map output by the layer, represents the true disparity value, represents the L1 loss function, which is defined as follows:

[0046]

[0047] Where, Represents the input variable of the L1 loss function.

[0048] Compared to related technologies, the present invention constructs a multi-scale cost volume containing gradient cues and fully integrates the cost volumes at different scales, resulting in a cost volume that integrates gradient cues and multi-scale information. Multi-scale information facilitates stereo matching in weakly textured areas, but in areas with fine structure, coarse scale information inevitably leads to erroneous matching points. The introduction of gradient cues explicitly extracts structural information in three-dimensional space, improving feature recognition and matching accuracy in areas with fine structure. Therefore, the binocular stereo matching method proposed in the present invention, which integrates gradient cues and multi-scale information, improves matching accuracy in both weakly textured and finely structured areas.

Brief Description of the Drawings

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can also derive other drawings based on these drawings without inventive work, among which:

[0051] Figure 1 A flowchart showing a binocular stereo matching method that integrates gradient cues and multi-scale information provided by the present invention;

[0052] Figure 2 The figure shows the effect of the method provided by the present invention on the KITTI 2012 test set. [Specific implementation method]

[0054] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present invention and to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are further described below in conjunction with the drawings of the present application.

[0055] See also Figure 1-2 The present invention provides a binocular stereo matching method that integrates gradient clues and multi-scale information, comprising the following steps:

[0056] S1: Get the left and right images from the left and right cameras.

[0057] The step S1 specifically includes: calibrating the left and right cameras, taking images with the calibrated left and right cameras respectively, and performing distortion correction and epipolar alignment on the images to obtain a resolution of left and right images.

[0058] The calibration process uses conventional techniques in this field. The calibration can obtain the intrinsic and extrinsic parameters of the left and right cameras, distortion parameters, and the rotational and translational position relationship between the left and right cameras.

[0059] S2: Construct a binocular stereo matching network, use the left and right images as inputs of the binocular stereo matching network, perform multi-scale feature extraction on each image, and obtain multi-scale left and right convolution feature maps; pass the left and right convolution feature maps through multiple convolution layers, nonlinear layers, and batch normalization layers in sequence, and then calculate the gradient corresponding to each feature point to obtain multi-scale left and right gradient feature maps; fuse the left and right convolution feature maps with the left and right gradient feature maps of the same resolution to obtain multi-scale left and right fusion feature maps;

[0060] The step S2 is specifically as follows:

[0061] S21: Construct a binocular stereo matching network, use the left and right images as the input of the binocular stereo matching network, perform multi-scale feature extraction, and obtain 、 、 Left and right convolution feature maps of resolution;

[0062] S22: Pass the left and right convolution feature maps through multiple convolution layers, nonlinear layers, and batch normalization layers in sequence, and then calculate the gradient corresponding to each feature point to obtain 、 、 The left and right gradient feature maps of the resolution are shown in Figure 2, where the gradient calculation process is expressed as:

[0063]

[0064] Indicates that it is located at the feature point The gradient characteristics at represents the gradient direction; Represents the feature map after multiple convolutional layers, nonlinear layers, and batch normalization layers. Indicates the channel, Indicates the total number of channels, Indicates along The direction of the neighboring pixels relative to the feature point The offset, Indicates absolute value.

[0065] S23: The left and right convolution feature maps and the left and right gradient feature maps with the same resolution are fused in turn through multiple convolution layers, nonlinear layers, and batch normalization layers to obtain 、 、 Left and right fused feature maps of different resolutions.

[0066] S3: Use the splicing operation to construct a spliced ​​cost volume using the left and right fused feature maps on each disparity plane; use the group correlation method to construct a group-related cost volume using the left and right convolution feature maps on each disparity plane; perform splicing operations on the spliced ​​cost volume and the group-related cost volume of the same resolution to obtain a multi-scale combined cost volume.

[0067] The construction process of the splicing cost body is expressed as follows:

[0068]

[0069] Where, Represents the concatenation operator, Indicates parallax, and Represent the left and right fusion feature maps respectively;

[0070] The construction process of the group-related cost body is expressed as:

[0071]

[0072] Where g represents the group number, Represents the total number of channels of the left and right convolution feature maps, Indicates the total number of groups, 、 Represent the left and right convolution feature maps of the g-th group respectively, represents the inner product operation;

[0073] The construction process of the combined cost body is expressed as follows:

[0074] ;

[0075] The multi-scale combined cost volume is expressed as: 、 、 , the dimensions are: 、 、 ,in, B is the batch size, is the number of channels, is the maximum parallax range.

[0076] Since the left and right fused feature maps contain gradient features, the resulting combined cost volume also contains gradient clues.

[0077] S4: Fusing the combined cost volumes at different scales to obtain a fused cost volume, and upsampling the fused cost volume to obtain a final cost volume with the same resolution as the left and right images;

[0078] The step S4 is specifically as follows:

[0079] S41: Combined cost body 、 、 Perform regularization;

[0080] S42: regularized cost body Downsample to a resolution of , and then with the regularized cost volume Perform splicing operation and merge the two;

[0081] S43: Downsample the cost volume fused in step S42 to a resolution of , and then with the regularized cost volume Perform splicing operation and merge the two;

[0082] S44: Upsample the cost volume fused in step S43 to a resolution of , get the fusion cost body , the dimensions are: ;

[0083] S45: Fusion cost body Upsample to a resolution of Then perform trilinear upsampling to a resolution of , and get the final cost body .

[0084] In step S45, the upsampling process is performed by the CUBG cost volume upsampling module proposed in Bilateral Grid Learning for Stereo Matching Networks. Under the guidance of the guided graph learned from the feature extraction module, the fusion cost volume is Upsample to 1 / 2 of the input image resolution, that is, the resolution is .

[0085] Fusion Cost Volume It also contains multi-scale information and gradient clues, which can better implement stereo matching in low-texture areas and fine-structure areas.

[0086] S5: Use soft argmin operation to output the disparity map.

[0087] The output disparity map is:

[0088]

[0089] Where, represents the softmax operation along the disparity dimension, Represents the output disparity map.

[0090] Furthermore, the binocular stereo matching method provided by the present invention that integrates gradient cues and multi-scale information further includes the following steps:

[0091] Training the binocular stereo matching network. During the training phase, the binocular stereo matching network outputs three levels of disparity maps and uses different training weights for full supervised learning: the first layer is composed of the combined cost volume after regularization in step S41. Output, output disparity map with gradient clues, aiming to retain detail features; the second layer is composed of the fused cost volume in step S44 Output, output disparity map with multi-scale information and gradient clues; the third layer is composed of the final cost volume in step S45 Output, the output learns a disparity map with finer and higher frequency features; the network is trained end-to-end using a smooth L1 loss function, and the loss function of the binocular stereo matching network is expressed as:

[0092]

[0093] Where, L represents the loss function, Indicates the i The weight coefficient of the disparity map output by the layer, represents the true disparity value, represents the L1 loss function, which is defined as follows:

[0094]

[0095] Where, Represents the input variable of the L1 loss function.

[0096] The method provided by the present invention is tested on the KITTI 2012 test set, and the obtained effect diagram is as follows: Figure 2 As shown; DispNetC, GCNet, iResNet-i2, PSMNet, and SegStereo were tested on the same test set, and the comparison results are shown in Table 1:

[0097] Table 1 Test results of multiple methods on the KITTI 2012 test set

[0098]

[0099] The sources of the above methods are:

[0100] DispNetC:《A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation》

[0101] GCNet: "End-to-End Learning of Geometry and Context for Deep StereoRegression"

[0102] iResNet-i2:《Learning for disparity estimation through featureconstancy》

[0103] PSMNet: "Pyramid Stereo Matching Network"

[0104] SegStereo: "SegStereo: Exploiting Semantic Information for DisparityEstimation".

[0105] As shown in Table 1, the method provided by the present invention achieves a lower percentage of pixel error. For unobstructed areas, the average pixel error is roughly the same as that of the prior art method, but it is even lower across the entire area. This demonstrates that the method provided by the present invention improves feature recognition and enhances matching accuracy in areas with subtle structures.

[0106] Compared to related technologies, the present invention constructs a multi-scale cost volume containing gradient cues and fully integrates the cost volumes at different scales, resulting in a cost volume that integrates gradient cues and multi-scale information. Multi-scale information facilitates stereo matching in weakly textured areas, but in areas with fine structure, coarse scale information inevitably leads to erroneous matching points. The introduction of gradient cues explicitly extracts structural information in three-dimensional space, improving feature recognition and matching accuracy in areas with fine structure. Therefore, the binocular stereo matching method proposed in the present invention, which integrates gradient cues and multi-scale information, improves matching accuracy in both weakly textured and finely structured areas.

[0107] The embodiments of the present invention are described in detail above, but the present invention is not limited to the described embodiments. It is apparent to those skilled in the art that various changes, modifications, substitutions, and variations made to these embodiments without departing from the principles and spirit of the present invention are still within the scope of protection of the present invention.

Claims

1. A binocular stereo matching method that integrates gradient cues and multi-scale information, characterized in that: The steps include: S1: Get the left and right images of the left and right cameras; S2: Construct a binocular stereo matching network, use the left and right images as inputs of the binocular stereo matching network, perform multi-scale feature extraction on each image, and obtain multi-scale left and right convolution feature maps; pass the left and right convolution feature maps through multiple convolution layers, nonlinear layers, and batch normalization layers in sequence, and then calculate the gradient corresponding to each feature point to obtain multi-scale left and right gradient feature maps; fuse the left and right convolution feature maps with the left and right gradient feature maps of the same resolution to obtain multi-scale left and right fusion feature maps; S3: Use the splicing operation to construct a spliced ​​cost volume using the left and right fused feature maps on each disparity plane; use the group correlation method to construct a group-related cost volume using the left and right convolution feature maps on each disparity plane; perform a splicing operation on the spliced ​​cost volume and the group-related cost volume of the same resolution to obtain a multi-scale combined cost volume; S4: Fusing the combined cost volumes at different scales to obtain a fused cost volume, and upsampling the fused cost volume to obtain a final cost volume with the same resolution as the left and right images; S5: Use soft argmin operation to output disparity map; The construction process of the splicing cost body is expressed as follows: Where, Represents the concatenation operator, Indicates parallax, and Represent the left and right fusion feature maps respectively; The construction process of the group-related cost body is expressed as: Where g represents the group number, Represents the total number of channels of the left and right convolution feature maps, Indicates the total number of groups, 、 Represent the left and right convolution feature maps of the g-th group respectively, represents the inner product operation; The construction process of the combined cost body is expressed as follows: ； The multi-scale combined cost volume is expressed as: 、 、 , the dimensions are: 、 、 ,in, B is the batch size, is the number of channels, is the maximum parallax range; The step S4 is specifically as follows: S41: Combined cost body 、 、 Perform regularization; S42: regularized cost body Downsample to a resolution of , and then with the regularized cost volume Perform splicing operation and merge the two; S43: Downsample the cost volume fused in step S42 to a resolution of , and then with the regularized cost volume Perform splicing operation and merge the two; S44: Upsample the cost volume fused in step S43 to a resolution of , get the fusion cost body , the dimensions are: ; S45: Fusion cost body Upsample to a resolution of Then perform trilinear upsampling to a resolution of , and get the final cost body .

2. The binocular stereo matching method for fusing gradient clues and multi-scale information according to claim 1, characterized in that: The step S1 is specifically as follows: Calibrate the left and right cameras, use the calibrated left and right cameras to capture images, perform distortion correction and epipolar alignment on the images, and obtain a resolution of left and right images.

3. The binocular stereo matching method for fusing gradient clues and multi-scale information according to claim 2, characterized in that: The step S2 is specifically as follows: S21: Construct a binocular stereo matching network, use the left and right images as the input of the binocular stereo matching network, perform multi-scale feature extraction, and obtain 、 、 Left and right convolution feature maps of resolution; S22: Pass the left and right convolution feature maps through multiple convolution layers, nonlinear layers, and batch normalization layers in sequence, and then calculate the gradient corresponding to each feature point to obtain 、 、 The left and right gradient feature maps of the resolution are shown in Figure 2, where the gradient calculation process is expressed as: Indicates that it is located at the feature point The gradient characteristics at represents the gradient direction; Represents the convolution feature map after multiple convolutional layers, nonlinear layers, and batch normalization layers. Indicates the channel, Indicates the total number of channels, Indicates along The direction of the neighboring pixels relative to the feature point The offset, Indicates absolute value; S23: The left and right convolution feature maps and the left and right gradient feature maps with the same resolution are fused in turn through multiple convolution layers, nonlinear layers, and batch normalization layers to obtain 、 、 Left and right fused feature maps of different resolutions.

4. The binocular stereo matching method for fusing gradient cues and multi-scale information according to claim 1, characterized in that: The disparity map output by the soft argmin operation is: Where, represents the softmax operation along the disparity dimension, Represents the output disparity map.

5. The binocular stereo matching method for fusing gradient cues and multi-scale information according to claim 4, characterized in that: The following steps are also included: Training the binocular stereo matching network. During the training phase, the binocular stereo matching network outputs three levels of disparity maps and uses different training weights for full supervised learning: the first layer is composed of the combined cost volume after regularization in step S41. Output: Output disparity map with gradient clues, aiming to preserve detail features; The second layer is composed of the fused cost volume in step S44 Output, output disparity map with multi-scale information and gradient clues; the third layer is composed of the final cost volume in step S45 Output, the output learns a disparity map with finer and higher frequency features; The network is trained end-to-end using a smooth L1 loss function. The loss function of the binocular stereo matching network is expressed as: Where, L represents the loss function, Indicates the i The weight coefficient of the disparity map output by the layer, represents the true disparity value, represents the L1 loss function, which is defined as follows: Where, Represents the input variable of the L1 loss function.

Images