Stereo matching method based on BGNet improvement

Abstract

The invention discloses a stereo matching method based on BGNet improvement, and the method comprises the steps: enriching channel information through employing a sub-pixel channel information enhancement module in a feature extraction network, processing cross-scale features through employing a channel attention guiding module, and obtaining a multi-scale feature map; constructing and fusing a multi-scale cost body by using grouping correlation, and performing cost aggregation on the low-resolution cost body; capturing multi-scale information in the image by using hole convolution of different sizes for feature maps of different hierarchies, and fusing to generate a guide map containing comprehensive information; under the guidance of the guide graph, a learnable bilateral grid algorithm is used for carrying out up-sampling on the cost body, parallax regression is used for obtaining an initial parallax image, a reconstruction error is calculated in a feature space, a residual error relative to the initial parallax image is obtained through a parallax refining module, and the sum of a residual error result and the initial parallax error forms a final parallax value; the methods are beneficial to enhancing the precision, the detail recovery capability and the robustness of the whole stereo matching process.

Description

Technical Field

[0001] The present invention relates to the field of computer binocular vision and stereo matching, and in particular to an improved stereo matching method based on BGNet. Background Art

[0002] Stereo matching calculates the disparity between pixels in a pair of rectified left and right images for depth estimation, and is a key technology in the fields of drone navigation, 3D reconstruction, and autonomous driving. Compared with tools such as LiDAR, it has lower costs, but there are still many problems in estimating depth through binocular vision, such as poor matching results in areas with weak textures, high reflections, and occlusions.

[0003] At present, the performance of binocular stereo matching algorithms based on deep learning has been greatly improved. In 2015, Zbontar et al. designed two network structures for image block matching, namely the fast structure MC-CNN-Fast and the accurate structure MC-CNN-Acc. Focusing on the design of more complex network structures to obtain rich semantic information, the accuracy of stereo matching has been effectively improved. Mayer et al. first proposed an end-to-end stereo matching network DispNet, which directly estimates the disparity map from the binocular image. A large number of studies have been carried out based on this structure and achieved good results. PSMNet uses a spatial pyramid pooling module when extracting features and uses a stacked hourglass structure to complete cost aggregation. CFNet uses the method of fusing cascaded cost volumes to improve the robustness of the matching algorithm. By estimating the uncertainty of different stages in the cascade, the disparity search range of the next stage is adaptively adjusted to improve the accuracy. Xu et al. designed a cost space upsampling module based on bilateral grid learning and proposed a real-time network BGNet, which saves a lot of time cost by directly upsampling the low-resolution cost volume and maintains considerable accuracy.

[0004] At present, the stereo matching algorithm based on deep learning has already surpassed the traditional method in terms of accuracy. With the maturity of mobile GPU computing platform and the continuous improvement of computing power, it has become possible to implement it on the edge, which puts forward higher requirements for the lightweight model. At the same time, how to improve the performance of the network, enhance the network's ability to understand the scene, and improve the accuracy of network matching under the premise of meeting the real-time requirements has become a problem that needs to be solved. Summary of the invention

[0005] The purpose of the present invention is to provide a stereo matching method based on an improved BGNet to address the existing problems of stereo matching. The feature extraction network used in the present invention incorporates more semantic information to enhance the understanding of objects, and uses context information to enrich the cost volume. At the same time, in the construction of the guidance map, the aggregation of feature maps at different levels more effectively guides the generation of high-quality cost volumes. In addition, this paper adds a disparity refinement module to correct errors in misestimation and occluded areas.

[0006] In order to achieve the above technical effects, the technical solution of the present invention is as follows:

[0007] The improved stereo matching method based on BGNet includes the following steps:

[0008] Step 1: Input the original left and right image pair into the feature extraction network, and reduce the image resolution to 1 / 8 of the original through convolution. Design a channel enhancement network to improve the expressiveness of features, use the sub-pixel channel information enhancement module to enrich channel information, and use the channel attention guidance module to process cross-scale features, and obtain multi-scale feature maps at the same time;

[0009] Step 2: Construct and fuse multi-scale cost volumes. For the feature maps with resolutions of 1 / 8, 1 / 16, and 1 / 32 of the original image obtained in step 1, use the group correlation method to construct a multi-scale cost volume, and fuse them to generate a cost volume with a size of D / 8×H / 8×W / 8×16;

[0010] Step 3: Cost aggregation is performed on the cost volume obtained in step 2, and a 3D hourglass structure is used to regularize the cost volume. Different sizes of dilated convolutions are used to generate guidance maps containing comprehensive information for the feature maps of different levels with 1 / 2 resolution obtained in the feature extraction part. Under the guidance of the guidance map, the cost volume is upsampled using a learnable bilateral grid algorithm to obtain a high-resolution cost volume;

[0011] Step 4: Use disparity regression to get the initial disparity map for the cost volume obtained in step 3, reconstruct the error in the feature space, get the residual relative to the initial disparity map through the disparity refinement module, calculate the sum of the residual result and the initial disparity, and generate the disparity map of the final stereo image pair. The overall framework is as follows: Figure 1 shown.

[0012] Furthermore, in step 1, after reducing the image resolution to 1 / 8 through convolution, the image is further downsampled, and a similar U-Net structure is used to finally obtain a 1 / 8 resolution feature map. In this structure, we use sub-pixel convolution instead of the original upsampling operation. For the feature map of size 1 / 32H×1 / 32W×C, 3×3 convolution is used to extract local information, and then sub-pixel convolution is used for upsampling. By shuffling pixels to increase the width and height, the shape H×W×C·r 2 The characteristics are rearranged into rH×rW×C, and the formula is defined as follows

[0013] P(F) x,y,c =F x / r,y / r,C·r·mod(y,r)+C·mod(x,r)+c

[0014] Where r represents the magnification factor, F represents the input feature, and P(F) x,y,c Represents the feature pixel output at the coordinate (x, y, c), using the sub-pixel context enhancement module on the highest level 1 / 32H×1 / 32W×C feature map.

[0015] This module is used to generate the integral image I. First, 3×3 convolution is used to extract local information, and then sub-pixel convolution is used for upsampling. Next, global average pooling is performed to obtain global context information, 1×1 convolution is used to compress the features, and then local information and global context information are fused by element-by-element summation.

[0016] Furthermore, the channel attention guidance module is used to process cross-scale features, and global average pooling and global maximum pooling are used for the integral image I. After being processed by the fully connected layer, the elements are summed up and the output features are merged using the sigmoid function. The process can be expressed as:

[0017] CA(z)=σ(Fc1(AvgPool(z))+Fc2(MaxPool(z))),

[0018] R i =CA(I)⊙P i

[0019] Among them, CA(z) is the channel attention function, σ is the sigmoid function, and i represents different scale levels;

[0020] The input and output feature maps are processed by the attention module, and the feature extraction networks of the left and right images share parameters. At the same time, the features of the left image at 1 / 8, 1 / 16, and 1 / 32 resolutions in the network are retained for cost volume construction.

[0021] Furthermore, in step 2, the feature maps of the three scales obtained in step 1 are used to construct group-related cost volumes. The formula of the group-related cost volume is as follows:

[0022]

[0023] Among them, i represents three different scales, <,> represents the inner product, d is the disparity value, and g is the group number. Represents the number of channels in the feature layer, N g represents the number of groups of partitioned features;

[0024] Use two 3D convolutions with skip connections to regularize the three cost volumes, and use the 3D convolution to fuse the three cost volumes using the encoder-decoder structure. The size of the fused cost volume is 1 / 8H×1 / 8W×1 / 8D×16, and the value of D is 192. The resolution is the original Figure 1 The cost volume of / 8 uses a simple 3D hourglass network to complete cost aggregation and obtain a low-resolution cost volume of size 1 / 8H×1 / 8W×1 / 8D×32.

[0025] Furthermore, the specific process of constructing the guide graph in step 3 is as follows:

[0026] The three levels of feature maps with 1 / 2 resolution obtained in the feature extraction part are denoted as F1, F2, and F3, and their number of channels is 32. For these three feature maps, three groups of convolution results are generated using dilated convolutions with different dilation rates. Then, these three groups of feature maps are interleaved and convolved to establish the dependency relationship between the three groups. The three groups of features are calculated using the weights of adaptive learning. The formula is as follows:

[0027]

[0028] where i = 1, 2, 3, w ij It is the weight of different feature maps, and is adaptively learned during the training process. Finally, the guide map G is generated through three feature maps;

[0029] G=Conv 1×1 (concat{F1,F2,F3})

[0030] Using two 1×1 convolutions, the obtained guidance map G is of size 1 / 2H × 1 / 2W × 1. Under the guidance of the guidance map, a learnable bilateral grid algorithm is used to upsample the cost volume to obtain a high-resolution cost volume.

[0031] Furthermore, the learnable bilateral grid algorithm in step 3 uses the slicing operation of the bilateral grid processing to upsample the cost volume at 1 / 8 resolution to high resolution. The bilateral grid has a total of 4 dimensions, width x, height y, disparity d and guide feature g, whose value can be expressed as B(x, y, d, g), created using a convolutional neural network. Under the guidance of the guide graph G, the four-dimensional bilateral grid is linearly interpolated, and the slicing operation can be described as:

[0032] C H (x,y,d)=B(sx,sy,sd,s G G(x,y)

[0033] Among them, s, s G ∈(0,1), s is the ratio of the width or height of the bilateral grid to the high-resolution cost volume, s G is the ratio of the grayscale of the grid to the grayscale of the guide map, C H (x,y,d) is the generated 3D high-resolution cost volume.

[0034] Furthermore, the specific execution process of step 4 is as follows:

[0035] First, we regress the initial disparity map from the high-resolution cost volume obtained in step 3. We predict the initial disparity using the softargmin method:

[0036]

[0037] Secondly, after obtaining the initial disparity, the reconstruction error is calculated in the feature space to measure the correctness of the initial disparity. The reconstructed error, the initial disparity map and the left image features are used as inputs, and the disparity refinement module is used to calculate the residual D relative to the initial disparity. r (x, y), add the residual and the initial disparity to get the final disparity map D of the stereo image pair f (x,y).

[0038] Finally, the smooth L1 loss function is used to train the entire network. The loss function is defined as follows:

[0039]

[0040] in,

[0041]

[0042] D gt (p) is the ground truth disparity of pixel p.

[0043] Compared with the prior art, the present invention has the following advantages:

[0044] 1. The present invention uses sub-pixel channel information enhancement in the feature extraction module, and uses high-level feature-rich channel information to enhance its representation ability. The channel attention guidance module is used to optimize the integration function at each scale, and multi-scale feature maps are used to generate cost volumes of different resolutions, and the cost volumes are fused to obtain context information.

[0045] 2. The present invention designs a guidance graph of multi-level group interaction, uses adaptive learning weights to increase the focus on important features, and assists the cost body in completing high-quality upsampling.

[0046] 3. The present invention adds a disparity refinement module to identify incorrect areas based on the initial disparity reconstruction error, which can better restore the detail information in the image and generate a more accurate disparity map. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a structural framework diagram of a stereo matching network based on the improved BGNet of the present invention;

[0048] Figure 2 A schematic diagram of the overall channel enhancement network structure in an embodiment of the present invention;

[0049] Figure 3 This is a structural diagram of a multi-scale cost volume fusion module in an embodiment of the present invention;

[0050] Figure 4 A schematic diagram of a guide diagram for multi-level group interaction in an embodiment of the present invention;

[0051] Figure 5 This is a comparison chart of the disparity results of the stereo matching algorithm of the present invention and the original model. Specific implementation plan

[0052] To make the technical solutions and advantages of the present invention more clear, the technical solutions will be fully described below in conjunction with the accompanying drawings in the embodiments of the present invention, but the present invention is not limited to the embodiments.

[0053] like Figure 1 As shown, the improved stereo matching method based on BGNet in this embodiment includes the following steps:

[0054] Step 1: Build a feature extraction network, input the original left and right image pairs, and reduce the image resolution to 1 / 8 of the original through convolution. Design a channel enhancement network to improve the expressiveness of features, use the sub-pixel channel information enhancement module to enrich channel information, and use the channel attention guidance module to process cross-scale features, and obtain multi-scale feature maps for subsequent construction of multi-scale cost volumes.

[0055] Step 2: Construct and fuse multi-scale cost volumes. For the feature maps with resolutions of 1 / 8, 1 / 16, and 1 / 32 of the original image obtained in step 1, use the group correlation method to construct a multi-scale cost volume, and then use multiple 3D convolutions to fuse them to generate a cost volume of size D / 8×H / 8×W / 8×16;

[0056] Step 3: Cost aggregation is performed on the cost volume obtained in step 2, and a simple 3D hourglass structure is used to regularize the cost volume. Different sizes of dilated convolutions are used to generate guidance maps containing comprehensive information for the feature maps of different levels with 1 / 2 resolution obtained in the feature extraction part. Under the guidance of the guidance map, the cost volume is upsampled using a learnable bilateral grid algorithm to obtain a high-resolution cost volume;

[0057] Step 4: Use disparity regression to obtain the initial disparity map for the cost volume obtained in step 3, then reconstruct the error in the feature space, obtain the residual relative to the initial disparity map through the disparity refinement module, calculate the sum of the residual result and the initial disparity, and generate the final disparity map of the stereo image pair.

[0058] In step 1 above, after reducing the image resolution to 1 / 8 through convolution, the image is further downsampled, and a channel enhancement network similar to the U-Net structure is used to finally obtain a 1 / 8 resolution feature map. The structure is as follows Figure 2 As shown in the figure. At the 1 / 32 resolution, sub-pixel convolution is used to replace the original upsampling operation in the feature map, and the remaining feature maps are processed vertically using 1×1 convolution. At the same time, sub-pixel context enhancement is used to enrich channel information. For the 1 / 32H×1 / 32W×C feature map, 3×3 convolution is used to extract local information, while reducing its number of channels to 1 / 2C, and then sub-pixel convolution is used for upsampling. Secondly, it proceeds to the upsampled image size, and the local information and global context information are fused by element-by-element summation to generate the integral image I.

[0059] For cross-scale features, a channel attention guidance module is used to process them, such as Figure 2 As shown in the figure, the input is the integral image I, which is subjected to global average pooling and global maximum pooling. After being processed by the fully connected layer, the elements are summed up and the output features are merged using the sigmoid function to generate new features. The process can be expressed as:

[0060] CA(z)=σ(Fc1(AvgPool(z))+Fc2(MaxPool(z))),

[0061] R i =CA(I)⊙P i

[0062] Among them, CA(z) is the channel attention function, σ is the sigmoid function, and i represents different scale levels;

[0063] The input and output feature maps are processed by the attention module, and the feature extraction networks of the left and right images share parameters. At the same time, the features of the left image at 1 / 8, 1 / 16, and 1 / 32 resolutions in the network are retained for cost volume construction.

[0064] like Figure 3 As shown in Figure 2, the specific method for constructing the fused multi-scale cost volume in step 2 is:

[0065] First, use the feature maps of the three scales obtained in step 1 to construct the group-related cost volume respectively. The formula of the group-related cost volume is as follows:

[0066]

[0067] Among them, i represents three different scales, <,> represents the inner product, d is the disparity value, and g is the group number. Represents the number of channels in the feature layer, N g represents the number of groups of partitioned features;

[0068] Use two 3D convolutions with skip connections to regularize the three cost volumes. Use a 3×3×3 convolution with a step size of 2 to downsample the cost volume at 1 / 8 resolution to 1 / 16, connect it to the cost volume at 1 / 16 resolution in the feature dimension, and use another 3D convolution to reduce the feature channels. Downsample it again, and then connect it to the cost volume at 1 / 32 resolution, also using a 3D convolution. Finally, use two transposed convolutions for upsampling to get the fused cost volume. The size of the cost volume is 1 / 8H×1 / 8W×1 / 8D×16, and the value of D is 192. The resolution is the original Figure 1 The cost volume of / 8 uses a simple 3D hourglass network to complete cost aggregation and obtain a low-resolution cost volume of size 1 / 8H×1 / 8W×1 / 8D×32.

[0069] In step 3, the three levels of feature maps with 1 / 2 resolution obtained from the feature extraction part are denoted as F1, F2, and F3, and their number of channels is 32. Figure 4 As shown in the figure, for F1, a dilated convolution with a convolution kernel of 3×3 and a dilation rate of 1 is used to reduce the number of channels to 8. For F2, a dilated convolution with a convolution kernel of 3×3 and a dilation rate of 2 is used to reduce the number of channels to 8. For F3, a dilated convolution with a convolution kernel of 3×3 and a dilation rate of 3 is used to reduce the number of channels to 8.

[0070] After generating three groups of convolution results, the three feature maps are interleaved and convolved to establish the dependency relationship between the three groups. The three groups of output features are calculated using the adaptively learned weights. The formula is as follows:

[0071]

[0072] where i = 1, 2, 3, w ij It is the weight of different feature maps, and is adaptively learned during the training process. Finally, the guide map G is generated through three feature maps;

[0073] G=Conv 1×1 (concat{F1,F2,F3})

[0074] Using two 1×1 convolutions, the obtained guidance map G is of size 1 / 2H × 1 / 2W × 1. Under the guidance of the guidance map, a learnable bilateral grid algorithm is used to upsample the cost volume to obtain a high-resolution cost volume.

[0075] For upsampling using the learnable bilateral grid algorithm in step 3, the specific operations are:

[0076] The slicing operation of bilateral grid processing is used to upsample the cost volume at 1 / 8 resolution to high resolution. The bilateral grid has 4 dimensions, width x, height y, disparity d and guide feature g. Its value can be expressed as B(x, y, d, g), which is created using a convolutional neural network. Under the guidance of the guide graph G, the four-dimensional bilateral grid is linearly interpolated. The slicing operation can be described as:

[0077] C H (x,y,d)=B(sx,sy,sd,s G G(x,y)

[0078] Among them, s, s G ∈(0,1), s is the ratio of the width or height of the bilateral grid to the high-resolution cost volume, s G is the ratio of the grayscale of the grid to the grayscale of the guide map, C H (x,y,d) is the generated 3D high-resolution cost volume.

[0079] For the high-resolution cost volume obtained in step 3, we regress the initial disparity map and predict the initial disparity value D using the soft argmin method. i (x,y):

[0080]

[0081] In step 4 of this embodiment, the initial disparity map is obtained by using a disparity refinement module to obtain a final disparity map, and the specific execution steps are as follows:

[0082] After obtaining the initial disparity, the reconstruction error r(x,y) is calculated in the feature space to measure the correctness of the initial disparity:

[0083] r(x,y)=|I L (x,y)-I R (x+D i (x,y),y)|

[0084] Among them, I L , I R are the left and right feature maps, (x, y) represents the position of the pixel.

[0085] The reconstruction error, initial disparity and left image features are concatenated together, as Figure 1 As shown, the residual is learned through an encoder-decoder structure network to obtain the residual D relative to the initial disparity r (x, y), add the residual and the initial disparity to get the final disparity map D of the stereo image pair f (x,y).

[0086] In summary, the present invention uses the stereo matching network BGNet as the benchmark model. In the feature extraction module, sub-pixel channel enhancement is used to enrich channel information. At the same time, the channel attention guidance module is used to process cross-scale features and improve the expressiveness of features. Multi-scale cost bodies are constructed and fused using group correlation, and cost aggregation is performed on low-resolution cost bodies. Different sizes of hole convolutions are used for feature maps of different levels to generate guidance maps containing comprehensive information. The weights of adaptive learning are used to increase the focus on important features and complete high-quality upsampling. Finally, the disparity refinement module is used to correct erroneous estimates, better restore the detail information in the image, and generate a more accurate disparity map.

[0087] Example

[0088] The stereo matching network of the present invention is verified using the KITTI dataset, runs under the PyTorch deep learning framework, uses two NVIDIA PTX 4080 GPUs to train the network, uses the Adam optimizer (β1=0.9 and β2=0.999), sets the batch size to 16, and sets the maximum disparity value to 192.

[0089] The experimental results of the algorithm of the present invention and other models are compared in Table 1.

[0090] Table 1 Results comparison

[0091]

[0092] As shown in Table 1, the matching accuracy of the proposed model on KITTI2012 and KITTI2015 exceeds that of the original model. Figure 5 The disparity maps of the present invention and the original model BGNet on two data sets are displayed, and the results show that the present invention can display smoother and more complete object structures.

[0093] The above examples have described the specific implementation methods of the present invention in detail. Other improvements and changes made by those skilled in the art without departing from the scope of the claims are all within the protection scope of the present invention.

Claims

1. A stereo matching method based on BGNet, characterized in that: The following steps are involved: Step 1: Input the original left and right image pair into the feature extraction network, and reduce the image resolution to 1 / 8 of the original through convolution. Design a channel enhancement network to improve the expressiveness of features, use the sub-pixel channel information enhancement module to enrich channel information, and use the channel attention guidance module to process cross-scale features, and obtain multi-scale feature maps at the same time; Step 2: Construct and fuse multi-scale cost volumes. For the feature maps with resolutions of 1 / 8, 1 / 16, and 1 / 32 of the original image obtained in step 1, use the group correlation method to construct a multi-scale cost volume, and fuse them to generate a cost volume with a size of D / 8×H / 8×W / 8×16; Step 3: Cost aggregation is performed on the cost volume obtained in step 2, and a 3D hourglass structure is used to regularize the cost volume. Different sizes of dilated convolutions are used to generate guidance maps containing comprehensive information for the feature maps of different levels with 1 / 2 resolution obtained in the feature extraction part. Under the guidance of the guidance map, the cost volume is upsampled using a learnable bilateral grid algorithm to obtain a high-resolution cost volume; Step 4: Use disparity regression to obtain the initial disparity map for the cost volume obtained in step 3, reconstruct the error in the feature space, obtain the residual relative to the initial disparity map through the disparity refinement module, calculate the sum of the residual result and the initial disparity, and generate the disparity map of the final stereo image pair.

2. The improved stereo matching method based on BGNet according to claim 1, characterized in that: In step 1, after reducing the image resolution to 1 / 8 through convolution, the image is further downsampled, and a U-Net-like structure is used to finally obtain a feature map with a resolution of 1 / 8. In this structure, the deep feature map contains rich semantic information. Sub-pixel convolution is used instead of the original upsampling operation. Sub-pixel convolution is an upsampling method that increases the width and height by shuffling pixels. 2 The characteristics are rearranged into rH×rW×C, and the formula is defined as follows: P(F) x,y,c =F x / r,y / r,C·r·mod(y,r)+C·mod(x,r)+c Where r represents the magnification factor, F represents the input feature, and P(F) x,y,c Represents the feature pixel output at the coordinate (x, y, c), using the sub-pixel context enhancement module on the highest level feature; Sub-pixel context enhancement is to use 3×3 convolution to extract local information from the 1 / 32H×1 / 32W×C feature map, and then use sub-pixel convolution for upsampling. Next, global average pooling is performed to obtain global context information, and 1×1 convolution is used to compress the features. Then, the local information and global context information are fused by element-by-element summation to generate the integral map I.

3. The improved stereo matching method based on BGNet according to claims 1 and 2, characterized in that: The channel attention guidance module is used to process cross-scale features, and global average pooling and global maximum pooling are used for the integral image I. After being processed by the fully connected layer, the elements are summed up and the output features are merged using the sigmoid function. The process can be expressed as: CA(z)=σ(Fc1(AvgPool(z))+Fc2(MaxPool(z))), R i =CA(I)⊙P i Among them, CA(z) is the channel attention function, σ is the sigmoid function, and i represents different scale levels; The input and output feature maps are processed by the attention module, and the feature extraction networks of the left and right images share parameters. At the same time, the features of the left image at 1 / 8, 1 / 16, and 1 / 32 resolutions in the network are retained for cost volume construction.

4. The improved stereo matching method based on BGNet according to claim 1, characterized in that: In step 2, we use the feature maps of the three scales obtained in step 1 to construct the group-related cost volume respectively. The formula of the group-related cost volume is as follows: Among them, i represents three different scales, <,> represents the inner product, d is the disparity value, and g is the group number. Represents the number of channels in the feature layer, N g represents the number of groups of partitioned features; Use 2 3D convolutions with skip connections to regularize the three cost volumes; The three cost volumes are fused using the encoder-decoder structure, and the size of the fused cost volume is 1 / 8H×1 / 8W×1 / 8D×16, and the value of D is 192. A simple 3D hourglass network is used to complete the cost aggregation for the cost volume with a resolution of 1 / 8 of the original image, and a low-resolution cost volume with a size of 1 / 8H×1 / 8W×1 / 8D×32 is obtained.

5. The improved stereo matching method based on BGNet according to claim 1, characterized in that: In step 3, the three levels of feature maps with 1 / 2 resolution obtained in the feature extraction part are recorded as F1, F2, and F3, and their number of channels is 32. For them, three groups of convolution results are generated using dilated convolutions with different dilation rates, and then these three groups of feature maps are interleaved and convolved to establish the dependency relationship between the three groups. The three groups of features are calculated using the adaptively learned weights. The formula is as follows: where i = 1, 2, 3, w ij It is the weight of different feature maps, and is adaptively learned during the training process. Finally, the guide map G is generated through three feature maps; G=Conv 1×1 (concat{F1,F2,F3}) Using two 1×1 convolutions, the obtained guidance map G is of size 1 / 2H × 1 / 2W × 1. Under the guidance of the guidance map, a learnable bilateral grid algorithm is used to upsample the cost volume to obtain a high-resolution cost volume.

6. The improved stereo matching method based on BGNet according to claim 5, characterized in that: The learnable bilateral grid algorithm uses the slicing operation of bilateral grid processing to upsample the cost volume at 1 / 8 resolution to high resolution. The bilateral grid has a total of 4 dimensions, width x, height y, disparity d and guide feature g, whose value can be expressed as B(x, y, d, g), created using a convolutional neural network. Under the guidance of the guide graph G, the four-dimensional bilateral grid is linearly interpolated. The slicing operation can be described as: C H (x,y,d)=B(sx,sy,sd,s G G(x,y)) Among them, s, s G ∈(0,1), s is the ratio of the width or height of the bilateral grid to the high-resolution cost volume, s G is the ratio of the grayscale of the grid to the grayscale of the guide map, C H (x,y,d) is the generated 3D high-resolution cost volume.

7. The improved stereo matching method based on BGNet according to claim 1, characterized in that: In step 4, the initial disparity map is regressed from the high-resolution cost volume obtained in step 3. We predict the initial disparity using the soft argmin method: After obtaining the initial disparity, the reconstruction error is calculated in the feature space to measure the correctness of the initial disparity. The reconstructed error, the initial disparity map and the left image features are used as inputs, and the disparity refinement module is used to calculate the residual D relative to the initial disparity. r (x, y), add the residual and the initial disparity to get the final disparity map D of the stereo image pair f (x,y).

Images