A binocular stereo matching method based on attention cascade cost neural network

Abstract

The present invention provides a binocular stereo matching method based on an attention cascade cost body neural network, belonging to the field of image processing technology. The present invention uses a binocular stereo data set for training and establishes an efficient stereo matching model. Among them, multi-scale feature extraction and cost body construction enrich the receptive field and improve the processing effect on repeated structures, weak textures or transparent objects and occluded areas. At the same time, the present invention introduces and improves the attention cascade cost body and cost body fusion method to reduce the redundant information in the cost body and the amount of network calculation, and maintains the balance of the amount of calculation, the amount of parameters and the speed.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing, and specifically is a binocular stereo matching method based on an attention cascade cost body neural network. Background Art

[0002] Binocular stereo vision is a key technology in machine vision. By calculating the pixel disparity of the same target in two images under different viewing angles and combining the calibration information of the binocular camera, the disparity is obtained, thereby obtaining the depth data of the target and realizing non-contact object distance measurement. Binocular stereo vision technology has been widely used in the fields of robot vision, aerial mapping, medical imaging and industrial inspection, and has always been a hot topic in machine vision research. Binocular stereo matching algorithm plays a decisive role in the performance of binocular stereo vision system. In recent years, convolutional neural networks have made significant progress in the field of target detection and recognition, and have had a profound impact on the field of image processing. Compared with traditional binocular stereo matching algorithms, algorithms based on convolutional neural networks have significantly improved in accuracy and speed, and have become the mainstream now.

[0003] At present, there are two main types of binocular stereo matching based on deep learning convolutional neural networks: one is a non-end-to-end algorithm and the other is an end-to-end algorithm. The non-end-to-end algorithm is to use a convolutional neural network to replace some steps in the traditional binocular stereo matching process. The end-to-end binocular stereo matching method means that the entire binocular stereo matching process is implemented by a convolutional neural network without the involvement of traditional algorithms. Compared with the non-end-to-end algorithm, the end-to-end algorithm can calculate the disparity more accurately. It uses the nonlinear characteristics and learning ability of the neural network to improve and optimize the key steps in the traditional algorithm, thereby achieving more accurate estimation results.

[0004] However, although the binocular stereo matching methods based on convolutional neural networks have made significant progress, the existing binocular stereo matching convolutional neural networks still have some challenging problems, such as difficulty in handling repeated structures, weak textures or transparent objects and occluded areas, the generated cost volume contains a lot of redundant information, and the network has a large amount of computational complexity. Summary of the invention

[0005] To improve the above problems, the present invention proposes a binocular stereo matching method based on an attention cascade cost body neural network. The receptive field is enriched through multi-scale feature extraction and cost body construction, thereby improving the processing effect of repeated structures, weak textures or transparent objects and occluded areas. The redundant information in the cost body and the amount of network calculation are reduced by introducing and improving the attention cascade cost body and cost body fusion method.

[0006] The technical solution of the present invention is as follows:

[0007] A binocular stereo matching method based on an attention cascade cost body neural network, the steps are as follows:

[0008] Step S1: Collect binocular stereo matching dataset.

[0009] Step S2: Preprocess the binocular stereo matching dataset, randomly crop the image to a size of H×W and normalize it to obtain the initial left and right images, where H and W are the height and width of the randomly cropped image.

[0010] Step S3: The initial left and right images are respectively placed into the weight-sharing feature extraction module to extract left and right feature maps of different sizes.

[0011] Step S3 further includes the following operations on the initial left and right images:

[0012] Step S3-1: Use three two-dimensional convolution modules to perform two-dimensional convolution operations on the initial left and right images. The two-dimensional convolution module consists of a two-dimensional convolution function, a batch normalization function, and a ReLU activation function. After the three two-dimensional convolution modules, a feature map F with a height of H / 2 and a width of W / 2 is obtained. 3-1-1 .

[0013] Step S3-2: F 3-1-1 Put it into two consecutive basic modules, where the basic module consists of two 2D convolution modules. The first 2D convolution module of the basic module consists of a 2D convolution function, a batch normalization function, and a ReLU activation function. The second 2D convolution module consists of a 2D convolution function and a batch normalization function. 3-1-1 After the first basic module, the output feature map F is H / 2 in height and W / 2 in width. 3-2-1 , F 3-2-1 After the second basic module, the feature map F with a height of H / 4 and a width of W / 4 is output. 3-2-2 .

[0014] Step S3-3: F 3-2-2 Copy it four times and pass it through four basic modules with different step lengths to obtain: feature maps with a height of H / 32 and a width of W / 32, feature maps with a height of H / 16 and a width of W / 16, feature maps with a height of H / 8 and a width of W / 8, and feature maps with a height of H / 4 and a width of W / 4; each route consists of two basic modules. The four feature maps are passed through the upsampling module in order from large to small to obtain feature maps of the same size as the feature map of the larger level, and added to the feature map of the larger level, and the new result is used to replace the original feature map. Finally, four feature maps are output, namely: feature map F with a height of H / 32 and a width of W / 32 3-3-1 , feature map F with a height of H / 16 and a width of W / 16 3-3-2, feature map F with a height of H / 8 and a width of W / 8 3-3-3 , a feature map F with a height of H / 4 and a width of W / 4 3-3-4 .

[0015] Step S4: Perform the following operations on the left and right feature maps of different sizes obtained by feature extraction in S3: Use the left and right feature maps F 3-3-1 、F 3-3-2 、F 3-3-3 The group-related cost volume V with a height of H / 32 and a width of W / 32 is constructed respectively 4-1 , the group-related cost volume V is H / 16 high and W / 16 wide 4-2 , the group-related cost volume V is H / 8 high and W / 8 wide 4-3 . Use the left and right feature maps F 3-3-4 Construct a cascaded cost volume V with a height of H / 4 and a width of W / 4 4-4 .

[0016] Step S5: The group-related cost volume V obtained in S4 4-1 、V 4-2 、V 4-3 Each of them passes through the three-dimensional convolution module to obtain a cost volume V with a height of H / 32 and a width of W / 32. 5-1 , the cost volume V is H / 16 in height and W / 16 in width 5-2 , the cost volume V is H / 8 high and W / 8 wide 5-3 The three-dimensional convolution module consists of a three-dimensional convolution function, a batch normalization function, and a ReLU activation function.

[0017] Step S6: The cost volume V obtained in S5 5-1 、V 5-2 、V 5-3 Put it into two consecutive hourglass modules to get a cost volume V with a height of H / 8 and a width of W / 8 6-2-4 .

[0018] The specific operations of step S6 further include:

[0019] Step S6-1: V 5-3 As the initial input of the first hourglass module, V 5-2 、V 5-1 In the first hourglass module, the initial input V 5-3 After a three-dimensional convolution function, the output cost volume V is H / 16 high and W / 16 wide. 6-1-1 , the cost body V 5-2 As input with V 6-1-1 Connect them in series to get a cost body V with a height of H / 16 and a width of W / 16 6-1-2Then set V 6-1-2 After two 3D convolution modules, we get a cost volume V with a height of H / 16 and a width of W / 16. 6-1-3 . V 6-1-3 After the three-dimensional convolution function, we get a cost volume V with a height of H / 16 and a width of W / 16. 6-1-4 . V 5-1 As input with V 6-1-4 Connect them in series to get a cost body V with a height of H / 16 and a width of W / 16 6-1-5 . V 6-5 After passing through the three-dimensional convolution function, batch normalization function, three-dimensional convolution module, three-dimensional transposed convolution function and batch normalization function in sequence, a cost volume V with a height of H / 16 and a width of W / 16 is obtained. 6-1-6 . V 6-1-3 After the three-dimensional convolution function and batch normalization function, V 6-1-6 Add them together, and pass the result through the ReLU activation function, the three-dimensional transposed convolution function, and the batch normalization function to obtain a cost volume V with a height of H / 8 and a width of W / 8 6-1-7 . V 6-7 With V 5-3 Add together the output of the first hourglass module, the cost volume V with a height of H / 8 and a width of W / 8 6-1-8 .

[0020] Step S6-2: V 6-1-8 As the input of the second hourglass module, it first passes through two 3D convolution modules to obtain a cost volume V with a height of H / 16 and a width of W / 16 6-2-1 . V 6-2-1 After two 3D convolution modules, 3D transposed convolution function, and batch normalization function, a cost volume V with a height of H / 16 and a width of W / 16 is obtained. 6-2-2 . V 6-2-1 After three-dimensional convolution function and batch normalization function, V 6-2-2 Add them together, and then pass through the ReLU activation function and the three-dimensional transposed convolution function to obtain a cost volume V with a height of H / 8 and a width of W / 8 6-2-3 . V 6-1-8 After the three-dimensional convolution function and V 6-2-3 Add them together and then pass through the ReLU activation function to get the output of the second hourglass module, the cost volume V with a height of H / 8 and a width of W / 8 6-2-4 .

[0021] Step S7: V 6-2-4 After the three-dimensional convolution module and the three-dimensional convolution function, the cost volume V with a height of H / 8 and a width of W / 8 is obtained. 7-1 .

[0022] Step S8: V 7-1The input upsampling module obtains a cost volume V with a height of H / 4 and a width of W / 4. 8-1 .

[0023] Step S9: V 8-1 Input the body attention propagation module, and obtain the attention weight with a height of H / 4 and a width of W / 4, which is used to filter and suppress the cascade cost body V 4-4 The irrelevant information in V is increased. The body attention propagation module is used to 8-1 Confidence estimation and sampling are performed to obtain confidence scores, disparity sampling and evaluation are performed to obtain matching scores, and finally these two scores are integrated through cross-propagation to obtain an attention weight with a height of H / 4 and a width of W / 4. Then, half of the scores with higher scores are selected, and finally an attention weight A with a height of H / 4 and a width of W / 4 is obtained. 9-3-2 .

[0024] The specific operations of step S9 further include:

[0025] Step S9-1: The confidence estimation and sampling operations in the body attention propagation module are as follows: First, V 8-1 The predicted disparity is obtained by performing disparity regression calculation, and then the variance calculation, confidence estimation and cross expansion are performed on the predicted disparity. Finally, the confidence score S with a height of H / 4 and a width of W / 4 is obtained. 9-1-1 .

[0026] The formula for parallax regression calculation is as follows:

[0027]

[0028] Where d is the parallax, D max is the maximum disparity value, σ(·) is the softmax function, c d To predict the cost, is the predicted disparity weighted by the probability of each disparity d. After disparity regression, the unsqueeze function is used to add a dimension to obtain an output with a height of H / 4 and a width of W / 4.

[0029] The formula for calculating the variance of the predicted disparity is as follows:

[0030]

[0031] Where U is the variance of the predicted disparity. The variance of the predicted disparity is calculated to obtain an output with a height of H / 4 and a width of W / 4.

[0032] The formula for confidence estimation is as follows:

[0033]

[0034] Where C is the confidence score, and are learnable parameters. After confidence estimation, the height is H / 4 and the width is W / 4.

[0035] The operation of cross expansion is to perform a two-dimensional convolution on the input using 5 convolution kernels, where only one element of each convolution kernel is 1 and the rest are 0. The output after cross expansion is H / 4 high and W / 4 wide.

[0036] Step S9-2: The operation of disparity sampling and evaluation in the volume attention propagation module is as follows: first, V 8-1 The predicted disparity is calculated by disparity regression, and then the predicted disparity is cross-expanded, the left and right feature maps are combined for spatial transformation, and the channel dimension is averaged to obtain a matching score S with a height of H / 4 and a width of W / 4. 9-2-1 .

[0037] The parallax regression calculation and cross expansion method are the same as S9-1, and the output V with a height of H / 4 and a width of W / 4 is obtained. 9-2-1 Then the left and right feature maps are combined for spatial transformation to obtain an output V with a height of H / 4 and a width of W / 4 9-2-2 . V 9-2-2 Take the average of the channel dimensions and get an output S with a height of H / 4 and a width of W / 4 9-2-1 .

[0038] Step S9-3: S 9-1-1 and S 9-2-1 The integration operation is as follows: first, S 9-1-1 and S 9-2-1 Multiply them to get an output with a height of H / 4 and a width of W / 4, and apply the softmax function to its disparity dimension. Then use the unsqueeze function to add a dimension to get an output S with a height of H / 4 and a width of W / 4. 9-2-3 At the same time, V 8-1 Perform three-dimensional cross expansion: Use five convolution kernels to perform three-dimensional convolution on the input, where only one element of each convolution kernel is 1 and the rest are 0. The output V after three-dimensional cross expansion is 9-3-1 The height is H / 4, the width is W / 4, and then S 9-2-3 With V 9-3-1 Multiply and sum the disparity dimensions to get an output V with a height of H / 4 and a width of W / 4 9-3-2 In the channel dimension V 9-3-2 Using the softmax function, we get an output A with a height of H / 4 and a width of W / 4. 9-3-1 Use the sort function and gather function to select half of the channels with the highest scores from all channels, and get the attention weight A with a height of H / 4 and a width of W / 4. 9-3-2 .

[0039] Step S10: V 4-4 The input structure is the same as the second hourglass module of S6, and the cost volume V with a height of H / 4 and a width of W / 4 is obtained. 10-1 . V 10-1 With A 9-3-2 Multiply them together to get a cost volume V with a height of H / 4 and a width of W / 4. 10-2 . V 10-2 The input structure is the same as the second hourglass module of S6, and the output V is H / 4 high and W / 4 wide. 10-3 .

[0040] Step S11: V 10-3 After the three-dimensional convolution module, we get an output V with a height of H / 4 and a width of W / 4. 11-1 .

[0041] Step S12: V 11-1 Use the squeeze function and disparity regression to calculate the predicted disparity D with a height of H / 4 and a width of W / 4 12-1 , for D 12-1 Using upsampling, we finally get the output predicted disparity D with a height of H and a width of W. 12-2 .

[0042] The above are the steps for reasoning. If it is training, the following steps are required:

[0043] Step S13: A 9-3-1 Use the squeeze function and disparity regression to calculate the predicted disparity D with a height of H / 4 and a width of W / 4 13-1 , for D 13-1 Using upsampling, we get an output predicted disparity D with a height of H and a width of W. 13-2 .

[0044] Step S14: V 10-1 After the three-dimensional convolution module, we get an output V with a height of H / 4 and a width of W / 4. 14-1 . For V 14-1 Use the squeeze function and disparity regression to calculate the predicted disparity D with a height of H / 4 and a width of W / 4 14-1 , for D 14-1 Using upsampling, we get an output predicted disparity D with a height of H and a width of W. 14-2 .

[0045] Step S15: Using D 12-2 , D 13-2 and D 14-2 Construct the final loss function to train the network.

[0046] The formula of the loss function is as follows:

[0047]

[0048] Among them, L F is the loss function. Yes D 12-2 The constant coefficient of Yes D 13-2 The constant coefficient of Yes D 14-2 The constant coefficient of . Yes D 12-2 , Yes D 13-2 , Yes D 14-2 , is the real parallax. L1 is the smooth L1 loss.

[0049] Beneficial effects of the present invention: Compared with the prior art, the present invention uses binocular stereo data sets for training and establishes an efficient stereo matching model. Among them, multi-scale feature extraction and cost volume construction enrich the receptive field and improve the processing effect on repeated structures, weak textures or transparent objects and occluded areas. At the same time, the present invention introduces and improves the attention cascade cost volume and cost volume fusion method to reduce the redundant information in the cost volume and the amount of network calculation, and maintains a balance between the amount of calculation, the amount of parameters and the speed. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solution of the present invention, the following briefly introduces the drawings required in the embodiments. In order to avoid blurring the key points of the present invention due to unnecessary details, only the parts closely related to the solution of the present invention are shown in the drawings, and the other details that are not closely related to the present invention are omitted.

[0051] Figure 1 It is a schematic diagram of the framework of the neural network training of the present invention.

[0052] Figure 2 Schematic diagram of predicted disparity.

[0053] The following specific embodiments will further illustrate the present invention in conjunction with the above-mentioned drawings. DETAILED DESCRIPTION

[0054] The technical solution of the present invention is further described below through the accompanying drawings.

[0055] like Figure 1 As shown, Figure 1 It is a schematic diagram of the framework of the neural network training of the present invention.

[0056] A binocular stereo matching method based on an attention cascade cost body neural network, the steps are as follows:

[0057] Step S1: Collect binocular stereo matching datasets, including SceneFlow, KITTI 2012 and KITTI 2015 public datasets.

[0058] Step S2: Preprocess the binocular stereo matching dataset, randomly crop the image to a size of H×W and normalize it to obtain the initial left and right images. H and W are the height and width of the randomly cropped image, where H is 256 and W is 512.

[0059] Step S3: The initial left and right images are respectively placed into the weight-sharing feature extraction module to extract left and right feature maps of different sizes.

[0060] Step S3 further includes the following operations on the initial left and right images:

[0061] Step S3-1: Use three two-dimensional convolution modules to perform two-dimensional convolution operations on the initial left and right images. The two-dimensional convolution module consists of a two-dimensional convolution function, a batch normalization function, and a ReLU activation function. The convolution kernels of the three two-dimensional convolution modules are all 3×3 in size. The first two-dimensional convolution module has a step size of 2, an input of 3 channels, and an output of 32 channels. The second and third two-dimensional convolution modules have a step size of 1, and both the input and output are 32 channels. After three two-dimensional convolution modules, a feature map F of size 32×H / 2×W / 2 is obtained. 3-1-1 .

[0062] Step S3-2: F 3-1-1 Put it into two consecutive basic modules, where the basic module consists of two two-dimensional convolution modules, and the convolution kernels of the two-dimensional convolution modules are all 3×3 in size. The first two-dimensional convolution module of the basic module consists of a two-dimensional convolution function, a batch normalization function, and a ReLU activation function. The number of input channels is the number of input channels of the basic module, and the number of output channels is the number of output channels of the basic module. The second two-dimensional convolution module consists of a two-dimensional convolution function and a batch normalization function, and the number of input and output channels is the number of output channels of the basic module. There are two types of basic modules. The first two-dimensional convolution module of the first basic module has a step size of 2, and the second two-dimensional convolution module has a step size of 1. The first two-dimensional convolution module of the second basic module has a step size of 2, and the second two-dimensional convolution module has a step size of 1. Its input and output are connected by a residual channel, where the residual channel is composed of a two-dimensional convolution function and a batch normalization function of a size of 1×1, and its output is the sum of the feature map of the input passing through the residual channel and the feature map passing through two two-dimensional convolution modules. F 3-1-1 After the first basic module, the output feature map F of size 64×H / 2×W / 2 is 3-2-1 , F 3-2-1 After the second basic module, the output feature map F of size 128×H / 4×W / 4 is3-2-2 .

[0063] Step S3-3: F 3-2-2 Replicate four times, and pass through four basic modules with different step sizes to obtain feature maps of 320×H / 32×W / 32, 320×H / 16×W / 16, 320×H / 8×W / 8, and 320×H / 4×W / 4, each route consisting of two basic modules. The four feature maps are passed through the upsampling module in descending order to obtain a feature map of the same size as the feature map of the next level, and added to the feature map of the next level, and the new result is used to replace the original feature map. The upsampling module consists of an upsampling function with a scaling factor of 2 and a two-dimensional convolution module with a convolution kernel size of 3×3 and a step size of 1. Finally, four feature maps are output, namely, feature map F of size 320×H / 32×W / 32. 3-3-1 , 320×H / 16×W / 16 feature map F 3-3-2 , 320×H / 8×W / 8 feature map F 3-3-3 , 320×H / 4×W / 4 feature map F 3-3-4 .

[0064] Step S4: Perform the following operations on the left and right feature maps of different sizes obtained by feature extraction in S3: Use the left and right feature maps F 3-3-1 、F 3-3-2 、F 3-3-3 The group-related cost volumes V of sizes 40×24×H / 32×W / 32, 40×24×H / 16×W / 16, and 40×24×H / 8×W / 8 are constructed respectively. 4-1 、V 4-2 、V 4-3 . Use the left and right feature maps F 3-3-4 The cascaded cost body V of 32×24×H / 4×W / 4 is constructed 4-4 .

[0065] Step S5: The group-related cost volume V obtained in S4 4-1 、V 4-2 、V 4-3 Each of them passes through the 3D convolution module to obtain the cost volume V of size 64×24×H / 32×W / 32, 64×24×H / 16×W / 16, and 32×24×H / 8×W / 8 respectively. 5-1 、V 5-2 、V 5-3 The three-dimensional convolution module consists of a three-dimensional convolution function, a batch normalization function, and a ReLU activation function. The convolution kernel size of the three-dimensional convolution function is 3×3×3 and the step size is 1.

[0066] Step S6: The cost volume V obtained in S5 5-1 、V5-2 、V 5-3 Put it into two consecutive hourglass modules to get a cost volume V of size 32×24×H / 8×W / 8 6-2-4 .

[0067] The specific operations of step S6 further include:

[0068] Step S6-1: V 5-3 As the initial input of the first hourglass module, V 5-2 、V 5-1 In the first hourglass module, the initial input V 5-3 After a 3D convolution function with a convolution kernel of 3×3×3 and a step size of 2, the output cost volume V is 64×24×H / 16×W / 16. 6-1-1 , the cost body V 5-2 As input with V 6-1-1 Connect them in series to get a cost body V of size 128×24×H / 16×W / 16 6-1-2 Then set V 6-1-2 After two 3D convolution modules with kernel size of 3×3×3 and stride of 1, the cost volume V with size of 64×24×H / 16×W / 16 is obtained. 6-1-3 . V 6-1-3 After a three-dimensional convolution function with a convolution kernel of 3×3×3 and a step size of 2, a cost volume V of size 128×24×H / 16×W / 16 is obtained. 6-1-4 . V 5-1 As input with V 6-1-4 Connect them in series to get a cost volume V of size 192×24×H / 16×W / 16 6-1-5 . V 6-5 After passing through the three-dimensional convolution function with a convolution kernel size of 3×3×3 and a step size of 1, the batch normalization function, the three-dimensional convolution module with a convolution kernel size of 3×3×3 and a step size of 1, the three-dimensional transposed convolution function with a convolution kernel size of 3×3×3 and a step size of 2, and the batch normalization function, a cost volume V of size 64×24×H / 16×W / 16 is obtained. 6-1-6 . V 6-1-3 After the three-dimensional convolution function with a convolution kernel of 3×3×3 and a step size of 2 and the batch normalization function, V 6-1-6 Add them together, and pass the result through the ReLU activation function, the three-dimensional transposed convolution function with a convolution kernel of 3×3×3 and a step size of 2, and the batch normalization function to obtain a cost volume V of size 32×24×H / 8×W / 8 6-1-7 . V 6-7 With V 5-3Add together the output of the first hourglass module, a cost volume V of size 32×24×H / 8×W / 8 6-1-8 .

[0069] Step S6-2: V 6-1-8 As the input of the second hourglass module, it first passes through two 3D convolution modules with a convolution kernel size of 3×3×3 to obtain a cost volume V of size 64×24×H / 16×W / 16 6-2-1 , where the stride of the first 3D convolution module is 2 and the stride of the second 3D convolution module is 1. 6-2-1 After two 3D convolution modules with a convolution kernel size of 3×3×3, a 3D transposed convolution function with a convolution kernel size of 3×3×3 and a step size of 2, and a batch normalization function, a cost volume V with a size of 64×24×H / 16×W / 16 is obtained. 6-2-2 , where the stride of the first three-dimensional convolution module is 2, and the stride of the second three-dimensional convolution module is 1. 6-2-1 After the convolution kernel size is 1×1×1, the step size is 1 three-dimensional convolution function, and the batch normalization function, V 6-2-2 Add them together, and then pass through the ReLU activation function, the three-dimensional transposed convolution function with a convolution kernel size of 3×3×3 and a step size of 2 to obtain a cost volume V of size 32×24×H / 8×W / 8 6-2-3 . V 6-1-8 After the convolution kernel size is 1×1×1 and the step size is 1, the three-dimensional convolution function and V 6-2-3 Add them together and then pass through the ReLU activation function to get the output of the second hourglass module, which is a cost volume V of size 32×24×H / 8×W / 8 6-2-4 .

[0070] Step S7: V 6-2-4 The cost volume V with a size of 1×24×H / 8×W / 8 is obtained by sequentially passing through a three-dimensional convolution module with a convolution kernel size of 3×3×3 and a step size of 1 and a three-dimensional convolution function. 7-1 .

[0071] Step S8: V 7-1 The input upsampling module obtains a cost volume V of size 1×48×H / 4×W / 4 8-1 , where the upsampling module uses trilinear interpolation.

[0072] Step S9: V 8-1 Input the attention propagation module and obtain the attention weight of size 1×24×H / 4×W / 4, which is used to filter and suppress the cascade cost body V 4-4 The irrelevant information in V is increased. The body attention propagation module is used to 8-1Confidence estimation and sampling are performed to obtain confidence scores, disparity sampling and evaluation are performed to obtain matching scores, and finally these two scores are integrated through cross-propagation to obtain an attention weight of 1×48×H / 4×W / 4. Then, half of them with higher scores are selected to finally obtain an attention weight A of 1×24×H / 4×W / 4. 9-3-2 .

[0073] The specific operations of step S9 further include:

[0074] Step S9-1: The confidence estimation and sampling operations in the body attention propagation module are: Figure 2 This is a schematic diagram of the predicted disparity. First, V 8-1 The predicted disparity is obtained by performing disparity regression calculation, and then the variance calculation, confidence estimation and cross expansion are performed on the predicted disparity, and finally the confidence score S of size 5×H / 4×W / 4 is obtained. 9-1-1 .

[0075] The formula for parallax regression calculation is as follows:

[0076]

[0077] Where d is the parallax, D max is the maximum disparity value, σ(·) is the softmax function, c d To predict the cost, is the predicted disparity weighted by the probability of each disparity d. After disparity regression, the unsqueeze function is used to increase one dimension to obtain an output of size 1×H / 4×W / 4.

[0078] The formula for calculating the variance of the predicted disparity is as follows:

[0079]

[0080] Where U is the variance of the predicted disparity. The variance of the predicted disparity is calculated to obtain an output of size 1×H / 4×W / 4.

[0081] The formula for confidence estimation is as follows:

[0082]

[0083] Where C is the confidence score, and is a learnable parameter. The size of the confidence estimate is 1×H / 4×W / 4.

[0084] The cross expansion operation is: use 5 convolution kernels of size 3×3 to perform two-dimensional convolution on the input, where only 1 of the 9 elements of each convolution kernel is 1, and the other 8 are 0. The positions of the 5 1 elements are respectively at the upper left corner, middle, lower right corner, lower left corner and upper right corner of the 5 convolution kernels. The output size after cross expansion is 5×H / 4×W / 4.

[0085] Step S9-2: The operation of disparity sampling and evaluation in the volume attention propagation module is as follows: first, V 8-1 The predicted disparity is calculated by disparity regression, and then the predicted disparity is cross-expanded, the left and right feature maps are combined for spatial transformation, and the channel dimension is averaged to obtain a matching score S of size 5×H / 4×W / 4 9-2-1 .

[0086] The parallax regression calculation and cross expansion method are the same as S9-1, and the output V with a size of 5×H / 4×W / 4 is obtained. 9-2-1 Then the left and right feature maps are combined for spatial transformation to obtain an output V of size 48×5×H / 4×W / 4 9-2-2 The left and right feature maps are the outputs of the initial left and right images, which are 48×H / 4×W / 4, obtained by passing two consecutive two-dimensional convolution modules with a convolution kernel size of 3×3 and a stride of 2. The spatial transformation is combined with V 9-2-1 The right feature map is normalized and deformed to a deformed right feature map F of size 48×5×H / 4×W / 4. 9-2-1 , normalize the left feature map and copy it 5 times to get the deformed left feature map F with a size of 48×5×H / 4×W / 4 9-2-2 Finally, F 9-2-2 With F 9-2-1 Multiply V 9-2-2 Taking the average of the channel dimension, we get an output S of size 5×H / 4×W / 4 9-2-1 .

[0087] Step S9-3: S 9-1-1 and S 9-2-1 The integration operation is as follows: first, S 9-1-1 and S 9-2-1 Multiply them together to get an output of size 5×H / 4×W / 4, and apply the softmax function to its disparity dimension. Then use the unsqueeze function to add one dimension to get an output S of size 5×1×H / 4×W / 4. 9-2-3 At the same time, V 8-1Perform a three-dimensional cross expansion and use five convolution kernels of size 3×3×3 to perform a three-dimensional convolution on the input. Among the 29 elements of each convolution kernel, only one is 1 and the other 28 are 0. The five 1 elements are located at the upper left corner, middle, lower right corner, lower left corner and upper right corner of the 0th layer of the five convolution kernels. The output V after three-dimensional cross expansion is 9-3-1 The size is 5×48×H / 4×W / 4, and then S 9-2-3 With V 9-3-1 Multiply and sum the disparity dimensions to get an output V of size 1×48×H / 4×W / 4 9-3-2 In the channel dimension V 9-3-2 Using the softmax function, we get an output A of size 1×48×H / 4×W / 4 9-3-1 Use the sort function and gather function to select the 24 channels with high scores from the 48 channels, and get the attention weight A of size 1×24×H / 4×W / 4 9-3-2 .

[0088] Step S10: V 4-4 The input structure is the same as the second hourglass module of S6, and the cost volume V with a size of 32×24×H / 4×W / 4 is obtained. 10-1 . V 10-1 With A 9-3-2 Multiply them together to get a cost volume V of size 32×24×H / 4×W / 4 10-2 . V 10-2 The input structure is the same as the second hourglass module of S6, and the output V is 32×24×H / 4×W / 4. 10-3 .

[0089] Step S11: V 10-3 After a three-dimensional convolution module with a convolution kernel size of 3×3×3 and a step size of 1, an output V of size 1×24×H / 4×W / 4 is obtained. 11-1 .

[0090] Step S12: V 11-1 Use the squeeze function and disparity regression to calculate the predicted disparity D of size H / 4×W / 4 12-1 , for D 12-1 Using upsampling, we finally get an output prediction disparity D of size H×W 12-2 .

[0091] The above are the steps for reasoning. If it is training, the following steps are required:

[0092] Step S13: A 9-3-1 Use the squeeze function and disparity regression to calculate the predicted disparity D of size H / 4×W / 413-1 , for D 13-1 Using upsampling, we get an output prediction disparity D of size H×W 13-2 .

[0093] Step S14: V 10-1 After a three-dimensional convolution module with a convolution kernel size of 3×3×3 and a step size of 1, an output V of size 1×24×H / 4×W / 4 is obtained. 14-1 . For V 14-1 Use the squeeze function and disparity regression to calculate the predicted disparity D of size H / 4×W / 4 14-1 , for D 14-1 Using upsampling, we get an output prediction disparity D of size H×W 14-2 .

[0094] Step S15: Using D 12-2 , D 13-2 and D 14-2 Construct the final loss function to train the network.

[0095] The formula of the loss function is as follows:

[0096]

[0097] Among them, L F is the loss function. Yes D 12-2 The constant coefficient of Yes D 13-2 The constant coefficient of Yes D 14-2 The constant coefficient of . Yes D 12-2 , Yes D 13-2 , Yes D 14-2 , is the real parallax. L1 is the smooth L1 loss.

[0098] The above embodiments are intended to help understand the method and core concept of the present invention. It is worth emphasizing that, for those skilled in the art, various improvements and adjustments can be made based on the principles of the present invention, and these improvements and adjustments should also be included in the scope of protection of the claims of the present invention.

[0099] The above description of the disclosed embodiments enables professionals and technicians in the field to implement or use the present invention. Various modifications of these embodiments are obvious to professionals and technicians in the field, so the general principles defined can be applied in other embodiments without departing from the spirit or scope of the present invention. Therefore, the scope of protection of the present invention is not limited to the embodiments shown herein, but should include the widest range consistent with the principles and innovative features disclosed herein.

Claims

1. A binocular stereo matching method based on an attention cascade cost body neural network, the steps are as follows: Step S1: Collect binocular stereo matching data set; Step S2: Preprocess the binocular stereo matching dataset, randomly crop the image to a size of H×W and normalize it to obtain the initial left and right images, where H and W are the height and width of the randomly cropped image; Step S3: Put the initial left and right images into the weight-sharing feature extraction module respectively to extract left and right feature maps of different sizes; Step S3 further includes the following operations on the initial left and right images: Step S3-1: Use three two-dimensional convolution modules to perform two-dimensional convolution operations on the initial left and right images; the two-dimensional convolution module is composed of a two-dimensional convolution function, a batch normalization function, and a ReLU activation function; after the three two-dimensional convolution modules, a feature map F with a height of H / 2 and a width of W / 2 is obtained 3-1-1 ; Step S3-2: F 3-1-1 Put it into two consecutive basic modules, where the basic module consists of two two-dimensional convolution modules; the first two-dimensional convolution module of the basic module consists of a two-dimensional convolution function, a batch normalization function and a ReLU activation function; the second two-dimensional convolution module consists of a two-dimensional convolution function and a batch normalization function; F 3-1-1 After the first basic module, the output feature map F is H / 2 in height and W / 2 in width. 3-2-1 , F 3-2-1 After the second basic module, the feature map F with a height of H / 4 and a width of W / 4 is output. 3-2-2 ; Step S3-3: F 3-2-2 Copy it four times, and pass it through four basic modules with different step lengths to obtain: feature maps with a height of H / 32 and a width of W / 32, feature maps with a height of H / 16 and a width of W / 16, feature maps with a height of H / 8 and a width of W / 8, and feature maps with a height of H / 4 and a width of W / 4; each route is composed of two basic modules; the four feature maps obtained are sequentially passed through the upsampling module in order from large to small to obtain a feature map of the same size as the feature map of a larger level, and are added to the feature map of the larger level, and the new result is used to replace the original feature map; finally, four feature maps are output, namely: feature map F with a height of H / 32 and a width of W / 32 3-3-1 , feature map F with a height of H / 16 and a width of W / 16 3-3-2 , feature map F with a height of H / 8 and a width of W / 8 3-3-3 , a feature map F with a height of H / 4 and a width of W / 4 3-3-4 ; Step S4: Perform the following operations on the left and right feature maps of different sizes obtained by feature extraction in S3: Use the left and right feature maps F 3-3-1 、F 3-3-2 、F 3-3-3 The group-related cost volume V with a height of H / 32 and a width of W / 32 is constructed respectively 4-1 , the group-related cost volume V is H / 16 high and W / 16 wide 4-2 , the group-related cost volume V is H / 8 high and W / 8 wide 4-3 ; Use the left and right feature maps F 3-3-4 Construct a cascaded cost volume V with a height of H / 4 and a width of W / 4 4-4 ; Step S5: The group-related cost volume V obtained in S4 4-1 、V 4-2 、V 4-3 Each of them passes through the three-dimensional convolution module to obtain a cost volume V with a height of H / 32 and a width of W / 32. 5-1 , the cost volume V is H / 16 in height and W / 16 in width 5-2 , the cost volume V is H / 8 high and W / 8 wide 5-3 ; The three-dimensional convolution module consists of a three-dimensional convolution function, a batch normalization function, and a ReLU activation function; Step S6: The cost volume V obtained in S5 5-1 、V 5-2 、V 5-3 Put it into two consecutive hourglass modules to get a cost volume V with a height of H / 8 and a width of W / 8 6-2-4 ; The specific operations of step S6 further include: Step S6-1: V 5-3 As the initial input of the first hourglass module, V 5-2 、V 5-1 In the subsequent processing of the first hourglass module, the initial input V 5-3 After a three-dimensional convolution function, the output cost volume V is H / 16 high and W / 16 wide. 6-1-1 , the cost body V 5-2 As input with V 6-1-1 Connect them in series to get a cost body V with a height of H / 16 and a width of W / 16 6-1-2 ; Then V 6-1-2 After two 3D convolution modules, we get a cost volume V with a height of H / 16 and a width of W / 16. 6-1-3 ; V 6-1-3 After the three-dimensional convolution function, we get a cost volume V with a height of H / 16 and a width of W / 16. 6-1-4 ; V 5-1 As input with V 6-1-4 Connect them in series to get a cost body V with a height of H / 16 and a width of W / 16 6-1-5 ; V 6-5 After passing through the three-dimensional convolution function, batch normalization function, three-dimensional convolution module, three-dimensional transposed convolution function and batch normalization function in sequence, a cost volume V with a height of H / 16 and a width of W / 16 is obtained. 6-1-6 ; V 6-1-3 After the three-dimensional convolution function and batch normalization function, V 6-1-6 Add them together, and pass the result through the ReLU activation function, the three-dimensional transposed convolution function, and the batch normalization function to obtain a cost volume V with a height of H / 8 and a width of W / 8 6-1-7 ; V 6-7 With V 5-3 Add together the output of the first hourglass module, the cost volume V with a height of H / 8 and a width of W / 8 6-1-8 ; Step S6-2: V 6-1-8 As the input of the second hourglass module, it first passes through two 3D convolution modules to obtain a cost volume V with a height of H / 16 and a width of W / 16 6-2-1 ; V 6-2-1 After two 3D convolution modules, 3D transposed convolution function, and batch normalization function, a cost volume V with a height of H / 16 and a width of W / 16 is obtained. 6-2-2 ; V 6-2-1 After three-dimensional convolution function and batch normalization function, V 6-2-2 Add them together, and then pass through the ReLU activation function and the three-dimensional transposed convolution function to obtain a cost volume V with a height of H / 8 and a width of W / 8 6-2-3 ; V 6-1-8 After the three-dimensional convolution function and V 6-2-3 Add them together and then pass through the ReLU activation function to get the output of the second hourglass module, the cost volume V with a height of H / 8 and a width of W / 8 6-2-4 ; Step S7: V 6-2-4 After the three-dimensional convolution module and the three-dimensional convolution function, the cost volume V with a height of H / 8 and a width of W / 8 is obtained. 7-1 ; Step S8: V 7-1 The input upsampling module obtains a cost volume V with a height of H / 4 and a width of W / 4. 8-1 ; Step S9: V 8-1 Input the body attention propagation module, and obtain the attention weight with a height of H / 4 and a width of W / 4, which is used to filter and suppress the cascade cost body V 4-4 The irrelevant information in V is increased, and the effective information is increased; the body attention propagation module is used to 8-1 Confidence estimation and sampling are performed to obtain confidence scores, disparity sampling and evaluation are performed to obtain matching scores, and finally these two scores are integrated through cross-propagation to obtain an attention weight with a height of H / 4 and a width of W / 4. Then, half of the scores with higher scores are selected, and finally an attention weight A with a height of H / 4 and a width of W / 4 is obtained. 9-3-2 ; The specific operations of step S9 further include: Step S9-1: The confidence estimation and sampling operations in the body attention propagation module are as follows: First, V 8-1 The predicted disparity is obtained by performing disparity regression calculation, and then the variance calculation, confidence estimation and cross expansion are performed on the predicted disparity. Finally, the confidence score S with a height of H / 4 and a width of W / 4 is obtained. 9-1-1 ; The formula for parallax regression calculation is as follows: Where d is the parallax, D max is the maximum disparity value, σ(·) is the softmax function, c d To predict the cost, is the predicted disparity obtained by weighting the probability of each disparity d; after disparity regression, the unsqueeze function is used to add a dimension to obtain an output with a height of H / 4 and a width of W / 4; The formula for calculating the variance of the predicted disparity is as follows: Where U is the variance of the predicted disparity; the variance of the predicted disparity is calculated to obtain an output with a height of H / 4 and a width of W / 4; The formula for confidence estimation is as follows: Where C is the confidence score, and are learnable parameters; the height after confidence estimation is H / 4 and the width is W / 4; The cross expansion operation is: use 5 convolution kernels to perform two-dimensional convolution on the input, where only one element of each convolution kernel is 1 and the rest are 0; the output after cross expansion is H / 4 high and W / 4 wide; Step S9-2: The operation of disparity sampling and evaluation in the volume attention propagation module is as follows: first, V 8-1 The predicted disparity is calculated by disparity regression, and then the predicted disparity is cross-expanded, the left and right feature maps are combined for spatial transformation, and the channel dimension is averaged to obtain a matching score S with a height of H / 4 and a width of W / 4. 9-2-1 ; The parallax regression calculation and cross expansion method are the same as S9-1, and the output V with a height of H / 4 and a width of W / 4 is obtained. 9-2-1 ; Then combine the left and right feature maps to perform spatial transformation to obtain an output V with a height of H / 4 and a width of W / 4 9-2-2 ; V 9-2-2 Take the average of the channel dimensions and get an output S with a height of H / 4 and a width of W / 4 9-2-1 ; Step S9-3: S 9-1-1 and S 9-2-1 The integration operation is as follows: first, S 9-1-1 and S 9-2-1 Multiply them to get an output with a height of H / 4 and a width of W / 4, and apply the softmax function to its disparity dimension; then use the unsqueeze function to add a dimension to get an output S with a height of H / 4 and a width of W / 4. 9-2-3 At the same time, V 8-1 Perform three-dimensional cross expansion: Use five convolution kernels to perform three-dimensional convolution on the input, where only one element of each convolution kernel is 1 and the rest are 0; the output V after three-dimensional cross expansion is 9-3-1 The height is H / 4, the width is W / 4, and then S 9-2-3 With V 9-3-1 Multiply and sum the disparity dimensions to get an output V with a height of H / 4 and a width of W / 4 9-3-2 ; In the channel dimension, V 9-3-2 Using the softmax function, we get an output A with a height of H / 4 and a width of W / 4. 9-3-1 ; Use the sort function and gather function to select half of the channels with the highest scores from all channels, and get the attention weight A with a height of H / 4 and a width of W / 4 9-3-2 ; Step S10: V 4-4 The input structure is the same as the second hourglass module of S6, and the cost volume V with a height of H / 4 and a width of W / 4 is obtained. 10-1 ; V 10-1 With A 9-3-2 Multiply them together to get a cost volume V with a height of H / 4 and a width of W / 4. 10-2 ; V 10-2 The input structure is the same as the second hourglass module of S6, and the output V is H / 4 high and W / 4 wide. 10-3 ; Step S11: V 10-3 After the three-dimensional convolution module, we get an output V with a height of H / 4 and a width of W / 4. 11-1 ; Step S12: V 11-1 Use the squeeze function and disparity regression to calculate the predicted disparity D with a height of H / 4 and a width of W / 4 12-1 , for D 12-1 Using upsampling, we finally get the output predicted disparity D with a height of H and a width of W. 12-2 ; The above are the steps for reasoning. If it is training, the following steps are required: Step S13: A 9-3-1 Use the squeeze function and disparity regression to calculate the predicted disparity D with a height of H / 4 and a width of W / 4 13-1 , for D 13-1 Using upsampling, we get an output predicted disparity D with a height of H and a width of W. 13-2 ; Step S14: V 10-1 After the three-dimensional convolution module, we get an output V with a height of H / 4 and a width of W / 4. 14-1 ; for V 14-1 Use the squeeze function and disparity regression to calculate the predicted disparity D with a height of H / 4 and a width of W / 4 14-1 , for D 14-1 Using upsampling, we get an output predicted disparity D with a height of H and a width of W. 14-2 ; Step S15: Using D 12-2 , D 13-2 and D 14-2 Construct the final loss function to train the network; The formula of the loss function is as follows: Among them, L F is the loss function; Yes D 12-2 The constant coefficient of Yes D 13-2 The constant coefficient of Yes D 14-2 The constant coefficient of Yes D 12-2 , Yes D 13-2 , Yes D 14-2 , is the real parallax; Smooth L1 is the smooth L1 loss.

Images