An infrared stereo matching method based on cost group attention and edge reinforcement

Abstract

The application discloses an infrared stereo matching method based on cost group attention and edge reinforcement, relates to the binocular vision technology field in computer vision, and solves the technical problem of low infrared stereo matching precision in a low-recognition environment.The technical solution points are as follows:multi-scale edge features of an original image are extracted based on a Canny edge detection algorithm and an edge feature extraction network;an edge reinforcement module is constructed in combination with the multi-scale edge features and original image features to obtain fusion features;the fusion features are used to construct a cost group attention module;and a predicted disparity map is obtained through cost aggregation and disparity regression.The edge reinforcement module and the cost group attention module are innovatively adopted, which can respectively supplement and focus on beneficial features in the infrared stereo matching process, so that better stereo matching precision is obtained.

Description

Technical Field

[0001] This application relates to the field of binocular vision technology in computer vision, and more particularly to an infrared stereo matching method based on cost group attention and edge enhancement. Background Technology

[0002] As a research hotspot in computer vision, stereo matching has wide applications in 3D reconstruction, autonomous driving, and mobile robotics. Currently, researchers tend to build an end-to-end stereo matching network based on deep learning methods, directly obtaining a predicted disparity map after inputting a corrected image pair. The core of this approach lies in the construction of the cost volume. Recent research shows that while correlation-based construction methods provide an effective way to measure feature similarity, they lose a significant amount of information because they only generate a single-channel correlation map at each disparity level. Furthermore, current stereo matching work is mostly based on visible light images. However, visible light sensors almost completely lose their effectiveness in low-light environments, while infrared images can overcome sensor failure caused by complex environmental changes, facilitating all-weather, multi-scene depth perception. However, compared to visible light images, infrared images have inherent defects such as unclear textures, blurred edges, and indistinct features. Directly transferring stereo matching algorithms based on visible light images to infrared images will make it difficult to guarantee the accuracy of stereo matching. Improving the accuracy of stereo matching is a problem that urgently needs to be solved. Summary of the Invention

[0003] This application provides an infrared stereo matching method based on cost group attention and edge enhancement, the technical purpose of which is to improve the accuracy of infrared stereo matching methods in low-resolution environments.

[0004] The above-mentioned technical objective of this application is achieved through the following technical solution:

[0005] An infrared stereo matching method based on cost group attention and edge enhancement includes:

[0006] S1: Extract multi-scale edge features from the original image using the Canny edge detection algorithm and an edge feature extraction network;

[0007] S2: Combine the multi-scale edge features and the original image features and perform edge enhancement on them to obtain fused features;

[0008] S3: Construct a cost group attention module based on the fusion features, obtain the group cost volume through the cost group attention module, and determine the optimal number of groups according to the training process;

[0009] S4: The predicted disparity map is obtained by cost aggregation and disparity regression of the group cost volume;

[0010] The beneficial effects of this application are as follows: The infrared stereo matching method based on cost group attention and edge enhancement described in this application proposes a cost group attention module and an edge enhancement strategy in the process of constructing the cost volume. This helps to solve the problems of information loss in the process of constructing the cost volume based on correlation methods and the lack of stereo matching information caused by unclear texture and indistinct features in infrared images, thereby improving the accuracy of infrared stereo matching in low-resolution environments. Attached Figure Description

[0011] Figure 1 This is a flowchart of the method described in this application;

[0012] Figure 2 This is a schematic diagram illustrating the principle of the edge enhancement module in this application;

[0013] Figure 3 This is a schematic diagram of the cost group attention module in this application;

[0014] Figure 4 This is a schematic diagram of the cost aggregation module and the disparity regression module in this application. Detailed Implementation

[0015] The technical solution of this application will be described in detail below with reference to the accompanying drawings.

[0016] like Figure 1 As shown, the infrared stereo matching method based on cost group attention and edge enhancement described in this application includes:

[0017] S1: The edge map of the original image is extracted using the Canny edge detection algorithm, and the edge map is input into the edge feature extraction network to obtain multi-scale edge features.

[0018] Specifically, step S1 includes:

[0019] S11: In the Batch dimension, the left image IM1 and the right image IMr of the original image are concatenated to obtain the image IM;

[0020] S12: Based on IM and using the cv2.Canny() function, obtain the stitched edge map IMe of the left and right images;

[0021] S13: Input IMe into the edge feature extraction network to obtain the multi-scale edge features.

[0022] The downsampling layer of the edge feature extraction network is constructed based on the depthwise separable convolution in MobileNet-v2, and its deconvolution layer is a regular convolution.

[0023] The dimensions of each layer of the multi-scale edge feature map are 16×S respectively.1 / 2 24×S 1 / 4 32×S 1 / 8 96×S 1 / 16 160×S 1 / 32 96×S 1 / 16 32×S 1 / 8 24×S 1 / 4 ; among which, 16×S 1 / 2 This indicates that the feature map is half the size of the original input image in both H and W dimensions, and that the feature map has 16 channels; 24×S 1 / 4 This indicates that the feature map is 1 / 4 the size of the original input image in both H and W dimensions, and that the feature map has 24 channels; 32×S 1 / 8 This indicates that the feature map is 1 / 8 the size of the original input image in both the H and W dimensions, and that the feature map has 32 channels; 96×S 1 / 16 This indicates that the feature map is 1 / 16th the size of the original input image in both the H and W dimensions, and that the feature map has 96 channels; 160×S 1 / 32 This indicates that the feature map is 1 / 32 the size of the original input image in both the H and W dimensions, and that the feature map has 160 channels; 96×S 1 / 16 This indicates that the feature map is 1 / 16th the size of the original input image in both H and W dimensions, and that the feature map has 96 channels; 32×S 1 / 8 This indicates that the feature map is 1 / 8 the size of the original input image in both the H and W dimensions, and that the feature map has 32 channels; 24×S 1 / 4 This indicates that the feature map is 1 / 4 the size of the original input image in both the H and W dimensions, and that the feature map has 24 channels; the dimension is 96×S. 1 / 16 32×S 1 / 8 24×S 1 / 4 The corresponding last three feature maps are denoted as feature map Fe. 1 / 16 Fe 1 / 8 Fe 1 / 4 .

[0024] S2: Combine the multi-scale edge features and the original image features and perform edge enhancement on them to obtain fused features.

[0025] Specifically, step S2 includes:

[0026] S21: Input the image IM into the feature extraction network for feature extraction, and obtain the feature maps of each layer of the original image with dimensions of 16×S. 1 / 2 24×S 1 / 4 32×S 1 / 8 96×S 1 / 16 and 160×S 1 / 32 , dimension 160×S 1 / 32 The corresponding last layer feature map is denoted as F. 1 / 32The downsampling layer of the feature extraction network is constructed based on depthwise separable convolutions in MobileNet-v2.

[0027] S22: Enhance feature map F using the edge enhancement module. 1 / 32 Edge enhancement is performed to obtain feature map f. 1 / 4 .

[0028] Among them, the edge enhancement module applies the feature map F 1 / 32 The process of edge enhancement is as follows Figure 2 As shown, it includes:

[0029] S221: For feature map F 1 / 32 Performing one deconvolution yields a dimension of 96×S. 1 / 16 Feature map F 1 / 16 ;

[0030] Feature map F 1 / 16 With feature map Fe 1 / 16 Concatenating along the channel dimension yields a dimension of 192×S. 1 / 16 Feature map Ff 1 / 16 ;

[0031] S222: For feature map Ff 1 / 16 Performing one deconvolution yields a dimension of 32×S. 1 / 8 Feature map F 1 / 8 ;

[0032] S223: Transfer feature map F 1 / 8 with Fe 1 / 8 Concatenating along the channel dimension yields a dimension of 64×S. 1 / 8 Feature map Ff 1 / 8 ;

[0033] S224: For feature map Ff 1 / 8 Performing one deconvolution yields a dimension of 48×S. 1 / 4 Feature map F 1 / 4 ;

[0034] S225: Transfer feature map F 1 / 4 With feature map Fe 1 / 4 Concatenating along the channel dimension yields a dimension of 72×S. 1 / 4 Feature map Ff 1 / 4 ;

[0035] S226: For feature map Ff 1 / 4 Performing a regular convolution yields a dimension-invariant feature map f. 1 / 4 .

[0036] S23: Using the split function to extract from feature map f 1 / 4The left fusion feature fl and the right fusion feature fr are separated from the data.

[0037] S3: Construct a cost group attention module based on the fusion features, obtain the group cost volume through the cost group attention module, and determine the optimal number of groups according to the training process.

[0038] Specifically, step S3 includes:

[0039] S31: Divide both the left fusion feature fl and the right fusion feature fr into N feature groups evenly along the channel dimension; each feature group has 72 / N channels, and the nth feature group fl n ,fr n It consists of the n×72 / N, n×72 / N+1, ..., (n+1)×72 / N-1 channels in fl and fr respectively; n≤N, where N is the optimal number of groups corresponding to the minimum loss after training;

[0040] S32: For each feature group, construct the correlation-based group cost volume according to equation (1), obtaining N dimensions [D max The cost volumes C0, C1, ..., C of [H / 4, W / 4] are given by [C0, C1, ..., C2]. N-1 ;

[0041]

[0042] Among them, D mmax The maximum disparity is indicated by < and >, which represent the inner product.

[0043] S33: After convolution, the left fusion feature fl is obtained as an N×S dimension. 1 / 4 Feature map fN;

[0044] S34: Decompose the feature map fN along the channel dimension into N group cost volume feature attention maps fN0, fN1, ..., fN N-1 According to equation (2), the cost volumes C0, C1, ..., C are determined. N-1 The correlation probability distribution in the data ultimately yields the group cost volume with feature attention.

[0045]

[0046] Where “·” represents broadcast multiplication; j∈[0,N-1].

[0047] Equations (1) and (2) above constitute the cost group attention module, as follows: Figure 3 As shown.

[0048] S4: The predicted disparity map is obtained by cost aggregation and disparity regression of the group cost volume.

[0049] Specifically, the process of cost aggregation and disparity regression is as follows: Figure 4 As shown, it includes:

[0050] The group cost volume The input is fed into the cost aggregation module, resulting in feature maps of each layer with dimensions [2N, D]. mmax [ / 8, H / 8, W / 8], [4N, D] mmax [ / 16, H / 16, W / 16], [8N, D] mmax [ / 32, H / 32, W / 32], [4N, D] mmax [ / 16, H / 16, W / 16], [2N, D] mmax [ / 8, H / 8, W / 8], [N, D] mmax [ / 4, H / 4, W / 4], [1, D] max / 4, H / 4, W / 4], with dimensions [1, D] max The last layer feature map corresponding to [H / 4, W / 4] is denoted as V. 1 / 4 ;

[0051] For feature map V 1 / 4 Upsampled to dimension [1, D] max [H, W], thus obtaining the cost volume V;

[0052] The cost volume V is transformed along the disparity dimension using softmax, and the final predicted disparity map Map is obtained by regression according to equation (3).

[0053]

[0054] Where k represents the disparity magnitude, p k represents the probability that the disparity level of a pixel is at k, and d represents the regressive disparity of the pixel.

[0055] Equation (3) above is the disparity regression module.

[0056] As a specific embodiment, the cost aggregation module consists of consecutive 3D convolutional layers.

[0057] The above are exemplary embodiments of this application, and the scope of protection of this application is defined by the claims and their equivalents.

Claims

1. An infrared stereo matching method based on cost group attention and edge enhancement, characterized in that, include: S1: Extract multi-scale edge features from the original image using the Canny edge detection algorithm and an edge feature extraction network; S2: Combine the multi-scale edge features and the original image features and perform edge enhancement to obtain fusion features, including left fusion feature fl and right fusion feature fr; The feature map F is processed by the edge enhancement module. 1 / 32 The process of edge enhancement includes: For feature map F 1 / 32 Performing one deconvolution yields a dimension of 96×S. 1 / 16 Feature map F 1 / 16 ; Feature map F 1 / 16 With feature map Fe 1 / 16 Concatenating along the channel dimension yields a dimension of 192×S. 1 / 16 Feature map Ff 1 / 16 ; For feature map Ff 1 / 16 Performing one deconvolution yields a dimension of 32×S. 1 / 8 Feature map F 1 / 8 ; Feature map F 1 / 8 With feature map Fe 1 / 8 Concatenating along the channel dimension yields a dimension of 64×S. 1 / 8 Feature map Ff 1 / 8 ; For feature map Ff 1 / 8 Performing one deconvolution yields a dimension of 48×S. 1 / 4 Feature map F 1 / 4 ; Feature map F 1 / 4 With feature map Fe 1 / 4 Concatenating along the channel dimension yields a dimension of 72×S. 1 / 4 Feature map Ff 1 / 4 ; For feature map Ff 1 / 4 Performing a regular convolution yields a dimension-invariant feature map f. 1 / 4 ; Among them, feature map F 1 / 32 The original image feature map has a dimension of 160×S. 1 / 32 The corresponding last layer feature map, feature map Fe 1 / 16 The multi-scale edge feature map has a dimension of 96×S. 1 / 16 The corresponding feature map, feature map Fe 1 / 8 The multi-scale edge feature map has a dimension of 32×S. 1 / 8 The corresponding feature map, feature map Fe 1 / 4 The multi-scale edge feature map has a dimension of 24×S. 1 / 4 The corresponding feature map; S3: Construct a cost group attention module based on the fusion features, obtain the group cost volume through the cost group attention module, and determine the optimal number of groups according to the training process; The construction of the cost group attention module includes: Both the left fusion feature fl and the right fusion feature fr are uniformly divided into N feature groups along the channel dimension; each feature group has 72 / N channels, and the nth feature group fl n ,fr n It consists of the n×72 / N, n×72 / N+1, ..., (n+1)×72 / N-1 channels in fl and fr respectively; n≤N, where N is the optimal number of groups corresponding to the minimum loss after training; For each feature group, construct the correlation-based group cost volume according to equation (1), resulting in N dimensions [D] max The cost volumes C0, C1, ..., C of [ / 4,H / 4,W / 4] are... N-1 ; Among them, D max The maximum disparity is represented by <,>, which represents the inner product. The left fusion feature fl is convolved to obtain a dimension of N×S. 1 / 4 Feature map fN; Decompose the feature map fN along the channel dimension into N group cost volume feature attention maps fN0, fN1, ..., fN N-1 According to equation (2), the cost volumes C0, C1, ..., C are determined. N-1 The correlation probability distribution in the data ultimately yields the group cost volume with feature attention. Where "·" represents broadcast multiplication; j∈[0,N-1]; Equations (1) and (2) constitute the cost group attention module; S4: The predicted disparity map is obtained by cost aggregation and disparity regression of the group cost volume.

2. The method as described in claim 1, characterized in that, The acquisition of the multi-scale edge features includes: In the Batch dimension, the left image IMi and the right image Imr of the original image are concatenated to obtain image IM; Based on IM, the stitching edge map Ime of the left and right images is obtained using the cv2.Canny() function; The Ime is input into the edge feature extraction network to obtain the multi-scale edge features; The downsampling layer of the edge feature extraction network is constructed based on the depthwise separable convolution in MobileNet-v2, and its deconvolution layer is a regular convolution. The dimension of the feature maps of each layer of the multi-scale edge feature is 16×S. 1 / 2 24×S 1 / 4 32×S 1 / 8 96×S 1 / 16 160×S 1 / 32 96×S 1 / 16 32×S 1 / 8 24×S 1 / 4 ; among which, 16×S 1 / 2 This indicates that the feature map is half the size of the original input image in both the H and W dimensions, and that the feature map has 16 channels; 24×S 1 / 4 This indicates that the feature map is 1 / 4 the size of the original input image in both H and W dimensions, and that the feature map has 24 channels; 32×S 1 / 8 This indicates that the feature map is 1 / 8 the size of the original input image in both the H and W dimensions, and that the feature map has 32 channels; 96×S 1 / 16 This indicates that the feature map is 1 / 16th the size of the original input image in both the H and W dimensions, and that the feature map has 96 channels; 160×S 1 / 32 This indicates that the feature map is 1 / 32 the size of the original input image in both the H and W dimensions, and that the feature map has 160 channels; 96×S 1 / 16 This indicates that the feature map is 1 / 16th the size of the original input image in both H and W dimensions, and that the feature map has 96 channels; 32×S 1 / 8 This indicates that the feature map is 1 / 8 the size of the original input image in both the H and W dimensions, and that the feature map has 32 channels; 24×S 1 / 4 This indicates that the feature map is 1 / 4 the size of the original input image in dimensions H and W, and the feature map has 24 channels; the dimension is 96×S. 1 / 16 32×S 1 / 8 24×S 1 / 4 The corresponding last three feature maps are denoted as feature map Fe. 1 / 16 Fe 1 / 8 Fe 1 / 4 .

3. The method as described in claim 2, characterized in that, The process of obtaining the fusion features includes: The image IM is input into a feature extraction network for feature extraction, resulting in feature maps of each layer of the original image with dimensions of 16×S. 1 / 2 24×S 1 / 4 32×S 1 / 8 96×S 1 / 16 and 160×S 1 / 32 , dimension 160×S 1 / 32 The corresponding last layer feature map is denoted as F. 1 / 32 The downsampling layer of the feature extraction network is constructed based on depthwise separable convolutions in MobileNet-v2. The feature map F is processed by the edge enhancement module. 1 / 32 Edge enhancement is performed to obtain feature map f. 1 / 4 ; The split function is used to extract from feature map f 1 / 4 The left fusion feature fl and the right fusion feature fr are separated from the data.

4. The method as described in claim 3, characterized in that, The cost aggregation and disparity regression process includes: The group cost volume The input is fed into the cost aggregation module, resulting in feature maps of each layer with dimensions [2N, D]. max / 8,H / 8,W / 8]、[4N,D max [ / 16,H / 16,W / 16]、[8N,D max [32,H / 32,W / 32]、[4N,D max / 16,H / 16,W / 16]、[2N,D max / 8,H / 8,W / 8]、[N,D max / 4,H / 4,W / 4]、[1,D max / 4,H / 4,W / 4], with dimensions [1,D max The last feature map corresponding to [H / 4, W / 4] is denoted as V. 1 / 4 ; For feature map V 1 / 4 Upsampled to dimension [1,D max [,H,W], to obtain the cost volume V; The cost volume V is transformed along the disparity dimension using softmax, and the final predicted disparity map Map is obtained by regression according to equation (3). Where k represents the disparity magnitude, p k represents the probability that the disparity level of a pixel is at k, and d represents the regressive disparity of the pixel.

5. The method as described in claim 4, characterized in that, The cost aggregation module consists of consecutive 3D convolutional layers.

Images