Lightweight stereo matching method and device for heat map

Abstract

The application discloses a kind of lightweight heat map stereo matching method and device, to solve the technical problem that existing lightweight heat map stereo matching method is prone to information loss when capturing subtle depth difference, resulting in the precision of final disparity result is poor.Method includes obtaining left eye, right eye heat image, input preset heat image stereo matching model (including multiscale fusion feature extraction, edge feature extraction, enhanced wavelet attention module and multiple cost aggregation residual blocks, convolution module).Multiscale module extracts features, generates left eye and right eye fusion feature map, and constructs initial cost volume;Edge module outputs left eye edge probability feature map, and combines initial cost volume to obtain intermediate cost volume.Wavelet attention module outputs attention weight with left eye fusion feature map;Cost aggregation residual block is multiplied by weight and intermediate cost volume, and initial disparity map is generated through disparity regression;Convolution module is combined with left eye edge probability feature map to refine, and finally generate target disparity map.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and image processing technology, and in particular to a lightweight thermal image stereo matching method and apparatus. Background Technology

[0002] Stereo matching is a core technology in computer vision for recovering three-dimensional structures from two-dimensional images. Essentially, it involves finding pixel-level correspondences (i.e., "matching points") between images from two perspectives (such as the left and right views of a binocular camera) and then calculating disparity (the positional deviation of pixels in the left and right views).

[0003] Most current stereo matching work focuses on RGB (Red, Green, Blue) image pairs. However, RGB cameras are susceptible to lighting conditions and struggle to operate effectively in smoky or low-light environments. On the other hand, thermal imaging cameras are virtually unaffected by ambient lighting, allowing them to operate effectively even when RGB cameras are insufficient. With the decreasing cost of thermal imaging cameras, their application prospects are expanding. However, thermal images typically lack texture, are noisy, and often have low resolution, posing significant challenges to stereo matching.

[0004] In pursuit of high computational efficiency, existing lightweight thermal image stereo matching methods often simplify the parameter scale and network depth in the feature extraction stage. However, thermal images themselves have characteristics such as sparse texture information, strong noise interference, and blurred temperature difference between the target and the background. This makes it easy for the model to lose information when capturing subtle depth differences, resulting in poor accuracy of the final parallax result. Summary of the Invention

[0005] This invention provides a lightweight thermal image stereo matching method and apparatus to solve the technical problem that existing lightweight thermal image stereo matching methods are prone to information loss when capturing subtle depth differences, resulting in poor accuracy of the final parallax results.

[0006] The first aspect of this invention provides a lightweight thermal image stereo matching method, comprising:

[0007] Acquire left-eye thermal image and right-eye thermal image, and input the left-eye thermal image and right-eye thermal image into a preset stereo matching model for thermal images. The preset stereo matching model for thermal images includes a multi-scale fusion feature extraction module, an edge feature extraction module, an enhanced wavelet attention module, multiple cost aggregation residual blocks and a convolution module.

[0008] The multi-scale fusion feature extraction module extracts multi-scale fusion features from the left and right thermal images to generate a left-eye fusion feature map and a right-eye fusion feature map. Based on the left-eye fusion feature map and the right-eye fusion feature map, an initial cost body is constructed.

[0009] The edge feature extraction module is used to extract edge features from the left eye thermal image, outputting a left eye edge probability feature map, and an intermediate cost body is generated based on the left eye edge probability feature map and the initial cost body.

[0010] The left eye fusion feature is used as the input to the enhanced wavelet attention module, and the attention weights are output.

[0011] An initial disparity map is generated using multiple cost aggregation residual blocks based on the attention weights and the intermediate cost volume;

[0012] The convolutional module refines the initial disparity map layer by layer based on the left eye edge probability feature map, the left eye thermal image, and the left eye fusion feature map to generate the target disparity map.

[0013] Optionally, the step of performing multi-scale fusion feature extraction on the left-eye thermal image and the right-eye thermal image through the multi-scale fusion feature extraction module to generate a left-eye fusion feature map and a right-eye fusion feature map includes:

[0014] Feature extraction is performed on the left and right thermal images respectively, and multi-scale left and right feature maps are output.

[0015] The multi-scale left-eye feature map and the multi-scale right-eye feature map are upsampled and fused multiple times to generate a left-eye fused feature map and a right-eye fused feature map.

[0016] Optionally, the step of using the edge feature extraction module to extract edge features from the left eye thermal image and outputting a left eye edge probability feature map includes:

[0017] The weighted Sobel algorithm is used to perform edge detection on the left eye thermal image, and the first left eye edge feature is output.

[0018] Average pooling is performed on the left eye thermal image to generate blurred feature values, and the blurred feature values ​​are subtracted from the left eye thermal image to determine the second left eye edge features;

[0019] The first left eye edge feature and the second left eye edge feature are fused together to generate the left eye fused edge feature;

[0020] The left eye fused edge features are subjected to convolution operation to output the first left eye convolution edge features, and the first left eye convolution edge features are nonlinearly mapped using the LeakyReLU activation function to output the first left eye nonlinear edge features.

[0021] The first left-eye nonlinear edge feature is convolved to output the second left-eye convolutional edge feature, and the LeakyReLU activation function is used to perform nonlinear mapping on the second left-eye convolutional edge feature to output the second left-eye nonlinear edge feature.

[0022] Perform a convolution operation on the second left-eye nonlinear edge feature to output the third left-eye convolution edge feature;

[0023] The Sigmoid activation function is used to linearly transform the edge features of the third left eye convolution, and the left eye edge probability feature map is output.

[0024] Optionally, generating an intermediate cost body based on the left eye edge probability feature map and the initial cost body includes:

[0025] Channel attention is calculated on the left eye edge probability feature map, and the left eye edge attention feature is output;

[0026] The left eye edge attention feature is multiplied by the initial cost volume to output the intermediate cost volume.

[0027] Optionally, the step of using the left eye fusion feature as input to the enhanced wavelet attention module and outputting attention weights includes:

[0028] Perform a two-dimensional convolution on the left eye fusion features to output the left eye two-dimensional convolution features;

[0029] The discrete wavelet transform decomposition algorithm is used to decompose the left eye two-dimensional convolutional features to generate multiple weighted frequency band components;

[0030] Global weight scaling is performed on the weights corresponding to each weighted frequency band component, and each weighted frequency band component after weight scaling is output.

[0031] The weighted frequency band components that have undergone weight scaling are spatially adaptively weighted to generate multiple secondary weighted frequency band components.

[0032] The multiple secondary weighted frequency band components are spliced ​​together to output the fused frequency band features;

[0033] The fused frequency band features are dynamically enhanced to generate enhanced frequency band features. The enhanced frequency band features and the fused frequency band features are then added element by element to output the summed frequency band features.

[0034] The summed frequency band features are upsampled to generate attention weights.

[0035] Optionally, the plurality of cost aggregation residual blocks include a first cost aggregation residual block, a second cost aggregation residual block, and a third cost aggregation residual block; the step of generating an initial disparity map using the plurality of cost aggregation residual blocks according to the attention weights and the intermediate cost body includes:

[0036] The intermediate cost body is aggregated using the first cost aggregation residual block to generate an aggregated cost body;

[0037] Multiply the aggregated cost body and the attention weight to output the weighted cost body;

[0038] The weighted cost volume is downsampled and aggregated using the second cost aggregation residual block to output a low-resolution cost volume.

[0039] The third cost aggregation residual block is used to downsample and aggregate the low-resolution cost volume, and the target cost volume is output.

[0040] The target cost volume is interpolated to output the interpolated cost volume. The interpolated cost volume is then multiplied by the attention weight to generate the aggregated cost volume. Disparity regression is then performed on the aggregated cost volume to output the initial disparity map.

[0041] Optionally, the cost aggregation residual block includes multiple residual blocks; each residual block includes a two-dimensional convolutional activation function module, a depthwise convolutional activation function module, and a two-dimensional convolutional linear output module; the feature processing of the residual block is specifically as follows:

[0042] The first cost body is determined by performing two-dimensional convolution and nonlinear mapping on the input cost body input to the cost aggregation residual block through a two-dimensional convolution activation function module.

[0043] The first cost body is subjected to deep convolution and nonlinear mapping using a deep convolution activation function module to generate the second cost body.

[0044] The second cost body is input into the deep convolution activation function module for deep convolution and nonlinear mapping, and the third cost body is output.

[0045] The third cost body is subjected to two-dimensional convolution by the two-dimensional convolution linear output module to generate the output cost body.

[0046] Optionally, the step of using the convolutional module to refine the initial disparity map layer by layer based on the left eye edge probability feature map, the left eye thermal image, and the left eye fusion feature map to generate a target disparity map includes:

[0047] Perform convolution operation on the left eye fusion feature map to output the first disparity refinement feature;

[0048] The first disparity refinement feature and the left eye edge probability feature map are regressively connected to generate the second disparity refinement feature;

[0049] The left eye thermal image is downsampled to generate detail features, and the second disparity refinement feature and the detail features are fused to generate a fused disparity feature;

[0050] The fused disparity features are sequentially upsampled and nonlinearly mapped to generate upsampling weights;

[0051] The initial disparity map is amplified by increasing the disparity value, and an amplified disparity map is output. The amplified disparity map is then weighted and averaged using the upsampling weights to generate the target disparity map.

[0052] Optionally, the model training process for the preset stereo matching model for thermal images specifically includes:

[0053] Obtain the heatmap dataset for model training;

[0054] A three-stage distillation learning approach is used to train the initial stereo matching model for thermal images based on the heat map dataset used for model training, thereby determining the trained preset stereo matching model for thermal images.

[0055] A lightweight thermal imaging stereo matching device is provided in a second aspect of the present invention, comprising:

[0056] The acquisition module is used to acquire the left-eye thermal image and the right-eye thermal image, and input the left-eye thermal image and the right-eye thermal image into a preset stereo matching model for thermal images. The preset stereo matching model for thermal images includes a multi-scale fusion feature extraction module, an edge feature extraction module, an enhanced wavelet attention module, multiple cost aggregation residual blocks and a convolution module.

[0057] The multi-scale fusion extraction module is used to extract multi-scale fusion features from the left-eye thermal image and the right-eye thermal image through the multi-scale fusion feature extraction module, generate a left-eye fusion feature map and a right-eye fusion feature map, and construct an initial cost body based on the left and right-eye fusion feature maps;

[0058] An edge feature extraction module is used to extract edge features from the left eye thermal image, output a left eye edge probability feature map, and generate an intermediate cost body based on the left eye edge probability feature map and the initial cost body.

[0059] An enhancement module is used to take the left eye fusion feature map as input to the enhanced wavelet attention module and output attention weights.

[0060] The aggregation module is used to generate an initial disparity map by using multiple cost aggregation residual blocks based on the attention weights and the intermediate cost body;

[0061] The refinement module is used to refine the initial disparity map layer by layer based on the left eye edge probability feature map using the convolution module to generate the target disparity map.

[0062] As can be seen from the above technical solutions, the present invention has the following advantages:

[0063] The above-mentioned technical solution of the present invention provides a lightweight stereo matching method for thermal images. It acquires left-eye and right-eye thermal images, inputs them into a pre-set stereo matching model for thermal images, and includes a multi-scale fusion feature extraction module, an edge feature extraction module, an enhanced wavelet attention module, multiple cost aggregation residual blocks, and a convolution module. The multi-scale fusion feature extraction module performs multi-scale fusion feature extraction on the left-eye and right-eye thermal images, generating left-eye fusion feature maps and right-eye fusion feature maps, and constructs an initial cost body based on these maps. The edge feature extraction module extracts edge features from the left-eye thermal image, outputting a left-eye edge probability feature map, and generates an intermediate cost body based on the left-eye edge probability feature map and the initial cost body. The left-eye fusion features are used as input to the enhanced wavelet attention module, which outputs attention weights. Multiple cost aggregation... The residual block generates an initial disparity map based on attention weights and intermediate cost volumes. A convolutional module refines the initial disparity map layer by layer using the left-eye edge probability feature map, the left-eye thermal image, and the left-eye fusion feature map to generate the target disparity map. Based on this approach, this invention achieves cross-scale feature aggregation through a multi-scale fusion feature extraction module, accurately preserving the depth correlation information of subtle textures and weak temperature difference regions at different scales in the thermal image. This avoids the problem of missing subtle depth information caused by single-scale feature extraction, laying a richer and more complete feature foundation for the subsequent construction of the initial cost volume. Simultaneously, this invention utilizes the left-eye edge probability feature map generated by the edge feature extraction module to explicitly enhance the depth difference signal at the edge contour of the thermal image, optimizing the edge region representation of the intermediate cost volume to reduce disparity shift. Furthermore, the introduction of enhanced wavelet transform in the cost aggregation module highlights edge information and suppresses noise, thereby significantly improving the accuracy of disparity results in thermal image stereo matching. Attached Figure Description

[0064] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 This is a flowchart illustrating the steps of a lightweight thermal image stereo matching method provided in Embodiment 1 of the present invention.

[0066] Figure 2 This is an overall framework diagram of a preset stereo matching model for thermal images provided in Embodiment 1 of the present invention;

[0067] Figure 3 This is a schematic diagram of the edge feature extraction module provided in Embodiment 1 of the present invention;

[0068] Figure 4 This is a schematic diagram of the structure of the enhanced wavelet attention module provided in Embodiment 1 of the present invention;

[0069] Figure 5 This is a schematic diagram of the structure of the residual block provided in Embodiment 1 of the present invention;

[0070] Figure 6 This is a schematic diagram of the binocular ranging principle provided in Embodiment 1 of the present invention;

[0071] Figure 7 This is a flowchart illustrating the steps of training a preset stereo matching model for thermal images, as provided in Embodiment 2 of the present invention.

[0072] Figure 8 The test results of the ACVNet model provided in Embodiment 2 of the present invention on the MS² stereo dataset;

[0073] Figure 9 The test results of the LightStereo model provided in Embodiment 2 of the present invention on the MS² stereo dataset;

[0074] Figure 10 The test results of the preset stereo matching model for thermal images provided in Embodiment 2 of the present invention on the MS² stereo dataset;

[0075] Figure 11 The test results of the preset stereo matching model for thermal images provided in Embodiment 2 of the present invention at night on the MS² stereo dataset;

[0076] Figure 12 This is a structural block diagram of a lightweight thermal imaging stereo matching device provided in Embodiment 3 of the present invention. Detailed Implementation

[0077] This invention provides a lightweight thermal image stereo matching method and apparatus to solve the technical problem that existing lightweight thermal image stereo matching methods are prone to information loss when capturing subtle depth differences, resulting in poor accuracy of the final parallax results.

[0078] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0079] Please see Figure 1 , Figure 1 This is a flowchart illustrating the steps of a lightweight thermal image stereo matching method provided in Embodiment 1 of the present invention.

[0080] This invention provides a lightweight thermal image stereo matching method, comprising:

[0081] Step 101: Obtain the left and right thermal images. Input the left and right thermal images into a preset stereo matching model for thermal images. The preset stereo matching model for thermal images includes a multi-scale fusion feature extraction module, an edge feature extraction module, an enhanced wavelet attention module, multiple cost aggregation residual blocks, and a convolution module.

[0082] It should be noted that you should refer to [link / reference]. Figure 2The stereo matching model for thermal images proposed in this invention is an improvement on the lightweight LightStereo framework. The overall process includes feature extraction, cost volume construction, edge attention enhancement, cost aggregation, disparity regression, and disparity refinement. The feature extraction stage uses the lightweight MobileNetV2 (MobileNetwork Version 2), progressively downsampling to 1 / 32 resolution and then upsampling and fusing to form a multi-scale fusion feature extraction module. The cost volume is constructed using the extracted 1 / 4 resolution left and right image features (i.e., left-eye fusion feature map and right-eye fusion feature map), with a maximum disparity of 192 / 4, or 48. Edge features (left-eye edge probability feature map) are extracted from the original left image and then applied to the cost volume through channel attention to obtain an intermediate cost volume. This process increases the cost volume's attention to edge information in the disparity dimension. In the subsequent cost aggregation stage, multiple cost aggregation residual blocks (using the classic inverted residual structure of MobileNetV2) are first used to extract high-level features from the cost volume. Then, an enhanced wavelet transform attention mechanism (enhanced wavelet attention module) is used to generate attention for the extracted multi-scale features to guide the cost volume, further enhancing the cost volume aggregation effect. Finally, disparity values ​​(initial disparity map) at 1 / 4 resolution are obtained through disparity regression. The disparity values ​​are then refined layer by layer on the extracted original features using the initially generated edge map through residual connections to obtain the final disparity values. That is, a convolutional module is used to refine the initial disparity map layer by layer based on the left eye edge probability feature map to generate the target disparity map.

[0083] Step 102: Extract multi-scale fusion features from the left and right thermal images using the multi-scale fusion feature extraction module to generate left and right fusion feature maps. Construct the initial cost body based on the left and right fusion feature maps.

[0084] Specifically, step 102 may include the following sub-steps S21-S22:

[0085] Step S21: Extract features from the left and right thermal images respectively, and output multi-scale left-eye feature maps and multi-scale right-eye feature maps;

[0086] Step S22: Perform multiple upsampling and fusion operations on the multi-scale left eye feature map and the multi-scale right eye feature map to generate the left eye fusion feature map and the right eye fusion feature map.

[0087] It should be noted that a pre-trained model (MobileNetV2) is loaded as a feature extractor. The left and right eye images (i.e., the left eye thermal image and the right eye thermal image) are input into the model for feature extraction, and multi-scale feature maps are output. The multi-scale feature maps are then passed through a lightweight feature pyramid (FPN) to upsample the deep features and fuse them with the shallow features to generate a feature pyramid rich in multi-scale contextual information. Finally, a set of feature maps extracted from different levels is output for subsequent tasks such as cost volume construction, aggregation, and disparity prediction.

[0088] Taking the left-eye fusion feature map processing as an example, MobileNetV2 extracts features from the left-eye thermal image to obtain a multi-scale feature map. This feature map includes a quarter-scale feature map, an eighth-scale feature map, a sixteenth-scale feature map, and a thirty-second-scale feature map. Next, the thirty-second-scale feature map is upsampled to obtain an upsampled one-sixteenth-scale feature map. This upsampled feature map is then concatenated with the first-sixteenth-scale feature map along the channel dimension to obtain an added one-sixteenth-scale feature map. This added one-sixteenth-scale feature map is then upsampled to obtain an upsampled one-eighth-scale feature map. This added one-eighth-scale feature map is then concatenated with the first-eighth-scale feature map along the channel dimension to obtain an added one-eighth-scale feature map. This added one-eighth-scale feature map is then upsampled to obtain an upsampled quarter-scale feature map. This upsampled quarter-scale feature map is then concatenated with the first-eighth-scale feature map along the channel dimension to obtain the left-eye fusion feature map. The right-eye fusion feature map can be obtained similarly.

[0089] Step 103: Use the edge feature extraction module to extract edge features from the left eye thermal image, output the left eye edge probability feature map, and generate the intermediate cost body based on the left eye edge probability feature map and the initial cost body.

[0090] It should be noted that the weighted Sobel algorithm is used to perform edge detection on the left-eye thermal image. Custom weights are used to enhance the response of edges with weak temperature differences in the thermal image, outputting the first left-eye edge feature that highlights the main contour. Then, average pooling is performed on the left-eye thermal image to generate blurred feature values. These blurred feature values ​​are subtracted from the original left-eye thermal image to capture subtle edge details, resulting in the second left-eye edge feature. Subsequently, the two types of edge features are fused through channel weighting to generate a left-eye fused edge feature that takes into account both primary and secondary edges. After multiple rounds of convolution operations and LeakyReLU nonlinear mapping to optimize the feature expression, the final left-eye edge probability feature map is output through the Sigmoid activation function. In this map, each pixel value corresponds to the confidence level of "this position is an edge", which can accurately mark blurred or weakly contrasting edge regions in the thermal image. When generating the intermediate cost volume, the left eye edge probability feature map is first adapted in terms of dimension (e.g., by adjusting the number of channels through 1×1 convolution) to make its spatial dimension consistent with the initial cost volume. Then, the confidence weight of the edge probability feature map is assigned to the initial cost volume by using element-wise multiplication: edge region pixels have a higher confidence weight, so their cost value is further enhanced to highlight the depth difference, while non-edge regions maintain a reasonable cost value to avoid excessive interference. Finally, an intermediate cost volume that can accurately characterize the depth matching relationship at the edge is generated, making up for the deficiency of the initial cost volume in paying insufficient attention to edge details.

[0091] Specifically, the process of extracting edge features from the left-eye thermal image using the edge feature extraction module and outputting the left-eye edge probability feature map can be executed through sub-steps S31-S37:

[0092] Step S31: Use the weighted Sobel algorithm to perform edge detection on the left eye thermal image and output the first left eye edge features;

[0093] Step S32: Perform average pooling on the left eye thermal image to generate blurred feature values, and subtract the blurred feature values ​​from the left eye thermal image to determine the second left eye edge features;

[0094] Step S33: Fuse the first left eye edge features and the second left eye edge features to generate the left eye fused edge features;

[0095] Step S34: Perform convolution operation on the left eye fused edge features to output the first left eye convolution edge features, and use the LeakyReLU activation function to perform nonlinear mapping on the first left eye convolution edge features to output the first left eye nonlinear edge features;

[0096] Step S35: Perform convolution operation on the first left eye nonlinear edge feature to output the second left eye convolution edge feature, and use the LeakyReLU activation function to perform nonlinear mapping on the second left eye convolution edge feature to output the second left eye nonlinear edge feature.

[0097] Step S36: Perform convolution operation on the second left-eye nonlinear edge features to output the third left-eye convolution edge features;

[0098] Step S37: Apply the Sigmoid activation function to perform a linear transformation on the edge features of the third left eye convolution, and output the left eye edge probability feature map.

[0099] It should be noted that you should refer to [link / reference]. Figure 3 The input left-eye thermal image is divided into two branches. The first branch first performs 5×5 mean blurring (simulating low-pass filtering), then calculates the absolute difference between the original image and the blurred image, i.e., average pooling is performed on the left-eye thermal image to generate blurred feature values. The blurred feature values ​​are then subtracted from the left-eye thermal image to determine the second left-eye edge features. This process can highlight "local gray-level abrupt change regions" in thermal images—the edges of objects in thermal images usually exhibit slowly changing gray-level gradients due to differences in thermal radiation. Blur difference can amplify this gradient difference, allowing even blurred edges to be captured.

[0100] Furthermore, the second branch, the Sobel algorithm (Sobel operator), calculates the gradients in the x and y directions respectively, and then calculates the gradient magnitude. This weighted Sobel algorithm is used to extract features from the left-eye thermal image, outputting the first left-eye edge features. This step strengthens the directional information of the edges and supplements the response of the blur difference to "steep edges" (the edges of some metals and high-temperature objects in thermal images may have significant gradients). Considering that the Sobel operator may introduce noise and lacks learnable capabilities, it is weighted by 0.3 and fused into the first branch. This preserves the overall trend of blurred edges while enhancing the local response of sharp edges, forming more robust edge candidate features. The convolutional layer adopts a simplified structure of "3 convolutional layers + LeakyReLU + Sigmoid," progressively extracting high-order edge features. This avoids parameter redundancy while effectively solving the gradient vanishing problem in weak edge regions of thermal images, and also increases the number of channels to compensate for the limited information in a single channel of the thermal image.

[0101] Furthermore, the process of generating an intermediate cost body based on the left eye edge probability feature map and the initial cost body can be performed through sub-steps S38-S39:

[0102] Step S38: Calculate channel attention on the left eye edge probability feature map and output the left eye edge attention feature;

[0103] Step S39: Multiply the left eye edge attention features and the initial cost volume to output the intermediate cost volume.

[0104] It should be noted that channel attention (SE channel attention, Squeeze-and-Excitation) is performed on the left eye edge probability feature map to output the left eye edge attention feature that enhances the key edge channels. Subsequently, this left eye edge attention feature is multiplied element-wise with the initial cost volume: due to the high weight value of the attention feature, the cost value of edge regions in the initial cost volume is significantly enhanced to highlight depth differences, while non-edge regions retain basic matching information through reasonable weights. Finally, an intermediate cost volume that can accurately enhance the depth correlation signal at the edge is generated, making up for the deficiency of insufficient attention to edge details in the initial cost volume.

[0105] Step 104: Use the left eye fusion feature map as input to the enhanced wavelet attention module and output the attention weights.

[0106] It should be noted that, firstly, a two-dimensional convolution is performed on the left eye fusion feature map. Key features are extracted through the convolution kernel, and the channel dimension is adjusted to output the right eye two-dimensional convolution feature map. Then, a discrete wavelet transform decomposition algorithm is used to decompose it, resulting in multiple weighted frequency band components containing low-frequency global trends and horizontal, vertical, and diagonal high-frequency details. This invention assigns a weight to each component; specifically, each of the four sub-bands is multiplied by a learnable weight, namely λ1, λ2, λ3, and λ4. Next, the learnable weights of each component are globally weighted (the weights are dynamically optimized as the network backpropagates, so that the weights of the depth-matching key frequency bands are adaptively increased), outputting the weighted frequency band components. Finally, spatial adaptive weighting is applied to different spatial regions of each component (such as edges and areas with weak temperature difference textures). Dynamic weighting generates secondary weighted frequency band components that balance frequency domain importance and spatial attention. Multiple secondary weighted frequency band components are then concatenated to aggregate multi-scale frequency domain and spatial information, resulting in fused frequency band features. Dynamic frequency enhancement is applied to these fused features to strengthen key frequency components for depth estimation, generating enhanced frequency band features. These enhanced features are then added element-wise to the fused frequency band features (introducing a residual mechanism to preserve original feature information), outputting the summed frequency band features. Finally, the summed frequency band features are upsampled to match their spatial dimension with the input left-eye fusion feature map, generating attention weights. This weight map accurately marks the most critical regions for binocular thermal image matching in the left-eye fusion feature map, providing dynamic attention guidance for subsequent cost aggregation.

[0107] Specifically, step 104 may include the following sub-steps S41-S47:

[0108] Step S41: Perform a two-dimensional convolution on the left eye fusion feature map to output the left eye two-dimensional convolution feature;

[0109] Step S42: The discrete wavelet transform decomposition algorithm is used to decompose the two-dimensional convolutional features of the left eye, generating multiple weighted frequency band components;

[0110] Step S43: Perform global weight scaling on the weights corresponding to each weighted frequency band component, and output each weighted frequency band component after weight scaling.

[0111] Step S44: Perform spatial adaptive weighting on each weighted frequency band component after weight scaling to generate multiple secondary weighted frequency band components;

[0112] Step S45: Perform feature stitching on multiple secondary weighted frequency band components and output the fused frequency band features;

[0113] Step S46: Perform dynamic frequency enhancement on the fused frequency band features to generate enhanced frequency band features, and add the enhanced frequency band features and the fused frequency band features element by element to output the added frequency band features;

[0114] Step S47: Upsample the summed frequency band features to generate attention weights.

[0115] It should be noted that you should refer to [link / reference]. Figure 4 The input image (left eye fused feature map) is first processed through a 1×1 convolution kernel and then subjected to DWT (Discrete Wavelet Transform). Based on db4 (Daubechies-4Wavelet) wavelet decomposition, the original image (left eye 2D convolutional feature) is decomposed into LL low-frequency component and Yh high-frequency component. The high-frequency component is further decomposed into three components: LH horizontal high-frequency component, HL vertical high-frequency component, and HH diagonal high-frequency component. The core of wavelet transform is "filter convolution + downsampling", the principle of which is as follows:

[0116] For a given signal x[n], the one-dimensional wavelet transform is performed using a low-pass filter with coefficients h[k] and a high-pass filter g[k] (k=0,1,...,L-1, where L is the filter length, e.g., L=8 for db4). The low-frequency component y low The calculation of [m] is as follows:

[0117] ;

[0118] High-frequency component y high The calculation of [m] is as follows:

[0119] ;

[0120] Where m is the output index, 2m corresponds to a downsampling step size of 2, ensuring that the output length is N / 2.

[0121] Furthermore, when processing two-dimensional images (such as thermal images), a "separable transformation" is employed:

[0122] Row Direction DWT: Apply the above one-dimensional DWT to each row of the image to obtain row low frequency and row high frequency;

[0123] Column Direction DWT: Apply a one-dimensional DWT to each column after the row direction DWT, and finally obtain:

[0124] LL (Low Frequency - Low Frequency): Both rows and columns are low frequency, corresponding to the overall trend of the image;

[0125] LH (Low Frequency-High Frequency): Row low frequency + column high frequency, corresponding to horizontal details (such as vertical edges).

[0126] HL (High Frequency-Low Frequency): Row high frequency + column low frequency, corresponding to vertical details (such as horizontal edges).

[0127] HH (High Frequency-High Frequency): Both rows and columns are high frequency, corresponding to details in the diagonal direction (such as slanted edges).

[0128] Furthermore, the four decomposed frequency band components are assigned learnable weight parameters to adjust the overall contribution of each frequency band through frequency band weighting. A spatially correlated weight matrix is ​​then generated via a convolutional network, and each frequency band is weighted a second time. The concatenated weighted frequency band components are then enhanced with residual connections to improve frequency band features, preserving the original information while injecting nonlinear transformations to further process the fused frequency band features. Finally, the model is upsampled to the size of the cost volume to generate an attention map (i.e., attention weights). This module decomposes features into different frequency bands through wavelet transform, and combined with adaptive weighting and dynamic enhancement, achieves fine-grained control over multi-scale features, helping to improve the model's ability to perceive image details.

[0129] Step 105: Using multiple cost aggregation residual blocks, generate an initial disparity map based on attention weights and intermediate cost volumes.

[0130] The multiple cost aggregation residual blocks include a first cost aggregation residual block, a second cost aggregation residual block, and a third cost aggregation residual block. Each of these blocks consists of multiple residual blocks, and the number of stacked residual blocks within each cost aggregation residual block can be set as needed; this invention does not limit this. Each residual block comprises a two-dimensional convolutional activation function module, a depthwise convolutional activation function module, and a two-dimensional convolutional linear output module. The two-dimensional convolutional activation function module consists of a 1×1 two-dimensional convolutional layer and a ReLU6 (Rectified LinearUnit 6) activation function layer. The depthwise convolutional activation function module consists of a 3×3 depthwise convolutional layer and a ReLU6 activation function layer. The two-dimensional convolutional linear output module consists of a 1×1 two-dimensional convolutional layer and a linear output layer (Linear). After performing a 1x1 convolutional layer on the input, the two-dimensional convolutional linear output module directly outputs the convolution result (i.e., linear output) without using any non-linear activation function.

[0131] Specifically, step 105 may include the following sub-steps S51-S55:

[0132] Step S51: Use the first cost aggregation residual block to aggregate the intermediate cost body to generate the aggregated cost body;

[0133] In this embodiment, the number of residual blocks in the first cost aggregation residual block can be set to 1, and the stride of the two 1×1 two-dimensional convolutional layers in the residual block is set to 1.

[0134] Step S52: Multiply the aggregate cost body and the attention weights to output the weighted cost body;

[0135] Step S53: Use the second cost aggregation residual block to downsample and aggregate the weighted cost volume, and output a low-resolution cost volume;

[0136] In this embodiment, the number of residual blocks in the second cost aggregation residual block can be set to 2. For the two 1×1 two-dimensional convolutional layers in the first residual block of the second cost aggregation residual block, the stride of the first 1×1 two-dimensional convolutional layer in the first residual block is set to 2, thereby completing the downsampling of the input features. The stride of the second 1×1 two-dimensional convolutional layer in the first residual block is set to 1. For the two 1×1 two-dimensional convolutional layers in the second residual block, the stride is set to 1 for both.

[0137] Step S54: Use the third cost aggregation residual block to downsample and aggregate the low-resolution cost volume, and output the target cost volume;

[0138] In this embodiment, the number of residual blocks in the third cost aggregation residual block can be set to 4. For the two 1×1 two-dimensional convolutional layers in the first residual block of the third cost aggregation residual block, the stride of the first 1×1 two-dimensional convolutional layer in the first residual block is set to 2, so as to downsample the input features. The stride of the second 1×1 two-dimensional convolutional layer in the first residual block is set to 1. The stride of the two 1×1 two-dimensional convolutional layers in the remaining residual blocks is set to 1.

[0139] Step S55: Interpolate the target cost volume, output the interpolated cost volume, multiply the interpolated cost volume with the attention weights to generate the aggregated cost volume, and perform disparity regression on the aggregated cost volume to output the initial disparity map.

[0140] It should be noted that the cost volume with a resolution of one-sixteenth (interpolated cost volume) is first interpolated and then multiplied with the attention weight to generate the aggregated cost volume. The "soft argmin" method is used to perform efficient disparity regression on the aggregated cost volume to obtain a disparity map with a resolution of one-quarter (initial disparity map).

[0141] Optionally, the cost aggregation residual block includes multiple residual blocks; each residual block includes a two-dimensional convolutional activation function module, a depthwise convolutional activation function module, and a two-dimensional convolutional linear output module; the feature processing of the residual block is as follows:

[0142] The first cost body is determined by performing two-dimensional convolution and nonlinear mapping on the input cost body to the cost aggregation residual block through a two-dimensional convolution activation function module.

[0143] The first cost body is generated by performing deep convolution and nonlinear mapping on the deep convolution activation function module.

[0144] The second cost body is input into the depthwise convolution activation function module for depthwise convolution and nonlinear mapping, and the third cost body is output.

[0145] The third cost body is generated by performing two-dimensional convolution on the two-dimensional convolution linear output module.

[0146] The input cost volume is the tensor data input to the cost aggregation residual block. It can be understood that the input cost volume can correspond to any tensor data input to the cost aggregation residual block for processing during the training or detection of a stereo matching model for thermal images.

[0147] The first cost body, the second cost body, and the third cost body are all intermediate tensor data generated in the cost aggregation residual block.

[0148] The output cost volume is the tensor data output by the cost aggregation residual block. It can be understood that it can correspond to any output tensor data of the cost aggregation residual block after image processing during the training or detection of the stereo matching model for thermal images.

[0149] It should be noted that you should refer to [link / reference]. Figure 5 The residual blocks proposed in this invention employ an improved residual block sequence from MobileNetV2, using depthwise separable convolutions to reduce the number of parameters and computational cost, while leveraging inverted residual structures to expand feature representation capabilities. Specifically, multiple sets of residual units formed by depthwise separable convolutions + pointwise convolutions (1×1 convolutions) are applied to the cost volume plane to progressively aggregate matching cost information. This structure maintains the network's lightweight characteristics while improving the cost aggregation effect. Disparity regression uses a conventional softargmin regression head to convert the aggregated cost volume into an initial disparity map.

[0150] Step 106: The initial disparity map is refined layer by layer using the convolution module based on the left eye edge probability feature map, the left eye thermal image, and the left eye fusion feature map to generate the target disparity map.

[0151] Specifically, step 106 may include the following sub-steps S61-S65:

[0152] Step S61: Perform convolution operation on the left eye fusion feature map to output the first disparity refinement feature;

[0153] Step S62: Perform residual connection on the first disparity refinement feature and the left eye edge probability feature map to generate the second disparity refinement feature;

[0154] Step S63: Downsample the left eye thermal image to generate detail features, and fuse the second disparity refinement feature and the detail feature to generate fused disparity features;

[0155] Step S64: Perform upsampling and nonlinear mapping on the fused disparity features sequentially to generate upsampling weights;

[0156] Step S65: Magnify the disparity values ​​of the initial disparity map, output the magnified disparity map, and use upsampling weights to perform a weighted average on the magnified disparity map to generate the target disparity map.

[0157] It should be noted that the convolutional module proposed in this invention consists of two 3×3 convolutional layers and one inverted convolutional layer. The first convolutional layer performs convolution operations on the input left-eye fusion feature map to obtain features for disparity refinement (i.e., the first disparity refinement feature). The first disparity refinement feature and the left-eye edge probability feature map are then residually connected to generate the second disparity refinement feature. The second convolutional layer with a stride of 2 downsamples the left-eye thermal image to generate low-level, high-resolution detail features, and the second disparity refinement feature and the detail features are then fused. (After upsampling the second disparity refinement feature, it is then concatenated with the detail feature along the channel dimension) to generate a fused disparity feature; the fused edge feature is upsampled by an inverted convolutional layer, and then the upsampled fused edge feature is non-linearly mapped by the Softmax activation function to obtain the upsampling weights. Finally, the disparity value of the initial disparity map is amplified (i.e., the initial disparity map is multiplied by 4 to amplify its disparity value by 4 times), and the amplified disparity map is output. The amplified disparity map is then weighted and averaged using the upsampling weights to generate the target disparity value (i.e., the target disparity map).

[0158] For comparison of technical effects, existing technologies can be referenced. Stereo matching is a core technology in computer vision for recovering three-dimensional structures from two-dimensional images. Essentially, it involves finding pixel-level correspondences (i.e., "matching points") between images from two perspectives (such as the left and right views of a binocular camera), and then calculating disparity (the positional deviation of pixels in the left and right views). The principle diagram is shown below. Figure 6 As shown, f and b are the camera's focal length and baseline, respectively (values ​​set during camera manufacturing). The depth value z is then derived from the parallax value using the following formula, where depth is inversely proportional to parallax:

[0159] ;

[0160] Where xl is the x-coordinate of the left eye image point; xr is the x-coordinate of the right eye image point; and d is the parallax.

[0161] The core objective of stereo matching is to generate accurate disparity maps. These disparity maps can be converted into depth maps using the focal lengths and baseline parameters of different cameras, thus enabling ranging. Therefore, disparity maps are fundamental to advanced tasks such as 3D scene reconstruction, object detection ranging, and motion estimation. Traditional stereo matching algorithms primarily rely on manually designed feature extraction algorithms or optimization functions for disparity prediction, consisting of four basic steps: pixel matching cost calculation, cost aggregation, disparity calculation, and disparity map post-processing. Due to the powerful feature extraction and learning capabilities of convolutional neural networks in computer vision, recent stereo matching algorithms have incorporated deep learning to build end-to-end stereo matching models, significantly improving the accuracy of stereo matching. For example, PSMNet (Pyramid Stereo Matching Network) integrates global contextual information using a pyramid structure and expands the regional contextual support of the cost volume through stacked hourglass 3D CNN (3D Convolutional Neural Network). ACVNet (Attention Concatenation Volume for Accurate and EfficientStereo Matching) suppresses irrelevant information and enhances the model's attention to important information through an attention weight generation module. MoCha-stereo learns a way to capture repetitive information from time series for the first time, solving the problem of lost feature channel information in traditional methods (leading to mismatched edge details), and designs a Motif Channel Correlation Volume (MCCV) and a Reconstruction Error Motif Penalty (REMP) module. LightStereo introduces a layer-by-layer refinement method and multi-scale convolutional attention to build a lightweight model, which better balances speed and accuracy.

[0162] However, the lack of real-world and synthetic thermal data makes developing robust and accurate thermal stereo matching algorithms more challenging. The MS² stereo dataset, a large-scale, multimodal real-world dataset, has very sparse available real disparity labels. The algorithm studied in this invention is validated on this dataset.

[0163] Most of the stereo matching models mentioned above are designed for RGB image pairs. Their robustness to thermal images is poor when transferred to them, and even high-accuracy models inevitably suffer from high parameter complexity, making them unsuitable for mobile devices. Existing lightweight models show low accuracy when transferred to thermal images, and these methods still have shortcomings in handling disparity map details, lacking explicit attention to edge contours and struggling to overcome the challenges of thermal images. In some datasets (such as the MS² stereo dataset), available real disparity labels are very sparse, posing a challenge to supervised training. Most existing methods are trained directly based on sparse labels, which may lead to inaccurate predictions of unlabeled regions, affecting the final performance. To achieve accurate stereo matching on thermal images while reducing model complexity and contributing to deployment on mobile devices, a lightweight stereo matching algorithm that integrates edge maps and enhanced wavelet transform attention is proposed. Furthermore, distillation learning is introduced into the training model to address the problem of sparse real disparity labels in the dataset.

[0164] To address the aforementioned issues, this invention proposes a lightweight heatmap-based stereo matching method. Through a layered, refined modular design, it first introduces an edge attention mechanism onto the constructed cost volume to enhance the model's focus on edge information. Secondly, an enhanced wavelet transform is introduced in the cost aggregation module to highlight edge information and suppress noise. Finally, in the disparity refinement stage, an edge map fusion feature map is designed to further refine the disparity. Simultaneously, an attention mechanism based on heatmap edges is introduced, employing wavelet transform with learnable weights to process frequency domain features, and specifically designed for the characteristics of heatmaps, forming a complete and novel algorithm framework. Compared with existing technologies, this invention significantly improves the accuracy and edge quality of stereo matching results on thermal images. While maintaining a lightweight model, the error metrics of the disparity map are significantly improved, especially in object edges and detail regions, where the false matching rate is greatly reduced.

[0165] In this embodiment of the invention, a lightweight stereo matching method for thermal images is provided. The method acquires left-eye and right-eye thermal images, inputs them into a pre-set stereo matching model for thermal images, and includes a multi-scale fusion feature extraction module, an edge feature extraction module, an enhanced wavelet attention module, multiple cost aggregation residual blocks, and a convolution module. The multi-scale fusion feature extraction module performs multi-scale fusion feature extraction on the left-eye and right-eye thermal images, generating left-eye fusion feature maps and right-eye fusion feature maps. An initial cost volume is constructed based on these feature maps. The edge feature extraction module extracts edge features from the left-eye thermal image, outputting a left-eye edge probability feature map. An intermediate cost volume is generated based on the left-eye edge probability feature map and the initial cost volume. The left-eye fusion feature is used as input to the enhanced wavelet attention module, which outputs attention weights. Multiple cost aggregation residual blocks are then used. The residual block generates an initial disparity map based on attention weights and intermediate cost volumes. A convolutional module refines this initial disparity map layer by layer using the left-eye edge probability feature map, the left-eye thermal image, and the left-eye fusion feature map to generate the target disparity map. Based on this approach, this invention achieves cross-scale feature aggregation through a multi-scale fusion feature extraction module. This accurately preserves the depth correlation information of subtle textures and weak temperature difference regions at different scales in the thermal image, avoiding the omission of subtle depth information caused by single-scale feature extraction. This lays a richer and more complete feature foundation for the subsequent construction of the initial cost volume. Simultaneously, the left-eye edge probability feature map generated by the edge feature extraction module explicitly enhances the depth difference signal at the edge contour of the thermal image, optimizes the edge region representation of the intermediate cost volume to reduce disparity shift, and introduces enhanced wavelet transform in the cost aggregation module to highlight edge information and suppress noise, thereby significantly improving the accuracy of disparity results in thermal image stereo matching.

[0166] For better explanation, refer to Figure 7 The diagram illustrates the steps of training a preset stereo matching model for thermal images according to Embodiment 2 of the present invention. This process may include the following steps:

[0167] Step 701: Obtain the heatmap dataset for model training;

[0168] Step 702: Using a three-stage distillation learning approach, train the initial stereo matching model for thermal images based on the heat map dataset used for model training, and determine the trained preset stereo matching model for thermal images.

[0169] The heatmap dataset used for model training is an existing heatmap dataset (such as the MS² multi-spectral stereo dataset).

[0170] The initial stereo matching model used for thermal images is a pre-set stereo matching model for thermal images to be trained.

[0171] It should be noted that this invention addresses the problem of very sparse real disparity labels available in existing heatmap datasets (such as the MS² stereo dataset) by constructing a three-stage distillation learning method:

[0172] (1) Teacher model training: The high-performance but computationally intensive teacher model Monster was first pre-trained on the large-scale synthetic dataset Sceneflow, and then fine-tuned on the sparsely labeled hot image dataset MS² to obtain a dense pseudo-label disparity map to compensate for the sparsity of the real labels.

[0173] (2) Student model distillation: The pseudo-label disparity map generated by the teacher model is used to supervise the training of the student model (initially used for stereo matching model of thermal image). The loss function adopts weighted L1 loss, and the final output is given higher weight.

[0174] (3) Sparse label fine-tuning: The student model after distillation is fine-tuned with real sparse labels to avoid the potential bias of the model relying too much on pseudo labels and improve the generalization of the real scene.

[0175] It is worth mentioning that the loss function consists of two parts: the main loss (the final disparity value) and the auxiliary loss (the disparity value obtained by disparity regression at 1 / 4 resolution). The smooth L1 loss function smooth_l1_loss is used, where the auxiliary loss is added to the main loss with a weight of 0.3.

[0176] Furthermore, this invention is trained on thermal images from the MS² stereo dataset. All experiments are implemented using PyTorch and trained and tested on a single NVIDIA 3090 GPU. The test results are comparable to, for example... Figures 8-10 The figure shows the performance of existing models and the proposed stereo matching model for thermal images monitored on the validation set during training. ACVNet (the best performing model, but with a large number of parameters and high computational complexity) and LightStereo (a lightweight model) are also shown. The figure demonstrates that the proposed model significantly improves matching at object edges.

[0177] Furthermore, Table 1 shows a benchmark test result on the MS² dataset, which provides test data for three scenarios: daytime, nighttime, and rainy days, distributed across different time periods of the day. This effectively represents various scene changes in the real world. It can be seen that the model proposed in this invention significantly improves the accuracy of the data with fewer model parameters and less complexity. While achieving accuracy comparable to better models (such as ACVNet), the computational complexity of this invention is much lower than theirs, achieving an excellent balance between lightweight design and accuracy, and contributing to deployment on mobile devices.

[0178] Table 1. Benchmark Results for the MS² Dataset

[0179]

[0180] Further, please refer to Figure 11 , Figure 11 This is a visualization of the test results of the model proposed in this invention on the MS² dataset. From top to bottom, it shows the original image (left), sparse ground truth values, and disparity map. A challenging nighttime thermal image was selected for testing, demonstrating that the model proposed in this invention performs well at the edges of object contours.

[0181] In this embodiment of the invention, addressing the problem of extremely sparse available real disparity labels in existing heatmap datasets, a three-stage distillation learning approach is constructed. A high-performance, computationally intensive teacher model generates dense pseudo-labels for student model training. Simultaneously, this invention effectively alleviates the problem of disparity edge blurring through edge attention, making the generated depth map clearer and more accurate at object edges. Multi-scale wavelet feature fusion improves the matching reliability of texture regions. Distillation learning fully utilizes the teacher model's knowledge, ensuring the model's generalization performance even under sparse labeling conditions. The overall solution improves matching accuracy while incurring only a minimal increase in computational overhead, making it highly suitable for practical deployment, balancing both accuracy and real-time requirements.

[0182] Please see Figure 12 , Figure 12 This is a structural block diagram of a lightweight thermal imaging stereo matching device provided in Embodiment 3 of the present invention.

[0183] The present invention provides a lightweight thermal imaging stereo matching device, comprising:

[0184] The acquisition module 1201 is used to acquire the left-eye thermal image and the right-eye thermal image, and input the left-eye thermal image and the right-eye thermal image into a preset stereo matching model for thermal images. The preset stereo matching model for thermal images includes a multi-scale fusion feature extraction module, an edge feature extraction module, an enhanced wavelet attention module, multiple cost aggregation residual blocks and a convolution module.

[0185] The multi-scale fusion extraction module 1202 is used to extract multi-scale fusion features from the left and right thermal images through the multi-scale fusion feature extraction module, generate left and right fusion feature maps, and construct an initial cost body based on the left and right fusion feature maps.

[0186] The edge feature extraction module 1203 is used to extract edge features from the left eye thermal image, output the left eye edge probability feature map, and generate an intermediate cost body based on the left eye edge probability feature map and the initial cost body.

[0187] Enhancement module 1204 is used to take the left eye fusion features as input to the enhanced wavelet attention module and output attention weights;

[0188] The aggregation module 1205 is used to generate an initial disparity map by aggregating multiple cost residual blocks based on attention weights and intermediate cost volumes.

[0189] The refinement module 1206 is used to refine the initial disparity map layer by layer using the convolution module based on the left eye edge probability feature map, the left eye thermal image, and the left eye fusion feature map to generate the target disparity map.

[0190] In one optional device embodiment, it further includes:

[0191] The first module is used to obtain the heatmap dataset for model training;

[0192] The second module is used to train the initial stereo matching model for thermal images based on the heat map dataset used for model training using a three-stage distillation learning method, and to determine the trained preset stereo matching model for thermal images.

[0193] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0194] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0195] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0196] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A lightweight thermal image stereo matching method, characterized in that, include: Acquire left-eye thermal image and right-eye thermal image, and input the left-eye thermal image and right-eye thermal image into a preset stereo matching model for thermal images. The preset stereo matching model for thermal images includes a multi-scale fusion feature extraction module, an edge feature extraction module, an enhanced wavelet attention module, multiple cost aggregation residual blocks and a convolution module. The multi-scale fusion feature extraction module extracts multi-scale fusion features from the left and right thermal images to generate a left-eye fusion feature map and a right-eye fusion feature map. Based on the left-eye fusion feature map and the right-eye fusion feature map, an initial cost body is constructed. The edge feature extraction module is used to extract edge features from the left eye thermal image, outputting a left eye edge probability feature map, and an intermediate cost body is generated based on the left eye edge probability feature map and the initial cost body. The left eye fusion feature is used as the input to the enhanced wavelet attention module, and the attention weights are output. An initial disparity map is generated using multiple cost aggregation residual blocks based on the attention weights and the intermediate cost volume; The convolutional module refines the initial disparity map layer by layer based on the left eye edge probability feature map, the left eye thermal image, and the left eye fusion feature map to generate the target disparity map.

2. The lightweight thermal image stereo matching method according to claim 1, characterized in that, The step of extracting multi-scale fusion features from the left and right eye thermal images using the multi-scale fusion feature extraction module to generate left and right eye fusion feature maps includes: Feature extraction is performed on the left and right thermal images respectively, and multi-scale left and right feature maps are output. The multi-scale left-eye feature map and the multi-scale right-eye feature map are upsampled and fused multiple times to generate a left-eye fused feature map and a right-eye fused feature map.

3. The lightweight thermal image stereo matching method according to claim 1, characterized in that, The step of using the edge feature extraction module to extract edge features from the left eye thermal image and outputting a left eye edge probability feature map includes: The weighted Sobel algorithm is used to perform edge detection on the left eye thermal image, and the first left eye edge feature is output. Average pooling is performed on the left eye thermal image to generate blurred feature values, and the blurred feature values ​​are subtracted from the left eye thermal image to determine the second left eye edge features; The first left eye edge feature and the second left eye edge feature are fused together to generate the left eye fused edge feature; The left eye fused edge features are subjected to convolution operation to output the first left eye convolution edge features, and the first left eye convolution edge features are nonlinearly mapped using the LeakyReLU activation function to output the first left eye nonlinear edge features. The first left-eye nonlinear edge feature is convolved to output the second left-eye convolutional edge feature, and the LeakyReLU activation function is used to perform nonlinear mapping on the second left-eye convolutional edge feature to output the second left-eye nonlinear edge feature. Perform a convolution operation on the second left-eye nonlinear edge feature to output the third left-eye convolution edge feature; The Sigmoid activation function is used to linearly transform the edge features of the third left eye convolution, and the left eye edge probability feature map is output.

4. The lightweight thermal image stereo matching method according to claim 1, characterized in that, The step of generating an intermediate cost body based on the left eye edge probability feature map and the initial cost body includes: Channel attention is calculated on the left eye edge probability feature map, and the left eye edge attention feature is output; The left eye edge attention feature is multiplied by the initial cost volume to output the intermediate cost volume.

5. The lightweight thermal image stereo matching method according to claim 1, characterized in that, The step of using the left eye fusion feature as input to the enhanced wavelet attention module and outputting attention weights includes: Perform a two-dimensional convolution on the left eye fusion features to output the left eye two-dimensional convolution features; The discrete wavelet transform decomposition algorithm is used to decompose the left eye two-dimensional convolutional features to generate multiple weighted frequency band components; Global weight scaling is performed on the weights corresponding to each weighted frequency band component, and each weighted frequency band component after weight scaling is output. The weighted frequency band components that have undergone weight scaling are spatially adaptively weighted to generate multiple secondary weighted frequency band components. The multiple secondary weighted frequency band components are spliced ​​together to output the fused frequency band features; The fused frequency band features are dynamically enhanced to generate enhanced frequency band features. The enhanced frequency band features and the fused frequency band features are then added element by element to output the summed frequency band features. The summed frequency band features are upsampled to generate attention weights.

6. The lightweight thermal image stereo matching method according to claim 1, characterized in that, The plurality of cost aggregation residual blocks include a first cost aggregation residual block, a second cost aggregation residual block, and a third cost aggregation residual block; the step of generating an initial disparity map using the plurality of cost aggregation residual blocks based on the attention weights and the intermediate cost body includes: The intermediate cost body is aggregated using the first cost aggregation residual block to generate an aggregated cost body; Multiply the aggregated cost body and the attention weight to output the weighted cost body; The weighted cost volume is downsampled and aggregated using the second cost aggregation residual block to output a low-resolution cost volume. The third cost aggregation residual block is used to downsample and aggregate the low-resolution cost volume, and the target cost volume is output. The target cost volume is interpolated to output the interpolated cost volume. The interpolated cost volume is then multiplied by the attention weight to generate the aggregated cost volume. Disparity regression is then performed on the aggregated cost volume to output the initial disparity map.

7. The lightweight thermal image stereo matching method according to claim 6, characterized in that, The cost aggregation residual block includes multiple residual blocks; each residual block includes a two-dimensional convolutional activation function module, a depthwise convolutional activation function module, and a two-dimensional convolutional linear output module; the feature processing of the residual block is specifically as follows: The first cost body is determined by performing two-dimensional convolution and nonlinear mapping on the input cost body input to the cost aggregation residual block through a two-dimensional convolution activation function module. The first cost body is subjected to deep convolution and nonlinear mapping using a deep convolution activation function module to generate the second cost body. The second cost body is input into the deep convolution activation function module for deep convolution and nonlinear mapping, and the third cost body is output. The third cost body is subjected to two-dimensional convolution by the two-dimensional convolution linear output module to generate the output cost body.

8. The lightweight thermal image stereo matching method according to claim 1, characterized in that, The step of refining the initial disparity map layer by layer based on the left eye edge probability feature map, the left eye thermal image, and the left eye fusion feature map using the convolution module to generate the target disparity map includes: Perform convolution operation on the left eye fusion feature map to output the first disparity refinement feature; The first disparity refinement feature and the left eye edge probability feature map are regressively connected to generate the second disparity refinement feature; The left eye thermal image is downsampled to generate detail features, and the second disparity refinement feature and the detail features are fused to generate a fused disparity feature; The fused disparity features are sequentially upsampled and nonlinearly mapped to generate upsampling weights; The initial disparity map is amplified by increasing the disparity value, and an amplified disparity map is output. The amplified disparity map is then weighted and averaged using the upsampling weights to generate the target disparity map.

9. The lightweight thermal image stereo matching method according to claim 1, characterized in that, The model training process for the preset stereo matching model for thermal images is as follows: Obtain the heatmap dataset for model training; A three-stage distillation learning approach is used to train the initial stereo matching model for thermal images based on the heat map dataset used for model training, thereby determining the trained preset stereo matching model for thermal images.

10. A lightweight thermal image stereo matching system, characterized in that, include: The acquisition module is used to acquire the left-eye thermal image and the right-eye thermal image, and input the left-eye thermal image and the right-eye thermal image into a preset stereo matching model for thermal images. The preset stereo matching model for thermal images includes a multi-scale fusion feature extraction module, an edge feature extraction module, an enhanced wavelet attention module, multiple cost aggregation residual blocks and a convolution module. The multi-scale fusion extraction module is used to extract multi-scale fusion features from the left-eye thermal image and the right-eye thermal image through the multi-scale fusion feature extraction module, generate a left-eye fusion feature map and a right-eye fusion feature map, and construct an initial cost body based on the left-eye fusion feature map and the right-eye fusion feature map; An edge feature extraction module is used to extract edge features from the left eye thermal image, output a left eye edge probability feature map, and generate an intermediate cost body based on the left eye edge probability feature map and the initial cost body. An enhancement module is used to take the left eye fusion features as input to the enhanced wavelet attention module and output attention weights. The aggregation module is used to generate an initial disparity map by using multiple cost aggregation residual blocks based on the attention weights and the intermediate cost body; The refinement module is used to refine the initial disparity map layer by layer using the convolution module based on the left eye edge probability feature map, the left eye thermal image, and the left eye fusion feature map to generate a target disparity map.

Images