Stereo matching method based on wavelet transform and superpixel segmentation

Abstract

The invention relates to the field of computer vision and image processing, in particular to a three-dimensional matching method based on wavelet transform and superpixel segmentation, which can improve the problems of poor depth estimation precision, blurred depth map contour and failure in multi-texture area depth estimation of the existing three-dimensional matching method. According to the method, different frequency domain sub-band information of the image is obtained by introducing a wavelet transform image processing method, the guiding effect of a low-frequency structure is enhanced, and the cost body aggregation process is participated; and meanwhile, fine modeling is carried out on the boundary of an image structure by combining a superpixel segmentation technology so as to enhance the integrity of an object contour in a depth image and the accuracy of parallax estimation. The method shows higher matching precision and boundary definition in a complex scene, has good calculation efficiency and deployment feasibility, and can provide reliable support for an embedded stereoscopic vision system with high precision and low resource consumption.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and image processing, and in particular to a stereo matching method based on wavelet transform and superpixel segmentation. Background Art

[0002] Humans have the ability to acquire spatial depth information through their two eyes. That is, by observing the parallax between left and right viewpoints, they can infer the three-dimensional positional relationship of objects. This ability is called stereo vision. Stereo matching is a computer vision technology that simulates the human binocular vision mechanism. Its goal is to use image pairs from two viewpoints of left and right cameras to calculate the parallax between each pixel, thereby restoring the depth information of the scene and achieving three-dimensional reconstruction. Stereo matching technology plays a key role in many application scenarios such as autonomous driving, robot navigation, augmented reality, and intelligent manufacturing, and is an important component of building machine three-dimensional perception systems. The typical stereo matching process usually consists of four stages: feature extraction, cost calculation, cost aggregation, and parallax optimization.

[0003] Existing stereo matching methods can be mainly divided into traditional methods and deep learning-based methods. Stereo matching based on traditional methods relies on manually designed features and matching strategies, and often uses classic algorithms such as grayscale consistency, block matching, SAD (sum of absolute differences), SGM (semi-global matching), etc. This type of method has high computational efficiency and is suitable for scenes with high real-time requirements, but it is prone to mismatching when dealing with texture loss, lighting changes, or occluded areas, and its robustness is not strong. Deep learning-based methods have developed rapidly in recent years, and have achieved mapping from image to disparity through end-to-end network structures, significantly improving matching accuracy and scene adaptability, and have become the current mainstream research direction.

[0004] Although the introduction of deep learning methods in recent years has significantly advanced stereo matching accuracy, existing methods still face numerous challenges when dealing with complex natural scenes. Current deep learning-based stereo matching methods still have limitations in multi-scale information fusion and context modeling. Existing methods typically employ simple feature pyramids or dilated convolutions to extract multi-scale features. These methods lack effective cross-scale interaction mechanisms, often resulting in the loss of geometric detail in low-resolution features and difficulty establishing long-range dependencies in high-resolution features, leading to unstable performance in complex scenes. When constructing the matching cost volume, most methods fail to fully utilize local structural boundary information. Due to the smoothing and aggregation of the cost volume, depth discontinuities at object edges are often blurred, resulting in "tailing" or "staircase effects" in depth maps, which compromises depth reconstruction accuracy. Current methods generally focus on spatial domain feature extraction and overlook the potential value of frequency domain features. They fail to effectively leverage the complementary advantages of low-frequency components in representing regional consistency and high-frequency components in capturing detailed features, reducing the robustness of the algorithms to illumination variations and noise interference. These issues hinder the accuracy and stability of depth estimation algorithms in complex scenes. Summary of the Invention

[0005] The present invention aims to improve the problems of poor depth estimation accuracy, blurred depth map contours, and failure of depth estimation in multi-texture areas in existing stereo matching methods. By introducing the image processing method of wavelet transform, the information of different frequency domain sub-bands of the image is obtained, the guiding role of the low-frequency structure is strengthened, and the process of cost volume aggregation is participated in; at the same time, the image structure boundary is finely modeled in combination with superpixel segmentation technology to enhance the integrity of the object contours in the depth map and the accuracy of disparity estimation. This method shows higher matching accuracy and boundary clarity in complex scenes, has good computational efficiency and deployment feasibility, and can provide reliable support for high-precision, low-resource-consumption embedded stereo vision systems.

[0006] A stereo matching method based on wavelet transform and superpixel segmentation is as follows:

[0007] The implementation of the present invention includes: constructing a deep learning network framework, training the deep learning network framework to obtain a stereo matching model, and specifically includes the following steps S1 and S2.

[0008] S1, construct a deep learning network framework based on wavelet transform and superpixel segmentation, the deep learning network framework includes a multi-scale feature extraction module, a wavelet transform-based attention module, a superpixel segmentation module, a multi-scale cost volume construction module, a multi-scale cost volume aggregation module, a cross-scale cost volume fusion module and a disparity regression module.

[0009] The multi-scale feature extraction module in step S1 is specifically as follows:

[0010] The present invention adopts ResNet combined with FPN as the feature extraction network skeleton, and the input of the multi-scale feature extraction module is the stereo image pair I L , I R , where I L ,I R ∈R B×3×H×W , respectively represent the left and right images of the stereo image pair, B is the batch size, 3 is the number of channels, and H×W is the original image resolution. The specific process of multi-scale feature extraction is as follows:

[0011] (1) Preliminary image feature extraction using ResNet. The left and right images of the stereo image pair are input into the residual network containing multiple residual blocks stacked in sequence for preliminary feature extraction. The extraction process consists of four stages: the first stage is to downsample the input image to The output feature map channel is 32; in the second stage, the convolution residual block with a step size of 2 is used for further downsampling, and the feature size is The output channels are 64; the feature size is not changed in the third stage, and the number of channels is increased to 128; the fourth stage uses dilated convolution with dilation = 2 to enhance the receptive field, while keeping the feature size unchanged and the number of channels still 128. The output feature maps of these four stages are F1, F2, F3, and F4 respectively, then:

[0012]

[0013] To enhance the multi-layer feature fusion effect, the present invention concatenates F2, F3, and F4 along the channel dimension to form a unified feature representation:

[0014]

[0015] After the initial feature extraction, the channel dimension is too high, resulting in high computational complexity. To reduce the computational complexity, the present invention introduces a two-stage dimensionality reduction convolution structure after the initial feature extraction, using 1×1 and 3×3 convolution kernels respectively, reducing the number of channels of the left and right feature maps from 320 to 128, and further to 32. The calculation process can be expressed as:

[0016]

[0017] where W1∈R 128×320×3×3 , represents a 3×3 convolution kernel that reduces the number of channels of the initial extracted feature map from 320 to 128, W2∈R 32×128×1×1 , which further reduces the 128-channel feature map to a 32-channel 1×1 convolution kernel. The symbol * represents the convolution operation, and ReLu(·) represents the ReLU activation function. It enhances the feature expression ability through nonlinear activation, cooperates with dimensionality reduction to filter redundant information, and improves computational efficiency. The feature extraction process can be written as:

[0018] F L =FeatureExtraction(I L ),F R =FeatureExtraction(I R )

[0019] The left and right feature maps

[0020] (2) Using FPN to construct multi-scale feature representation. In order to improve the perception ability of deep learning networks to contextual semantic information of different scales, the present invention introduces the feature pyramid network FPN, which constructs multi-scale feature representation by downsampling layer by layer. FPN includes horizontal connections, top-down paths and feature refinement parts. Specifically, FPN is used to construct multi-scale feature representation. L ,F R As the initial input of FPN, corresponding to large-scale features A 3×3 convolution with a step size of 2 and a 1×1 convolution are combined to Downsample to obtain mid-scale features Use the same method to Downsampling to obtain small-scale features in, Finally, the output of FPN can be expressed as:

[0021]

[0022] Features of various scales have different receptive fields and structural details. Specifically, small-scale features Emphasis on global context and high-level semantic information; mesoscale Achieve a balance between semantic and structural information, large-scale features It retains more edge details and texture information, which can meet the requirements for details and resolution in cost volume construction and disparity estimation.

[0023] The wavelet transform-based attention module in step S1 is specifically as follows:

[0024] (1) Discrete wavelet decomposition. For the input image I L Perform learnable discrete wavelet transform decomposition, the decomposition process is as follows:

[0025] ① Image preprocessing. To ensure the correct size of wavelet subband after downsampling, the I L Perform reflection filling to extend the border of the image by mirroring the edge pixels of the image so that I L Both the height and width are even:

[0026] Ipadded =Pad(I L ,mode='reflect')

[0027] ② Wavelet decomposition. For the filled image I padded Perform two-dimensional discrete wavelet decomposition, use learnable 2×2 convolution kernel weights for convolution operation, and downsample with a step size of 2. The calculation formula is: LL = (I padded *W LL )↓2,LH=(I padded *W LH )↓2,HL=(I padded *W HL )↓2,HH=(I padded *W HH )↓2, where W LL ,W LH ,W HL ,W HH ∈R 2×2 It is a filter bank generated by the Haar wavelet basis, which is learnable. ↓2 represents downsampling with a step size of 2. This operation uses unfold to expand the matrix to improve computational efficiency:

[0028] I unfold =Unfold(I padded ,kernel_size=2,stride=2)

[0029] After wavelet decomposition, the sizes are all The low frequency sub-band LL and three high frequency sub-bands LH, HL, HH.

[0030] (2) Subband attention generation. In order to enhance the discriminative ability of wavelet subbands, a subband attention mechanism is introduced, which mainly includes three parts: subband splicing, attention weight generation, and subband enhancement.

[0031] ① Sub-band splicing. The sub-bands in the four directions are spliced ​​along the channel dimension. The splicing process can be expressed as:

[0032]

[0033] ② Attention weight generation. First, a 3×3 group convolution with groups=3 is used to independently extract cross-subband context features of the RGB channels to ensure the independence of each channel. Nonlinear characteristics are then introduced through ReLU activation. Subsequently, cross-channel features are fused through 1×1 convolution. Finally, a normalized attention weight A in the range of [0, 1] is generated using the Sigmoid function. Specifically, it can be expressed as:

[0034]

[0035] Where A is the normalized attention weight and σ(·) is the Sigmoid function.

[0036] ③Sub-band enhancement. Apply attention weight to F concat , perform channel-by-channel multiplication operation to achieve the enhancement of the sub-band salient area. The sub-band enhancement process can be expressed as:

[0037] F att =F concat ⊙A

[0038] Among them, F att is the enhanced feature map, and ⊙ represents the channel element-by-element multiplication.

[0039] (3) Sub-band feature fusion. First, a 3×3 convolutional layer is used to fusion the F att Extract local features, then use ReLU activation function to introduce nonlinear characteristics, then use 3×3 convolution layer to extract local features and fuse cross-channel information, and finally perform group normalization to alleviate the normalization instability problem caused by small batch training. The output size is The fusion feature F DWT . F DWT The operation process can be expressed as:

[0040] F DWT =GN(Conv 3×3 (ReLU(Conv 3×3 (F att ))))

[0041] Where GN represents group normalization. DWT The information of each sub-band of wavelet transform is retained, which is convenient for participating in multi-scale cost volume aggregation.

[0042] (4) Feature reorganization and channel adjustment. In order to effectively fuse wavelet transform features into multi-scale cost volumes, F DWT Downsampling to the size of the multi-scale cost volume and adjusting its number of channels through convolution can be expressed as:

[0043]

[0044] Among them, Interp represents interpolation downsampling, for F DWT Resize to match the small, medium, and large cost volume sizes H s ×W s 、H m ×W m 、H l ×W l , that is, respectively Conv 1×1Represents a 1×1 convolution operation, used to adjust F DWT The number of channels, after adjustment The number of channels are 12, 24, and 48 respectively.

[0045] The superpixel segmentation module in step S1 is specifically as follows:

[0046] (1) Edge probability map generation. In order to make full use of the structural information of the binocular image, this module uses the improved SLIC algorithm to generate the edge probability map of the left image I. L With the right picture I R Perform operations and jointly construct structure-aware superpixel segmentation. The specific contents are as follows:

[0047] ① Feature-guided edge extraction. L and I R Perform convolution processing separately to extract edge perception features of the left and right images:

[0048]

[0049] Among them, the convolution kernel size is 3×3; the number of input left and right image channels is 3, and the number of output channels is 16; the ReLU activation function is used to enhance the left and right perception features and nonlinear expression capability.

[0050] ②Edge prediction. and Based on this, the edge probability maps E of the left and right images are generated respectively. L , E R , which can be expressed as:

[0051]

[0052] Here, σ is the Sigmoid function, which is used to compress the output to the interval [0, 1]. The final result is an edge probability map of size B × 1 × H × W.

[0053] (2) Edge enhancement map generation. The edge probability map is spliced ​​with the left and right image feature maps to generate the corresponding left and right edge enhancement maps. It can be expressed as:

[0054]

[0055] Where Concat(·) represents the concatenation operation on the channel dimension, It aims to inject edge structure information into large-scale features and participate in the construction process of large-scale cost volumes.

[0056] The multi-scale cost volume construction module in step S1 is specifically as follows:

[0057] Construct the cost volume at three scales: small, medium, and large. and the feature map on the right As the basis, * represents the three scales s, m, and l. The correlation between the left and right features is calculated by pixel-by-pixel translation operation in the disparity dimension to obtain the multi-scale basic cost volume. The calculation formula is as follows:

[0058]

[0059] Among them, Corr represents the pixel-by-pixel dot product correlation calculation to generate a multi-scale cost volume. They correspond to small, medium and large scale basic cost volumes respectively.

[0060] The multi-scale cost volume aggregation module is specifically as follows:

[0061] In order to improve the cost volume's ability to distinguish texture repetition, weak texture and occluded areas, wavelet transform features are introduced at three scales respectively. This module aggregates superpixel edge information to enhance the cost volume's expression ability, achieving structural enhancement and semantic supplementation.

[0062] ① Small-scale cost volume aggregation. The wavelet features of the high-frequency subband HH are used as structural details to supplement them, mapped to the cost space through 1×1 convolution, and added element-by-element with the small-scale basic cost volume to obtain the aggregated small-scale cost volume C s , which can be expressed as:

[0063]

[0064] in, Represents the HH subband features obtained by wavelet transform, Conv 1×1 Used for channel matching and semantic alignment.

[0065] ②Mesoscale cost volume aggregation. The midscale cost volume is added element by element with the horizontal LH and vertical HL subband features of the wavelet transform module to supplement the edge direction information. The aggregation method is as follows:

[0066]

[0067] in It is the splicing feature of LH and HL at the mesoscale.

[0068] ③ Large-scale cost volume aggregation. The large-scale aggregated cost volume is constructed by adding the large-scale basic cost volume and the low-frequency sub-band LL features of the wavelet transform element by element:

[0069]

[0070] in, is the LL subband feature of the wavelet transform module.

[0071] The cross-scale cost volume fusion module in step S1 is specifically as follows:

[0072] (1) Feature extraction of multi-scale aggregate cost volume based on 3D hourglass network structure. The deep feature extraction network based on 3D hourglass network structure is used to extract the aggregate cost volume C of small, medium and large scales respectively. s 、C m 、C l , achieving deep fusion and context modeling of multi-scale aggregated cost volume information. Each hourglass structure contains symmetrical encoder and decoder paths. The encoder achieves feature compression and extracts global semantics through layer-by-layer 3D convolution; the decoder achieves structural reconstruction through 3D transposed convolution and integrates shallow detail features from jump connections, thereby obtaining a cost volume representation that combines structural perception and semantic understanding. The multi-scale cost volume feature extraction process can be expressed as:

[0073] D s =Hourglass3D(C s ),D m =Hourglass3D(C m ),D l =Hourglass3D(C l )

[0074] in, They respectively represent the small, medium and large scale cost volume feature representations after optimization by the 3D hourglass network, which have stronger three-dimensional context perception capabilities and provide basic support for cross-scale information fusion.

[0075] (2) Cross-scale fusion mechanism integrates multi-layer information. To fully utilize the complementarity of features at different scales, this paper designs an adaptive cross-scale fusion mechanism to achieve efficient fusion of high-level semantic information and low-level structural details. The implementation process is as follows:

[0076] First, the small-scale feature D s Upsampled to the mesoscale spatial size by bilinear interpolation, and the mesoscale feature D m Alignment can be expressed as:

[0077]

[0078] in, UpSample(·) represents bilinear interpolation upsampling, and then the upsampled Input to the convolution layer to obtain mid-scale enhanced features and with D m Jointly generate attention weight A m , which can be expressed as:

[0079]

[0080] Among them, σ represents the Sigmoid activation function, which is used to generate the normalized attention weight, and then the salient area is enhanced by channel-by-channel multiplication, and fused with the original mid-scale features in the form of residuals to obtain the fused mid-scale representation It can be expressed as:

[0081]

[0082] Next, Further upsampling to the large-scale feature D l Aligned spatial dimensions:

[0083]

[0084] Further convolution operation is performed to obtain large-scale enhanced features Generate attention weight A l , by element-wise addition With A l Generate large-scale fusion features, which can be expressed as:

[0085]

[0086] The fused large-scale feature volume It also has multi-scale contextual semantics and fine structure information, providing rich feature support for disparity estimation.

[0087] The disparity regression module in step S1 is specifically as follows:

[0088] (1) Cost volume upsampling. The fused cost volume By bilinear interpolation, it is restored to the original resolution of the input image, providing a cost volume consistent with the input image size for the subsequent Softmax normalization and disparity regression steps. The specific expression is:

[0089]

[0090] in, Indicates the disparity value index, in the present invention, d max Take 192; x∈[0,H-1], represents the pixel coordinates in the width direction of the image; y∈[0,W-1], represents the pixel coordinates in the height direction of the image, and Interp represents the bilinear interpolation operation.

[0091] (2) Softmax normalization. Softmax normalization operation is performed along the disparity dimension to generate a disparity probability distribution map, which can be expressed as:

[0092]

[0093] in, Indicates that at each pixel position (x, y), it belongs to each disparity layer d∈[0,d max ], exp represents the exponential transformation operation.

[0094] (3) Expected value regression. Based on probability graph Take a weighted average of the disparity to achieve sub-pixel disparity estimation:

[0095]

[0096] Disparity map after regression As the final depth estimation output.

[0097] S2, the network framework described in S1 is subjected to multiple rounds of iterative training using supervised learning to obtain a disparity estimation model. During training, the left and right images in the training set are used as input, and forward propagation is performed to calculate the predicted disparity map, which is compared with the corresponding real disparity map. The training error is calculated through the loss function, and the network parameters are updated using the backpropagation algorithm.

[0098] Specifically, the training process includes the following steps:

[0099] S2.1 Dataset selection.

[0100] This paper uses the KITTI2012 and KITTI2015 datasets from the KITTI public dataset for training. Both are widely used in stereo matching research in fields such as autonomous driving and computer vision. Each dataset is divided into a training set and a test set. The KITTI2012 dataset contains 194 training image pairs and 195 test image pairs with an image resolution of 1226×370; the KITTI2015 dataset contains 200 training image pairs and 200 test image pairs with an image resolution of 1242×375. Compared to KITTI2012, KITTI2015 improves on its data acquisition method, using lidar to generate real disparity maps and introducing more dynamic objects and semantic information, further increasing the complexity and authenticity of the data.

[0101] S2.2 Optimizer configuration and learning rate strategy adjustment.

[0102] The Adam optimizer is used to perform end-to-end training on the network parameters, and the initial learning rate is set to η0 = 1×10 -4 , and adopts a phased dynamic learning rate scheduling strategy, including warm-up phase, constant phase and decay phase, to improve the stability and generalization performance of training.

[0103] During the initial warm-up phase of training, the learning rate increases linearly from 0 to η0 to prevent gradient oscillation. During the mid-training constant phase, η0 remains constant to accelerate model feature learning. During the late-training decay phase, the learning rate decays linearly to a smaller value at a fixed rate to improve model stability on the validation set. The optimizer parameters are set to β1 = β2 = 0.9 to prevent drastic gradient fluctuations.

[0104] S2.3 Construction of loss function.

[0105] Introducing the joint loss function L during training total , to simultaneously optimize the global accuracy and edge structure consistency of disparity prediction, defined as follows:

[0106] L total =αL smooth-L1 +βL multi-scale

[0107] Among them, α and β are weight coefficients, which adjust L smooth-L1 , L multi-scale The contribution of two losses to the total loss. smooth-L1 is the disparity smoothing loss function, which predicts the disparity map within the valid disparity mask area. and the true disparity map D gt Perform pixel-by-pixel regression, the expression is as follows:

[0108]

[0109] Where N is the number of effective pixels, is the predicted disparity of the i-th pixel, D gti is the true disparity of the i-th pixel. In order to further improve the feature matching ability of the model at different scales, the present invention introduces a multi-scale feature loss L multi-scale , the specific calculation is as follows:

[0110]

[0111] Among them, *∈s,m,l, s,m,l represent small, medium and large scale cost volumes respectively, λ s is the weight coefficient of each loss, represents scalar multiplication, is the correlation loss on scale *, and the calculation formula is:

[0112]

[0113] Among them, N · Indicates the total number of valid pixels in the area where the true disparity is greater than 0 at * scale, and ||·||2 is used to calculate and The Euclidean distance between them, Corr(·) represents the correlation calculation, Represents the left and right feature maps at * scale.

[0114] S2.4 Iterative training.

[0115] In each round of training, the image data is read from the training set in batch as the smallest unit, including the left image I L , right image I R And the corresponding real disparity map D gt . Will I L , I R Input the network, perform the forward propagation process, and output the predicted disparity map

[0116] Then, based on the predicted disparity map Results and real disparity map D gt The difference between them is used to calculate the joint loss function L total , the present invention adopts weighted L smooth-L1 The loss function is the main loss term, that is, α>β, and D is selected gt The effective area >0 is included in the calculation to enhance the fitting ability of the effective depth area. The calculated loss L total Backpropagation is used to obtain gradient information, and the model parameters are updated using the Adam optimizer. Training is repeated until all training samples in the current round are fully traversed, and the average error metric of this round of training is calculated for performance evaluation.

[0117] After each round of training, model validation is started immediately to evaluate the generalization ability of the currently saved training model on the validation set. During the validation phase, the validation set data is read in sequence and input into the current model for forward reasoning to obtain the predicted disparity map. By comparing the corresponding real disparity map D gt For comparison, the following five error indicators are calculated: average endpoint error EPE, D1-error, and accuracy indicators T1, T2, and T3 within 1, 2, and 3 pixel thresholds. These indicators can comprehensively measure the prediction accuracy of the model.

[0118] This invention uses a conditional mechanism to save models, preserving the best-performing model. During iterative training, the five error metrics corresponding to each training round can be used to determine whether to save the model trained in the current round. This invention uses D1-error to determine whether to save a model. Specifically, if the D1-error obtained in a given round of verification is less than the historical minimum, the D1-error value and the corresponding round number are recorded, and the trained model is saved.

[0119] It should be noted that the performance test indicators EPE, D1-error, T1, T2, and T3 used in the present invention are as follows:

[0120] ①EPE (End-Point Error) is the endpoint error, which is used to measure the predicted disparity map and the true disparity map D gt The mean absolute error between is defined as follows:

[0121]

[0122] Where N is the total number of valid pixels, is the predicted disparity of the i-th pixel, D gti is the true disparity of the i-th pixel.

[0123] ②D1-error is the three-pixel error rate. This metric emphasizes accuracy when the error is significant and is suitable for evaluating the robustness of the algorithm under large error tolerance. D1-error can be used to measure the proportion of pixels where the prediction error exceeds 3 pixels or exceeds 5% of the true disparity. It is defined as follows:

[0124]

[0125] Among them, [·] means that the value is 1 when the Boolean expression is true, otherwise it is 0. is the predicted disparity of the i-th pixel, D gti is the true disparity of the i-th pixel.

[0126] T1, T2, and T3 are threshold error rates, which represent the percentage of pixels where the absolute error between the predicted disparity and the true disparity exceeds 1 pixel, 2 pixels, and 3 pixels, respectively. This reflects the model's ability to control error distribution at different thresholds. The smaller the values ​​of T1, T2, and T3, the higher the accuracy of the model at the corresponding threshold. The calculation formula is as follows:

[0127]

[0128] Where T1, T2, and T3 represent the ratios of errors exceeding 1, 2, and 3 pixels in the edge region, respectively; [·] indicates that the value is 1 when the Boolean expression holds, and 0 otherwise; is the predicted disparity of the i-th pixel; D gti is the true disparity of the i-th pixel.

[0129] The beneficial effects of the present invention are:

[0130] ① The proposed method introduces a wavelet transform module to perform multi-scale frequency-domain decomposition of the input image, effectively extracting low-frequency structural information and high-frequency texture details. This module, combined with a sub-band attention mechanism, dynamically adjusts the response weights of each sub-band, enhancing the proposed deep learning network's ability to perceive image structure, edges, and texture changes. This makes the proposed method more suitable for stereo matching tasks in areas with sparse or repetitive textures.

[0131] ② This invention uses a superpixel segmentation module to partition images into regions, effectively maintaining the continuity and regional consistency of object edges, and providing a more refined prior spatial structure for cost volume construction. This module allows the disparity estimation process to pay more attention to structural boundaries, reducing the risk of mismatches caused by blurred image edges or object occlusions. This improves the accuracy and clarity of depth map edges and enhances the robustness of stereo matching in complex scenes.

[0132] ③ This invention combines multi-scale cost volume construction with a multi-scale cost volume aggregation module, integrating wavelet and superpixel enhancement features to construct a deep learning network framework that considers both local and global contextual information. This structure significantly improves disparity regression accuracy, maintaining high stability and accuracy in the presence of interference factors such as illumination changes, perspective differences, and image noise. It is suitable for practical applications requiring high precision in 3D scene understanding, such as autonomous driving, robotic perception, and industrial inspection, and has broad application potential. BRIEF DESCRIPTION OF THE DRAWINGS

[0133] Figure 1 This is a diagram of the deep learning network architecture of the present invention, outlining the interrelationships between the various network modules and data flow. The superpixel segmentation module participates in the cost volume construction process, the wavelet-based attention module participates in the cost volume aggregation process, and the cross-scale cost volume fusion module performs weighted addition and fusion of large-, medium-, and small-scale cost volumes. The fused cost volume is the same size as the large-scale cost volume.

[0134] Figure 2 This is a schematic diagram of the wavelet-based attention module of the present invention, illustrating the embedding method of the wavelet transform module. The left image generates subband features in different frequency domains after wavelet transform processing. LL participates in the aggregation of large-scale cost volumes, LH and HL are concatenated to participate in the aggregation of medium-scale cost volumes, and HH participates in the aggregation of small-scale cost volumes.

[0135] Figure 3 This is a schematic diagram of the superpixel segmentation module of the present invention, describing its embedding method. After superpixel segmentation, the left and right images are downsampled to the size of the large-scale feature map and concatenated with the left and right large-scale feature maps, respectively, to form the large-scale cost volume.

[0136] Figure 4This is a schematic diagram of the disparity prediction output of the present invention, showing the disparity prediction map generated by the stereo matching network model of the present invention. In this diagram, high pixel grayscale values ​​indicate large disparity and close objects; low pixel grayscale values ​​indicate small disparity and far objects.

[0137] Figure 5 This is a visualization diagram of the error in the present invention. Dark blue indicates an error close to 0, indicating an accurate prediction result; light blue corresponds to a smaller error, indicating a slight but acceptable deviation; red represents the maximum error, typically occurring in areas of prediction failure or severe mismatch.

[0138] Figure 6 This is the ablation experiment results of the present invention on the KITTI2015 dataset, comparing the impact of different module combinations on performance.

[0139] Figure 7 This is a performance table of the present invention and other stereo matching methods on the KITTI2012 and KITTI2015 datasets, with the main evaluation indicators being D1-error and EPE. DETAILED DESCRIPTION

[0140] The following will be combined with the accompanying drawings to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in a variety of different configurations.

[0141] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but is merely intended to represent selected embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0142] Figure 1-Figure 7 A specific embodiment of the present invention is a stereo matching method based on wavelet transform and superpixel segmentation:

[0143] S1, construct a deep learning network framework based on wavelet transform and superpixel segmentation, the deep learning network framework includes a multi-scale feature extraction module, a wavelet transform-based attention module, a superpixel segmentation module, a multi-scale cost volume construction module, a multi-scale cost volume aggregation module, a cross-scale cost volume fusion module and a disparity regression module.

[0144] The deep learning network framework structure described in S1 is as shown in the attached Figure 1 As shown. Figure 2 , Attachment Figure 3The structural schematic diagrams of the attention module based on wavelet transform and the structural schematic diagrams of the superpixel segmentation module are further given.

[0145] The multi-scale feature extraction module described in S1 is as follows:

[0146] The present invention adopts ResNet combined with FPN as the feature extraction network skeleton, and the input of the multi-scale feature extraction module is the stereo image pair I L , I R , where I L ,I R ∈R B×3×H×W , respectively represent the left and right images of the stereo image pair, B is the batch size, 3 is the number of channels, and H×W is the original image resolution. The specific process of multi-scale feature extraction is as follows:

[0147] (1) Preliminary image feature extraction using ResNet. The left and right images of the stereo image pair are input into the residual network containing multiple residual blocks stacked in sequence for preliminary feature extraction. The extraction process consists of four stages: the first stage is to downsample the input image to The output feature map channel is 32; in the second stage, the convolution residual block with a step size of 2 is used for further downsampling, and the feature size is The output channels are 64; the feature size is not changed in the third stage, and the number of channels is increased to 128; the fourth stage uses dilated convolution with dilation = 2 to enhance the receptive field, while keeping the feature size unchanged and the number of channels still 128. The output feature maps of these four stages are F1, F2, F3, and F4 respectively, then:

[0148]

[0149] To enhance the multi-layer feature fusion effect, the present invention concatenates F2, F3, and F4 along the channel dimension to form a unified feature representation:

[0150]

[0151] After the initial feature extraction, the channel dimension is too high, resulting in high computational complexity. To reduce the computational complexity, the present invention introduces a two-stage dimensionality reduction convolution structure after the initial feature extraction, using 1×1 and 3×3 convolution kernels respectively, reducing the number of channels of the left and right feature maps from 320 to 128, and further to 32. The calculation process can be expressed as:

[0152]

[0153] where W1∈R 128×320×3×3 , represents a 3×3 convolution kernel that reduces the number of channels of the initial extracted feature map from 320 to 128, W2∈R32×128×1×1 , which further reduces the 128-channel feature map to a 32-channel 1×1 convolution kernel. The symbol * represents the convolution operation, and ReLu(·) represents the ReLU activation function. It enhances the feature expression ability through nonlinear activation, cooperates with dimensionality reduction to filter redundant information, and improves computational efficiency. The feature extraction process can be written as:

[0154] F L =FeatureExtraction(I L ),F R =FeatureExtraction(I R )

[0155] The left and right feature maps

[0156] (2) Using FPN to construct multi-scale feature representation. In order to improve the perception ability of deep learning networks to contextual semantic information of different scales, the present invention introduces the feature pyramid network FPN, which constructs multi-scale feature representation by downsampling layer by layer. FPN includes horizontal connections, top-down paths and feature refinement parts. Specifically, FPN is used to construct multi-scale feature representation. L ,F R As the initial input of FPN, corresponding to large-scale features A 3×3 convolution with a step size of 2 and a 1×1 convolution are combined to Downsample to obtain mid-scale features Use the same method to Downsampling to obtain small-scale features in, Finally, the output of FPN can be expressed as:

[0157]

[0158] Features of various scales have different receptive fields and structural details. Specifically, small-scale features Emphasis on global context and high-level semantic information; mesoscale Achieve a balance between semantic and structural information, large-scale features It retains more edge details and texture information, which can meet the requirements for details and resolution in cost volume construction and disparity estimation.

[0159] The wavelet transform-based attention module described in S1 is as follows:

[0160] (1) Discrete wavelet decomposition. For the input image I L Perform learnable discrete wavelet transform decomposition, the decomposition process is as follows:

[0161] ① Image preprocessing. To ensure the correct size of wavelet subband after downsampling, the I L Perform reflection filling to extend the border of the image by mirroring the edge pixels of the image so that I L Both the height and width are even:

[0162] I padded =Pad(I L ,mode='reflect')

[0163] ② Wavelet decomposition. For the filled image I padded Perform two-dimensional discrete wavelet decomposition, use learnable 2×2 convolution kernel weights for convolution operation, and downsample with a step size of 2. The calculation formula is: LL = (I padded *W LL )↓2,LH=(I padded *W LH )↓2,HL=(I padded *W HL )↓2,HH=(I padded *W HH )↓2, where W LL ,W LH ,W HL ,W HH ∈R 2×2 It is a filter bank generated by the Haar wavelet basis, which is learnable. ↓2 represents downsampling with a step size of 2. This operation uses unfold to expand the matrix to improve computational efficiency:

[0164] I unfold =Unfold(I padded ,kernel_size=2,stride=2)

[0165] After wavelet decomposition, the sizes are all The low frequency sub-band LL and three high frequency sub-bands LH, HL, HH.

[0166] (2) Subband attention generation. In order to enhance the discriminative ability of wavelet subbands, a subband attention mechanism is introduced, which mainly includes three parts: subband splicing, attention weight generation, and subband enhancement.

[0167] ① Sub-band splicing. The sub-bands in the four directions are spliced ​​along the channel dimension. The splicing process can be expressed as:

[0168]

[0169] ② Attention weight generation. First, a 3×3 group convolution with groups=3 is used to independently extract cross-subband context features of the RGB channels to ensure the independence of each channel. Nonlinear characteristics are then introduced through ReLU activation. Subsequently, cross-channel features are fused through 1×1 convolution. Finally, a normalized attention weight A in the range of [0, 1] is generated using the Sigmoid function. Specifically, it can be expressed as:

[0170]

[0171] Where A is the normalized attention weight and σ(·) is the Sigmoid function.

[0172] ③Sub-band enhancement. Apply attention weight to F concat , perform channel-by-channel multiplication operation to achieve the enhancement of the sub-band salient area. The sub-band enhancement process can be expressed as:

[0173] F att =F concat ⊙A

[0174] Among them, F att is the enhanced feature map, and ⊙ represents the channel element-by-element multiplication.

[0175] (3) Sub-band feature fusion. First, a 3×3 convolutional layer is used to fusion the F att Extract local features, then use ReLU activation function to introduce nonlinear characteristics, then use 3×3 convolution layer to extract local features and fuse cross-channel information, and finally perform group normalization to alleviate the normalization instability problem caused by small batch training. The output size is The fusion feature F DWT . F DWT The operation process can be expressed as:

[0176] F DWT =GN(Conv 3×3 (ReLU(Conv 3×3 (F att ))))

[0177] Where GN represents group normalization. DWT The information of each sub-band of wavelet transform is retained, which is convenient for participating in multi-scale cost volume aggregation.

[0178] (4) Feature reorganization and channel adjustment. In order to effectively fuse wavelet transform features into multi-scale cost volumes, F DWT Downsampling to the size of the multi-scale cost volume and adjusting its number of channels through convolution can be expressed as:

[0179]

[0180]

[0181] Among them, Interp represents interpolation downsampling, for F DWT Resize, Conv 1×1 Represents a 1×1 convolution operation, used to adjust F DWT The number of channels, after adjustment The number of channels are 12, 24, and 48 respectively.

[0182] The superpixel segmentation module described in S1 is as follows:

[0183] (1) Edge probability map generation. In order to make full use of the structural information of the binocular image, this module uses the improved SLIC algorithm to generate the edge probability map of the left image I. L With the right picture I R Perform operations and jointly construct structure-aware superpixel segmentation. The specific contents are as follows:

[0184] ① Feature-guided edge extraction. L and I R Perform convolution processing separately to extract edge perception features of the left and right images:

[0185]

[0186] Among them, the convolution kernel size is 3×3; the number of input left and right image channels is 3, and the number of output channels is 16; the ReLU activation function is used to enhance the left and right perception features and nonlinear expression capability.

[0187] ②Edge prediction. and Based on this, the edge probability maps E of the left and right images are generated respectively. L , E R , which can be expressed as:

[0188]

[0189] Here, σ is the Sigmoid function, which is used to compress the output to the interval [0, 1]. The final result is an edge probability map of size B × 1 × H × W.

[0190] (2) Edge enhancement map generation. The edge probability map is spliced ​​with the left and right image feature maps to generate the corresponding left and right edge enhancement maps. It can be expressed as:

[0191]

[0192] Where Concat(·) represents the concatenation operation on the channel dimension, It aims to inject edge structure information into large-scale features and participate in the construction process of large-scale cost volumes.

[0193] The multi-scale cost volume construction module described in S1 is as follows:

[0194] Construct the cost volume at three scales: small, medium, and large. and the feature map on the right As the basis, * represents the three scales s, m, and l. The correlation between the left and right features is calculated by pixel-by-pixel translation operation in the disparity dimension to obtain the multi-scale basic cost volume. The calculation formula is as follows:

[0195]

[0196] Among them, Corr represents the pixel-by-pixel dot product correlation calculation to generate a multi-scale cost volume. They correspond to small, medium and large scale basic cost volumes respectively.

[0197] The multi-scale cost volume aggregation module is specifically as follows:

[0198] In order to improve the cost volume's ability to distinguish texture repetition, weak texture and occluded areas, wavelet transform features are introduced at three scales respectively. This module aggregates superpixel edge information to enhance the cost volume's expression ability, achieving structural enhancement and semantic supplementation.

[0199] ① Small-scale cost volume aggregation. The wavelet features of the high-frequency subband HH are used as structural details to supplement them, mapped to the cost space through 1×1 convolution, and added element-by-element with the small-scale basic cost volume to obtain the aggregated small-scale cost volume C s , which can be expressed as:

[0200]

[0201] in, Represents the HH subband features obtained by wavelet transform, Conv 1×1 Used for channel matching and semantic alignment.

[0202] ②Mesoscale cost volume aggregation. The midscale cost volume is added element by element with the horizontal LH and vertical HL subband features of the wavelet transform module to supplement the edge direction information. The aggregation method is as follows:

[0203]

[0204] in It is the splicing feature of LH and HL at the mesoscale.

[0205] ③ Large-scale cost volume aggregation. The large-scale aggregated cost volume is constructed by adding the large-scale basic cost volume and the low-frequency sub-band LL features of the wavelet transform element by element:

[0206]

[0207] in, is the LL subband feature of the wavelet transform module.

[0208] The cross-scale cost volume fusion module described in S1 is as follows:

[0209] (1) Feature extraction of multi-scale aggregate cost bodies based on 3D hourglass network structure. A deep feature extraction network based on 3D hourglass network structure is used to extract the aggregate cost bodies C at small, medium and large scales. s 、C m 、C l , achieving deep aggregation and context modeling of multi-scale aggregated cost volume information. Each hourglass structure contains symmetrical encoder and decoder paths. The encoder achieves feature compression and extracts global semantics through layer-by-layer 3D convolution; the decoder achieves structural reconstruction through 3D transposed convolution and integrates shallow detail features from jump connections, thereby obtaining a cost volume representation that combines structural perception and semantic understanding. The multi-scale cost volume feature extraction process can be expressed as:

[0210] D s =Hourglass3D(C s ),D m =Hourglass3D(C m ),D l =Hourglass3D(C l )

[0211] in, They respectively represent the small, medium and large scale cost volume feature representations after optimization by the 3D hourglass network, which have stronger three-dimensional context perception capabilities and provide basic support for cross-scale information fusion.

[0212] (2) Cross-scale fusion mechanism integrates multi-layer information. To fully utilize the complementarity of features at different scales, this paper designs an adaptive cross-scale fusion mechanism to achieve efficient fusion of high-level semantic information and low-level structural details. The implementation process is as follows:

[0213] First, the small-scale feature D s Upsampled to the mesoscale spatial size by bilinear interpolation, and the mesoscale feature D m Alignment can be expressed as:

[0214]

[0215] in, UpSample(·) represents bilinear interpolation upsampling, and then the upsampled Input to the convolution layer to obtain mid-scale enhanced features and with D m Jointly generate attention weight A m , which can be expressed as:

[0216]

[0217] Among them, σ represents the Sigmoid activation function, which is used to generate the normalized attention weight, and then the salient area is enhanced by channel-by-channel multiplication, and fused with the original mid-scale features in the form of residuals to obtain the fused mid-scale representation It can be expressed as:

[0218]

[0219] Next, Further upsampling to the large-scale feature D l Aligned spatial dimensions:

[0220]

[0221] Further convolution operation is performed to obtain large-scale enhanced features Generate attention weight A l , by element-wise addition With A l Generate large-scale fusion features, which can be expressed as:

[0222]

[0223] The fused large-scale feature volume It also has multi-scale contextual semantics and fine structure information, providing rich feature support for disparity estimation.

[0224] The disparity regression module described in S1 is as follows:

[0225] (1) Cost volume upsampling. The fused cost volume By bilinear interpolation, it is restored to the original resolution of the input image, providing a cost volume consistent with the input image size for the subsequent Softmax normalization and disparity regression steps. The specific expression is:

[0226]

[0227] in, Indicates the disparity value index, in the present invention, d max Take 192; x∈[0,H-1], which represents the pixel coordinates in the width direction of the image;

[0228] y∈[0,W-1] represents the pixel coordinates in the height direction of the image, and Interp represents the bilinear interpolation operation.

[0229] (2) Softmax normalization. Perform Softmax normalization along the disparity dimension to generate a disparity probability distribution map:

[0230]

[0231] in, Indicates that at each pixel position (x, y), it belongs to each disparity layer d∈[0,d max ], exp represents the exponential transformation operation.

[0232] (3) Expected value regression. Based on probability graph Take a weighted average of the disparity to achieve sub-pixel disparity estimation:

[0233]

[0234] Disparity map after regression As the final depth estimation output.

[0235] S2, the network framework described in S1 is subjected to multiple rounds of iterative training using supervised learning to obtain a disparity estimation model. During training, the left and right images in the training set are used as input, and forward propagation is performed to calculate the predicted disparity map, which is compared with the corresponding real disparity map. The training error is calculated through the loss function, and the network parameters are updated using the backpropagation algorithm.

[0236] Specifically, the training process includes the following steps:

[0237] S2.1 Dataset selection.

[0238] This paper uses two subsets of the KITTI public dataset, KITTI2012 and KITTI2015, as training data for the network model. This dataset is widely used in stereo matching, autonomous driving perception, and computer vision research tasks, and is highly authoritative and representative in the international academic community.

[0239] The KITTI2012 dataset contains 194 training image pairs and 195 test image pairs, with an image resolution of 1226×370. The KITTI2015 dataset contains 200 training image pairs and 200 test image pairs, with an image resolution of 1242×375. Compared to KITTI2012, KITTI2015 features improvements in data acquisition, using lidar to generate realistic disparity maps and introducing more dynamic objects and semantic information, further increasing the complexity and realism of the data.

[0240] S2.2 Optimizer configuration and learning rate strategy adjustment.

[0241] The Adam optimizer is used to perform end-to-end training on the network parameters, and the initial learning rate is set to η0 = 1×10 -4 , and adopts a phased dynamic learning rate scheduling strategy, including warm-up phase, constant phase and decay phase, to improve the stability and generalization performance of training.

[0242] During the initial warm-up phase of training, the learning rate increases linearly from 0 to η0 to prevent gradient oscillation. During the mid-training constant phase, η0 remains constant to accelerate model feature learning. During the late-training decay phase, the learning rate decays linearly to a smaller value at a fixed rate to improve model stability on the validation set. The optimizer parameters are set to β1 = β2 = 0.9 to prevent drastic gradient fluctuations.

[0243] S2.3 Construction of loss function.

[0244] Introducing the joint loss function L during training total , to simultaneously optimize the global accuracy and edge structure consistency of disparity prediction, defined as follows:

[0245] L total =αL smooth-L1 +βL multi-scale

[0246] Among them, α and β are weight coefficients, which adjust L smooth-L1 , L multi-scale The contribution of two losses to the total goal. L smooth-L1 is the disparity smoothing loss function, which predicts the disparity map within the valid disparity mask area. and the true disparity map D gt Perform pixel-by-pixel regression, the expression is as follows:

[0247]

[0248] Where N is the number of effective pixels, is the predicted disparity of the i-th pixel, D gtiis the true disparity of the i-th pixel. In order to further improve the feature matching ability of the model at different scales, the present invention introduces a multi-scale feature loss L multi-scale , the specific calculation is as follows:

[0249]

[0250] Among them, *∈s,m,l, s,m,l represent small, medium and large scale cost volumes respectively, λ s is the weight coefficient of each loss, represents scalar multiplication, is the correlation loss on scale *, and the calculation formula is:

[0251]

[0252] Among them, N * Indicates the total number of valid pixels in the area where the true disparity is greater than 0 at * scale, and ||·||2 is used to calculate and The Euclidean distance between them, Corr(·) represents the correlation calculation, Represents the left and right feature maps at * scale.

[0253] S2.4 Iterative training.

[0254] In each round of training, the image data is read from the training set in batch as the smallest unit, including the left image I L , right image I R And the corresponding real disparity map D gt . Will I L , I R Input the network, perform the forward propagation process, and output the predicted disparity map

[0255] Then, based on the predicted disparity map Results and real disparity map D gt The difference between them is used to calculate the joint loss function L total , the present invention adopts weighted L smooth-L1 The loss function is the main loss term, that is, α>β, and D is selected gt The effective area >0 is included in the calculation to enhance the fitting ability of the effective depth area. The calculated loss L total Backpropagation is used to obtain gradient information, and the model parameters are updated using the Adam optimizer. Training is repeated until all training samples in the current round are fully traversed, and the average error metric of this round of training is calculated for performance evaluation.

[0256] After each round of training, model validation is started immediately to evaluate the generalization ability of the currently saved training model on the validation set. During the validation phase, the validation set data is read in sequence and input into the current model for forward reasoning to obtain the predicted disparity map. By comparing the corresponding real disparity map D gt For comparison, the following five error indicators are calculated: average endpoint error EPE, D1-error, and accuracy indicators T1, T2, and T3 within 1, 2, and 3 pixel thresholds. These indicators can comprehensively measure the prediction accuracy of the model.

[0257] This invention uses a conditional mechanism to save models, preserving the best-performing model. During iterative training, the five error metrics corresponding to each training round can be used to determine whether to save the model trained in the current round. This invention uses D1-error to determine whether to save a model. Specifically, if the D1-error obtained in a given round of verification is less than the historical minimum, the D1-error value and the corresponding round number are recorded, and the trained model is saved.

[0258] It should be noted that the performance test indicators EPE, D1-error, T1, T2, and T3 used in the present invention are as follows:

[0259] ①EPE (End-Point Error) is the endpoint error, which is used to measure the predicted disparity map and the true disparity map D gt The mean absolute error between is defined as follows:

[0260]

[0261] Where N is the total number of valid pixels, is the predicted disparity of the i-th pixel, D gti is the true disparity of the i-th pixel.

[0262] ②D1-error is the three-pixel error rate. This metric emphasizes accuracy when the error is significant and is suitable for evaluating the robustness of the algorithm under large error tolerance. D1-error can be used to measure the proportion of pixels where the prediction error exceeds 3 pixels or exceeds 5% of the true disparity. It is defined as follows:

[0263]

[0264] Among them, [·] means that the value is 1 when the Boolean expression is true, otherwise it is 0. is the predicted disparity of the i-th pixel, D gti is the true disparity of the i-th pixel.

[0265] ③T1, T2, and T3 are threshold error rates, which represent the percentage of pixels where the absolute error between the predicted disparity and the true disparity exceeds 1 pixel, 2 pixels, and 3 pixels, respectively. This reflects the model's ability to control error distribution at different thresholds. The smaller the values ​​of T1, T2, and T3, the higher the accuracy of the model at the corresponding threshold. The calculation formula is as follows:

[0266]

[0267] Where T1, T2, and T3 represent the ratios of errors exceeding 1, 2, and 3 pixels in the edge region, respectively; [·] indicates that the value is 1 when the Boolean expression holds, and 0 otherwise; is the predicted disparity of the i-th pixel; D gti is the true disparity of the i-th pixel.

[0268] This example uses the NVIDIA GeForce RTX 3090 graphics card as the computing platform, configured with driver version CUDA 12.4 and PyTorch 2.x deep learning framework. All experiments were completed in a single-card environment, and nvidia-smi real-time monitoring was used to ensure that the GPU temperature was stable below 80°C to avoid performance throttling. The method of the present invention designed an ablation experiment based on the wavelet transform attention module and superpixel segmentation module, as well as a performance evaluation experiment, respectively. Figure 6 Figure 7 shown.

[0269] Figure 6 The ablation experiment results of the present invention on the KITTI2015 dataset are presented, and the effects of different module combinations on performance are compared. The experimental indicators include endpoint error EPE, disparity error D1-error, and T1, T2, and T3. The results show that when the wavelet transform attention module is used alone, the EPE drops from 0.799 to 0.773, and the performance is improved; when the superpixel segmentation module is used alone, the EPE rises to 0.999, and the model endpoint error increases significantly. The best effect is achieved when the two are used together, with the EPE dropping to 0.733, indicating that there is a synergistic optimization effect between the wavelet transform-based attention module and the superpixel segmentation module. The bold data in the figure are the best performance indicators.

[0270] Figure 7The performance of our method was compared with other stereo matching methods on the KITTI 2012 and KITTI 2015 datasets, with D1-error and EPE as the primary evaluation metrics. 300 training rounds of models were tested on the corresponding test sets. The results show that on the KITTI 2012 dataset, our method achieved the best performance, with a D1-error of 1.06% and an EPE of 0.45 pixels. This performance outperforms traditional methods such as MC-CNN and GCNet, as well as advanced methods such as GANet-deep and AcfNet. On the KITTI 2015 dataset, our method achieved a D1-error of 0.73% and an EPE of 2.4 pixels, outperforming mainstream methods such as GANet-deep and PSMNet. Experimental results demonstrate that our method exhibits good matching accuracy on both datasets, with particularly significant improvements on the KITTI 2012 dataset. Data missing for some methods on specific datasets are marked with a minus sign. The bolded data represent the best performance indicators.

[0271] Figure 7 The stereo matching methods and performance index data compared with the method of the present invention are all from relevant literature:

[0272] [1]J.Zbontar,Y.LeCun,et al.,Stereo matching by training aconvolutional neural network to compare image patches,J.Mach.Learn.Res.17(1)(2016)2287–2318

[0273] [2] F. Zhang, V. Prisacariu, R. Yang, PHSTorr, GA-Net: guided aggregationnet for end-to-end stereo matching. Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 185–194.

[0274] [3]A.Kendall,H.Martirosyan,S.Dasgupta,P.Henry,R.Kennedy,A.Bachrach,A.Bry,End-to-endlearning of geometry and context for deep stereoregression.Proceedings of the IEEE International Conference on ComputerVision,2017,pp.66–75

[0275] [4]Z.Wu,X.Wu,X.Zhang,S.Wang,L.Ju,Semantic stereo matching withpyramid cost volumes.Proceedings of theIEEE / CVF International Conference onComputer Vision,2019,pp.7484–7493.

[0276] [5]G.Yang,H.Zhao,J.Shi,Z.Deng,J.Jia,SegStereo:exploiting semanticinformation for disparity estimation.Proceedings of the European Conferenceon Computer Vision(ECCV),2018,pp.636–651.

[0277] [6]Y.Zhang,Y.Chen,X.Bai,S.Yu,K.Yu,Z.Li,K.Yang,Adaptive unimodal costvolume filtering for deep stereo matching.Proceedings of the AAAI Conferenceon Artificial Intelligence Vol.34,2020,pp.12926–12934.

[0278] [7]J.-R.Chang,Y.-S.Chen,Pyramid stereo matching network.Proceedingsof the IEEE Conference on Computer Vision and Pattern Recognition,2018,pp.5410–5418.

[0279] [8]X.Guo,K.Yang,W.Yang,X.Wang,H.Li,Group-wi se correlation stereonetwork.Proceedings of the IEEE / CVF Conference on Computer Vision and PatternRecognition,2019,pp.3273–3282.

[0280] [9]G.Xu,J.Cheng,P.Guo,X.Yang,Attention concatenation volume foraccurate and efficient stereo matching.Proceedings of the IEEE / CVF Conferenceon Computer Vision and Pattern Recognition,2022,pp.12981–12990.

[0281]

[10] Z.Shen,Y.Dai,Z.Rao,CFNet:cascade and fused cost volume for robuststereo matching.Proceedings of the IEEE / CVF Conference on Computer Vision andPattern Recognition,2021,pp.13906–13915.

[0282]

[11] Zhao, Y., Zhong, H., Jia, B., & Li, H. (2023). A fast stereo matching network based on temporal attention and 2D convolution. Signal Processing: Image Communication, 117, 116535. https: / / doi.org / 10.1016 / j.image.2023.116535.

[0283] Figure 4 The predicted disparity map for the image named 000010_11.png in the KITTI2015 dataset for stereo matching. Figure 5 The disparity error map after visualization of the image. The visualization of the error disparity map is based on the error mapping function Calculate the prediction error of each pixel and convert it into a color image, which can be expressed as:

[0284]

[0285] Where (x, y) represents the image pixel coordinates, and ColorMap(·) represents the color mapping function, which maps the normalized error values ​​using pseudo-color encoding in the RGB color space. The visualization uses color to represent the error magnitude, with different colors corresponding to different prediction error sizes. Warmer colors indicate larger prediction errors, while cooler colors indicate smaller errors and more accurate predictions.

[0286] The method proposed in this paper constructs a deep learning network framework based on wavelet transform and superpixel segmentation to improve the accuracy of disparity estimation in stereo matching tasks. This deep learning network framework consists of a feature extraction module, a wavelet-based attention module, a superpixel segmentation module, a multi-scale cost volume construction module, a multi-scale cost volume aggregation module, a cross-scale cost volume fusion module, and a disparity regression module. These modules work together to efficiently model the multi-scale and multi-structural information of an image. The feature extraction module uses ResNet combined with FPN to extract multi-layered image representations at different scales, providing rich feature support for multi-scale cost volume construction. The wavelet-based attention module, combined with a sub-band attention mechanism, enhances the representation of structural edge features in the image. The superpixel segmentation module, by introducing a feature-guided structure, effectively segments semantically consistent regions within the image, enhancing the edge-preserving capability of disparity estimation. The multi-scale cost volume construction module performs correlation operations on multi-scale features to generate cost volumes at different scales. The output of the superpixel segmentation module is incorporated into the large-scale cost volume construction process. The multi-scale cost volume aggregation module aggregates information from different subbands of the wavelet transform-based attention module with the corresponding cost volumes, effectively utilizing image frequency domain information. The cross-scale cost volume fusion module effectively integrates redundant and complementary information between features at different scales, improving matching robustness and edge transition smoothness. Finally, the disparity regression module outputs highly accurate disparity estimation results.

[0287] The stereo matching method provided by the embodiment of the present invention not only fully exploits the potential information of the image in the frequency domain and structure domain, but also improves the generalization ability and precision performance of the network, meeting the requirements for accurate disparity estimation of complex textures and structural edges in various stereo vision tasks.

[0288] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limiting. Other modifications or equivalent substitutions made to the technical solution of the present invention by ordinary technicians in this field should be included in the scope of the claims of the present invention as long as they do not depart from the spirit and scope of the technical solution of the present invention.

Claims

1. A stereo matching method based on wavelet transform and superpixel segmentation, characterized in that: The following steps are involved: S1, constructing a deep learning network framework based on wavelet transform and superpixel segmentation, wherein the deep learning network framework includes a multi-scale feature extraction module, a wavelet transform-based attention module, a superpixel segmentation module, a multi-scale cost volume construction module, a multi-scale cost volume aggregation module, a cross-scale cost volume fusion module, and a disparity regression module; S2 uses supervised learning to perform multiple rounds of training iterations on the network framework of S1 to obtain a disparity estimation model. During training, the left and right images in the training set are used as input, and forward propagation is performed to calculate the predicted disparity map, which is compared with the corresponding real disparity map. The training error is calculated through the loss function, and the network parameters are updated using the backpropagation algorithm.

2. The stereo matching method based on wavelet transform and superpixel segmentation according to claim 1, characterized in that: The operation method of the multi-scale feature extraction module in S1 is: The left and right images of the stereo image pair are input into the residual network containing multiple residual blocks for preliminary feature extraction. The extraction process consists of four stages: the first stage is to downsample the input image to The output feature map channel is 32; In the second stage, the convolution residual block with a step size of 2 is used for further downsampling, and the feature size is The output channels are 64; the third stage does not change the feature size, and the number of channels is increased to 128; the fourth stage uses a dilated convolution with dilation = 2 to enhance the receptive field, keep the feature size unchanged, and the number of channels is still 128. The output feature maps of these four stages are F1, F2, F3, and F4 respectively, then: To enhance the multi-layer feature fusion effect, F2, F3, and F4 are concatenated along the channel dimension to form a unified feature representation: After the initial feature extraction, the channel dimension is too high and the computational complexity is high. To reduce the computational complexity, a two-stage dimensionality reduction convolution structure is introduced after the initial feature extraction. 1×1 and 3×3 convolution kernels are used respectively to reduce the number of channels of the left and right feature maps from 320 to 128, and then further to 32. The computational process can be expressed as: where W1∈R 128×320×3×3 , represents a 3×3 convolution kernel that reduces the number of channels of the initial extracted feature map from 320 to 128, W2∈R 32×128×1×1 , which means that the number of channels of the feature map processed by W1 is further reduced to 32 channels of 1×1 convolution kernel, the symbol * represents the convolution operation, and ReLu(·) represents the ReLU activation function; the final feature extraction process can be recorded as: F L =FeatureExtraction(I L ),F R =FeatureExtraction(I R ) The left and right feature maps F L , F L ,F R As the initial input of FPN, corresponding to large-scale features A 3×3 convolution with a step size of 2 and a 1×1 convolution are combined to Downsample to obtain mid-scale features Use the same method to Downsampling to obtain small-scale features in, Finally, the output of the FPN module can be expressed as:

3. The stereo matching method based on wavelet transform and superpixel segmentation according to claim 1, characterized in that: The operation method of the multi-scale cost volume construction module in S1 is: Construct the cost volume at three scales: small, medium, and large. and the feature map on the right As the basis, * represents the three scales s, m, and l. The correlation between the left and right features is calculated by pixel-by-pixel translation operation in the disparity dimension to obtain the multi-scale basic cost volume. The calculation formula is as follows: Among them, Corr represents the pixel-by-pixel dot product correlation calculation to generate a multi-scale cost volume. They correspond to small, medium and large scale basic cost volumes respectively.

4. The stereo matching method based on wavelet transform and superpixel segmentation according to claim 1, characterized in that: The operation method of the multi-scale cost volume aggregation module in S1 is as follows: In order to fuse super-pixel edge information and enhance the cost volume expression capability, this module introduces wavelet transform features at three scales respectively; When aggregating the small-scale cost volume, the wavelet features of the high-frequency subband HH are used as structural details to supplement them, which are mapped to the cost space through 1×1 convolution and added element-by-element with the small-scale basic cost volume to obtain the aggregated small-scale cost volume C. s , which can be expressed as: in, Represents the HH subband features obtained by wavelet transform, Conv 1×1 Used for channel matching and semantic alignment; When aggregating the mid-scale cost volume, the sub-band features of the high-frequency sub-band LH and HL are used. As the image edge detail information, its size is adjusted to the size of the mid-scale cost space through 1×1 convolution, and then aggregated element by element with the mid-scale basic cost volume. The process can be expressed as: When aggregating large-scale cost volumes, the wavelet features of the low-frequency subband LL are used As a supplement to image structure enhancement, the size is adjusted to the medium-large cost space size through 1×1 convolution, and the large and medium-scale cost volumes are added and aggregated. The process can be expressed as:

5. The stereo matching method based on wavelet transform and superpixel segmentation according to claim 1, characterized in that: The operation method of the cross-scale cost volume fusion module in S1 is as follows: For small, medium and large scale aggregation cost bodies C respectively s 、C m 、C l , achieving deep fusion and context modeling of multi-scale aggregated cost volume information; each hourglass structure contains symmetrical encoder and decoder paths. The encoder achieves feature compression and extracts global semantics through layer-by-layer 3D convolution; the decoder achieves structural reconstruction through 3D transposed convolution and integrates shallow detail features in jump connections, thereby obtaining a cost volume representation that combines structural perception and semantic understanding; the multi-scale cost volume feature extraction process can be expressed as: D s =Hourglass3D(C s ),D m =Hourglass3D(C m ),D l =Hourglass3D(C l ) in, They represent the small, medium, and large-scale cost volume feature representations after optimization by the 3D hourglass network, respectively. They have stronger three-dimensional context perception capabilities and provide basic support for cross-scale information fusion. First, the small-scale feature D s Upsampled to the mesoscale spatial size by bilinear interpolation, and the mesoscale feature D m Alignment: in, UpSample(·) represents bilinear interpolation upsampling, and then the upsampled Input to the convolution layer to obtain mid-scale enhanced features and with D m Jointly generate attention weight A m , which can be expressed as: Among them, σ represents the Sigmoid activation function, which is used to generate the normalized attention weight, and then the salient area is enhanced by channel-by-channel multiplication, and fused with the original mid-scale features in the form of residuals to obtain the fused mid-scale representation It can be expressed as: Next, Further upsampling to the large-scale feature D l Aligned spatial dimensions: Further convolution operation is performed to obtain large-scale enhanced features Generate attention weight A l , by element-wise addition With A l Generate large-scale fusion features, which can be expressed as: The fused large-scale feature volume It also has multi-scale contextual semantics and fine structure information, providing rich feature support for disparity estimation.

6. The stereo matching method based on wavelet transform and superpixel segmentation according to claim 1, characterized in that: The operation method of the disparity regression module in S1 is as follows: The fused cost body By bilinear interpolation, it is restored to the original resolution of the input image, providing a cost volume consistent with the input image size for the subsequent Softmax normalization and disparity regression steps. The specific expression is: in, Indicates the disparity value index, d max Take 192; x∈[0,H-1], represents the pixel coordinates in the width direction of the image; y∈[0,W-1], represents the pixel coordinates in the height direction of the image, Interp represents the bilinear interpolation operation; Next, Perform Softmax normalization along the disparity dimension D to generate a disparity probability distribution map: in, Indicates that at each pixel position (x, y), it belongs to each disparity layer d∈[0,d max ], exp represents the exponential transformation operation; Finally, based on the probability map Take a weighted average of the disparity to achieve sub-pixel disparity estimation: Disparity map after regression As the final depth estimation output.

7. The stereo matching method based on wavelet transform and superpixel segmentation according to claim 1, characterized in that: The operation method of the wavelet transform-based attention module in S1 is: Input stereo image pair Left image I L , before wavelet decomposition, reflection filling is performed to extend the boundary of the image by mirroring the edge pixels of the image, so that I L Both the height and width are even: I padded =Pad(I L ,mode='reflect') Use learnable 2×2 convolution kernel weights for convolution operation and downsample with a stride of 2; The calculation formula is: LL = (I padded *W LL )↓2,LH=(I padded *W LH )↓2,HL=(I padded *W HL )↓2,HH=(I padded *W HH )↓2, where W LL ,W LH ,W HL ,W HH ∈R 2×2 It is a filter bank generated by the Haar wavelet basis, which is learnable. ↓2 represents downsampling with a step size of 2. This operation uses unfold to expand the matrix to improve computational efficiency: I unfold =Unfold(I padded ,kernel_size=2,stride=2) After wavelet decomposition, the sizes are all The low-frequency sub-band LL and three high-frequency sub-bands LH, HL, and HH are combined in the channel dimension: Then, a 3×3 group convolution with groups=3 is used to independently extract cross-subband context features of the RGB three channels to ensure the independence of each channel. Nonlinear characteristics are introduced through ReLU activation. Then, cross-channel features are fused through 1×1 convolution. Finally, a normalized attention weight map in the range of [0, 1] is generated using the Sigmoid function. It can be expressed as: Where A is the normalized attention weight, σ(·) is the Sigmoid function; Apply the attention weight to F concat , perform channel-by-channel multiplication operations to enhance the salient areas: F att =F concat ⊙A Among them, F att is the enhanced feature map, ⊙ represents the channel-by-channel element multiplication; Then, sub-band enhancement is performed. First, a 3×3 convolutional layer is used to enhance F att Extract local features, then use ReLU activation function to introduce nonlinear characteristics, use 3×3 convolution layer to extract local features and fuse cross-channel information, and finally perform group normalization to alleviate the normalization instability problem caused by small batch training. The output size is The fusion feature F DWT , F DWT The operation process can be expressed as: Among them, GN represents group normalization, F DWT The information of each sub-band of wavelet transform is retained, which is convenient for participating in the multi-scale cost volume aggregation. In order to effectively aggregate the wavelet transform features into the multi-scale cost volume, F DWT Downsampling to the size of the multi-scale cost volume and adjusting its number of channels through convolution can be expressed as: Among them, Interp represents interpolation downsampling, for F DWT Resize to match the small, medium, and large cost volume sizes H s ×W s 、H m ×W m 、H l ×W l , that is, respectively Conv 1×1 Represents a 1×1 convolution operation, used to adjust F DWT The number of channels, after adjustment The number of channels are 12, 24, and 48 respectively.

8. The stereo matching method based on wavelet transform and superpixel segmentation according to claim 1, characterized in that: The operation method of the superpixel segmentation module in S1 is: For left and right images I L and I R Perform convolution processing separately to extract edge perception features of the left and right images: Among them, the convolution kernel size is 3×3; the number of input left and right image channels is 3, and the number of output channels is 16; the ReLU activation function is used to enhance the left and right perception features and Nonlinear expression ability; exist and Based on this, the edge probability maps E of the left and right images are generated respectively. L , E R , which can be expressed as: Where σ is the Sigmoid function, which is used to compress the output to the interval [0, 1], and finally obtain an edge probability map of size B×1×H×W; The edge probability map is channel-joined with the left and right image feature maps to generate the corresponding left and right edge enhancement maps. It can be expressed as: Where Concat(·) represents the concatenation operation on the channel dimension, It aims to inject edge structure information into large-scale features and participate in the construction process of large-scale cost volumes.

9. The stereo matching method based on wavelet transform and superpixel segmentation according to claim 1, characterized in that: The method for training the depth estimation network framework in S2 is: The Adam optimizer is used to train the network parameters end-to-end, and the initial learning rate is set to η0 = 1×10 -4 , and adopts a phased dynamic learning rate scheduling strategy, including warm-up phase, constant phase and decay phase, to improve the stability and generalization performance of training; During the warm-up phase at the beginning of training, the learning rate increases linearly from 0 to η0 to prevent gradient oscillation; during the constant phase in the middle of training, η0 remains unchanged to accelerate model feature learning; In the decay phase of the later training phase, the learning rate is linearly decayed to a smaller value at a fixed ratio to improve the stability of the model on the validation set. The optimizer parameters are set to β1 = β2 = 0.9 to prevent drastic fluctuations in the gradient. Introducing the joint loss function L during training total , to simultaneously optimize the global accuracy and edge structure consistency of disparity prediction, defined as follows: L total =αL smooth-L1 +βL multi-scale Among them, α and β are weight coefficients, which adjust L smooth-L1 , L multi-scale The contribution of two losses to the total loss; L smooth-L1 is the disparity smoothing loss function, which predicts the disparity map within the valid disparity mask area. and the true disparity map D gt Perform pixel-by-pixel regression, the expression is as follows: Where N is the number of effective pixels, is the predicted disparity of the i-th pixel, D gti is the true disparity of the i-th pixel; in order to further improve the feature matching ability of the model at different scales, a multi-scale feature loss L is introduced multi-scale , the specific calculation is as follows: Among them, *∈s,m,l, s,m,l represent small, medium and large scale cost volumes respectively, λ s is the weight coefficient of each loss, represents scalar multiplication, is the correlation loss on scale *, and the calculation formula is: Among them, N * Indicates the total number of valid pixels in the area where the true disparity is greater than 0 at * scale, and ||·||2 is used to calculate and The Euclidean distance between them, Corr(·) represents the correlation calculation, Represents the left and right feature maps at * scale; in each round of training, the image data is read from the training set in sequence with batch as the minimum unit, including the left image I L , right image I R And the corresponding real disparity map D gt ; will I L , I R Input the network, perform the forward propagation process, and output the predicted disparity map Then, based on the predicted disparity map Results and real disparity map D gt The difference between them is used to calculate the joint loss function L total , using weighted L smooth-L1 The loss function is the main loss term, that is, α>β, and D is selected gt The effective area >0 participates in the calculation to enhance the fitting ability of the effective depth area; the calculated loss L total Backpropagation is used to obtain gradient information, and the model parameters are updated once using the Adam optimizer. Training is iterated until all training samples in the current round are fully traversed, and the average error index of this round of training is calculated for performance evaluation. After each round of training is completed, the model verification process is immediately started to evaluate the generalization ability of the currently saved training model on the verification set; in the verification stage, the verification set data is read in sequence and input into the current model for forward reasoning to obtain the predicted disparity map. By comparing the corresponding real disparity map D gt For comparison, the following five error indicators are calculated: average endpoint error EPE, D1-error, and accuracy indicators T1, T2, and T3 within 1, 2, and 3 pixel thresholds. These indicators can comprehensively measure the prediction accuracy of the model; a conditional mechanism is used to save the model to save the best performance model; the five error indicators corresponding to each training round in iterative training can be used to determine whether to save the model obtained by the current round of training, and D1-error is used to determine whether to save the model. Specifically: when the D1-error obtained in a round of verification is less than the historical minimum value, the numerical value of the D1-error and the corresponding round number are recorded, and the trained model is saved.

Images