Cross-domain binocular stereo matching method based on multi-scale information dynamic fusion and feature deviation correction

Abstract

The invention provides a cross-domain binocular stereo matching method based on multi-scale information dynamic fusion and feature deviation correction, and belongs to the field of computer vision. Specifically, the model adaptively fuses a convolutional neural network (CNN) and a Transform structure according to dynamic change of a scene so as to capture local detail information and global context dependence at the same time. Secondly, through a multi-stage progressive cost body fusion and aggregation mechanism, full fusion of multi-scale matching cost bodies is promoted, redundant information is effectively inhibited, and matching precision is improved; and finally, guiding the model to learn more stable domain invariant representation by gradually reducing the deviation between different domain features, thereby remarkably enhancing the cross-domain robustness and adaptability. Compared with the most advanced method for performing cross-domain experiments in different scenes, the method disclosed by the invention shows better performance compared with an existing advanced model: 1) the cross-domain stereo matching precision is remarkably improved in a complex scene, and stronger domain migration capability and generalization stability are shown; and 2) while the cross-domain robustness is maintained, relatively high accuracy and calculation efficiency are maintained in a non-cross-domain environment, and double consideration of cross-domain and non-cross-domain performance is realized.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and more specifically, to a cross-domain binocular stereo matching method based on dynamic fusion of multi-scale information and feature deviation correction. Background Art

[0002] In the development of modern society, humans rely on the visual system to perceive the environment and obtain rich semantic information and three-dimensional structure from it. Similarly, computer vision technology uses visual sensors (such as cameras) to simulate the functions of the human eye, analyzing and processing captured two-dimensional images to achieve the ability to approximate the human visual system. Binocular stereo matching, a core task in computer vision, aims to estimate the disparity from a pair of rectified stereo images to obtain the three-dimensional information required for scene reconstruction. Binocular stereo matching methods have a wide range of applications in fields such as autonomous driving, robot navigation, and target tracking.

[0003] In recent years, convolutional neural network-based stereo matching methods have become mainstream, and they can be divided into two main categories: non-end-to-end algorithms and end-to-end algorithms. Non-end-to-end stereo matching algorithms typically follow the traditional four-step process, including matching cost calculation, cost aggregation, disparity calculation, and disparity post-processing. In these methods, convolutional neural networks (CNNs) are primarily used to optimize matching cost calculation or disparity optimization, without participating in the entire process. In contrast, end-to-end stereo matching methods directly learn the mapping from the input stereo image to the final disparity map, eliminating the hand-crafted cost calculation and optimization steps. Typical end-to-end networks include GC-Net, PSMNet, and GANet. These methods typically use 3D convolution or attention mechanisms to learn deep feature representations across views, and achieve more accurate disparity estimation through end-to-end training.

[0004] However, although a lot of research has been devoted to improving matching accuracy, how to achieve high accuracy on datasets in the same domain while ensuring that the model still has good generalization capabilities in different domains remains an important challenge. Summary of the Invention

[0005] To address these issues, the present invention proposes a cross-domain binocular stereo matching method based on dynamic fusion of multi-scale information and feature deviation correction. This method improves feature robustness by adaptively fusing local and global features based on the scene. Furthermore, a novel cost volume aggregation method is introduced to effectively utilize small-scale cost volume information and dynamically optimize the construction of high-scale cost volumes, thereby improving the efficiency of multi-scale information fusion. Finally, by targeted processing of regions with large feature deviations, the interference of noisy matching points on disparity prediction is effectively suppressed. Through this effective integration, efficient cross-domain generalization is achieved.

[0006] The technical solutions of the present invention are as follows:

[0007] 1. A cross-domain binocular stereo matching method based on dynamic fusion of multi-scale information and feature deviation correction, the steps are as follows:

[0008] Step 1: Collect binocular stereo matching datasets, including Scene Flow dataset, KITTI 2012 dataset, and KITTI 2015 dataset;

[0009] Step 2: Preprocess the image pairs for binocular stereo matching. Specifically, crop the images to 256×512, perform random brightness, contrast, and gamma adjustments, and use random rotation and random occlusion strategies. The height and width of the cropped images are H and W, respectively.

[0010] Step 3. After obtaining the preprocessed left and right image pairs, the stereo matching model will sequentially execute the local-global feature collaborative fusion module to extract multi-scale features, construct the cost volume, perform progressive layered cost volume aggregation, perform disparity regression on the regularized cost volume to generate the initial disparity, and finally refine the disparity using uncertainty estimation based on feature consistency.

[0011] In the cross-domain stereo matching method based on dynamic fusion of multi-scale information and feature deviation correction, the feature extraction adopts the designed local-global feature collaborative fusion method, which includes three components:

[0012] Step 1. The first component is the multi-scale local feature extraction module. Its input is the left and right view images, both of which are four-dimensional tensors of shape (B, C, H, W). The multi-scale local feature extraction module uses the pre-trained semantic segmentation network ResNet-101 to extract local features from the input left and right images respectively. The purpose is to introduce prior knowledge in other data scenarios in advance to provide more robust information for subsequent feature extraction. It outputs four corresponding four-dimensional tensors respectively, whose feature scales are 1 / 4, 1 / 8, and 1 / 16 of the feature scales of the left and right input images, respectively, and are denoted as F 2-1-1 , F 2-1-2 , F 2-1-3 ;

[0013] Step 2. The second component is the global feature extraction module, whose input is the left and right view images. The global feature extraction module first divides the left and right view images into blocks through a projection layer with a convolution kernel size of 4 and a stride of 4, generating an embedding vector sequence of 1 / 16 of the original image scale. It then passes it through a four-layer stacked self-attention structure, each layer containing a four-head attention mechanism. Finally, a context-rich serialized feature map F with only 1 / 16 of the original image scale is extracted. 2-2-1 ;

[0014] Step 3: The third component is the adaptive scene perception feature fusion module, whose inputs are the local features F extracted by the multi-scale feature extraction module. 2-1-3 and the global context feature F extracted by the global feature extraction module 2-2-1 . For the local feature F of the input 2-1-3 and global context feature F 2-2-1 The specific fusion operations include:

[0015] Step 3-1: First, serialize the feature graph F 2-2-1 Perform dimension permutation and reorganization operations to reconstruct it into a standard four-dimensional tensor. Secondly, a hierarchical feature conversion strategy is used to transform the local feature F 2-1-3 Perform refinement processing, that is, build a primary feature extraction module through a cascaded 3×3Conv-BN-ReLU structure, and finally connect a 1×1Conv-BN layer to form a composite feature encoder for feature F 2-3-1 to refine;

[0016] Step 3-2: Local feature F after feature alignment 2-3-1 and global context feature F 2-2-1 Splicing along the channel dimension to obtain the joint feature F 2-3-2 In order to further enhance the information interaction between features of different receptive fields, the 3×3Conv-BN-ReLU structure is used to combine the joint features F 2-3-2 Perform feature extraction and further perform channel compression using 1×1 Conv-BN to reduce the amount of subsequent calculations;

[0017] Step 3-3: In order to enrich the statistical information of the features, the feature F after feature union is 2-3-2 Apply global maximum pooling and global average pooling respectively to extract feature representations under different perspectives, which are denoted as F 2-3-3 and F 2-3-4 In view of the complementarity between the two, they are spliced ​​along the channel dimension to form a mixed statistical feature F 2-3-5 ,This feature can adaptively capture more discriminative information in different scenarios,,providing a more solid foundation for subsequent dynamic weight allocation;

[0018] Step 3-4: For the obtained mixed statistical feature F 2-3-5 , this method uses a 1×1Conv-BN layer to reduce the channel dimension, and then generates a dynamic weight coefficient W with probabilistic interpretation through the Sigmoid function 2-3-1 and W 2-3-2 , and finally the feature weight W 2-3-1 and W 2-3-2 Multiply the original local features F2-1-3 and global context feature F 2-2-1 Perform dynamic weighting and then add the two weighted features to obtain the final adaptive fusion feature F 2-3-6 .

[0019] Step 4: The adaptive fusion feature F with a height of 1 / 16H and a width of 1 / 16W is obtained 2-3-6 Through a two-dimensional transposed convolution module, we get an upsampled feature F with a height of 1 / 8H and a width of 1 / 8W. 2-4-1 . Then it is combined with the obtained low-level features F of the same scale 2-1-2 The concatenation operation is performed along the channel dimension to obtain high-level semantic information while fully preserving the underlying geometric details, and finally obtain a feature F with a height of 1 / 8H and a width of 1 / 8W 2-4-2 Then, the feature F 2-4-2 It is also put into the two-dimensional transposed convolution module to obtain the upsampled feature F with a height of 1 / 4H and a width of 1 / 4W 2-4-3 , and then compare it with the obtained low-level feature F of the same scale 2-1-1 The concatenation operation is performed along the channel dimension, and finally the feature F with a feature height of 1 / 4H and a width of 1 / 4W is obtained. 2-4-4 ;

[0020] Step 5: The obtained feature F is only 1 / 16, 1 / 8 and 1 / 4 of the original image resolution 2-3-6 、F 2-4-2 、F 2-4-4 Further feature extraction operations are performed separately to lay the foundation for the subsequent construction of the series cost volume and the group-related cost volume. The further operations are as follows:

[0021] Step 5-1, first of all, the feature F 2-3-6 、F 2-4-2 、F 2-4-4 Perform a 3×3 convolution operation respectively, and then use the channel and spatial attention modules to weight the features used to construct the series cost body, thereby enhancing useful features and suppressing redundant information, and obtaining the feature F used for subsequent construction of the series cost body 2-5-1 、F 2-5-2 and F 2-5-3 ;

[0022] Step 5-2, feature F 2-3-6 、F 2-4-2 、F 2-4-4 More detailed information is extracted through a 3×3 convolution and a 1×1 convolution respectively to obtain the feature F 2-5-4 、F 2-5-5 、F 2-5-6 Used for the subsequent construction of group-related cost bodies.

[0023] Step 6: In order to provide more discriminative features for subsequent disparity refinement, the feature F 2-4-4 Perform a 3×3 and 1×1 ordinary convolution to obtain the refined left feature F for subsequent disparity refinement 2-6-1 and refine the right feature F 2-6-2 .

[0024] According to the cross-domain stereo matching method based on dynamic fusion of multi-scale information and correction of feature deviation, this method constructs a series cost volume and a group-related cost volume respectively, and combines the two to obtain a cost volume with complementary advantages, which includes the following steps:

[0025] Step 1: The obtained left and right features F 2-5-1 、F 2-5-2 、F 2-5-3 The series cost body is constructed separately to obtain a series cost body V with a height of 1 / 16H and a width of 1 / 16W. 3-1-1 , a series value body V with a height of 1 / 8H and a width of 1 / 8W 3-1-2 , a series value body V with a height of 1 / 4H and a width of 1 / 4W 3-1-3 The construction formula of the series cost body is as follows:

[0026]

[0027] Among them, concat represents the concatenation operation. and represents the left and right features of the corresponding scale, <,> represents the inner product operation, and d represents all disparity levels.

[0028] Step 2: The obtained left and right features F 2-5-4 、F 2-5-5 、F 2-5-6 The group-related cost volume V is constructed separately to obtain a group-related cost volume V with a height of 1 / 16H and a width of 1 / 16W. 3-2-1 , the group-related cost volume V is 1 / 8H high and 1 / 8W wide 3-2-2 , the group-related cost volume V is 1 / 4H high and 1 / 4W wide 3-2-3 The formula for constructing the group-related cost body is as follows:

[0029]

[0030] in, and Represents left and right features, N c Indicates the number of feature channels, N g represents the number of groups, <,> represents the inner product operation, and d represents all disparity levels;

[0031] Step 3: Connect the obtained series cost volume and the group-related cost volume in series to obtain a cost volume V with a height of 1 / 16H and a width of 1 / 16W. 3-3-1 , the cost body V is 1 / 8H high and 1 / 8W wide 3-3-2 , the cost body V is 1 / 4H high and 1 / 4W wide 3-3-3 . Its construction formula is as follows:

[0032] V 3-3-i =concat(V 3-1-i ,V 3-2-i )

[0033] Where concat represents a concatenation operation.

[0034] According to the cross-domain stereo matching method based on dynamic fusion of multi-scale information and feature deviation correction, this method aggregates cost volumes in a progressive layered manner. The results of the regularized small-resolution cost volume are dynamically fused into the next stage of cost volume aggregation to achieve efficient multi-scale cost volume fusion and aggregation. The method includes the following steps:

[0035] Step 1: In the 1 / 16 resolution cost volume aggregation stage, the obtained 1 / 16 resolution cost volume V 3-3-1 Regularization is performed continuously through two compact codecs. First, a downsampled 3D convolution with a stride of 2 and a 3×3 normal 3D convolution are performed to obtain a cost volume V with a height of 1 / 32H and a width of 1 / 32W. 4-1-1 Then, it passes through a three-dimensional transposed convolution function and a 3×3 ordinary three-dimensional convolution to the cost volume V 4-1-1 Regularization is performed to obtain a cost volume V with a height of 1 / 16H and a width of 1 / 16W 4-1-2 Then the same encoding and decoding regularization operation is performed again, and finally the cost volume V with a height of 1 / 16H and a width of 1 / 16W after two compact encoding and decoding regularizations is obtained. 4-1-2 This cost volume is used in subsequent stages to provide fine geometric information for higher-scale cost volumes. On the other hand, it is upsampled to the original image resolution and then subjected to disparity regression to obtain the initial disparity D1.

[0036] Step 2: In the 1 / 8 resolution cost volume aggregation stage, the obtained 1 / 8 resolution cost volume is subjected to the same operation as step 1. The difference is that when performing the first encoding and decoding downsampling, the downsampled cost volume V 3-3-2 and the regularized cost body V 4-1-2 Perform a series operation to better utilize the refined geometric features of the small-scale resolution cost volume, and finally obtain the cost volume V 4-2-1On the one hand, it is used to provide fine geometric information for higher-scale cost volumes in subsequent stages, and on the other hand, it is upsampled to the original image resolution and then performs disparity regression to obtain the initial disparity D2;

[0037] Step 3: In the 1 / 4 resolution cost volume aggregation stage, the obtained 1 / 4 resolution cost volume is subjected to two consecutive ordinary convolutions to obtain the cost volume V 4-3-1 , and the obtained cost volume V 4-2-1 Perform a three-dimensional transposed convolution function to generate a cost volume V with a height of 1 / 4H and a width of 1 / 4W 4-3-2 , and then the cost body V 4-3-2 and V obtained in step 2 4-2-1 The purpose of the concatenation operation is to integrate the low-resolution cost volume information into the higher-scale cost volume to retain local detail information while alleviating the noise amplification problem. Subsequently, two consecutive ordinary convolutions are performed to obtain the cost volume V 4-3-3 , and finally passes through two compact encoders and decoders to fully regularize. And it is worth noting that when the first compact encoder downsamples, it is again combined with the cost volume V 4-2-1 The concatenation is performed to enable high-level global features to guide low-level features from top to bottom to optimize the geometric reasoning ability. After four consecutive ordinary convolutions, the first compact codec, and the second compact codec, the regularized results are upsampled to the original resolution size, and then disparity regression is performed to obtain the initial disparities D3, D4, and D5.

[0038] In step 4, disparity predictions at each stage participate in the loss calculation, serving as intermediate supervisory signals to dynamically adjust the results of feature co-optimization at different resolutions during training. Guided by gradient backpropagation, networks at different levels can gradually focus on geometric representations at their corresponding scales, ensuring efficient utilization of multi-scale information and achieving progressive optimization of disparity predictions.

[0039] According to the cross-domain stereo matching method based on dynamic fusion of multi-scale information and correction of feature deviation, the feature consistency-based uncertainty estimation disparity refinement module is mainly divided into three components, which include the following steps:

[0040] Step 1: The first component is to construct the uncertainty estimation map guided by feature consistency. To quantify the reliability of disparity prediction, we first construct the uncertainty map U based on the alignment error of the left and right features. Specifically, based on the obtained initial disparity D5, we use differentiable bilinear interpolation to obtain the right feature map F. 2-5-8 Perform reverse deformation to obtain the deformed right feature map F aligned with the left view 5-1-1 Then the deformed right feature F is compared by cosine similarity. 5-1-1 And the obtained refined left feature F2-5-7 A similarity calculation is performed, followed by a negation operation, to obtain a preliminary uncertainty map U. A larger uncertainty weight indicates a higher feature alignment error and a lower reliability of the corresponding regional disparity prediction. To further enhance noise suppression, learnable parameters α and β are introduced to nonlinearly adjust the weight map. Finally, a sigmoid function is used to normalize the uncertainty values ​​to ensure a stable output range and good numerical adaptability.

[0041] Step 2: The second component is to fuse multi-source information. In this stage, the initial disparity D4, the left feature F 2-5-7 , original left image I, left feature F 2-5-7 The reconstruction error Σ is obtained by comparing the difference between the right feature and the deformed feature F 2-5-7 and the deformed right feature F 5-1-1 The warped volume W is obtained by constructing the correlation volume. These multi-source information are connected in series according to the channel dimension to obtain the fusion feature F 5-2-1 , to provide more comprehensive information for the subsequent disparity refinement process. 2-5-7 It is used to supplement the high-level semantic features of the image; the original left image I helps to preserve the original image information; the reconstruction error ∑ can represent the inconsistency between the left and right features, providing guidance for the uncertainty map U to accurately locate the area with large deviations; the warping volume W can effectively narrow the residual search range and reduce the computational complexity;

[0042] Step 3. In order to optimize the areas with higher uncertainty in a targeted manner, this method also designs a cascaded dilated convolution module (with dilation rates of 1, 2, 4, and 1, respectively) to gradually expand the receptive field and capture multi-scale context information to enhance the feature expression ability of these areas. Specifically, a 3×3 convolution is first applied to the fusion feature F5-2-1 to reduce the channel dimension, and then it is fed into the cascaded dilated convolution module to convert the fusion feature F5-2-1 with reduced channel number into a 3×3 convolution. 5-2-1 Element-wise multiplication is performed with the uncertainty map U to achieve weighted feature-guided feature enhancement. Next, the optimized features are concatenated with the initial disparity map and subjected to dilated convolutions to gradually correct the deviations in the initial disparity map. Through multiple iterative optimizations, the rectified feature map is further refined through 3×3 and 1×1 convolutional blocks to generate the final refined residual, which is then residually concatenated with the initial disparity D5 to obtain the optimized disparity D6.

[0043] According to the cross-domain stereo matching method based on dynamic fusion of multi-scale information and feature deviation correction, the final loss function is constructed using the obtained initial disparity and refined disparity D1, D2, D3, D4, D5 and D6 to train the network. The loss function is calculated as follows:

[0044]

[0045] Among them, Smooth L1 represents the smooth L1 loss; D gt represents the real parallax; w j is the weight of the j-th disparity map estimation, and the weights are set to 0.5, 0.5, 0.5, 0.7, 1.0, and 1.3 respectively.

[0046] Compared with the prior art, the present invention has the following beneficial effects: the present invention uses a public binocular stereo dataset for training to establish a stereo matching model with strong cross-domain generalization capabilities. Among them, the local-global feature collaborative fusion module adaptively fuses local and global features according to the scene, providing more general scene information for subsequent cost body construction. Secondly, the multi-scale cost body is progressively and hierarchically aggregated to efficiently fuse the fine-grained geometric features and global semantic information in the low-resolution cost body, thereby enhancing cross-scale geometric consistency. Finally, the present invention introduces a parallax refinement method for uncertainty estimation based on feature consistency, uses feature consistency to evaluate matching confidence, and specifically optimizes unstable matching areas, thereby effectively suppressing noise interference and improving the reliability of disparity prediction. Through the effective collaborative operation of each module, the cross-domain generalization capability is greatly improved while maintaining single-domain accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 Schematic diagram of the design framework of the cross-domain binocular stereo matching method based on dynamic fusion of multi-scale information and feature deviation correction of the present invention

[0048] Figure 2 Schematic diagram of the structure of the local-global feature collaborative fusion module in the present invention

[0049] Figure 3 This is a schematic diagram of the structure of the progressive layered cost volume aggregation module in the present invention.

[0050] Figure 4 Schematic diagram of the structure of the uncertainty estimation disparity refinement method based on feature consistency in the present invention

[0051] Figure 5 Comparison of the cross-domain generalization results of stereo matching between the present invention and related technologies on the Scene Flow to KITTI 2012 public dataset

[0052] Figure 6 Comparison of the cross-domain generalization results of stereo matching between the present invention and related technologies on the Scene Flow to KITTI 2015 public dataset

[0053] Figure 7The following is a comparison chart showing the performance of stereo matching algorithms in this aspect compared with those in related technologies on the KITTI 2015 public dataset. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solutions and advantages of the present invention more clearly described, the present invention is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below may be combined with each other as long as they do not conflict with each other.

[0055] The technical solution of the present invention is further described below with reference to the accompanying drawings.

[0056] like Figure 1 As shown, Figure 1 This is a schematic diagram of the framework of the neural network training of the present invention.

[0057] A cross-domain binocular stereo matching method based on dynamic fusion of multi-scale information and feature deviation correction is proposed. The steps are as follows:

[0058] Step 1: Collect binocular stereo matching datasets, including Scene Flow, KITTI 2012, and KITTI 2015 public datasets.

[0059] Step 2: Preprocess the image pairs for binocular stereo matching. Specifically, crop the images to 256×512, perform random brightness, contrast, and gamma adjustments, and use random rotation and random occlusion strategies. The height and width of the cropped images are H and W, respectively.

[0060] Step 3. After obtaining the preprocessed left and right image pairs through steps 1 and 2, the stereo matching model will sequentially execute the local-global feature collaborative fusion module to extract multi-scale features, construct the cost volume, perform progressive layered cost volume aggregation, perform disparity regression on the regularized cost volume to generate the initial disparity, and finally refine the disparity using uncertainty estimation based on feature consistency.

[0061] In order to make up for the deficiency of global context information extracted by local features, this method introduces a lightweight Transformer and an adaptive fusion method. The designed local-global feature collaborative fusion module is as follows: Figure 2 As shown in the figure, it consists of three components: a multi-scale local feature extraction module, a global feature extraction module, and an adaptive scene-aware feature fusion module to extract scene-adaptive features:

[0062] Step 1. The first component is the multi-scale local feature extraction module. Its input is the left and right view images, both of which are four-dimensional tensors of shape (B, C, H, W). The multi-scale local feature extraction module uses the pre-trained semantic segmentation network ResNet-101 to extract local features from the input left and right images respectively. The purpose is to introduce prior knowledge in other data scenarios in advance to provide more robust information for subsequent feature extraction. It outputs four corresponding four-dimensional tensors respectively, whose feature scales are 1 / 4, 1 / 8, and 1 / 16 of the feature scales of the left and right input images, respectively, and are denoted as F 2-1-1 , F 2-1-2 , F 2-1-3 ;

[0063] Step 2. The second component is the global feature extraction module, whose input is the left and right view images. The global feature extraction module first divides the left and right view images into blocks through a projection layer with a convolution kernel size of 4 and a stride of 4, generating an embedding vector sequence of 1 / 16 of the original image scale. It then passes it through a four-layer stacked self-attention structure, each layer containing a four-head attention mechanism. Finally, a context-rich serialized feature map F with only 1 / 16 of the original image scale is extracted. 2-2-1 ;

[0064] Step 3: The third component is the adaptive scene perception feature fusion module, whose inputs are the local features F extracted by the multi-scale feature extraction module. 2-1-3 and the global context feature F extracted by the global feature extraction module 2-2-1 . For the local feature F of the input 2-1-3 and global context feature F 2-2-1 The specific fusion operations include:

[0065] Step 3-1: First, serialize the feature graph F 2-2-1 Perform dimension permutation and reorganization operations to reconstruct it into a standard four-dimensional tensor. Secondly, a hierarchical feature conversion strategy is used to transform the local feature F 2-1-3 Perform refinement processing, that is, build a primary feature extraction module through a cascaded 3×3Conv-BN-ReLU structure, and finally connect a 1×1Conv-BN layer to form a composite feature encoder for feature F 2-3-1 to refine;

[0066] Step 3-2: Local feature F after feature alignment 2-3-1 and global context feature F 2-2-1 Splicing along the channel dimension to obtain the joint feature F 2-3-2 In order to further enhance the information interaction between features of different receptive fields, the 3×3Conv-BN-ReLU structure is used to combine the joint features F 2-3-2Perform feature extraction and further perform channel compression using 1×1 Conv-BN to reduce the amount of subsequent calculations;

[0067] Step 3-3: In order to enrich the statistical information of the features, the feature F after feature union is 2-3-2 Apply global maximum pooling and global average pooling respectively to extract feature representations under different perspectives, which are denoted as F 2-3-3 and F 2-3-4 In view of the complementarity between the two, they are spliced ​​along the channel dimension to form a mixed statistical feature F 2-3-5 ,This feature can adaptively capture more discriminative information in different scenarios,,providing a more solid foundation for subsequent dynamic weight allocation;

[0068] Step 3-4: For the obtained mixed statistical feature F 2-3-5 , this method uses a 1×1Conv-BN layer to reduce the channel dimension, and then generates a dynamic weight coefficient W with probabilistic interpretation through the Sigmoid function 2-3-1 and W 2-3-2 , and finally the feature weight W 2-3-1 and W 2-3-2 Multiply the original local features F 2-1-3 and global context feature F 2-2-1 Perform dynamic weighting and then add the two weighted features to obtain the final adaptive fusion feature F 2-3-6 .

[0069] Step 4: The adaptive fusion feature F with a height of 1 / 16H and a width of 1 / 16W is obtained 2-3-6 Through a two-dimensional transposed convolution module, we get an upsampled feature F with a height of 1 / 8H and a width of 1 / 8W. 2-4-1 . Then it is combined with the obtained low-level features F of the same scale 2-1-2 The concatenation operation is performed along the channel dimension to obtain high-level semantic information while fully preserving the underlying geometric details, and finally obtain a feature F with a height of 1 / 8H and a width of 1 / 8W 2-4-2 Then, the feature F 2-4-2 It is also put into the two-dimensional transposed convolution module to obtain the upsampled feature F with a height of 1 / 4H and a width of 1 / 4W 2-4-3 , and then compare it with the obtained low-level feature F of the same scale 2-1-1 The concatenation operation is performed along the channel dimension, and finally the feature F with a feature height of 1 / 4H and a width of 1 / 4W is obtained. 2-4-4 ;

[0070] Step 5: The obtained feature F is only 1 / 16, 1 / 8 and 1 / 4 of the original image resolution 2-3-6 、F 2-4-2、F 2-4-4 Further feature extraction operations are performed separately to lay the foundation for the subsequent construction of the series cost volume and the group-related cost volume. The further operations are as follows:

[0071] Step 5-1, first of all, the feature F 2-3-6 、F 2-4-2 、F 2-4-4 Perform a 3×3 convolution operation respectively, and then use the channel and spatial attention modules to weight the features used to construct the series cost body, thereby enhancing useful features and suppressing redundant information, and obtaining the feature F used for subsequent construction of the series cost body 2-5-1 、F 2-5-2 and F 2-5-3 ;

[0072] Step 5-2: Feature F 2-3-6 、F 2-4-2 、F 2-4-4 More detailed information is extracted through a 3×3 convolution and a 1×1 convolution respectively to obtain the feature F 2-5-4 、F 2-5-5 、F 2-5-6 Used for the subsequent construction of group-related cost bodies.

[0073] Step 6: In order to provide more discriminative features for subsequent disparity refinement, the feature F 2-4-4 Perform a 3×3 and 1×1 ordinary convolution to obtain the refined left feature F for subsequent disparity refinement 2-6-1 and refine the right feature F 2-6-2 .

[0074] According to the cross-domain stereo matching method based on dynamic fusion of multi-scale information and feature deviation correction, this method constructs a series cost volume and a group-related cost volume respectively, and combines the two to obtain a cost volume with complementary advantages:

[0075] Step 1: The obtained left and right features F 2-5-1 、F 2-5-2 、F 2-5-3 The series cost body is constructed separately to obtain a series cost body V with a height of 1 / 16H and a width of 1 / 16W. 3-1-1 , a series value body V with a height of 1 / 8H and a width of 1 / 8W 3-1-2 , a series value body V with a height of 1 / 4H and a width of 1 / 4W 3-1-3 The construction formula of the series cost body is as follows:

[0076]

[0077] Among them, concat represents the concatenation operation. and represents the left and right features of the corresponding scale, <,> represents the inner product operation, and d represents all disparity levels.

[0078] Step 2: The obtained left and right features F 2-5-4 、F 2-5-5 、F 2-5-6 The group-related cost volume V is constructed separately to obtain a group-related cost volume V with a height of 1 / 16H and a width of 1 / 16W. 3-2-1 , the group-related cost volume V is 1 / 8H high and 1 / 8W wide 3-2-2 , the group-related cost volume V is 1 / 4H high and 1 / 4W wide 3-2-3 The formula for constructing the group-related cost body is as follows:

[0079]

[0080] in, and Represents left and right features, N c Indicates the number of feature channels, N g represents the number of groups, <,> represents the inner product operation, and d represents all disparity levels.

[0081] Step 3: Connect the obtained series cost volume and the group-related cost volume in series to obtain a cost volume V with a height of 1 / 16H and a width of 1 / 16W. 3-3-1 , the cost body V is 1 / 8H high and 1 / 8W wide 3-3-2 , the cost body V is 1 / 4H high and 1 / 4W wide 3-3-3 . Its construction formula is as follows:

[0082] V 3-3-i =concat(V 3-1-i ,V 3-2-i )

[0083] Where concat represents a concatenation operation.

[0084] After the multi-scale cost volume is constructed, this method will fuse and aggregate the multi-scale cost volume through the designed progressive hierarchical aggregation method. The proposed progressive hierarchical cost volume aggregation module is as follows: Figure 3 As shown in the figure, the cost volume of each scale is regularized, and the results of each scale regularization are dynamically integrated into the cost volume aggregation process of the next stage to achieve efficient multi-scale cost volume fusion and aggregation:

[0085] Step 1: In the 1 / 16 resolution cost volume aggregation stage, the obtained 1 / 16 resolution cost volume V 3-3-1Regularization is performed continuously through two compact codecs. First, a downsampled 3D convolution with a stride of 2 and a 3×3 normal 3D convolution are performed to obtain a cost volume V with a height of 1 / 32H and a width of 1 / 32W. 4-1-1 Then, it passes through a three-dimensional transposed convolution function and a 3×3 ordinary three-dimensional convolution to the cost volume V 4-1-1 Regularization is performed to obtain a cost volume V with a height of 1 / 16H and a width of 1 / 16W 4-1-2 Then the same encoding and decoding regularization operation is performed again, and finally the cost volume V with a height of 1 / 16H and a width of 1 / 16W after two compact encoding and decoding regularizations is obtained. 4-1-2 This cost volume is used in subsequent stages to provide fine geometric information for higher-scale cost volumes. On the other hand, it is upsampled to the original image resolution and then subjected to disparity regression to obtain the initial disparity D1.

[0086] Step 2: In the 1 / 8 resolution cost volume aggregation stage, the obtained 1 / 8 resolution cost volume is subjected to the same operation as step 1. The difference is that when performing the first encoding and decoding downsampling, the downsampled cost volume V 3-3-2 and the regularized cost body V 4-1-2 Perform a series operation to better utilize the refined geometric features of the small-scale resolution cost volume, and finally obtain the cost volume V 4-2-1 On the one hand, it is used to provide fine geometric information for higher-scale cost volumes in subsequent stages, and on the other hand, it is upsampled to the original image resolution and then performs disparity regression to obtain the initial disparity D2;

[0087] Step 3: In the 1 / 4 resolution cost volume aggregation stage, the obtained 1 / 4 resolution cost volume is subjected to two consecutive ordinary convolutions to obtain the cost volume V 4-3-1 , and the obtained cost volume V 4-2-1 Perform a three-dimensional transposed convolution function to generate a cost volume V with a height of 1 / 4H and a width of 1 / 4W 4-3-2 , and then the cost body V 4-3-2 and V obtained in step 2 4-2-1 The purpose of the concatenation operation is to integrate the low-resolution cost volume information into the higher-scale cost volume to retain local detail information while alleviating the noise amplification problem. Subsequently, two consecutive ordinary convolutions are performed to obtain the cost volume V 4-3-3 , and finally passes through two compact encoders and decoders to fully regularize. And it is worth noting that when the first compact encoder downsamples, it is again combined with the cost volume V 4-2-1The concatenation is performed to enable high-level global features to guide low-level features from top to bottom to optimize the geometric reasoning ability. After four consecutive ordinary convolutions, the first compact codec, and the second compact codec, the regularized results are upsampled to the original resolution size, and then disparity regression is performed to obtain the initial disparities D3, D4, and D5.

[0088] In step 4, disparity predictions at each stage participate in the loss calculation, serving as intermediate supervisory signals to dynamically adjust the results of feature co-optimization at different resolutions during training. Guided by gradient backpropagation, networks at different levels can gradually focus on geometric representations at their corresponding scales, ensuring efficient utilization of multi-scale information and achieving progressive optimization of disparity predictions.

[0089] After the initial disparity prediction result is obtained by multi-scale cost volume aggregation, the rough disparity map will be refined. The refinement process is as follows: Figure 4 As shown in Figure 2, the feature consistency-based uncertainty estimation disparity refinement module is mainly divided into three components:

[0090] Step 1: The first component is to construct the uncertainty estimation map guided by feature consistency. To quantify the reliability of disparity prediction, we first construct the uncertainty map U based on the alignment error of the left and right features. Specifically, based on the obtained initial disparity D5, we use differentiable bilinear interpolation to obtain the right feature map F. 2-5-8 Perform reverse deformation to obtain the deformed right feature map F aligned with the left view 5-1-1 Then the deformed right feature F is compared by cosine similarity. 5-1-1 And the obtained refined left feature F 2-5-7 A similarity calculation is performed, followed by a negation operation, to obtain a preliminary uncertainty map U. A larger uncertainty weight indicates a higher feature alignment error and a lower reliability of the corresponding regional disparity prediction. To further enhance noise suppression, learnable parameters α and β are introduced to nonlinearly adjust the weight map. Finally, a sigmoid function is used to normalize the uncertainty values ​​to ensure a stable output range and good numerical adaptability.

[0091] Step 2: The second component is to fuse multi-source information. In this stage, the initial disparity D4, the left feature F 2-5-7 , original left image I, left feature F 2-5-7 The reconstruction error Σ is obtained by comparing the difference between the right feature and the deformed feature F 2-5-7 and the deformed right feature F 5-1-1 The warped volume W is obtained by constructing the correlation volume. These multi-source information are connected in series according to the channel dimension to obtain the fusion feature F 5-2-1 , to provide more comprehensive information for the subsequent disparity refinement process.2-5-7 It is used to supplement the high-level semantic features of the image; the original left image I helps to preserve the original image information; the reconstruction error ∑ can represent the inconsistency between the left and right features, providing guidance for the uncertainty map U to accurately locate the area with large deviations; the warping volume W can effectively narrow the residual search range and reduce the computational complexity;

[0092] Step 3. In order to optimize the areas with higher uncertainty in a targeted manner, this method also designs a cascaded dilated convolution module (with dilation rates of 1, 2, 4, and 1, respectively) to gradually expand the receptive field and capture multi-scale context information to enhance the feature expression ability of these areas. Specifically, a 3×3 convolution is first applied to the fusion feature F5-2-1 to reduce the channel dimension, and then it is fed into the cascaded dilated convolution module to convert the fusion feature F5-2-1 with reduced channel number into a 3×3 convolution. 5-2-1 Element-wise multiplication is performed with the uncertainty map U to achieve weighted feature-guided feature enhancement. Next, the optimized features are concatenated with the initial disparity map and subjected to dilated convolutions to gradually correct the deviations in the initial disparity map. Through multiple iterative optimizations, the rectified feature map is further refined through 3×3 and 1×1 convolutional blocks to generate the final refined residual, which is then residually concatenated with the initial disparity D5 to obtain the optimized disparity D6.

[0093] According to the cross-domain stereo matching method based on dynamic fusion of multi-scale information and feature deviation correction, the final loss function is constructed using the obtained initial disparity and refined disparity D1, D2, D3, D4, D5 and D6 to train the network. The loss function is calculated as follows:

[0094]

[0095] Among them, Smooth L1 represents the smooth L1 loss; D gt represents the real parallax; w j is the weight of the j-th disparity map estimation, and the weights are set to 0.5, 0.5, 0.5, 0.7, 1.0, and 1.3 respectively.

[0096] In order to illustrate that the method of the present invention can achieve good accuracy in both non-cross-domain scenarios and cross-domain scenarios, tests were conducted on multiple data sets. In the cross-domain scenario, the test results obtained by the present invention were compared with the currently commonly used methods, and various state-of-the-art methods such as GwcNet, PCWNet, CFNet, Guard-Net, etc. The comparison results are shown in Table A and Table B. The present invention achieved the best results in the cross-domain test from synthetic data sets to real data sets. The comparison chart is as follows Figure 5 and Figure 6In addition, the performance of the present invention is compared with various state-of-the-art methods in non-cross-domain scenarios. The comparison results are shown in Table C. The results show that the present method can maintain excellent single-domain accuracy while maintaining good cross-domain accuracy. Figure 7 The experimental results show that this method has a good balance between cross-domain and non-cross-domain.

[0097] The above embodiments are intended to help understand the method and core concept of the present invention. It is worth emphasizing that, for those skilled in the art, based on the principles of the present invention, various improvements and adjustments can be made, and these improvements and adjustments should also be included in the scope of protection of the claims of the present invention.

[0098] The above description of the disclosed embodiments enables one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be applied to other embodiments without departing from the spirit or scope of the present invention. Therefore, the scope of protection of the present invention is not limited to the embodiments shown herein, but rather encompasses the broadest scope consistent with the principles and innovative features disclosed herein.

[0099] Table A Cross-domain performance comparison from the synthetic dataset Scene Flow to the real dataset KITTI 2012

[0100]

[0101] Table B Cross-domain performance comparison from the synthetic dataset Scene Flow to the real dataset KITTI 2015

[0102]

[0103] Table C Performance comparison on Scene Flow dataset

[0104]

Claims

1. A cross-domain binocular stereo matching method based on dynamic fusion of multi-scale information and feature deviation correction, comprising the steps of: Step 1: The stereo matching model will sequentially execute the local-global feature collaborative fusion module to extract multi-scale features, construct a cost volume, perform progressive layered cost volume aggregation, perform disparity regression on the regularized cost volume to generate initial disparity, and finally refine the disparity using uncertainty estimation based on feature consistency.

2. The cross-domain stereo matching method based on dynamic fusion of multi-scale information and correction of feature deviation according to claim 1, characterized in that: The local-global feature collaborative fusion module consists of three submodules, and its processing flow includes the following steps: Step 1: Construct a multi-scale local feature extraction submodule. This submodule takes the left and right view images as input, and the input image shape is a four-dimensional tensor of (B, C, H, W). The multi-scale local feature extraction module uses the pre-trained semantic segmentation network ResNet-101 to extract local features from the input left and right images respectively, and outputs the corresponding four four-dimensional tensors respectively, whose feature scales correspond to 1 / 4, 1 / 8, and 1 / 16 of the feature scales of the left and right input images respectively, which are denoted as F 2-1-1 , F 2-1-2 , F 2-1-3 ; Step 2: Construct a global feature extraction submodule. This submodule also takes the left and right view images as input. First, the left and right view images are divided into blocks through a projection layer with a convolution kernel size of 4 and a stride of 4 to generate an embedding vector sequence with a scale of 1 / 16 of the original image. Then, this embedding vector sequence is passed through a four-layer stacked self-attention structure, with each layer containing a four-head attention mechanism. Finally, a context-rich sequential feature map F with a scale of only 1 / 16 of the original image is extracted. 2-2-1 ; Step 3: Construct an adaptive scene perception feature fusion submodule. This submodule uses the local feature F obtained in step 1 2-1-3 and the global context feature F extracted in step 2 2-2-1 As input, feature fusion operation is performed. The specific fusion operation includes the following steps: Step 3-1: First, serialize the feature graph F 2-2-1 Perform dimension permutation and reorganization operations to reconstruct it into a standard four-dimensional tensor. Secondly, a hierarchical feature conversion strategy is used to transform the local feature F 2-1-3 Perform refinement processing, that is, build a primary feature extraction module through a cascaded 3×3Conv-BN-ReLU structure, and finally connect a 1×1Conv-BN layer to form a composite feature encoder for feature F 2-3-1 to refine; Step 3-2: Align the local features F after step 3-1 features 2-3-1 and global context feature F 2-2-1 Splicing along the channel dimension to obtain the joint feature F 2-3-2 In order to further enhance the information interaction between features of different receptive fields, the 3×3Conv-BN-ReLU structure is used to combine the joint features F 2-3-2 Perform feature extraction and further perform channel compression using 1×1 Conv-BN to reduce the amount of subsequent calculations; Step 3-3: In order to enrich the statistical information of the features, the feature F after the feature combination in step 3-2 is 2-3-2 Apply global maximum pooling and global average pooling respectively to extract feature representations under different perspectives, which are denoted as F 2-3-3 and F 2-3-4 In view of the complementarity between the two, they are spliced ​​along the channel dimension to form a mixed statistical feature F 2-3-5 ,This feature can adaptively capture more discriminative information in different scenarios,,providing a more solid foundation for subsequent dynamic weight allocation; Step 3-4: For the mixed statistical feature F obtained in step 3-3 2-3-5 , use the 1×1Conv-BN layer to reduce the channel dimension, and then generate a dynamic weight coefficient W with probabilistic interpretation through the Sigmoid function 2-3-1 and W 2-3-2 , and finally the feature weight W 2-3-1 and W 2-3-2 Multiply by the original local features F 2-1-3 and global context feature F 2-2-1 Perform dynamic weighting and then add the two weighted features to obtain the final adaptive fusion feature F 2-3-6 . Step 4: The adaptive fusion feature F with a height of 1 / 16H and a width of 1 / 16W obtained in step 3 is 2-3-6 Through a two-dimensional transposed convolution module, we get an upsampled feature F with a height of 1 / 8H and a width of 1 / 8W. 2-4-1 Then combine it with the low-level feature F of the same scale obtained in step 1 2-1-2 The concatenation operation is performed along the channel dimension to obtain high-level semantic information while fully preserving the underlying geometric details, and finally obtain a feature F with a height of 1 / 8H and a width of 1 / 8W 2-4-2 Then, the feature F 2-4-2 It is also put into the two-dimensional transposed convolution module to obtain the upsampled feature F with a height of 1 / 4H and a width of 1 / 4W 2-4-3 , and then compare it with the low-level feature F of the same scale obtained in step 1 2-1-1 The concatenation operation is performed along the channel dimension, and finally the feature F with a feature height of 1 / 4H and a width of 1 / 4W is obtained. 2-4-4 ; Step 5: The feature F obtained in steps 3 and 4 is only 1 / 16, 1 / 8 and 1 / 4 of the original image resolution. 2-3-6 、F 2-4-2 、F 2-4-4 Further feature extraction operations are performed separately to lay the foundation for the subsequent construction of the series cost volume and the group-related cost volume, which includes the following steps: Step 5-1, first of all, the feature F 2-3-6 、F 2-4-2 、F 2-4-4 Perform a 3×3 convolution operation respectively, and then use the channel and spatial attention modules to weight the features used to construct the series cost body, thereby enhancing useful features and suppressing redundant information, and obtaining the feature F used for subsequent construction of the series cost body 2-5-1 、F 2-5-2 and F 2-5-3 ; Step 5-2, feature F 2-3-6 、F 2-4-2 、F 2-4-4 More detailed information is extracted through a 3×3 convolution and a 1×1 convolution respectively to obtain the feature F 2-5-4 、F 2-5-5 、F 2-5-6 Used for the subsequent construction of group-related cost bodies. Step 6: In order to provide more discriminative features for subsequent disparity refinement, the feature F 2-4-4 Perform a 3×3 and 1×1 ordinary convolution to obtain the refined left feature F for subsequent disparity refinement 2-6-1 and refine the right feature F 2-6-2 .

3. The cross-domain stereo matching method based on dynamic fusion of multi-scale information and correction of feature deviation according to claim 1 or 2, characterized in that: Constructing a serial cost body and a group-related cost body separately, and combining the two to obtain a cost body with complementary advantages, which includes the following steps: Step 1: The left and right features F obtained in step 5 of claim 2 2-5-1 、F 2-5-2 、F 2-5-3 The series cost body is constructed separately to obtain a series cost body V with a height of 1 / 16H and a width of 1 / 16W. 3-1-1 , a series value body V with a height of 1 / 8H and a width of 1 / 8W 3-1-2 , a series value body V with a height of 1 / 4H and a width of 1 / 4W 3-1-3 ; Step 2: The left and right features F obtained in step 5 of claim 2 2-5-4 、F 2-5-5 、F 2-5-6 The group-related cost volume V is constructed separately to obtain a group-related cost volume V with a height of 1 / 16H and a width of 1 / 16W. 3-2-1 , the group-related cost volume V is 1 / 8H high and 1 / 8W wide 3-2-2 , the group-related cost volume V is 1 / 4H high and 1 / 4W wide 3-2-3 ; Step 3: Connect the series cost volume obtained in step 1 and step 2 and the group-related cost volume to obtain a cost volume V with a height of 1 / 16H and a width of 1 / 16W. 3-3-1 , the cost body V is 1 / 8H high and 1 / 8W wide 3-3-2 , the cost body V is 1 / 4H high and 1 / 4W wide 3-3-3 .

4. The cross-domain stereo matching method based on dynamic fusion of multi-scale information and correction of feature deviation according to any one of claims 1 to 3, characterized in that: Cost volumes are aggregated in a progressive layered manner. The results of the regularized small-resolution cost volume are dynamically fused into the cost volume aggregation process of the next stage to achieve efficient multi-scale cost volume fusion and aggregation, which includes the following steps: Step 1: In the 1 / 16 resolution cost volume aggregation stage, the 1 / 16 resolution cost volume V obtained in step 3 of claim 3 is 3-3-1 Regularization is performed continuously through two compact codecs. First, a downsampled 3D convolution with a stride of 2 and a 3×3 normal 3D convolution are performed to obtain a cost volume V with a height of 1 / 32H and a width of 1 / 32W. 4-1-1 Then, it passes through a three-dimensional transposed convolution function and a 3×3 ordinary three-dimensional convolution to the cost volume V 4-1-1 Regularization is performed to obtain a cost volume V with a height of 1 / 16H and a width of 1 / 16W 4-1-2 Then the same encoding and decoding regularization operation is performed again, and finally the cost volume V with a height of 1 / 16H and a width of 1 / 16W after two compact encoding and decoding regularizations is obtained. 4-1-2 This cost volume is used in subsequent stages to provide fine geometric information for higher-scale cost volumes. On the other hand, it is upsampled to the original image resolution and then subjected to disparity regression to obtain the initial disparity D1. Step 2: At the 1 / 8 resolution cost volume aggregation stage, the same operation as step 1 is performed on the 1 / 8 resolution cost volume obtained in step 3 of claim 3. The difference is that when performing the first encoding and decoding downsampling, the downsampled cost volume V 3-3-2 and the regularized cost body V 4-1-2 Perform a series operation to better utilize the refined geometric features of the small-scale resolution cost volume, and finally obtain the cost volume V 4-2-1 On the one hand, it is used to provide fine geometric information for higher-scale cost volumes in subsequent stages, and on the other hand, it obtains the initial disparity D2 by upsampling to the original image resolution and then performing disparity regression. Step 3: In the 1 / 4 resolution cost volume aggregation stage, the 1 / 4 resolution cost volume obtained in step 3 of claim 3 is subjected to two consecutive ordinary convolutions to obtain the cost volume V 4-3-1 , and the cost volume V obtained in step 2 4-2-1 Perform a three-dimensional transposed convolution function to generate a cost volume V with a height of 1 / 4H and a width of 1 / 4W 4-3-2 , then perform a series operation on V4-3-2 and V4-2-1 obtained in step 2 to integrate the low-resolution cost volume information into the higher-scale cost volume to retain local detail information while alleviating the noise amplification problem. Subsequently, perform two consecutive ordinary convolutions to obtain the cost volume V 4-3-3 , and finally passes through two compact codecs for full regularization. It is also worth noting that when the first compact encoder is downsampled, it is again connected in series with the cost body V4-2-1 to achieve the top-down guidance of low-level features by high-level global features to optimize the geometric reasoning ability. After four consecutive ordinary convolutions, the first compact codec, and the second compact codec, the regularized results are upsampled to the original resolution size, and then disparity regression is performed to obtain the initial disparities D3, D4, and D5. In step 4, disparity predictions at each stage participate in the loss calculation, serving as intermediate supervisory signals to dynamically adjust the results of feature co-optimization at different resolutions during training. Guided by gradient backpropagation, networks at different levels can gradually focus on geometric representations at their corresponding scales, ensuring efficient utilization of multi-scale information and achieving progressive optimization of disparity predictions.

5. The cross-domain stereo matching method based on dynamic fusion of multi-scale information and correction of feature deviation according to any one of claims 1 to 4, characterized in that: The feature consistency-based uncertainty estimation disparity refinement module is mainly divided into three components, which includes the following steps: Step 1: The first component is to construct an uncertainty estimation map guided by feature consistency. To quantify the reliability of disparity prediction, we first construct an uncertainty map M based on the alignment error of the left and right features. Specifically, based on the initial disparity D5 obtained in claim 4, we use differentiable bilinear interpolation to interpolate the right feature map F obtained in claim 2. 2-5-8 Perform reverse deformation to obtain the deformed right feature map F aligned with the left view 5-1-1 Then the deformed right feature F is compared by cosine similarity. 5-1-1 and the refined left feature F obtained in claim 2 2-5-7 Perform similarity calculations and then perform inversion operations to obtain a preliminary uncertainty map U. A larger uncertainty weight indicates a higher feature alignment error and a lower reliability of the corresponding regional disparity prediction. To further enhance noise suppression capabilities, learnable parameters α and β are introduced to perform nonlinear adjustments to the weight map. Step 2: The second component is to fuse multi-source information. In this stage, the initial disparity D4, the left feature F 2-5-7 , original left image I, left feature F 2-5-7 The reconstruction error Σ is obtained by comparing the difference between the right feature and the deformed feature F 2-5-7 and the deformed right feature F 5-1-1 The warped volume W is obtained by constructing the correlation volume. These multi-source information are connected in series according to the channel dimension to obtain the fusion feature F 5-2-1 , to provide more comprehensive information for the subsequent disparity refinement process. 2-5-7 It is used to supplement the high-level semantic features of the image; the original left image I helps to preserve the original image information; the reconstruction error ∑ can represent the inconsistency between the left and right features, providing guidance for the uncertainty map U to accurately locate the area with large deviations; the warping volume W can effectively narrow the residual search range and reduce the computational complexity; Step 3. In order to optimize the areas with higher uncertainty in a targeted manner, this method also designs a cascaded dilated convolution module (with dilation rates of 1, 2, 4, and 1, respectively) to gradually expand the receptive field and capture multi-scale context information to enhance the feature expression ability of these areas. Specifically, a 3×3 convolution is first applied to the fusion feature F5-2-1 to reduce the channel dimension, and then it is fed into the cascaded dilated convolution module to convert the fusion feature F5-2-1 with reduced channel number into a 3×3 convolution. 5-2-1 Element-wise multiplication is performed with the uncertainty map U to achieve weighted feature-guided feature enhancement. Next, the optimized features are concatenated with the initial disparity map and subjected to dilated convolutions to gradually correct the deviations in the initial disparity map. Through multiple iterative optimizations, the rectified feature map is further refined through 3×3 and 1×1 convolutional blocks to generate the final refined residual, which is then residually concatenated with the initial disparity D5 to obtain the optimized disparity D6.

6. The cross-domain stereo matching method based on dynamic fusion of multi-scale information and correction of feature deviation according to claims 1 to 5, characterized in that: The final loss function is constructed using the initial disparity and refined disparity D1, D2, D3, D4, D5 and D6 obtained in claims 4 and 5 to train the network. The loss function is calculated as follows: Among them, Smooth L1 represents the smooth L1 loss; D gt represents the real parallax; w j is the weight of the j-th disparity map estimation, and the weights are set to 0.5, 0.5, 0.5, 0.7, 1.0, and 1.3 respectively.

Images