A stereo matching network fusing domain generalization and feature similarity

Abstract

The application relates to the technical field of stereo matching, in particular to a stereo matching network fusing domain generalization and feature similarity. The steps are as follows: S1: using a domain generalization method to improve a feature extraction module of the stereo matching network; S2: introducing a feature similarity module to measure the visual similarity of left and right image features as consistency information input into the network; and S3: using a deformable attention mechanism in a cost aggregation module. The stereo matching network fusing domain generalization and feature similarity provided by the application improves the feature robustness by introducing a domain generalization method into the feature extraction module, designs a feature similarity module to comprehensively match the visual and semantic information of images to reduce pixel errors, and uses a deformable attention module to enhance the processing of complex regions, so that the prediction accuracy of the network for weak texture and complex edge regions is effectively improved. The network is realized based on 2D convolution, and is more efficient than 3D convolution.

Description

Technical Field

[0001] This invention relates to the field of stereo matching technology, and in particular to a stereo matching network that integrates domain generalization and feature similarity. Background Technology

[0002] Stereo matching is a crucial step in binocular vision ranging, typically involving searching corresponding pixels in paired images captured simultaneously by two cameras and then calculating their disparity. In recent years, deep learning-based stereo matching methods have been increasingly applied to stereo matching tasks due to their lower error rate and higher robustness. Deep learning-based stereo matching methods mainly include four important modules: feature extraction, cost volume computation, matching cost aggregation, and disparity optimization.

[0003] The first end-to-end stereo matching network performs cost aggregation on the left and right images using 2D convolution, and then calculates disparity.

[0004] Researchers have combined left and right feature maps to construct a cost quantity and added a 3D convolutional layer to the cost aggregation module, which effectively reduced the pixel error rate. However, because the feature extraction module is only linearly stacked, the receptive field of the extracted features is greatly limited. The pyramid stereo matching network adds a spatial pyramid pooling module to optimize the feature extraction module for handling textureless regions. It also improves the performance of the cost aggregation module by using multiple hourglass modules with a structure similar to 3D-UNet.

[0005] GA-Net, based on the semi-global matching concept in traditional stereo matching methods, proposes an improved cost aggregation module using semi-global guided aggregation and local guided aggregation structures, significantly reducing pixel errors in disparity maps. The Group-Related Stereo Network redesigns its cost module, proposing a method combining group joint cost and concatenated cost, utilizing adjacent contextual feature vectors to enable the network to understand richer semantic information and reduce the pixel error rate. While these networks all reduce the network error rate, their feature extraction modules are not robust to weakly textured regions or other complex situations, resulting in low feature extraction accuracy. Furthermore, the extensive use of stacked 3D convolutions leads to speed issues.

[0006] AANet proposes an adaptive cost aggregation module that can be used both within the same scale and across different scales, significantly improving inference speed while sacrificing a small amount of accuracy. The Hierarchical Neural Architecture Search (NAS) deep stereo matching network uses Neural Architecture Search (NAS) to search for parameters and structures in the search space for both the feature extraction module and the matching network, resulting in a substantial reduction in error rate; however, the network requires significant computational resources.

[0007] CFNet proposes a fusion cost method and a cascaded cost method to optimize for disparity imbalance in different datasets, greatly enhancing the robustness of the network.

[0008] Most of the aforementioned stereo matching networks focus on improving the cost aggregation module, while giving less consideration to the importance of the feature extraction module.

[0009] To address this issue, we designed a stereo matching network that integrates domain generalization and feature similarity to provide an alternative technical solution. Summary of the Invention

[0010] Therefore, it is necessary to provide a stereo matching network that integrates domain generalization and feature similarity to address the aforementioned technical problems.

[0011] To solve the above-mentioned technical problems, the present invention adopts the following technical solution:

[0012] A stereo matching network that integrates domain generalization and feature similarity, comprising the following steps:

[0013] S1: Improve the feature extraction module of the stereo matching network using the domain generalization method;

[0014] S2: Introduce a feature similarity module to measure the visual similarity of features between the left and right images as consistency information input into the network;

[0015] S3: Use deformable attention mechanisms in the cost aggregation module.

[0016] As a preferred embodiment of the stereo matching network that integrates domain generalization and feature similarity provided by the present invention, in step S1, a ResNet-50 model pre-trained on the ImageNet dataset is used as the basis for the feature extraction module.

[0017] The shallow feature extraction part of ResNet-50 is combined with U-Net to fuse the generalization features of the pre-trained model.

[0018] Replace part of Batch Norm with Instance Norm.

[0019] As a preferred embodiment of the stereo matching network that integrates domain generalization and feature similarity provided by the present invention, in step S2, a feature similarity module is introduced to measure the visual similarity of features in the left and right images as consistency information input into the network. The steps are as follows:

[0020] Set the left and right camera image pixels and neighbor set to ;

[0021] Where P represents the left camera feature image pixel p and its neighboring point set, and Q represents the right camera feature image pixel q in the same row and its neighboring point set;

[0022]

[0023] in, The similarity threshold is set to a default value of 0.5, meaning that when the similarity between two pixels reaches 0.5, the two points are considered the best similarity pair.

[0024] As a preferred embodiment of the stereo matching network that integrates domain generalization and feature similarity provided by the present invention, the degree of visual similarity between pixel sets P and Q is expressed as follows:

[0025]

[0026] in, As the visual similarity weights are calculated, the visual similarity of features decreases when the distribution difference between P and Q increases, and increases when the distributions are more similar.

[0027] As a preferred embodiment of the stereo matching network that integrates domain generalization and feature similarity provided by the present invention, semantic similarity is defined as the similarity of features in two images after linear transformation. The expression for the degree of semantic similarity is as follows:

[0028]

[0029] Where P and Q are the sets of neighboring pixels, For semantic similarity weights, Let F be the scaling function, p and q be the pixels in the pixel set, F be a linear function, and M and N be the number of pixels in the pixel set.

[0030] As a preferred embodiment of the stereo matching network that integrates domain generalization and feature similarity provided by the present invention, in step S3, the deformable attention mechanism is an attention mechanism added to the deformable convolution.

[0031] As a preferred embodiment of the stereo matching network that integrates domain generalization and feature similarity provided by the present invention, the deformable convolution expression is as follows:

[0032]

[0033] in, This represents the cost of aggregating pixel p within the disparity search space d. This represents the total number of sampling points, where k represents the k-th sampling point. This represents the aggregate weight of the k-th sampling point. This represents the fixed offset of the k-th sampling point during the cost aggregation process. Indicates the adjustment used in the cost aggregation process. Learnable additional offset at the sampling location This represents the improved adjustment coefficient. A fixed amount to control the basic range of the adjustment coefficient.

[0034] As a preferred embodiment of the stereo matching network that integrates domain generalization and feature similarity provided by the present invention, an attention mechanism is added to deformable convolution.

[0035] Define the input cost as The definition of deformable attention is as follows:

[0036]

[0037] in, This represents a deformable attention feature, where 1 indicates the first layer of deformable attention. This represents the deformable attention weight matrix of the first layer. This represents the weight matrix of the deformable attention layer at the k-th sampling point. This represents the learnable additional offset of the attention mechanism.

[0038] It is clear without a doubt that the technical solution described above in this application can solve the technical problem that this application aims to address.

[0039] Meanwhile, through the above technical solutions, the present invention has at least the following beneficial effects:

[0040] This invention provides a stereo matching network that integrates domain generalization and feature similarity. By introducing a domain generalization method into the feature extraction module to improve feature robustness, designing a feature similarity module to integrate visual and semantic information of the image for matching to reduce pixel errors, and using a deformable attention module to enhance the processing of complex regions, the network effectively improves the prediction accuracy of the network for weak texture and complex edge regions. This network is based on 2D convolution, which is more efficient than 3D convolution. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 This is a schematic diagram of the feature extraction module of the present invention;

[0043] Figure 2 This is a schematic diagram of the characteristic visual similarity structure of the present invention;

[0044] Figure 3 This is a diagram of the DSD-Net network structure of the present invention;

[0045] Figure 4 This is a schematic diagram of the cost aggregation module of the present invention;

[0046] Figure 5 This is a comparison diagram of the network performance of the present invention. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0048] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0049] It should be noted that, unless otherwise specified, the embodiments and features and technical solutions in the present invention can be combined with each other.

[0050] It should be noted that similar labels and letters in the following figures indicate similar items. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures.

[0051] Reference Figures 1-5 A stereo matching network that integrates domain generalization and feature similarity.

[0052] 1. The feature extraction module of the stereo matching network is improved by adopting the domain generalization method to enhance the robustness and generalization ability of feature extraction.

[0053] A ResNet-50 model pre-trained on the ImageNet dataset is used as the basis for the feature extraction module, which can leverage the generalization ability of the massive classification data of ImageNet. However, directly applying the pre-trained model will reduce accuracy, so a feature optimization structure is designed for adaptation.

[0054] This architecture employs an improved U-Net structure, combining the shallow feature extraction part of ResNet-50 with U-Net to fuse the generalization features of the pre-trained model. Simultaneously, Instance Norm replaces part of Batch Norm, enhancing the robustness of feature extraction and making it unaffected by style variations. This design leverages the advantages of the pre-trained model while incorporating adaptation optimizations, ensuring the feature extraction module retains content information while enhancing its robustness to style changes. The feature extraction module structure is as follows: Figure 1 As shown.

[0055] The improved U-Net diagram shows function replacements in the first two layers; the BatchNorm is not changed in the deeper layers to preserve content-related information. Considering model speed, the retraining process for the feature extraction module only runs during model training, while the inference process is not retrained, thus not slowing down the inference process.

[0056] 2. By incorporating a visual similarity module that considers feature consistency into the stereo matching network, and designing this module using the BBS algorithm, the visual similarity of features between the left and right images is measured as consistency information input into the network. This complements the differences in features, enabling the network to obtain a more comprehensive feature representation and enhancing its robust matching ability in complex situations such as occlusion. This, in turn, improves the matching accuracy and reduces errors. This invention defines a visual feature similarity structure for measuring the degree of visual similarity of features in paired images from left and right cameras, as defined below:

[0057] Set the left and right camera image pixels and neighbor set to , where P is the left camera feature image pixel p and its neighboring point set, and Q is the right camera feature image pixel q in the same row and its neighboring point set.

[0058] (1)

[0059] in, The similarity threshold is set to a default value of 0.5, meaning that when the similarity between two pixels reaches 0.5, the two points are considered the best similar pair. The visual similarity between pixel sets P and Q is defined as follows:

[0060] (2)

[0061] in, For visual similarity weights, the visual similarity of features decreases as the distribution difference between P and Q increases, and increases as the distributions become more similar. The feature similarity module is as follows: Figure 2 As shown, the darker the color, the higher the visual similarity between the left and right feature maps.

[0062] In addition to visual similarity, the features also consider semantic similarity between left and right views, meaning that the semantics expressed by the same object in two images are consistent. Semantic similarity is defined as the similarity of features in two images after a linear transformation. The degree of semantic similarity is defined as follows:

[0063] (3)

[0064] Where P and Q are the sets of neighboring pixels, For semantic similarity weights, Let F be the scaling function, p and q be the pixels in the pixel set, F be a linear function, and M and N be the number of pixels in the pixel set. We assume that when the semantic similarity increases, the feature semantic similarity increases, and when the semantic similarity decreases, the feature semantic similarity decreases. The feature similarity module consists of a visual similarity structure and a semantic similarity structure. The proportion of the two structures is controlled by similarity weights, with visual similarity having a relatively high weight. The feature similarity module is obtained through weighted fusion.

[0065] 3. By adjusting the sampling range adjustment coefficient in deformable convolution, its adaptive sampling range adjustment becomes smoother, enabling better focus on the effective receptive field and reducing the decrease in target recognition accuracy caused by sampling position deviations. This enhances the ability of deformable convolution to handle complex edges and weakly textured regions. The cost is defined as... Where D represents the maximum disparity in the search space, and W and H represent the width and height of the feature image, respectively, the improved deformable convolution is defined as follows:

[0066] (4)

[0067] in, This represents the cost of aggregating pixel p within the disparity search space d. This represents the total number of sampling points, where k represents the k-th sampling point. This represents the aggregate weight of the k-th sampling point. This represents the fixed offset of the k-th sampling point during the cost aggregation process. Indicates the adjustment used in the cost aggregation process. Learnable additional offset at the sampling location This represents the improved adjustment coefficient, used to adjust the sampling range of the deformable convolution so that it can better focus on the effective receptive field. A fixed amount to control the basic range of the adjustment coefficient.

[0068] Building upon deformable convolution, a deformable attention module is proposed to better handle weakly textured regions and complex edge regions in stereo matching networks, thereby improving matching accuracy. Unlike existing methods that only utilize deformable convolution to handle geometric deformations, this invention incorporates an attention mechanism into deformable convolution, allowing the convolution kernel to focus more on effective features and avoiding insufficient feature learning caused by uniform weight distribution. The sampling range adjustment coefficient of deformable convolution is improved, resulting in smoother adjustment. The single-scale deformable attention module is extended to multiple scales, constructing a cross-scale attention structure that allows for the learning and fusion of features at different scales, leading to better learning of multi-scale pixel location features. The sampling range and weight matrix of the deformable kernel are designed according to different scales to reduce computational cost. The input cost is defined as... The definition of deformable attention is as follows:

[0069] (5)

[0070] in, This represents a deformable attention feature, where 1 indicates the first layer of deformable attention. This represents the deformable attention weight matrix of the first layer. This represents the weight matrix of the deformable attention layer at the k-th sampling point. This represents the learnable additional offset of the attention mechanism. We apply deformable attention modules to the cost aggregation part of the stereo matching network.

[0071] 4. Network structure as follows Figure 3 As shown, the feature extraction module improves generalization ability through domain generalization method; the feature similarity module integrates visual and semantic similarity information; the cost matching and aggregation module constructs a multi-scale feature structure to improve matching accuracy; and finally, the disparity optimization module performs optimization.

[0072] By enhancing feature extraction through domain generalization and optimizing the matching process from multiple aspects, such as multi-scale feature fusion and the introduction of similarity information, the various components of the entire matching network can work together to more accurately match complex regions in the image, thereby improving the overall matching performance of the network.

[0073] Figure 4 This is a single-scale cost aggregation module that uses an improved deformable convolutional module and a deformable attention module. During the network downsampling process, the deformable convolutional module can effectively improve the network's attention to the effective receptive field, thereby capturing more information features. The deformable attention module focuses more on weak texture features and complex features. Its cross-scale attention structure effectively conveys cross-scale feature information and contextual information.

[0074] 5. Technical Effects

[0075] (1) The ablation experiment results are shown in Table 1. The Baseline network refers to... Figure 3 The network shown replaces the feature extraction module with a U-Net structure, removes the feature similarity module and the deformable attention module in the cost matching module, and uses an 8:2 partitioning of the KITTI dataset. The experimental data in Table 1 demonstrates that all three proposed structures can improve the baseline network to some extent.

[0076] Table 1 Ablation Experiment

[0077] Analysis of the data in Table 1 shows that, compared to the Baseline network, this invention has significant improvements in the following aspects:

[0078] 1) After improving the feature extraction module, the EPE index decreased by 0.67px, 0.72px and 0.11px on each dataset, respectively. This indicates that the introduction of the domain generalization method improved the robustness of feature extraction and provided better feature representation for subsequent matching.

[0079] 2) Based on 1, a feature similarity module was added, and the EPE index was further reduced by 0.38px, 0.39px and 0.14px. This is because a comprehensive judgment of both visual and semantic similarity of the image was added, which enhanced the robustness to occlusion and other situations.

[0080] 3) Based on 2, a deformable attention module was added, and the EPE index continued to decrease by 0.18px, 0.26px and 0.04px. This is mainly because the module enhances the processing capability of weak textures and complex edge areas.

[0081] By adding each module step by step and conducting ablation experiments, the improvements made by this invention in feature extraction, similarity judgment and attention mechanism are demonstrated. Compared with the original baseline network, it has achieved significant improvements on various datasets, reduced the disparity error of stereo matching, and verified the effectiveness of this invention.

[0082] (2) The performance comparison experiment of different convolutional network models in three datasets is shown in Table 2 below. 3D indicates that the network model uses 3D convolution, and 2D indicates that the network model uses 2D convolution.

[0083] Table 2 Comparative Experiments

[0084] In Table 2, 3D indicates the 3D convolution used in the network model, and 2D indicates the 2D convolution used in the network model. From the comparative experiments in Table 2, we can see that on the Sceneflow dataset, GANet and LEAStereo networks using 3D convolution outperform other networks. The EPE of our invention is only 0.03 lower than these two networks, but our EPE is higher than the other 3D convolutional network models. Among the network models using 2D convolution, our invention improves performance by 0.06 compared to AANet and 0.02 compared to FADNet. On the Middlebury dataset, our invention improves performance by 1.4 compared to GCNet, 20.4 compared to PSMNet, 3.2 compared to GANet, and 20 compared to GwcNet. While this invention performs worse than some 3D convolutional network models on the KITTI dataset, its inference time is significantly faster than that of GANet (1.8s), LEAStereo (0.3s), and CFNet (0.18s), compared to 0.08s. This is 20 times faster than GANet, 3 times faster than LEAStereo, and 2 times faster than CFNet. Although its inference speed is slower than FADNet and AANet, this invention outperforms FADNet in all five metrics listed in Table 2 and surpasses AANet in four of them. Therefore, it is considered that this invention offers a certain improvement over AANet. The comparative experiments in Table 2 verify the effectiveness of this invention.

[0085] (3) In addition to objective data indicators, this invention also conducted a qualitative analysis of the network performance. The visualization results of the stereo matching of each network on the Sceneflow dataset and the KITTI2015 dataset are shown below. Figure 5 As shown, GANet, which applies 3D convolution, performs best, generating images with more accurate disparity. Compared to PSMNet and AANet, the objects in the generated disparity images are more complete in shape and more accurate in detail. It can be seen that the stereo matching performance of the network of this invention is better than the two.

[0086] The preferred embodiments of the present invention disclosed above are merely illustrative of the invention. These preferred embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the content of this specification. This specification selects and specifically describes these embodiments to better explain the principles and practical applications of the invention, thereby enabling those skilled in the art to better understand and utilize the invention. The invention is limited only by the claims and their full scope and equivalents.

Claims

1. A method for constructing a stereo matching network that integrates domain generalization and feature similarity, characterized in that, The steps are as follows: S1: Improve the feature extraction module of the stereo matching network using the domain generalization method; S2: Introduce a feature similarity module to measure the visual similarity of features between the left and right images as consistency information input into the network; The feature similarity module consists of a visual similarity structure and a semantic similarity structure. The proportion of the two structures is controlled by similarity weights, with the visual similarity weight being relatively high. The feature similarity module is obtained by weighted fusion. S3: Use deformable attention mechanisms in the cost aggregation module; In step S1, a ResNet-50 model pre-trained on the ImageNet dataset is used as the basis for the feature extraction module; The shallow feature extraction part of ResNet-50 is combined with U-Net to fuse the generalization features of the pre-trained model. Replace some Batch Norms with Instance Norms; In step S2, a feature similarity module is introduced to measure the visual similarity of features between the left and right images, which is then input into the network as consistency information. The steps are as follows: Set the left and right camera image pixels and neighbor set to ; Where P represents the left camera feature image pixel p and its neighboring point set, and Q represents the right camera feature image pixel q in the same row and its neighboring point set; in, The similarity threshold is set to a default value of 0.5, meaning that when the similarity between two pixels reaches 0.5, the two points are considered the best similarity pair. The degree of visual similarity between pixel sets P and Q is expressed as follows: in, As the visual similarity weights are calculated, the visual similarity of features decreases when the distribution difference between P and Q increases, and the visual similarity of features increases when the distributions are more similar. Semantic similarity is defined as the similarity of features in two images after a linear transformation. The expression for the degree of semantic similarity is as follows: in, For semantic similarity weights, The scaling function is F, which is a linear function, and M and N are the number of pixels in the set. In step S3, the deformable attention mechanism is to add an attention mechanism to the deformable convolution; Define the input cost as The definition of deformable attention is as follows: in, This represents a deformable attention feature, where 1 indicates the first layer of deformable attention. This represents the deformable attention weight matrix of the first layer. This represents the weight matrix of the deformable attention layer at the k-th sampling point. This represents the learnable additional offset of the attention mechanism.

2. The method for constructing a stereo matching network that integrates domain generalization and feature similarity according to claim 1, characterized in that, The deformable convolution expression is as follows: in, This represents the cost of aggregating pixel p within the disparity search space d. This represents the total number of sampling points, where k represents the k-th sampling point. This represents the aggregate weight of the k-th sampling point. This represents the fixed offset of the k-th sampling point during the cost aggregation process. Indicates the adjustment used in the cost aggregation process. Learnable additional offset at the sampling location This represents the improved adjustment coefficient. A fixed amount to control the basic range of the adjustment coefficient.

Images