Stereo matching system and method based on bidirectional weighted path and frequency fusion

Abstract

The application provides a kind of stereo matching system and method based on bidirectional weighted path and frequency fusion, it is related to computer vision and graphics technical field, the system includes feature extraction module, bidirectional weighted path module, matching cost volume construction module, context extraction module, nonlinear frequency fusion module, iteration update module.The application eliminates multi-scale feature representation ambiguity and noise by bidirectional weighted path, combines nonlinear frequency fusion to optimize context information, significantly improves the precision and efficiency of stereo matching.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision and graphics, and particularly relates to a stereo matching system and method based on bidirectional weighted path and frequency fusion. BACKGROUND

[0002] In the field of computer vision and graphics, disparity estimation is one of the key technologies to realize three-dimensional space perception and reconstruction. Especially in binocular vision systems, the images of the same object are captured by two cameras with different positions. Due to the difference in the projection positions of the object on the imaging planes of the two cameras, this difference is called disparity. Based on the principle of triangulation, combined with the known baseline distance of the camera and the disparity information, the depth information of the object can be accurately calculated. This technology plays a crucial role in applications such as three-dimensional reconstruction, target detection and positioning, and has great application potential in the fields of robotics, autonomous driving, augmented reality, etc.

[0003] Stereo matching, as the core step in binocular disparity estimation, aims to establish the accurate correspondence between pixels in two-view images. Traditional stereo matching methods rely on manually designed features and matching strategies. These methods perform well when dealing with simple scenes, but their matching results are often unsatisfactory when facing weak texture or repetitive texture regions. In order to improve the matching accuracy, in recent years, learning-based methods have gradually become a research hotspot. Convolutional Neural Networks (CNN) are widely used in stereo matching due to their strong feature extraction ability, replacing the handcrafted features and matching strategies in traditional methods.

[0004] In learning-based stereo matching methods, there are mainly two categories: aggregation-based methods and iterative-based methods. Aggregation-based methods estimate the cost aggregation function by designing various convolutional neural networks, thereby optimizing the matching results. For example, GCNet uses a 3D convolutional neural network to regularize and aggregate the four-dimensional matching cost volume; GwcNet performs cost aggregation by calculating the correlation between feature groups; AANet adopts an adaptive cost aggregation method to alleviate the edge thickening problem. However, these methods are limited in their application when dealing with high-resolution scenes due to the high computational and storage costs of 3D convolution.

[0005] The iterative-based methods optimize the matching results by iteratively updating the disparity values, which use the values of the local matching cost volume indexed from the relevant matching cost volume to continuously update the disparity values. For example, RAFT-Stereo indexes the local matching cost volume by constructing the full pair correlation, but the full pair correlation lacks non-local information, leading to poor performance in handling local blur regions. To improve this, MoCha-Stereo proposes Motif Channel correlation and optimizes the disparity through error-based penalties; DLNR uses Long Short-Term Memory Network (LSTM) as the updater, improving the flexibility of the iterative process; and the stereo matching method based on adaptive frequency information selection further improves the matching accuracy by enabling the network to capture information from multiple frequencies during the iterative process through selective recurrent units.

[0006] However, existing iterative-based methods usually directly utilize multi-scale features generated by feature networks when constructing the matching cost volume. These multi-scale features introduce a large amount of representation ambiguity and noise due to scale differences, affecting the accuracy of feature representation and leading to incorrect disparity estimation. In addition, existing models only utilize the same receptive field when capturing global context information, ignoring the diversity of global information feature representation, leading to poor performance of the model in edge and detail regions.

[0007] Therefore, in order to further improve the accuracy and robustness of stereo matching, new techniques and methods need to be developed to more effectively handle multi-scale features, reduce representation ambiguity and noise, and fully utilize the diversity of global information to improve the performance of the model in complex scenes. SUMMARY

[0008] To this end, the embodiments of the present application provide a stereo matching method and system based on bidirectional weighted path and frequency fusion, which is used to solve the problem of inaccurate disparity estimation caused by representation ambiguity and noise of multi-scale features and insufficient global information capture in the prior art.

[0009] To solve the above problems, the embodiments of the present application provide a stereo matching system based on bidirectional weighted path and frequency fusion, which comprises:

[0010] A feature extraction module for extracting multi-scale features of a left-right view pair;

[0011] A bidirectional weighted path module for processing the multi-scale features to eliminate representation ambiguity and noise, wherein the bidirectional weighted path module comprises a forward weighted path and a reverse weighted path, the forward weighted path reconstructs global feature representation by fusing multi-scale features; and the reverse weighted path reconstructs high-scale features based on the global feature representation and low-scale features in turn;

[0012] The matching cost body construction module is used to construct the matching cost body based on the processed multi-scale features.

[0013] The context extraction module is used to extract multi-scale context features from the left view;

[0014] The nonlinear frequency fusion module is used to perform convolution operations with different kernel sizes on the multi-scale context features to obtain feature information with different receptive fields, and to fuse the feature information to obtain fused context feature information.

[0015] The iterative update module is used to iteratively update the disparity value based on the matching cost body and the fused context feature information to obtain the final disparity map.

[0016] Preferably, the feature extraction module uses EfficientNetv2-m as the backbone network to extract multi-scale feature pyramids for the left and right views, wherein the feature pyramids include feature maps of different resolutions.

[0017] Preferably, the forward weighted path obtains a global feature representation by fusing multi-scale features, specifically including:

[0018] First, multi-scale features are pooled to the same scale and then coupled to obtain coupled global features. This process is represented as follows:

[0019] ;

[0020] In the formula, This indicates an average pooling operation; Indicates a connection operation; Represents the global features after coupling; This indicates the result obtained after inputting the left or right view into the backbone network. Original features at scale Due to global features The scale is the same as that of the 1 / 32 feature, therefore No average pooling operation is required;

[0021] Then, the coupled global features are convolved, and channel attention and spatial attention are applied for denoising to obtain the global feature representation. The process is as follows:

[0022] ;

[0023] ;

[0024] ;

[0025] ;

[0026] ;

[0027] In the formula, Indicates the convolution operation; Represents the global features after convolution; This represents the Sigmoid activation function; Indicates a fully connected operation; This represents the max pooling operation; Represents channel attention weights; ⊙ represents element-wise dot product; Indicates spatial attention weights; This represents the global feature representation.

[0028] Preferably, the reverse weighted path reconstructs high-scale features sequentially based on global feature representation and low-scale features, specifically including:

[0029] First, represent the global features. Decoupling into global features at different scales can be represented as follows:

[0030] ;

[0031] In the formula, This indicates a sharding operation, which is about to... Divide into 4 equal parts based on the number of channels; Corresponding to the left or right view respectively Global features at a certain scale ;

[0032] Then, the decoupled low-scale features are fused with the high-scale features to be reconstructed to achieve representation enhancement;

[0033] ;

[0034] In the formula, Indicates an upsampling operation; This indicates the result obtained after inputting the left or right view into the backbone network. Original features at scale ; Indicates decoupling Scale characteristics, i.e. Lower-level scale features; Represents the global features after fusion;

[0035] Next, channel attention and spatial attention are applied to the fused global features for denoising, resulting in denoised fused features. The process is as follows:

[0036] ;

[0037] ;

[0038] ;

[0039] ;

[0040] wherein, denotes the de-noised fusion feature;

[0041] Finally, the de-noised fusion feature is fused with the decoupled high-scale feature to reconstruct the high-scale feature, and the process is represented as:

[0042] ;

[0043] wherein, denotes the reconstructed feature.

[0044] Preferably, the matching cost volume construction module is configured to construct a matching cost volume based on the processed multi-scale feature, and the process is represented as:

[0045] ;

[0046] wherein, denotes the inner product; denotes the disparity; denotes the number of feature channels; denotes the number of groups; denotes the matching cost volume; denotes the matching cost when the position is , the disparity is , and the group is ; denote the feature maps of the left image and the right image at a specific group and scale, respectively.

[0047] Preferably, the context extraction module is configured to generate multi-scale context features with the same number of channels for a given left view using an EfficientNetv2-M network , wherein the number of channels of the multi-scale context features is all 128.

[0048] Preferably, the non-linear frequency fusion module is configured to perform convolution operations with different kernel sizes on the context features to obtain feature information of different receptive fields, and fuse the feature information to obtain fused context feature information, specifically including:

[0049] First, convolution operations with different kernel sizes are performed on the multi-scale context features to obtain feature information of different receptive fields, and the feature information is fused, and the process is represented as:

[0050] ;

[0051] ;

[0052] ;

[0053] wherein, denotes the multi-scale context feature, ; and denote and convolution operation of the convolution kernel; and denote the feature information after the convolution operation of the and convolution kernel; denotes the fusion feature;

[0054] Then, the attention weight in the fusion information is extracted using the attention mechanism, and the process is represented as:

[0055] ;

[0056] ;

[0057] ;

[0058] wherein, denotes the convolution operation; denotes the Sigmoid activation function; denotes the full connection operation; denotes the average pooling operation; denotes the maximum pooling operation; denotes the channel attention weight; and denotes the element dot product; denotes the nonlinear fusion attention weight;

[0059] Finally, considering that the attention is concentrated in the high-frequency information area, it is used for the information of the smaller kernel, while the opposite weight is applied to the information of the larger kernel, so as to realize the feature enhancement of the context information, and obtain the fused context feature information, and the process is represented as:

[0060] ;

[0061] wherein, denotes the fused context feature information.

[0062] Preferably, the iterative updating module is configured to iteratively update the disparity value based on the matching cost volume and the fused context feature information, and specifically comprises:

[0063] Based on the matching cost volume and the fused context feature information, the disparity map is obtained using a dynamic updater based on a gated recurrent unit, and the hidden information is fused using attention weights. The iterative update process is implemented by the following formula:

[0064] ;

[0065] ;

[0066] ;

[0067] ;

[0068] In the formula, and respectively represent the gated recurrent unit operation using , convolution kernels; represents the matching cost volume; represents the context feature information; represents the hidden information at the th iteration; represents the disparity at the th iteration; represents the element-wise product; represents the attention weight of the nonlinear fusion; and respectively represent the hidden information after the gated recurrent unit operation using , convolution kernels at the th iteration; represents the fused hidden information; represents the convolution operation; represents the activation function.

[0069] Preferably, in the process of iteratively updating the disparity value, the smooth L1 loss and the L1 loss are used together to constitute the loss function, so as to optimize the initial disparity and the disparity prediction in the iterative process, and the process is represented as:

[0070] ;

[0071] ;

[0072] ;

[0073] In the formula, represents the loss of the predicted initial disparity; represents the true value disparity; represents the initial disparity; denotes a smooth L1 loss; denotes a loss of all predictions produced by the iteration process; denotes a penalty term; denotes a predicted disparity at the denotes a predicted disparity at the denotes a total number of iterations; denotes an L1 loss; denotes a total loss.

[0074] The embodiment of the present application also provides a stereo matching method based on bidirectional weighted path and frequency fusion, which adopts the stereo matching system based on bidirectional weighted path and frequency fusion and specifically comprises the following steps:

[0075] extracting multi-scale features of the left view and the right view by using a feature extraction module;

[0076] processing the multi-scale features by using a bidirectional weighted path module to eliminate representation ambiguity and noise;

[0077] constructing a matching cost volume based on the processed multi-scale features by using a matching cost volume construction module;

[0078] extracting multi-scale context features of the left view by using a context extraction module;

[0079] performing convolution operations of different kernel sizes on the multi-scale context features by using a nonlinear frequency fusion module, obtaining feature information of different receptive fields, and fusing the feature information to obtain fused context feature information;

[0080] iteratively updating disparity values based on the matching cost volume and the fused context feature information by using an iterative updating module to obtain a final disparity map.

[0081] From the above technical solutions, the present application has the following beneficial effects:

[0082] (1) Significantly improving stereo matching precision: by using the proposed new stereo matching method based on bidirectional weighted path and frequency fusion, the present application can eliminate representation ambiguity and noise caused by multi-scale features, thereby significantly improving the precision of stereo matching.

[0083] (2) Optimizing context information and improving performance in challenging regions: the nonlinear frequency fusion module fuses information from different frequency receptive fields, retains useful details, and uses attention as a nonlinear fusion weight, thereby optimizing context information and improving performance in challenging image regions.

[0084] (3) Enhanced edge and low-texture region representation: the present application uses attention weights to nonlinearly fuse information from different receptive fields, enhancing the representation of context information in edge and low-texture regions, solving the problem of insensitivity of context information to these regions. BRIEF DESCRIPTION OF DRAWINGS

[0085] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments will be briefly described below. The features and advantages of the present application can be more clearly understood by referring to the drawings. The drawings are schematic and should not be understood as any limitation on the present application. For those skilled in the art, other drawings can be obtained from these drawings without creative labor. Among them:

[0086] Figure 1 Structure diagram of a stereo matching system based on bidirectional weighted path and frequency fusion provided in the embodiments;

[0087] Figure 2 Structure diagram of a stereo matching system based on bidirectional weighted path and frequency fusion provided in the embodiments;

[0088] Figure 3 Flowchart of a stereo matching method based on bidirectional weighted path and frequency fusion provided in the embodiments;

[0089] Figure 4 KITTI dataset comparison chart in the embodiments, wherein (a) is an image to be predicted; (b) is the result of the stereo matching method based on iterative geometric coding volume; and (c) is the result of the method of the present application;

[0090] Figure 5 Generalization performance result comparison chart in the embodiments, wherein the first line is an image to be predicted; the second line is the result of the stereo matching method based on iterative geometric coding volume; and the third line is the result of the method of the present application. DETAILED DESCRIPTION

[0091] In order to make the purpose, technical solutions and advantages of the embodiments of the present application more clear, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0092] Embodiment one

[0093] In order to solve the problem of inaccurate disparity estimation caused by representation ambiguity and noise of multi-scale features and insufficient global information capture in the prior art. Figure 1 As shown in the figure, the embodiment of the present application proposes a stereo matching system based on bidirectional weighted path and frequency fusion, which comprises:

[0094] The feature extraction module 100 is used to extract multi-scale features of the left and right view pairs.

[0095] The bidirectional weighted path module 200 is used to process the multi-scale features to eliminate representation ambiguity and noise.

[0096] The matching cost volume construction module 300 is used to construct a matching cost volume based on the processed multi-scale features.

[0097] The context extraction module 400 is used to extract multi-scale context features of the left view.

[0098] The nonlinear frequency fusion module 500 is used to perform convolution operations of different kernel sizes on the multi-scale context features to obtain feature information of different receptive fields, and fuse the feature information to obtain fused context feature information.

[0099] The iterative update module 600 is used to iteratively update the disparity value based on the matching cost volume and the fused context feature information to obtain a final disparity map.

[0100] From the above technical solution, the present application proposes a stereo matching system based on bidirectional weighted path and frequency fusion. The system obtains multi-scale features of left and right view pairs through a feature extraction module, effectively eliminates representation ambiguity and noise caused by multi-scale features through a bidirectional weighted path module, and enhances the accuracy and robustness of the features. A matching cost volume construction module constructs a matching cost volume based on the processed features, providing a reliable basis for subsequent disparity estimation. At the same time, a context extraction module and a nonlinear frequency fusion module jointly optimize context information, fuse feature information of different receptive fields through convolution operations of different kernel sizes and attention weights, and improve the disparity estimation accuracy in edge and low texture regions. Finally, an iterative update module iteratively updates the disparity value based on the matching cost volume and the optimized context feature information to obtain a high-quality final disparity map. The system significantly improves the accuracy and efficiency of stereo matching, and provides a new effective method for solving the problem of representation ambiguity and noise caused by multi-scale features.

[0101] The technical solution of the present application will be described below in conjunction with Figure 2 The technical solution of the present application will be described below in conjunction with

[0102] In this embodiment, the feature extraction module 100 uses EfficientNetv2-M as the backbone network. For a given left-right view pair, the module extracts feature maps with a size of 1 / 32 of the original image resolution using the backbone network. Then, it is sequentially restored to feature maps with sizes of 1 / 16, 1 / 8 and 1 / 4 of the original image resolution through upsampling operations, thereby obtaining a multi-scale feature pyramid , . The obtained multi-scale features are used as input into the bidirectional weighted path module 200 to eliminate the ambiguity and noise of the features. In this embodiment, the bidirectional weighted path module 200 is composed of a forward weighted path and a backward weighted path, where the forward weighted path obtains a global feature representation by fusing the multi-scale features; the backward weighted path reconstructs high-scale features based on the global feature representation and low-scale features in turn.

[0103] Specifically, in view of the deficiency that the multi-scale features have feature representation ambiguity, the embodiment designs a forward weighted path. First, the multi-scale features are pooled to the same scale (1 / 32 scale) and then coupled to obtain coupled global features, and the calculation process is shown in formula (1). Since the coupled global features have only one scale, there is no representation ambiguity.

[0104] (1)

[0105] In the formula, represents the average pooling operation; represents the connection operation; represents the coupled global feature; represents the original feature of the left or right view input into the backbone network at the 1 / 32 scale, . Since the global feature has the same scale as the 1 / 32 feature, it does not need an average pooling operation.

[0106] Although the coupled global feature has consistent representation, noise still exists. In view of this deficiency, the forward weighted path convolves the coupled global feature and applies channel attention and spatial attention to denoise, thereby obtaining a global feature representation, and the process is represented as:

[0107] ;

[0108] ;

[0109] ;

[0110] ;

[0111] ​ (2)

[0112] wherein, denotes the convolution operation; denotes the global feature after convolution; denotes the Sigmoid activation function; denotes the full connection operation; denotes the max-pooling operation; denotes the channel attention weight; denotes the element-wise product; denotes the spatial attention weight; denotes the global feature representation.

[0113] Further, to reconstruct the feature, the backward weighting path firstly decouples the global feature representation into different scale global features, which is represented as:

[0114] (3)

[0115] wherein, denotes the slicing operation, i.e., splitting into 4 parts averagely by channel number; corresponds to the global feature of the left or right view at different scales, respectively.

[0116] Since the low scale feature has strong representation ability, the backward weighting path fuses the decoupled low scale feature with the high scale feature to be reconstructed in turn, achieving representation enhancement, which is represented as:

[0117] (4)

[0118] wherein, denotes the up-sampling operation; denotes the decoupled low scale feature, i.e., the lower level scale feature; denotes the fused global feature.

[0119] Next, similar to the forward weighting path, the backward weighting path uses channel attention and spatial attention to denoise the fused global feature , obtaining the denoised fused feature , which is represented as:

[0120] ;

[0121] ;

[0122] ; ​

[0123] (5)

[0124] Finally, the reverse weighted path fuses the denoised fused features with the decoupled high-scale features to reconstruct the high-scale features. The process is represented as follows:

[0125] (6)

[0126] In the formula, Indicates the reconstruction features.

[0127] During the reconstruction process, due to These are the lowest-scale features and do not need to be fused with lower-scale features; that is, they can be directly... Substituting into formula (5) yields the denoised fused features, and substituting into formula (6) yields the reconstructed features. Then, repeating formulas (4)-(6) allows for the sequential reconstruction of feature maps at scales of 1 / 16, 1 / 8, and 1 / 4. Due to the reconstructed features... It eliminates ambiguity and denoises, possesses strong feature representation capabilities and noise suppression capabilities, and helps to construct accurate matching cost volumes.

[0128] In this embodiment, after the bidirectional weighted path module 200, the matching cost volume construction module 300 is based on the processed multi-scale features (i.e., reconstructed features). The process of constructing the matching cost body is represented as follows:

[0129] (7)

[0130] In the formula, Indicates the inner product; Indicates parallax; Indicates the number of feature channels; Indicates the number of groups; Represents the matching cost body; Indicates the location Parallax is Grouped into Matching cost at that time; These represent the feature maps of the left and right images under specific groupings and scales, respectively.

[0131] The matching cost body constructed in this embodiment Used for regressing initial disparity And retrieve the local matching cost to iteratively update the disparity.

[0132] In the embodiment, in the stereo matching system, the semantic information and the spatial information around the pixel point in the left view are obtained by the context extraction module 400 to assist the matching decision. For a given left image, the EfficientNetv2-M network is used to generate multi-scale context features with the same number of channels , where the number of channels of the multi-scale context features is all 128.

[0133] Further, in order to obtain the context information representing the edge and detailed area better, the nonlinear frequency fusion module 500 is designed to fuse the information from the receptive fields of different frequencies, so that the system fully utilizes the edge and detail information.

[0134] Specifically, the nonlinear frequency fusion module 500 is used to perform convolution operations with different kernel sizes on the context features, obtain feature information of different receptive fields, and fuse the feature information to obtain fused context feature information, specifically including:

[0135] First, the multi-scale context features are subjected to convolution operations with different kernel sizes to obtain feature information of different receptive fields, and the feature information is fused, and the process is represented as:

[0136] ;

[0137] ;

[0138] (8)

[0139] wherein, represents the multi-scale context features, ; and respectively represent and convolution operations of the convolution kernels; and respectively represent the feature information after the convolution operations of the and convolution kernels; represents the fused features.

[0140] Then, in order to obtain a robust result, an attention mechanism is used to extract attention weights in the fused information, and the process is represented as:

[0141] ;

[0142] ;

[0143] (9)

[0144] In the formula, This represents the attention weights for nonlinear fusion.

[0145] Finally, considering that attention is focused on high-frequency information regions, this invention applies them to information with smaller kernels, while applying the opposite weight to information with larger kernels, thereby achieving feature enhancement of contextual information and obtaining fused contextual feature information. The process is represented as follows:

[0146] (10)

[0147] In the formula, This represents the fused contextual feature information.

[0148] In this embodiment, the iterative update module 600 is used to iteratively update the disparity value based on the matching cost volume and the fused contextual feature information to obtain the final disparity map, specifically including:

[0149] Based on the matching cost volume and the fused contextual feature information, a dynamic updater based on gated recurrent units (GRU) is used to obtain the disparity map, and hidden information is fused using attention weights. The iterative update process is implemented through the following formula:

[0150] ;

[0151] ;

[0152] ;

[0153] (11)

[0154] In the formula, and They respectively represent the use of , Gated recurrent unit operations of convolution kernels; Represents contextual feature information; Indicates the first Hidden information during the next iteration; Indicates the first The disparity of the next iteration; and They represent the first time. In the next iteration, use , Hidden information after the gated recurrent unit operation of the convolution kernel; This indicates hidden information related to the fusion; This represents the activation function.

[0155] Furthermore, during the iterative update of disparity values, this invention uses a loss function composed of smoothed L1 loss and L1 loss to optimize the initial disparity and disparity prediction during the iteration process, specifically including:

[0156] First, the loss for the predicted initial disparity is calculated using smoothed L1 loss. As shown in formula (12):

[0157] (12)

[0158] In the formula, Indicates the true parallax; Indicates the initial disparity; This represents smoothed L1 loss.

[0159] Then, the loss of all predictions generated by the iterative process is calculated using L1 loss. As shown in formula (13):

[0160] (13)

[0161] In the formula, The penalty is set to 0.9 in this embodiment. It is the first The disparity predicted in the next iteration It represents the total number of iterations.

[0162] Finally, the sum of these two parts constitutes the total loss, as shown in formula (14).

[0163] (14)

[0164] In the formula, This indicates the total loss.

[0165] Example 2

[0166] like Figure 3 As shown, this invention provides a stereo matching method based on bidirectional weighted path and frequency fusion. This method employs the stereo matching system based on bidirectional weighted path and frequency fusion described in Embodiment 1 above, and specifically includes:

[0167] S1: Use the feature extraction module to extract multi-scale features from the left and right views;

[0168] S2: Multi-scale features are processed through a bidirectional weighted path module to eliminate representation ambiguity and noise;

[0169] S3: Based on the processed multi-scale features, the matching cost body is constructed using the matching cost body construction module;

[0170] S4: extracting multi-scale context features of the left view using a context extraction module;

[0171] S5: performing convolution operations with different kernel sizes on the multi-scale context features using a nonlinear frequency fusion module, obtaining feature information of different receptive fields, and fusing the feature information to obtain fused context feature information;

[0172] S6: based on the matching cost volume and the fused context feature information, iteratively updating the disparity value through an iterative update module to obtain a final disparity map.

[0173] The stereo matching method based on bidirectional weighted path and frequency fusion of the embodiment is used to realize the stereo matching system based on bidirectional weighted path and frequency fusion described above, so the specific embodiments in the stereo matching method based on bidirectional weighted path and frequency fusion can be seen in the embodiment part of the stereo matching system based on bidirectional weighted path and frequency fusion described above, for example, steps S1, S2, S3, S4, S5, and S6 respectively use the feature extraction module 100, the bidirectional weighted path module 200, the matching cost volume construction module 300, the context extraction module 400, the nonlinear frequency fusion module 500, and the iterative update module 600 of the stereo matching system based on bidirectional weighted path and frequency fusion described above, so the specific embodiments can refer to the description of the respective embodiment parts. In order to avoid redundancy, it will not be described here.

[0174] In order to further illustrate the advantages of the present application, specific experiments will be described below.

[0175] 1. Introduction of data set

[0176] Four public stereo matching data sets are used in this experiment to pre-train and fine-tune the stereo matching method based on bidirectional weighted path and frequency fusion.

[0177] (1) Scene Flow: a synthetic data set containing 35454 pairs of training sets and 4370 pairs of test sets. The finalpass version is used in this experiment, which contains images with realistic settings such as motion blur, defocus blur, and noise.

[0178] (2) KITTI 2012 and KITTI 2015: data sets of real driving scenes. KITTI 2012 contains 194 training image pairs and 195 test image pairs, while KITTI 2015 includes 200 training image pairs and 200 test image pairs. Both provide sparse ground truth disparities obtained from laser radar.

[0179] (3) Middlebury: An indoor dataset, providing 15 pairs of training pairs and 15 pairs of test pairs, some images are taken under consistent lighting or color conditions, and all pictures have three different resolutions.

[0180] (4) ETH3D: Contains 27 pairs of stereo gray images from indoor and outdoor scenes, used to test the generalization performance of the model.

[0181] 2. Implementation details

[0182] The experiment realizes the stereo matching method based on bidirectional weighted path and frequency fusion under the PyTorch framework, and uses the AdamW optimizer. All experiments are carried out in an environment equipped with two NVIDIA Tesla V100 GPUs. The experimental settings include 22 update iterations and data augmentation. On the Scene Flow dataset, the experiment is trained for 200k steps, the batch size is 8, the single-cycle learning rate is 2x10 -4 , and the image is randomly cropped to 320x1152.

[0183] 3. Algorithm comparison

[0184] The experiment compares the stereo matching method based on bidirectional weighted path and frequency fusion with the most advanced method on SceneFlow, KITTI 2012 and KITTI 2015. The results show that the method performs well on these datasets. As shown in Table 1, on the Scene Flow dataset, the method achieves new state-of-the-art results in EPE and 0.5 pixel error metrics, with EPE of 0.42, which is 10.64% higher than the stereo matching method based on iterative geometric coding volume. The 0.5 pixel error index is 7.77%, and the 1 pixel error index is 4.86%, ranking second only to Stereo-Risk. Compared with the classic CFNet, the accuracy of the stereo matching method based on bidirectional weighted path and frequency fusion is 2 times higher. As shown in Tables 2 and 3, on the KITTI 2012 and KITTI 2015 benchmarks, the method also achieves excellent performance. Figure 4 The results of KITTI 2012 and KITTI 2015 are shown, which show that the method performs very well in textureless and detailed areas.

[0185] Table 1 Comparison with state-of-the-art algorithms on Scene Flow dataset

[0186]

[0187] Table 2 Comparison with state-of-the-art algorithms on KITTI 2015 dataset

[0188]

[0189] Table 3 Comparison with state-of-the-art algorithms on KITTI 2012 dataset

[0190]

[0191] 4. Generalization test

[0192] Since it is difficult to obtain ground truth on real datasets, generalization performance is crucial. We conduct generalization test on Middlebury and ETH3D datasets. As shown in Table 4, the stereo matching method based on bi-directional weighted path and frequency fusion achieves state-of-the-art performance in all cases. On the Middlebury dataset, the method performs well in full resolution, half resolution, and quarter resolution. On the ETH3D dataset, the method has a 1 -pixel error rate of 4.3. Figure 5 It is shown that the method performs better in detail areas compared to the stereo matching method based on iteratively encoded volume.

[0193] Table 4 Test results of generalization performance

[0194]

[0195] Those skilled in the art will understand that embodiments of the present application can be provided as methods, systems, or computer program products. Thus, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, etc.) embodying computer readable program code.

[0196] The present application is described in terms of flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It will be understood that each flow and / or block of the flowcharts and / or block diagrams, and combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processing system, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowcharts and / or block diagrams block or blocks. Figure 1 The flow and / or block Figure 1 The means for performing the function specified by the flow and / or block

[0197] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The functions specified in one or more boxes. These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable apparatus for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0198] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. A stereo matching system based on bidirectional weighted path and frequency fusion, characterized in that, The method comprises the following steps: a feature extraction module for extracting multi-scale features of a left view and a right view; a bidirectional weighted path module for processing the multi-scale features to eliminate ambiguity and noise, wherein the bidirectional weighted path module comprises a forward weighted path and a reverse weighted path, the forward weighted path couples multi-scale features after being pooled to the same scale to obtain coupled global features, performs convolution on the coupled global features, and applies channel attention and spatial attention for denoising to obtain global feature representation, and the reverse weighted path reconstructs high-scale features in sequence based on the global feature representation and low-scale features; a matching cost volume construction module for constructing a matching cost volume based on the processed multi-scale features; a context extraction module for extracting multi-scale context features of the left view; a nonlinear frequency fusion module for performing convolution operations of different kernel sizes on the multi-scale context features to obtain feature information of different receptive fields, fusing the feature information, extracting attention weights in the fused information using an attention mechanism, considering that attention is concentrated in a high-frequency information region, and using the attention weights for information of a smaller kernel and applying opposite weights for information of a larger kernel to achieve feature enhancement of context information, and obtaining fused context feature information; an iterative update module for iteratively updating a disparity value based on the matching cost volume and the fused context feature information to obtain a final disparity map.

2. The stereo matching system based on bidirectional weighted path and frequency fusion of claim 1, wherein, The feature extraction module uses EfficientNetv2-M as a backbone network to extract a multi-scale feature pyramid of the left view and the right view, and the feature pyramid comprises feature maps of different resolutions.

3. The stereo matching system based on bidirectional weighted path and frequency fusion of claim 1, wherein, The forward weighted path couples multi-scale features after being pooled to the same scale to obtain coupled global features, performs convolution on the coupled global features, and applies channel attention and spatial attention for denoising to obtain global feature representation, and specifically comprises the following steps: First, the multi-scale features are coupled after being pooled to the same scale to obtain coupled global features, and the process is represented as: ； In the formula, represents an average pooling operation; represents a concatenation operation; represents a coupled global feature; represents the original feature obtained after the left or right view input backbone network at the scale of 1 / 32, Since the global feature is the same as the scale of the 1 / 32 feature, the average pooling operation is not needed; Then, the coupled global features are convolved, and channel attention and spatial attention are applied for denoising to obtain global feature representation, and the process is represented as: ； ； ； ； ； wherein, denotes a convolution operation; denotes global features after convolution; denotes a Sigmoid activation function; denotes a full connection operation; denotes a max-pooling operation; denotes channel attention weights; denotes element-wise product; denotes spatial attention weights; denotes global feature representation.

4. The stereo matching system based on bidirectional weighted path and frequency fusion of claim 3, wherein, The reverse weighted path reconstructs high-scale features in sequence based on the global feature representation and low-scale features, and specifically comprises the following steps: First, the global feature representation is decoupled into global features of different scales, the process of which is represented as: ； In the formula, denotes a slicing operation, i.e. the is divided into 4 parts by averaging over the channel number; respectively correspond to the left or right view global features at a scale, ; Then, the decoupled low-scale features are fused with the high-scale features to be reconstructed to achieve feature enhancement, and the process is represented as: ； wherein, denotes an up-sampling operation; denotes the original feature at scale s after the left or right view input backbone network; denotes the original feature at scale s after the left or right view input backbone network; ; denotes the decoupled feature at scale s, i.e. the lower level scale feature; denotes the decoupled feature at scale s, i.e. the lower level scale feature; denotes the decoupled feature at scale s, i.e. the lower level scale feature; denotes the fused global feature; Next, channel attention and spatial attention are applied to the fused global features for denoising to obtain denoised fused features, and the process is represented as: ； ； ； ； In the formula, denotes the fusion feature after denoising; Finally, the denoised fused features are fused with the decoupled high-scale features to reconstruct the high-scale features, and the process is represented as: ； In the formulae, denotes a reconstruction feature.

5. The stereo matching system based on bidirectional weighted path and frequency fusion of claim 1, wherein, The matching cost volume construction module is used to construct a matching cost volume based on the processed multi-scale features, and the process is represented as: ； wherein denotes an inner product; denotes a disparity; denotes a number of feature channels; denotes a number of groups; denotes a matching cost volume; denotes a matching cost at position , disparity , and group ; denotes a feature map of the left and right images at a certain group and scale, respectively.

6. The stereo matching system based on bidirectional weighted path and frequency fusion of claim 1, wherein, The context extraction module is configured to generate, for a given left view, multi-scale context features with the same number of channels by using an EfficientNetv2-M network , wherein the number of channels of the multi-scale context features is all 128.

7. The stereo matching system based on bidirectional weighted path and frequency fusion of claim 1, wherein, The nonlinear frequency fusion module is configured to perform convolution operations of different kernel sizes on the context features, obtain feature information of different receptive fields, and fuse the feature information, use an attention mechanism to extract attention weights in the fused information, consider that the attention is concentrated in a high-frequency information region, and use the attention weights for information of a smaller kernel, while applying opposite weights to information of a larger kernel, so as to realize feature enhancement of the context information, and obtain fused context feature information, and specifically includes the following steps. First, convolution operations of different kernel sizes are performed on the multi-scale context features, feature information of different receptive fields is obtained, and the feature information is fused, and the process is represented as follows: ； ； ； In the formula, represent multi-scale context features, and respectively represent and convolution operations of convolution kernels; and respectively represent feature information after the and convolution operations of convolution kernels; represent fused features;​ Then, an attention mechanism is used to extract attention weights in the fused information, and the process is represented as follows: ； ； ； wherein, denotes a convolution operation; denotes a Sigmoid activation function; denotes a full connection operation; denotes an average pooling operation; denotes a max pooling operation; denotes a channel attention weight; denotes an element-wise product; denotes a nonlinear fused attention weight; Finally, considering that the attention is concentrated in a high-frequency information region, the attention weights are used for information of a smaller kernel, while opposite weights are applied to information of a larger kernel, so as to realize feature enhancement of the context information, and obtain fused context feature information, and the process is represented as follows: ； In the formula, represents the context feature information after fusion.

8. The stereo matching system based on bidirectional weighted path and frequency fusion of claim 1, wherein, The iterative updating module is configured to iteratively update the disparity value based on the matching cost volume and the fused context feature information, and specifically includes the following steps: Based on the matching cost volume and the fused context feature information, a dynamic updater based on a gated recurrent unit is used to obtain a disparity map, and hidden information is fused by using attention weights, and the iterative updating process is realized by the following formula: ； ； ； ； In the formula, and respectively represent the gating recurrent unit operation using , convolution kernel; represent the matching cost body; represent the context feature information; represent the hidden information at the first iteration; represent the disparity at the first iteration; represent the element dot product; represent the nonlinear fused attention weight; and respectively represent the hidden information after the gating recurrent unit operation using , convolution kernel at the first iteration; represent the fused hidden information; represent the convolution operation; represent the activation function.

9. The stereo matching system based on bidirectional weighted path and frequency fusion of claim 1, wherein, In the process of iteratively updating the disparity value, a smooth L1 loss and an L1 loss are used to jointly constitute a loss function, so as to optimize the initial disparity and the disparity prediction in the iterative process, and the process is represented as follows: ； ； ； wherein, denotes the loss of the predicted initial disparity; denotes the ground truth disparity; denotes the initial disparity; denotes the smooth L1 loss; denotes the loss of all predictions resulting from the iterative process; denotes the penalty term; denotes the predicted disparity at the th iteration; denotes the total number of iterations; denotes the L1 loss; denotes the total loss.

10. A stereo matching method based on bidirectional weighted path and frequency fusion, characterized in that, The method adopts the stereo matching system based on bidirectional weighted path and frequency fusion according to any one of claims 1 to 9, and specifically includes the following steps: A feature extraction module is used to extract multi-scale features of the left view and the right view; A bidirectional weighted path module is used to process the multi-scale features to eliminate ambiguity and noise; Based on the processed multi-scale features, a matching cost volume construction module is used to construct a matching cost volume; A context extraction module is used to extract multi-scale context features of the left view; A nonlinear frequency fusion module is used to perform convolution operations of different kernel sizes on the multi-scale context features, obtain feature information of different receptive fields, and fuse the feature information, and obtain fused context feature information; Based on the matching cost volume and the fused context feature information, an iterative updating module is used to iteratively update the disparity value, and a final disparity map is obtained.

Images