Stereo matching system and method based on bidirectional weighted path and frequency fusion

Abstract

The invention provides a stereo matching system and method based on bidirectional weighted path and frequency fusion, and relates to the technical field of computer vision and graphics, and the system comprises a feature extraction module, a bidirectional weighted path module, a matching cost body construction module, a context extraction module, a nonlinear frequency fusion module and an iteration updating module. According to the method, multi-scale feature representation ambiguity and noise are eliminated through a bidirectional weighting path, context information is optimized in combination with nonlinear frequency fusion, and the precision and efficiency of stereo matching are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision and graphics, and in particular to a stereo matching system and method based on bidirectional weighted path and frequency fusion. Background Art

[0002] In the field of computer vision and graphics, disparity estimation is one of the key technologies for achieving three-dimensional space perception and reconstruction. Especially in binocular vision systems, images of the same object are captured by two cameras at different positions. Since the projection position of the object on the imaging plane of the two cameras is different, this difference is called disparity. Based on the principle of triangulation, combined with the known camera baseline distance and disparity information, the depth information of the object can be accurately calculated. This technology plays a vital role in applications such as three-dimensional reconstruction, target detection and positioning, and shows great application potential in robotics, autonomous driving, augmented reality and other fields.

[0003] Stereo matching, as the core step in binocular disparity estimation, aims to establish the precise correspondence between pixels in two perspective images. Traditional stereo matching methods rely on manually designed features and matching strategies. These methods perform well when dealing with simple scenes, but their matching results are often unsatisfactory when facing weak texture or repeated texture areas. In order to improve matching accuracy, learning-based methods have gradually become a research hotspot in recent years. Convolutional neural networks (CNNs) are widely used in stereo matching due to their powerful feature extraction capabilities to replace the manual features and matching strategies in traditional methods.

[0004] Among the learning-based stereo matching methods, there are two main categories: aggregation-based methods and iteration-based methods. Aggregation-based methods estimate the cost aggregation function by designing various convolutional neural networks to optimize the matching results. For example, GCNet uses a 3D convolutional neural network to perform regularized aggregation on the four-dimensional matching cost volume; GwcNet performs cost aggregation by calculating the correlation between feature groups; and AANet uses an adaptive cost aggregation method to alleviate the edge thickening problem. However, these methods are limited in their application when dealing with high-resolution scenes due to the high computational and storage costs of 3D convolutions.

[0005] Iteration-based methods optimize the matching results by cyclically updating the disparity values. They use the values ​​of the local matching cost volume indexed from the relevant matching cost volume to continuously update the disparity values. For example, RAFT-Stereo indexes the local matching cost volume by constructing full-pair correlation, but the full-pair correlation lacks non-local information, resulting in poor performance in handling local blurred areas. To improve this, MoCha-Stereo proposes Motif channel correlation and optimizes disparity through error-based penalties; DLNR uses a long short-term memory network (LSTM) as an updater to improve the flexibility of the iterative process; and the stereo matching method based on adaptive frequency information selection uses selective cyclic units to enable the network to capture information from multiple frequencies during the iteration process, further improving the matching accuracy.

[0006] However, existing iteration-based methods usually directly use the multi-scale features generated by the feature network when constructing the matching cost volume. These multi-scale features introduce a lot of representation ambiguity and noise due to scale differences, which affects the accuracy of feature representation and leads to incorrect disparity estimation. In addition, when capturing global context information, existing models only use the same receptive field, ignoring the diversity of global information feature representation, resulting in poor performance of the model in edge and detail areas.

[0007] Therefore, in order to further improve the accuracy and robustness of stereo matching, new technologies and methods need to be developed to more effectively process multi-scale features, reduce representation ambiguity and noise, and make full use of the diversity of global information to improve the performance of the model in complex scenes. Summary of the invention

[0008] To this end, an embodiment of the present invention provides a stereo matching method and system based on a bidirectional weighted path and frequency fusion, which is used to solve the problems of representation ambiguity, noise caused by multi-scale features in the prior art, and inaccurate disparity estimation caused by insufficient global information capture.

[0009] In order to solve the above problems, an embodiment of the present invention provides a stereo matching system based on bidirectional weighted path and frequency fusion, the system comprising:

[0010] A feature extraction module for extracting multi-scale features of left and right view pairs;

[0011] A bidirectional weighted path module is used to process the multi-scale features and eliminate representation ambiguity and noise, wherein the bidirectional weighted path module includes a forward weighted path and a reverse weighted path, the forward weighted path obtains a global feature representation by fusing multi-scale features; the reverse weighted path sequentially reconstructs high-scale features based on the global feature representation and low-scale features;

[0012] A matching cost volume construction module is used to construct a matching cost volume based on the processed multi-scale features;

[0013] A context extraction module, used to extract multi-scale context features of the left view;

[0014] A nonlinear frequency fusion module, used to perform convolution operations with different kernel sizes on the multi-scale context features, obtain feature information of different receptive fields, and fuse the feature information to obtain fused context feature information;

[0015] The iterative updating module is used to iteratively update the disparity value based on the matching cost volume and the fused context feature information to obtain a final disparity map.

[0016] Preferably, the feature extraction module uses EfficientNetv2-m as a backbone network to extract multi-scale feature pyramids of the left view and the right view, and the feature pyramid includes feature maps of different resolutions.

[0017] Preferably, the forward weighted path obtains a global feature representation by fusing multi-scale features, specifically comprising:

[0018] First, the multi-scale features are pooled to the same scale and then coupled to obtain the coupled global features. The process is expressed as:

[0019] f G =Concat{AP(f l(r),4 ),AP(f l(r),8 ),AP(f l(r),16 ),f l(r),32};

[0020] In the formula, AP represents the average pooling operation; Concat represents the concatenation operation; f G represents the global characteristics after coupling; f l(r),i represents the original features at 1 / i scale obtained after the left or right view is input into the backbone network, i∈{4,8,16,32}; due to the global feature f G The same scale as the 1 / 32 feature, so f l(r),32 No average pooling operation is required;

[0021] Then, the coupled global features are convolved, and channel attention and spatial attention are applied for denoising to obtain the global feature representation. The process is expressed as:

[0022] f′ G =Conv(f G );

[0023] W c =σ(MLP(AP(F M))+MLP(MP(F M )));

[0024] f′ G =f′ G ⊙W c ;

[0025] W s =σ(Conv(Concat(AP(f′ G ),MP(f′ G ))));

[0026] F G =f′ G ⊙W s ;

[0027] Where Conv represents the convolution operation; f′ G represents the global feature after convolution; σ represents the Sigmoid activation function; MLP represents the fully connected operation; MP represents the maximum pooling operation; W c represents the channel attention weight; ⊙ represents the element dot product; W s represents the spatial attention weight; F G Represents the global feature representation.

[0028] Preferably, the reverse weighted path sequentially reconstructs high-scale features based on the global feature representation and low-scale features, specifically comprising:

[0029] First, the global feature representation F G Decoupling into global features of different scales, the process is expressed as:

[0030] f′ l(r),i =Split(F G );

[0031] In the formula, Split represents the sharding operation, that is, F G Divide into 4 parts according to the number of channels; f′ l(r),i Corresponding to left or right view respectively Figure 1 / Global features at scale i, i∈{4,8,16,32};

[0032] Then, the decoupled low-scale features are fused with the high-scale features to be reconstructed to achieve representation enhancement;

[0033] F M =f l(r),i +US(F l(r),i / 2 );

[0034] Where US represents the upsampling operation; f l(r),iF represents the original features at the 1 / i scale obtained after the left or right view is input into the backbone network, i∈{4,8,16,32}; l(r),i / 2 represents the decoupled 1 / (i / 2) scale feature, that is, the scale feature one level lower than 1 / i; F M Represents the global features after fusion;

[0035] Next, channel attention and spatial attention are applied to the fused global features for denoising to obtain the denoised fused features. The process is expressed as:

[0036] W c =σ(MLP(AP(F M ))+MLP(MP(F M )));

[0037] F M =F M ⊙W c ;

[0038] W s =σ(Conv(Concat(AP(F M ),MP(F M ))));

[0039] F AM =F M ⊙W s ;

[0040] In the formula, F AM Represents the fused features after denoising;

[0041] Finally, the denoised fusion features are fused with the decoupled high-scale features to reconstruct the high-scale features. The process is expressed as:

[0042] F l(r),i =F AM ⊙f′ l(r),i ;

[0043] In the formula, F l(r),i Represents the reconstruction feature.

[0044] Preferably, the matching cost volume construction module is used to construct a matching cost volume based on the processed multi-scale features, and the process is expressed as follows:

[0045]

[0046] Where <,> represents the inner product; d represents the disparity; N c Represents the number of feature channels; N grepresents the number of groups; C represents the matching cost volume; C(g,d,x,y) represents the matching cost at position (x,y), disparity d, and group g; Represent the feature maps of the left and right images under specific groups and scales, respectively.

[0047] Preferably, the context extraction module is used to generate multi-scale context features c with the same number of channels using an EfficientNetv2-M network for a given left view. i , i∈{4,8,16}, where the multi-scale context feature c i The number of channels is 128.

[0048] Preferably, the nonlinear frequency fusion module is used to perform convolution operations with different kernel sizes on the context features to obtain feature information of different receptive fields, and fuse the feature information to obtain fused context feature information, specifically including:

[0049] First, convolution operations with different kernel sizes are performed on the multi-scale context features to obtain feature information of different receptive fields and fuse the feature information. The process is expressed as:

[0050]

[0051] In the formula, c i Represents multi-scale context features, i∈{4,8,16}; and Represent the convolution operations of 1×1 and 3×3 convolution kernels respectively; and They represent the feature information after convolution operation with 1×1 and 3×3 convolution kernels respectively; F L Indicates fusion features;

[0052] Then, the attention mechanism is used to extract the attention weights in the fusion information, and the process is expressed as:

[0053] W c =σ(MLP(AP(F L ))+MLP(MP(F L )));

[0054] F L =F L ⊙W c ;

[0055] W = σ(Conv(Concat(AP(F L ),MP(F L ))));

[0056] Where Conv represents the convolution operation; σ represents the Sigmoid activation function; MLP represents the fully connected operation; AP represents the average pooling operation; MP represents the maximum pooling operation; W c represents the channel attention weight; ⊙ represents the element dot product; W represents the attention weight of nonlinear fusion;

[0057] Finally, considering that the attention is focused on the high-frequency information area, it is used for the information of the smaller kernel, and the opposite weight is applied to the information of the larger kernel, so as to enhance the features of the context information and obtain the fused context feature information. The process is expressed as:

[0058]

[0059] In the formula, Represents the fused context feature information.

[0060] Preferably, the iterative updating module is used to iteratively update the disparity value based on the matching cost volume and the fused context feature information, specifically comprising:

[0061] Based on the matching cost volume and the fused context feature information, a dynamic updater based on a gated recurrent unit is used to obtain the disparity map, and the attention weight is used to fuse the hidden information. The iterative update process is implemented by the following formula:

[0062]

[0063] d t =Conv(ReLU(Conv(h t )));

[0064] In the formula, and They represent the gated recurrent unit operations using k1 and k2 convolution kernels respectively; C represents the matching cost volume; F C Represents context feature information; h t represents the hidden information at the tth iteration; d t represents the disparity of the t-th iteration; ⊙ represents the element-wise dot product; W represents the attention weight of nonlinear fusion; and They represent the hidden information after the gated recurrent unit operation with the k1 and k2 convolution kernels at the t+1th iteration; h t+1 Represents the fused hidden information; Conv represents the convolution operation; ReLU represents the activation function.

[0065] Preferably, in the process of iteratively updating the disparity value, the smooth L1 loss and the L1 loss are used to jointly constitute a loss function to optimize the initial disparity and the disparity prediction in the iterative process, and the process is expressed as:

[0066]

[0067]

[0068] Where, L init represents the loss of the predicted initial disparity; d gt represents the true value disparity; d gt represents the initial disparity; represents smooth L1 loss; L iter represents the loss of all predictions generated by the iterative process; γ represents the penalty term; d i represents the predicted disparity at the i-th iteration; N represents the total number of iterations; ‖‖1 represents the L1 loss; L total Represents the total loss.

[0069] The embodiment of the present invention further provides a stereo matching method based on bidirectional weighted path and frequency fusion, which adopts the above-mentioned stereo matching system based on bidirectional weighted path and frequency fusion, and specifically includes:

[0070] Use a feature extraction module to extract multi-scale features of the left view and the right view;

[0071] Processing the multi-scale features through a bidirectional weighted path module to eliminate representation ambiguity and noise;

[0072] Based on the processed multi-scale features, a matching cost volume is constructed using a matching cost volume construction module;

[0073] Use the context extraction module to extract multi-scale context features of the left view;

[0074] Using a nonlinear frequency fusion module to perform convolution operations with different kernel sizes on the multi-scale context features, obtain feature information of different receptive fields, and fuse the feature information to obtain fused context feature information;

[0075] Based on the matching cost volume and the fused context feature information, the disparity value is iteratively updated by an iterative updating module to obtain a final disparity map.

[0076] It can be seen from the above technical solutions that the present invention has the following beneficial effects:

[0077] (1) Significantly improve stereo matching accuracy: Through the proposed novel stereo matching method based on bidirectional weighted path and frequency fusion, the present invention can eliminate representation ambiguity and noise caused by multi-scale features, thereby significantly improving the stereo matching accuracy.

[0078] (2) Optimizing contextual information and improving performance in challenging areas: The nonlinear frequency fusion module fuses information from different frequency receptive fields, retains useful details, and uses attention as the weight of nonlinear fusion to optimize the contextual information and improve the performance in challenging image areas.

[0079] (3) Enhanced representation of edge and low-texture regions: The present invention uses attention weights to nonlinearly fuse information from different receptive fields, thereby enhancing the representation of contextual information in edge and low-texture regions and solving the problem that contextual information is insensitive to these regions. BRIEF DESCRIPTION OF THE DRAWINGS

[0080] In order to more clearly illustrate the implementation cases of the present invention or the technical solutions in the prior art, the following is a brief description of the drawings required for use in the embodiments. By referring to the drawings, the features and advantages of the present invention will be more clearly understood. The drawings are schematic and should not be understood as limiting the present invention in any way. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0081] Figure 1 A schematic diagram of the structure of a stereo matching system based on a bidirectional weighted path and frequency fusion provided in an embodiment;

[0082] Figure 2 A schematic diagram of the overall architecture of a stereo matching system based on a bidirectional weighted path and frequency fusion provided in an embodiment;

[0083] Figure 3 A flowchart of a stereo matching method based on bidirectional weighted path and frequency fusion provided in an embodiment;

[0084] Figure 4 It is a comparison diagram of the KITTI data set in the embodiment, wherein (a) is the image to be predicted; (b) is the result of the stereo matching method based on iterative geometric coder; (c) is the result of the method of the present invention;

[0085] Figure 5 It is a comparison chart of the generalization performance results in the embodiments, wherein the first line is the image to be predicted; the second line is the result of the stereo matching method based on the iterative geometric coder; and the third line is the result of the method of the present invention. DETAILED DESCRIPTION

[0086] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0087] Embodiment 1

[0088] In order to solve the problem of inaccurate disparity estimation caused by ambiguity and noise caused by multi-scale features in the prior art and insufficient global information capture. Figure 1 As shown, an embodiment of the present invention proposes a stereo matching system based on a bidirectional weighted path and frequency fusion, the system comprising:

[0089] A feature extraction module 100, for extracting multi-scale features of a left and right view pair;

[0090] A bidirectional weighted path module 200 for processing multi-scale features and eliminating representation ambiguity and noise;

[0091] A matching cost volume construction module 300 is used to construct a matching cost volume based on the processed multi-scale features;

[0092] A context extraction module 400, configured to extract multi-scale context features of the left view;

[0093] The nonlinear frequency fusion module 500 is used to perform convolution operations with different kernel sizes on multi-scale context features, obtain feature information of different receptive fields, and fuse the feature information to obtain fused context feature information;

[0094] The iterative updating module 600 is used to iteratively update the disparity value based on the matching cost volume and the fused context feature information to obtain a final disparity map.

[0095] It can be seen from the above technical scheme that the present invention proposes a stereo matching system based on bidirectional weighted path and frequency fusion. The system obtains multi-scale features of left and right view pairs through a feature extraction module, and uses a bidirectional weighted path module to effectively eliminate the representation ambiguity and noise caused by multi-scale features, thereby enhancing the accuracy and robustness of the features. The matching cost volume construction module constructs a matching cost volume based on the processed features, providing a reliable basis for subsequent disparity estimation. At the same time, the context extraction module and the nonlinear frequency fusion module jointly optimize the context information, and fuse the feature information of different receptive fields through convolution operations of different kernel sizes and attention weights, thereby improving the accuracy of disparity estimation in edge and low-texture areas. Finally, the iterative update module iteratively updates the disparity value based on the matching cost volume and the optimized context feature information to obtain a high-quality final disparity map. The system significantly improves the accuracy and efficiency of stereo matching, and provides a new and effective method for solving the representation ambiguity and noise problems caused by multi-scale features.

[0096] Combine the following Figure 2 The technical solution of the present invention is described.

[0097] In this embodiment, the feature extraction module 100 uses EfficientNetv2-M as the backbone network. For a given left and right view pair, the module uses the backbone network to extract feature maps with a size of 1 / 32 of the original image resolution. Then, through upsampling operations, it is restored to feature maps with sizes of 1 / 16, 1 / 8, and 1 / 4 of the original image resolution in turn, thereby obtaining a multi-scale feature pyramid f l(r),i , i∈{4,8,16,32}. The obtained multi-scale features are used to input into the bidirectional weighted path module 200 to eliminate the ambiguity and noise of the features.

[0098] In this embodiment, the bidirectional weighted path module 200 consists of a forward weighted path and a reverse weighted path, wherein the forward weighted path obtains a global feature representation by fusing multi-scale features; and the reverse weighted path sequentially reconstructs high-scale features based on the global feature representation and low-scale features.

[0099] Specifically, in view of the deficiency of feature representation ambiguity in multi-scale features, this embodiment designs a forward weighted path. First, the multi-scale features are pooled to the same scale (1 / 32 scale) and then coupled to obtain the coupled global features. The calculation process is shown in formula (1). Since the coupled global features have only one scale, there is no representation ambiguity.

[0100] f G =Concat{AP(f l(r),4 ),AP(f l(r),8 ),AP(f l(r),16 ),f l(r),32 (1)

[0101] In the formula, AP represents the average pooling operation; Concat represents the concatenation operation; f G represents the global characteristics after coupling; f l(r),i represents the original features at 1 / i scale obtained after the left or right view is input into the backbone network, i∈{4,8,16,32}. Since the global feature f G The same scale as the 1 / 32 feature, so f l(r),32 No average pooling operation is required.

[0102] Although the coupled global features have a consistent representation, noise still exists. To address this shortcoming, the forward weighted path convolves the coupled global features and applies channel attention and spatial attention to denoise them to obtain the global feature representation. The process is expressed as:

[0103] f′ G =Conv(f G );

[0104] W c =σ(MLP(AP(F M ))+MLP(MP(F M )));

[0105] f′ G =f′ G ⊙W c ;

[0106] W s =σ(Conv(Concat(AP(f′ G ),MP(f′ G ))));

[0107] F G =f′ G ⊙W s (2)

[0108] Where Conv represents the convolution operation; f′ G represents the global feature after convolution; σ represents the Sigmoid activation function; MLP represents the fully connected operation; MP represents the maximum pooling operation; W c represents the channel attention weight; ⊙ represents the element dot product; W s represents the spatial attention weight; F G Represents the global feature representation.

[0109] Furthermore, to reconstruct the features, the reverse weighted path first transforms the global feature representation F G Decoupling into global features of different scales, the process is expressed as:

[0110] f′ l(r),i =Split(F G ) (3)

[0111] In the formula, Split represents the sharding operation, that is, F G Divide into 4 parts according to the number of channels; f′ l(r),i Corresponding to left or right view respectively Figure 1 / Global features at scale i, i∈{4,8,16,32}.

[0112] Since low-scale features have strong representation capabilities, the reverse weighted path successively fuses the decoupled low-scale features with the high-scale features to be reconstructed to achieve representation enhancement. The process is expressed as:

[0113] F M =f l(r),i +US(F l(r),i / 2 (4)

[0114] Where US represents the upsampling operation; F i(r),i / 2 represents the decoupled 1 / (i / 2) scale feature, that is, the scale feature one level lower than 1 / i; F M Represents the fused global features.

[0115] Next, similar to the forward weighted path, the reverse weighted path uses channel attention and spatial attention to denoise the fused global features and obtain the denoised fused feature F AM , the process is expressed as:

[0116] W c =σ(MLP(AP(F M ))+MLP(MP(F M )));

[0117] F M =F M ⊙W c ;

[0118] W s =σ(Conv(Concat(AP(F M ),MP(F M ))));

[0119] F AM =F M ⊙W s (5)

[0120] Finally, the reverse weighted path fuses the denoised fusion features with the decoupled high-scale features to reconstruct the high-scale features. The process is expressed as:

[0121] F l(r),i =FAM ⊙f′ l(r),i (6)

[0122] In the formula, F l(r),i Represents the reconstruction feature.

[0123] During the reconstruction process, due to f l(r),32 It is the lowest scale feature and does not need to be fused with lower features, that is, directly f l(r),32 Substitute into formula (5) to get the denoised fusion feature, and substitute into formula (6) to get the reconstructed feature. Then, repeat the operations of formula (4)-(6), and the feature maps of 1 / 16, 1 / 8 and 1 / 4 scales can be reconstructed in turn. l(r),i Eliminating representation ambiguity and denoising, with strong feature representation ability and noise suppression ability, helps to construct an accurate matching cost body.

[0124] In this embodiment, after the bidirectional weighted path module 200, the matching cost volume construction module 300 is based on the processed multi-scale features (i.e., the reconstructed features F l(r),i ) to construct the matching cost body, and the process is expressed as:

[0125]

[0126] Where <,> represents the inner product; d represents the disparity; N c Represents the number of feature channels; N g represents the number of groups; C represents the matching cost volume; C(g,d,x,y) represents the matching cost at position (x,y), disparity d, and group g; Represent the feature maps of the left and right images under specific groups and scales, respectively.

[0127] The matching cost volume C constructed in this embodiment is used to regress the initial disparity d0 and retrieve the local matching cost to cyclically update the disparity.

[0128] In this embodiment, in the stereo matching system, the context extraction module 400 obtains the semantic information and spatial information around the pixels in the left view, thereby assisting the matching decision. For a given left image, this embodiment uses the EfficientNetv2-M network to generate multi-scale context features c with the same number of channels. i , i∈{4,8,16}, where the multi-scale context feature c i The number of channels is 128.

[0129] Furthermore, in order to obtain context information that better represents edge and detail areas, the present invention designs a nonlinear frequency fusion module 500 to fuse information from receptive fields of different frequencies, so that the system of the present invention can fully utilize edge and detail information.

[0130] Specifically, the nonlinear frequency fusion module 500 is used to perform convolution operations with different kernel sizes on the context features, obtain feature information of different receptive fields, and fuse the feature information to obtain fused context feature information, which specifically includes:

[0131] First, convolution operations with different kernel sizes are performed on the multi-scale context features to obtain feature information of different receptive fields and fuse the feature information. The process is expressed as:

[0132]

[0133] In the formula, c i Represents multi-scale context features, i∈{4,8,16}; and Represent the convolution operations of 1×1 and 3×3 convolution kernels respectively; and They represent the feature information after convolution operation with 1×1 and 3×3 convolution kernels respectively; F L Represents fusion features.

[0134] Then, in order to obtain robust results, the attention mechanism is used to extract the attention weights in the fusion information, and the process is expressed as:

[0135] W c =σ(MLP(AP(F L ))+MLP(MP(F L )));

[0136] F L =F L ⊙W c ;

[0137] W = σ(Conv(Concat(AP(F L ),MP(F L )))) (9)

[0138] Where W represents the attention weight of nonlinear fusion.

[0139] Finally, considering that attention is focused on high-frequency information areas, the present invention uses it for information with smaller kernels, and applies opposite weights to information with larger kernels, thereby achieving feature enhancement of context information and obtaining fused context feature information. The process is expressed as follows:

[0140]

[0141] In the formula, Represents the fused context feature information.

[0142] In this embodiment, the iterative updating module 600 is used to iteratively update the disparity value based on the matching cost volume and the fused context feature information to obtain the final disparity map, specifically including:

[0143] Based on the matching cost volume and the fused context feature information, a dynamic updater based on a gated recurrent unit (GRU) is used to obtain the disparity map, and the attention weight is used to fuse the hidden information. The iterative update process is implemented by the following formula:

[0144]

[0145] d t =Conv(ReLU(Conv(h t ))) (11)

[0146] In the formula, and Respectively represent the gated recurrent unit operation using k1 and k2 convolution kernels; F C Represents context feature information; h t represents the hidden information at the tth iteration; d t represents the disparity of the t-th iteration; and They represent the hidden information after the gated recurrent unit operation with the k1 and k2 convolution kernels at the t+1th iteration respectively; h t+1 Represents the fused hidden information; ReLU represents the activation function.

[0147] Furthermore, in the process of iteratively updating the disparity value, the present invention uses the smooth L1 loss and the L1 loss to jointly constitute a loss function to optimize the initial disparity and the disparity prediction in the iterative process, specifically including:

[0148] First, the loss L of the predicted initial disparity is calculated using the smooth L1 loss iter , as shown in formula (12):

[0149]

[0150] Where, d gt represents the true disparity; d0 represents the initial disparity; represents the smoothed L1 loss.

[0151] Then, the loss L for all predictions produced by the iterative process is calculated using the L1 loss iter , as shown in formula (13):

[0152]

[0153] In the formula, γ represents the penalty term, which is set to 0.9 in this embodiment, and di is the predicted disparity at the i-th iteration, and N is the total number of iterations.

[0154] Finally, the sum of these two parts constitutes the total loss, as shown in formula (14),

[0155]

[0156] Where, L total Represents the total loss.

[0157] Embodiment 2

[0158] like Figure 3 As shown, the present invention provides a stereo matching method based on bidirectional weighted path and frequency fusion, which adopts the stereo matching system based on bidirectional weighted path and frequency fusion of the above-mentioned embodiment 1, and specifically includes:

[0159] S1: Use the feature extraction module to extract multi-scale features of the left and right views;

[0160] S2: Process multi-scale features through a bidirectional weighted path module to eliminate representation ambiguity and noise;

[0161] S3: Based on the processed multi-scale features, a matching cost volume is constructed using a matching cost volume construction module;

[0162] S4: Use the context extraction module to extract multi-scale context features of the left view;

[0163] S5: Use the nonlinear frequency fusion module to perform convolution operations with different kernel sizes on the multi-scale context features to obtain feature information of different receptive fields, and fuse the feature information to obtain fused context feature information;

[0164] S6: Based on the matching cost volume and the fused context feature information, the disparity value is iteratively updated through the iterative update module to obtain the final disparity map.

[0165] A stereo matching method based on a bidirectional weighted path and frequency fusion in this embodiment is used to implement the aforementioned stereo matching system based on a bidirectional weighted path and frequency fusion. Therefore, the specific implementation method of the stereo matching method based on a bidirectional weighted path and frequency fusion can be seen in the embodiment part of the stereo matching system based on a bidirectional weighted path and frequency fusion in the previous text. For example, steps S1, S2, S3, S4, S5, and S6 respectively adopt the feature extraction module 100, bidirectional weighted path module 200, matching cost volume construction module 300, context extraction module 400, nonlinear frequency fusion module 500, and iterative update module 600 of the above-mentioned stereo matching system based on a bidirectional weighted path and frequency fusion. Therefore, its specific implementation method can refer to the description of the corresponding embodiments of each part. In order to avoid redundancy, it will not be repeated here.

[0166] In order to further illustrate the advantages of the present invention, a description is given below in conjunction with specific experiments.

[0167] 1. Dataset Introduction

[0168] This experiment uses four public stereo matching datasets to pre-train and fine-tune a stereo matching method based on bidirectional weighted paths and frequency fusion.

[0169] (1) Scene Flow: A synthetic dataset containing 35,454 training pairs and 4,370 test pairs. This experiment uses the final pass version, which contains images with realistic settings such as motion blur, defocus blur, and noise.

[0170] (2) KITTI 2012 and KITTI 2015: datasets of real driving scenes. KITTI 2012 contains 194 training image pairs and 195 test image pairs, while KITTI 2015 contains 200 training image pairs and 200 test image pairs. Both provide sparse ground truth differences obtained from LiDAR.

[0171] (3) Middlebury: An indoor dataset that provides 15 training pairs and 15 test pairs. Some images are taken under consistent lighting or color conditions, and all images have three different resolutions.

[0172] (4) ETH3D: Contains 27 pairs of stereo grayscale images from indoor and outdoor scenes, used to test the generalization performance of the model.

[0173] 2. Implementation details

[0174] This experiment implements a stereo matching method based on bidirectional weighted paths and frequency fusion in the PyTorch framework, and uses the AdamW optimizer. All experiments are performed in an environment equipped with two NVIDIA Tesla V100 GPUs. The experimental setting includes 22 update iterations and data augmentation. On the Scene Flow dataset, this experiment is trained for 200k steps, with a batch size of 8 and a single-cycle learning rate of 2×10 -4 , the images are randomly cropped to 320×1152.

[0175] 3. Algorithm comparison

[0176] This experiment compares the stereo matching method based on bidirectional weighted path and frequency fusion with the state-of-the-art methods on SceneFlow, KITTI 2012 and KITTI 2015. The results show that the method performs well on these datasets. As shown in Table 1, on the Scene Flow dataset, the method achieves new state-of-the-art results in both EPE and 0.5 pixel error metrics, with an EPE of 0.42, which is 10.64% higher than the stereo matching method based on iterative geometric encoder. The 0.5 pixel error metric is 7.77%, and the 1 pixel error metric is 4.86%, ranking second only to Stereo-Risk. Compared with the classic CFNet, the stereo matching method based on bidirectional weighted path and frequency fusion is 2 times more accurate than it. As shown in Tables 2 and 3, the method also achieves excellent performance on the KITTI 2012 and KITTI 2015 benchmarks. Figure 4 Results from KITTI 2012 and KITTI 2015 are shown, showing that our method performs very well in both texture-free and detailed regions.

[0177] Table 1 Comparison with the most advanced algorithms on the Scene Flow dataset

[0178]

[0179] Table 2 Comparison with the most advanced algorithms on the KITTI 2015 dataset

[0180]

[0181] Table 3 Comparison with the most advanced algorithms on the KITTI 2012 dataset

[0182]

[0183] 4. Generalization Test

[0184] Given that it is difficult to obtain true values ​​for real datasets, generalization performance is crucial. This experiment conducted generalization tests on the Middlebury and ETH3D datasets. As shown in Table 4, the stereo matching method based on bidirectional weighted paths and frequency fusion achieves state-of-the-art performance in all cases. On the Middlebury dataset, the method performs well at full resolution, half resolution, and quarter resolution. On the ETH3D dataset, the method achieves a 1-pixel error rate of 4.3. Figure 5 It is shown that this method exhibits better performance in detail areas compared to stereo matching methods based on iterative geometric encoding.

[0185] Table 4 Test results of generalization performance

[0186]

[0187] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0188] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0189] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, so that the instructions executed on the computer or other programmable device provide for implementing the process in the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0190] Obviously, the above embodiments are merely examples for clear explanation and are not intended to limit the implementation methods. For those skilled in the art, other different forms of changes or modifications can be made based on the above description. It is not necessary and impossible to list all the implementation methods here. The obvious changes or modifications derived from them are still within the protection scope of the invention.

Claims

1. A stereo matching system based on bidirectional weighted path and frequency fusion, characterized in that: include: A feature extraction module for extracting multi-scale features of left and right view pairs; A bidirectional weighted path module is used to process the multi-scale features and eliminate representation ambiguity and noise, wherein the bidirectional weighted path module includes a forward weighted path and a reverse weighted path, the forward weighted path obtains a global feature representation by fusing multi-scale features; the reverse weighted path sequentially reconstructs high-scale features based on the global feature representation and low-scale features; A matching cost volume construction module is used to construct a matching cost volume based on the processed multi-scale features; A context extraction module, used to extract multi-scale context features of the left view; A nonlinear frequency fusion module, used to perform convolution operations with different kernel sizes on the multi-scale context features, obtain feature information of different receptive fields, and fuse the feature information to obtain fused context feature information; The iterative updating module is used to iteratively update the disparity value based on the matching cost volume and the fused context feature information to obtain a final disparity map.

2. The stereo matching system based on bidirectional weighted path and frequency fusion according to claim 1, characterized in that: The feature extraction module uses EfficientNetv2-M as a backbone network to extract multi-scale feature pyramids of left and right views, wherein the feature pyramids include feature maps of different resolutions.

3. The stereo matching system based on bidirectional weighted path and frequency fusion according to claim 1, characterized in that: The forward weighted path obtains a global feature representation by fusing multi-scale features, specifically including: First, the multi-scale features are pooled to the same scale and then coupled to obtain the coupled global features. The process is expressed as: f G =Concat{AP(f l(r),4 ),AP(f l(r),8 ),AP(f l(r),16 ),f l(r),32 }; In the formula, AP represents the average pooling operation; Concat represents the concatenation operation; f G represents the global characteristics after coupling; f l(r),i represents the original features at 1 / i scale obtained after the left or right view is input into the backbone network, i∈{4,8,16,32}; due to the global feature f G The same scale as the 1 / 32 feature, so f l(r),32 No average pooling operation is required; Then, the coupled global features are convolved, and channel attention and spatial attention are applied for denoising to obtain the global feature representation. The process is expressed as: f′ G =Conv(f G ); W c =σ(MLP(AP(F M ))+MLP(MP(F M ))); f′ G =f′ G ⊙W c ; W s =σ(Conv(Concat(AP(f′ G ),MP(f′ G )))); F G =f′ G ⊙W s ; Where Conv represents the convolution operation; f′ G represents the global feature after convolution; σ represents the Sigmoid activation function; MLP represents the fully connected operation; MP represents the maximum pooling operation; W c represents the channel attention weight; ⊙ represents the element dot product; W s represents the spatial attention weight; F G Represents the global feature representation.

4. The stereo matching system based on bidirectional weighted path and frequency fusion according to claim 1, characterized in that: The reverse weighted path sequentially reconstructs high-scale features based on the global feature representation and low-scale features, specifically including: First, the global feature representation F G Decoupling into global features of different scales, the process is expressed as: f′ l(r),i =Split(F G ); In the formula, Split represents the sharding operation, that is, F G Divide into 4 parts according to the number of channels; f′ l(r),i Corresponding to the global features of the left or right view at 1 / i scale, i∈{4,8,16,32}; Then, the decoupled low-scale features are fused with the high-scale features to be reconstructed to achieve representation enhancement. The process is expressed as; F M =f l(r),i +US(F l(r),i / 2 ); Where US represents the upsampling operation; f l(r),i F represents the original features at the 1 / i scale obtained after the left or right view is input into the backbone network, i∈{4,8,16,32}; l(r),i / 2 represents the decoupled 1 / (i / 2) scale feature, that is, the scale feature one level lower than 1 / i; F M Represents the global features after fusion; Next, channel attention and spatial attention are applied to the fused global features for denoising to obtain the denoised fused features. The process is expressed as: W c =σ(MLP(AP(F M ))+MLP(MP(F M ))); F M =F M ⊙W c ; W s =σ(Conv(Concat(AP(F M ),MP(F M )))); F AM =F M ⊙W s ; In the formula, F AM represents the fused features after denoising; Finally, the denoised fusion features are fused with the decoupled high-scale features to reconstruct the high-scale features. The process is expressed as: F l(r),i =F AM ⊙f′ l(r),i ; In the formula, F l(r),i Represents the reconstruction feature.

5. The stereo matching system based on bidirectional weighted path and frequency fusion according to claim 1, characterized in that: The matching cost volume construction module is used to construct a matching cost volume based on the processed multi-scale features, and its process is expressed as follows: Where <,> represents the inner product; d represents the disparity; N c Represents the number of feature channels; N g represents the number of groups; C represents the matching cost volume; C(g,d,x,y) represents the matching cost at position (x,y), disparity d, and group g; Represent the feature maps of the left and right images under specific groups and scales, respectively.

6. The stereo matching system based on bidirectional weighted path and frequency fusion according to claim 1, characterized in that: The context extraction module is used to generate multi-scale context features c with the same number of channels for a given left view using the EfficientNetv2-M network. i , i∈{4,8,16}, where the multi-scale context feature c i The number of channels is 128.

7. The stereo matching system based on bidirectional weighted path and frequency fusion according to claim 1, characterized in that: The nonlinear frequency fusion module is used to perform convolution operations with different kernel sizes on the context features to obtain feature information of different receptive fields, and fuse the feature information to obtain fused context feature information, specifically including: First, convolution operations with different kernel sizes are performed on the multi-scale context features to obtain feature information of different receptive fields and fuse the feature information. The process is expressed as: In the formula, c i Represents multi-scale context features, i∈{4,8,16}; and Represent the convolution operations of 1×1 and 3×3 convolution kernels respectively; and They represent the feature information after convolution operation with 1×1 and 3×3 convolution kernels respectively; F L Indicates fusion features; Then, the attention mechanism is used to extract the attention weights in the fusion information, and the process is expressed as: W c =σ(MLP(AP(F L ))+MLP(MP(F L ))); F L =F L ⊙W c ; W=σ(Conv(Concat(AP(F L ),MP(F L )))); In the formula, Conv represents the convolution operation; σ represents the Sigmoid activation function; MLP represents the fully connected operation; AP represents the average pooling operation; MP represents the maximum pooling operation; W c represents the channel attention weight; ⊙ represents the element dot product; W represents the attention weight of nonlinear fusion; Finally, considering that the attention is focused on the high-frequency information area, it is used for the information of the smaller kernel, and the opposite weight is applied to the information of the larger kernel, so as to enhance the features of the context information and obtain the fused context feature information. The process is expressed as: In the formula, Represents the fused context feature information.

8. The stereo matching system based on bidirectional weighted path and frequency fusion according to claim 1, characterized in that: The iterative updating module is used to iteratively update the disparity value based on the matching cost volume and the fused context feature information, specifically including: Based on the matching cost volume and the fused context feature information, a dynamic updater based on a gated recurrent unit is used to obtain the disparity map, and the attention weight is used to fuse the hidden information. The iterative update process is implemented by the following formula: d t =Conv(ReLU(Conv(h t ))); In the formula, and They represent the gated recurrent unit operations using k1 and k2 convolution kernels respectively; C represents the matching cost volume; F C Represents context feature information; h t represents the hidden information at the tth iteration; d t represents the disparity of the t-th iteration; ⊙ represents the element-wise dot product; W represents the attention weight of nonlinear fusion; and They represent the hidden information after the gated recurrent unit operation with the k1 and k2 convolution kernels at the t+1th iteration respectively; h t+1 Represents the fused hidden information; Conv represents the convolution operation; ReLU represents the activation function.

9. The stereo matching system based on bidirectional weighted path and frequency fusion according to claim 1, characterized in that: In the process of iteratively updating the disparity value, the smooth L1 loss and the L1 loss are used to form a loss function to optimize the initial disparity and the disparity prediction in the iterative process. The process is expressed as: Where, L init represents the loss of the predicted initial disparity; d gt represents the true value disparity; d gt represents the initial disparity; represents smooth L1 loss; L iter represents the loss of all predictions generated by the iterative process; γ represents the penalty term; d i represents the predicted disparity at the i-th iteration; N represents the total number of iterations; ‖‖1 represents the L1 loss; L total Represents the total loss.

10. A stereo matching method based on bidirectional weighted path and frequency fusion, characterized in that: The method adopts the stereo matching system based on bidirectional weighted path and frequency fusion according to any one of claims 1 to 9, and specifically comprises: Use a feature extraction module to extract multi-scale features of the left view and the right view; Processing the multi-scale features through a bidirectional weighted path module to eliminate representation ambiguity and noise; Based on the processed multi-scale features, a matching cost volume is constructed using a matching cost volume construction module; Use the context extraction module to extract multi-scale context features of the left view; Using a nonlinear frequency fusion module to perform convolution operations with different kernel sizes on the multi-scale context features, obtain feature information of different receptive fields, and fuse the feature information to obtain fused context feature information; Based on the matching cost volume and the fused context feature information, the disparity value is iteratively updated by an iterative updating module to obtain a final disparity map.

Images