A method and apparatus for constructing a deep learning-based binocular stereo matching recursive network

Abstract

This invention relates to the field of computer vision technology, specifically providing a method and apparatus for constructing a deep learning-based binocular stereo matching recurrent network. In the feature extraction stage, an adaptive spatial pyramid pooling layer based on kernel-normalized convolutional layers is proposed to extract multi-scale features from the image. In the matching cost calculation stage, a novel attention method combined with regularized cost volume is used to generate a more accurate similarity measure. In the main network framework, a kernel-convolution-based gated recurrent unit (GRU) is used as an iterative update operator to retrieve features from the stereo matching cost volume and iteratively update the disparity. This allows the GRU to consider a broader context around each pixel, promoting the fusion of local and global information, providing a more comprehensive understanding of the feature space, and enhancing the network's reasoning ability regarding scene structures in larger regions.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, specifically providing a method and apparatus for constructing a deep learning-based binocular stereo matching recursive network. Background Technology

[0002] Stereo matching has always been a research hotspot in binocular vision. Binocular cameras capture two viewpoint images of the same scene from the left and right perspectives, and stereo matching algorithms are used to obtain disparity maps, which in turn yield depth maps. Depth maps have a wide range of applications, recording the distances of objects in a scene from the camera, and are widely used in robot navigation, smart healthcare, parameter detection in micro-operating systems, 3D measurement, and virtual reality. Traditional stereo matching algorithms generally consist of four steps: calculating matching costs, cost aggregation, disparity calculation, and disparity thinning. However, in open-world environments, traditional methods still struggle to handle various conditions such as image noise, uneven lighting, and large areas without texture. How to handle images in real-world scenes, especially outdoor images, requires specific consideration. In recent years, with the support of large synthetic datasets, stereo matching methods based on convolutional neural networks have significantly improved the accuracy of disparity estimation. However, due to various real-world challenges, achieving both high accuracy and efficiency is crucial for practical applications. Depth estimation for some special structures (such as thin objects and ill-conditioned regions with sharp edges) remains challenging.

[0003] In response to the above needs, there are currently many relevant solutions both domestically and internationally.

[0004] Chinese patent publication CN116245928A discloses a 3D reconstruction method based on binocular stereo matching, combining binocular stereo matching algorithms, triangulation algorithms, and surface texture mapping. After stereo calibration of the binocular camera to obtain its intrinsic and extrinsic parameters, stereo correction is performed. Then, an optimized semi-global stereo matching algorithm generates a disparity map of the corresponding scene objects. After obtaining the disparity information, the point cloud information of the object's disparity transformation is triangulated using a triangulation algorithm. Finally, texture mapping technology is used to reconstruct the 3D surface of the object. This invention uses a traditional semi-global stereo matching algorithm; however, in open-world environments, these traditional methods still cannot effectively handle various situations such as image noise, uneven lighting, and large areas without texture. Targeted considerations are needed for handling images in real-world scenes, especially outdoor images.

[0005] Chinese patent publication CN117635989A proposes a binocular stereo matching method based on an improved CFNet. It utilizes an optimized EfficientNetV2-M model as the feature extraction network to extract multi-scale feature maps corresponding to the original left and right images. Multi-scale cost volumes are constructed and fused using group correlation and concatenation. Guided cost volume excitation is added to a 3D convolutional module to guide cost aggregation. A cost self-reorganization strategy is used to redistribute the cost amount in the aggregated cost volume, followed by disparity regression to generate an initial disparity map. A cascaded cost volume is constructed and combined with the initial disparity map to refine the disparity in a coarse-to-fine manner, ultimately obtaining a refined disparity map. This method uses a 2D convolutional neural network to extract features to form a cost volume, which is then fed into a cost aggregation and regularization module composed of 2D or 3D convolutional neural networks for processing. Compared to iterative structures based on GRU, this method often ignores a large amount of high-frequency information, resulting in a relatively blurry disparity map lacking detail.

[0006] Similar to Chinese patent publication CN117808862A, Chinese patent publication CN117635989A also proposes a binocular stereo matching method based on convolutional neural networks. It uses depthwise separable convolutions to reduce the number of parameters in binocular stereo matching and improve computational speed, and employs an attention mechanism to enhance feature extraction and cost aggregation capabilities, further optimizing before disparity regression. Although the specific backend operations differ, they share the same problems.

[0007] Most current algorithms use 2D CNNs to extract features, which are then fed into a cost volume. This cost volume is then fed into a cost aggregation and regularization module composed of 2D or 3D CNNs. During this process, a large amount of high-frequency information is ignored, resulting in a relatively blurry disparity map lacking detail. Therefore, current methods perform poorly in recovering thin objects and sharp edges, and due to limited generalization ability, learning-based algorithms are often less effective on specific datasets than in real-world scenarios. Summary of the Invention

[0008] To address the aforementioned problems, this invention provides a method and apparatus for constructing a deep learning-based binocular stereo matching recursive network, which enhances the pixel-level representativeness of images and improves the overall generalization ability of the network.

[0009] This invention provides a method for constructing a deep learning-based binocular stereo matching recursive network, comprising:

[0010] An adaptive multi-scale feature extraction module is constructed, which includes a context encoder for extracting multi-scale context features and a feature encoder for the left and right images. The context encoder is composed of a series of residual blocks and downsampling layers based on the stereo matching network IGEV-Stereo. It is used to initialize the hidden state of the update operator and inject a gated recurrent unit (GRU) during each iteration of the update operator. The feature encoder uses multiple parallel convolutional layers with different sampling rates to extract features and completes feature fusion by combining kernel convolution normalization and adaptive bias.

[0011] A matching attention mechanism is constructed to build an initial cost body, generate matching attention weights on the initial cost body, and input the final stereo matching cost body into a GRU-based update operator for iterative update of the disparity map.

[0012] A kernel-based GRU is used as the iterative update operator to retrieve features from the stereo matching cost volume and update the disparity iteratively through regression.

[0013] The smoothing loss is used as the training loss function to calculate the smoothed L1 loss between the predicted value and the true value in the entire prediction sequence, and the exponentially weighted L1 distance is used as the loss function.

[0014] As a preferred approach, the feature fusion is achieved by combining kernel convolutional normalization and adaptive bias, including:

[0015] Adaptive bias steps:

[0016] Variable convolution is extended to multi-scale pyramid pooling layers, and a learnable bias offset is used to form an adaptive search window for multi-scale extraction of image features. This achieves adaptive sampling of the input feature map without increasing memory and computational costs. Specifically, in feature extraction, any point in the feature map y... Relative to the initial image x The mapping relationship between points is as follows:

[0017] (1);

[0018] Apply bias to convolution Then, the mapping relationship becomes:

[0019] (2);

[0020] Kernel convolution normalization feature fusion steps:

[0021] The normalized unit is randomly dropped to obtain the dropout unit. Then, the original normalized unit is normalized by calculating the mean and variance of the dropout unit. The hyperparameters include kernel size k, stride s, padding d, and dropout probability p.

[0022] (3);

[0023] (4);

[0024] in, To exit the operation, U is the original normalized unit. For those leaving the organization, and These are the mean and variance for dropout units, respectively. It is a newly normalized unit.

[0025] As a preferred embodiment, the construction of the matching attention mechanism involves constructing an initial cost volume, generating matching attention weights on the initial cost volume, and inputting the final stereo matching cost volume into a GRU-based update operator for iterative disparity map updates, including:

[0026] The initial cost volume is calculated using group correlation. Following the vector dot product approach, at one disparity level, the channels of the left and right feature matrices are evenly grouped along the channel dimension, and the vector dot product is calculated group by group. After the dot product of two four-dimensional matrices, the mean is calculated in the dimension with dim=2, resulting in a three-dimensional matrix. After traversing all disparity levels, the resulting matrices are split and concatenated to obtain the group correlation volume. If there are a total of Channels divide the matrices into If we consider a set of features, then the correlation between all feature groups g and all disparity levels d can be expressed as:

[0027] (5);

[0028] The initial 3D cost volume is traversed using a cross-attention process from both planar and spatial directions, and correlation calculations are performed. D The 3D cost bodies are respectively for Parallax-level attention in planar computation and for The epipolar attention of planar computation, after being refined by the MAR matching attention module, is used to input the cost volume into the GRU iterative update to further refine the disparity map, where W is the width, H is the height, and D is the depth.

[0029] As a preferred approach, the initial 3D cost volume is subjected to a cross-attention traversal from both planar and spatial directions, and the correlation is calculated. D The 3D cost bodies are respectively for Parallax-level attention in planar computation and for The epipolar attention in planar computation, after being refined by the MAR matching attention module, is used to input the cost volume into the GRU iterative update to further refine the disparity map, including:

[0030] At parallax level ( ) to win A plane is traversed using a cross-attention traversal, that is, for any spatial location within it... According to Calculate the position of the central cross relative to its row or column. and Relevant weights ,but Includes The mapping information corresponds to any point in this cross-shaped positional relationship; then, repeating the above operation once more, for any other position in space... It can always be calculated that it corresponds to Relevant weights At this point, the plane, after attention refinement, becomes:

[0031] (6);

[0032] Then calculate its arbitrary spatial position The attention-related weights for each view difference level are determined based on the epipolar constraint principle, since binocular images are usually left and right eye images. ( ) to win On a plane, repeat the above operation to obtain...

[0033] (7);

[0034] The matching attention formula can be further obtained as follows:

[0035] (8);

[0036] The final correlation can be expressed as:

[0037] (9).

[0038] As a preferred embodiment, the step of employing a kernel-based GRU as an iterative update operator to retrieve features from the stereo matching cost volume and iteratively update the disparity through regression includes:

[0039] The matching cost volume is regressed to the initial disparity using the soft argmin function. The disparity is updated through three GRU iterative update layers. Each GRU iterative update layer contains three kernel convolutions. The activation function is the sigmoid function. The relevance, disparity and context features are concatenated and injected into the GRU.

[0040] Update the hidden state and use the new hidden state to predict the parallax update.

[0041] As a preferred embodiment, the method of using smoothing loss as the training loss function to calculate the smoothed L1 loss between the predicted and true values ​​in the entire prediction sequence, and using exponentially weighted L1 distance as the loss function, includes:

[0042] Smoothing loss is used as the training loss function to calculate the smoothed L1 loss between the predicted and true values ​​throughout the entire prediction sequence, and exponentially weighted L1 distance is used as the loss function for all predicted disparities. Given ground truth parallax The loss function is defined as

[0043] (10);

[0044] , (11).

[0045] On the other hand, the present invention also provides a device for constructing a deep learning-based binocular stereo matching recursive network, comprising:

[0046] An adaptive multi-scale feature extraction module includes a context encoder for extracting multi-scale context features and a feature encoder for the left and right images. The context encoder is based on the IGEV-Stereo stereo matching network and consists of a series of residual blocks and downsampling layers. It is used to initialize the hidden state of the update operator and inject a gated recurrent unit (GRU) during each iteration of the update operator. The feature encoder uses multiple parallel convolutional layers with different sampling rates to extract features and combines kernel convolution normalization and adaptive bias to complete feature fusion.

[0047] The matching attention mechanism constructs an initial cost body, generates matching attention weights on the initial cost body, and inputs the final stereo matching cost body into a GRU-based update operator for iterative update of the disparity map.

[0048] An iterative update operator is used to retrieve features from the stereo matching cost volume by employing a kernel-based GRU as the iterative update operator and iteratively update the disparity through regression.

[0049] The loss function is used as the training loss function, which calculates the smoothed L1 loss between the predicted value and the true value in the entire prediction sequence, and uses the exponentially weighted L1 distance as the loss function.

[0050] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0051] This invention provides a method and apparatus for constructing a deep learning-based binocular stereo matching recurrent network. In the feature extraction stage, an adaptive spatial pyramid pooling layer based on kernel-normalized convolutional layers is proposed to extract multi-scale features from the image, fully considering the spatial correlation between pixels, further improving the performance of the deep network, extracting rich contextual features, enabling the network to better handle overexposure, underexposure, and weak texture regions, and enabling the network to learn error correction capabilities. In the matching cost calculation stage, a novel attention method (MAR) is used in conjunction with regularized cost volume to generate a more accurate similarity measure. This effectively utilizes the effective information of the binocular images and greatly reduces the difficulty for the network to find the correct matching point in the search space. In the main network framework, a kernel-convolution-based gated recurrent unit (GRU) is used as an iterative update operator to retrieve features from the stereo matching cost volume and iteratively update the disparity. This allows the GRU to consider a wider range of context around each pixel, promoting the fusion of local and global information, providing a more comprehensive understanding of the feature space, especially in challenging regions such as occlusion and homogeneous textures, enhancing the network's reasoning ability for scene structures in larger areas. Attached Figure Description

[0052] Figure 1 This is a flowchart of a method for constructing a deep learning-based binocular stereo matching recursive network according to an embodiment of the present invention.

[0053] Figure 2 This is a schematic diagram illustrating the composition of a method for constructing a deep learning-based binocular stereo matching recursive network according to an embodiment of the present invention.

[0054] Figure 3 This is a schematic diagram illustrating how the offset changes the formation of the feature extraction box in the construction method of the deep learning-based binocular stereo matching recursive network provided by an embodiment of the present invention.

[0055] Figure 4This is a schematic diagram of feature maps under different dilation rates in the method for constructing a deep learning-based binocular stereo matching recursive network according to an embodiment of the present invention.

[0056] Figure 5 This is a schematic diagram illustrating the application of cross-attention traversal to the initial 3D cost volume from both planar and spatial directions in the construction method of a deep learning-based binocular stereo matching recursive network provided by an embodiment of the present invention.

[0057] Figure 6 This is a structural block diagram of a device for constructing a deep learning-based binocular stereo matching recursive network according to an embodiment of the present invention. Detailed Implementation

[0058] In the following description, embodiments of the invention will be described with reference to the accompanying drawings. In the description below, the same modules are denoted by the same reference numerals. Where the same reference numerals are used, their names and functions are also the same. Therefore, their detailed description will not be repeated.

[0059] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and do not constitute a limitation thereof.

[0060] Combination Figure 1 As shown, this invention provides a method for constructing a deep learning-based binocular stereo matching recursive network, comprising:

[0061] S101. Construct an adaptive multi-scale feature extraction module. The adaptive multi-scale feature extraction module includes a context encoder for extracting multi-scale context features and a feature encoder for the left and right images. The context encoder is composed of a series of residual blocks and downsampling layers based on the stereo matching network IGEV-Stereo. It is used to initialize the hidden state of the update operator and inject a gated recurrent unit (GRU) during each iteration of the update operator. The feature encoder uses multiple parallel convolutional layers with different sampling rates to extract features and completes feature fusion by combining kernel convolution normalization and adaptive bias.

[0062] S102. Construct a matching attention mechanism, construct an initial cost body, generate matching attention weights for the initial cost body, and input the final stereo matching cost body into a GRU-based update operator for iterative update of the disparity map.

[0063] S103. A kernel-based GRU is used as the iterative update operator to retrieve features from the stereo matching cost volume and update the disparity iteratively through regression.

[0064] S104. Using smoothing loss as the training loss function, calculate the smoothing L1 loss between the predicted value and the true value in the entire prediction sequence, and use the exponentially weighted L1 distance as the loss function.

[0065] It should be noted that the order of S101 to S104 can be adjusted as needed, and there is no specific limitation.

[0066] Furthermore, the feature fusion process, which combines kernel convolutional normalization and adaptive bias, includes:

[0067] Adaptive bias steps:

[0068] Variable convolution is extended to multi-scale pyramid pooling layers, and a learnable bias offset is used to form an adaptive search window for multi-scale extraction of image features. This achieves adaptive sampling of the input feature map without increasing memory and computational costs. Specifically, in feature extraction, any point in the feature map y... Relative to the initial image x The mapping relationship between points is as follows:

[0069] (1);

[0070] Apply bias to convolution Then, the mapping relationship becomes:

[0071] (2);

[0072] Kernel convolution normalization feature fusion steps:

[0073] The normalized unit is randomly dropped to obtain the dropout unit. Then, the original normalized unit is normalized by calculating the mean and variance of the dropout unit. The hyperparameters include kernel size k, stride s, padding d, and dropout probability p.

[0074] (3);

[0075] (4);

[0076] in, To exit the operation, U is the original normalized unit. For those leaving the organization, and These are the mean and variance for dropout units, respectively. It is a newly normalized unit.

[0077] Furthermore, the construction of the matching attention mechanism, including the construction of an initial cost body, the generation of matching attention weights on the initial cost body, and the input of the final stereo matching cost body into a GRU-based update operator for iterative updating of the disparity map, includes:

[0078] The initial cost volume is calculated using group correlation. Following the vector dot product approach, at one disparity level, the channels of the left and right feature matrices are evenly grouped along the channel dimension, and the vector dot product is calculated group by group. After the dot product of two four-dimensional matrices, the mean is calculated in the dimension with dim=2, resulting in a three-dimensional matrix. After traversing all disparity levels, the resulting matrices are split and concatenated to obtain the group correlation volume. If there are a total of Channels divide the matrices into If we consider a set of features, then the correlation between all feature groups g and all disparity levels d can be expressed as:

[0079] (5);

[0080] The initial 3D cost volume is traversed using a cross-attention process from both planar and spatial directions, and correlation calculations are performed. D The 3D cost bodies are respectively for Parallax-level attention in planar computation and for The epipolar attention of planar computation, after being refined by the MAR matching attention module, is used to input the cost volume into the GRU iterative update to further refine the disparity map, where W is the width, H is the height, and D is the depth.

[0081] Furthermore, the initial 3D cost volume is subjected to a cross-attention traversal from both planar and spatial directions, and the correlation calculation is performed. D The 3D cost bodies are respectively for Parallax-level attention in planar computation and for The epipolar attention in planar computation, after being refined by the MAR matching attention module, is used to input the cost volume into the GRU iterative update to further refine the disparity map, including:

[0082] At parallax level ( ) to win A plane is traversed using a cross-attention traversal, that is, for any spatial location within it... According to Calculate the position of the central cross relative to its row or column. and Relevant weights ,but Includes The mapping information corresponds to any point in this cross-shaped positional relationship; then, repeating the above operation once more, for any other position in space... It can always be calculated that it corresponds to Relevant weights At this point, the plane, after attention refinement, becomes:

[0083] (6);

[0084] Then calculate its arbitrary spatial position The attention-related weights for each view difference level are determined based on the epipolar constraint principle, since binocular images are usually left and right eye images. ( ) to win On a plane, repeat the above operation to obtain...

[0085] (7);

[0086] The matching attention formula can be further obtained as follows:

[0087] (8);

[0088] The final correlation can be expressed as:

[0089] (9).

[0090] Furthermore, the step of using a kernel-based GRU as an iterative update operator to retrieve features from the stereo matching cost volume and iteratively update the disparity through regression includes:

[0091] The matching cost volume is regressed to the initial disparity using the soft argmin function. The disparity is updated through three GRU iterative update layers. Each GRU iterative update layer contains three kernel convolutions. The activation function is the sigmoid function. The relevance, disparity and context features are concatenated and injected into the GRU.

[0092] Update the hidden state and use the new hidden state to predict the parallax update.

[0093] Furthermore, the step of using smoothing loss as the training loss function to calculate the smoothed L1 loss between the predicted and true values ​​in the entire prediction sequence, and using exponentially weighted L1 distance as the loss function, includes:

[0094] Smoothing loss is used as the training loss function to calculate the smoothed L1 loss between the predicted and true values ​​throughout the entire prediction sequence, and exponentially weighted L1 distance is used as the loss function for all predicted disparities. Given ground truth parallax The loss function is defined as

[0095] (10);

[0096] , (11).

[0097] Combination Figure 2 As shown, in order to better understand the construction method of the deep learning-based binocular stereo matching recursive network provided by the present invention, another embodiment will be described below.

[0098] The construction method of the deep learning-based binocular stereo matching recursive network mainly includes an adaptive multi-scale feature extraction module based on kernel-normalized convolutional layers, a matching cost calculation module based on a matching attention mechanism using rotational cross calculation, an iterative update operator based on kernel convolution, and finally, a loss function. Specifically:

[0099] (1) Feature Extraction Module: An adaptive multi-scale feature extraction module based on kernel-normalized convolutional layers. The feature extractor consists of two parts: a context encoder for extracting multi-scale context features and a feature encoder for the left and right images. The context encoder, based on IGEV-Stereo (stereo matching network), consists of a series of residual blocks and downsampling layers, used to initialize the hidden state of the update operator and inject GRUs (gated recurrent units) during each iteration of the update operator. The feature encoder, inspired by the successful spatial pyramid pooling method in image segmentation, utilizes dilated convolutions and multiple parallel convolutional layers with different sampling rates to extract features. It combines kernel convolution normalization and adaptive bias to implement a variant of this scheme, namely the KMP module.

[0100] (2) Matching attention mechanism: A matching cost calculation refinement module based on the rotating cross-computation matching attention mechanism. The initial 3D cost volume is traversed by cross-interference from both planar and spatial directions, which reduces computational complexity and memory usage while enhancing pixel-level representativeness.

[0101] (3) Iterative update operator: GRU iterative update operator based on kernel convolution. The gated recurrent unit (GRU) based on kernel convolution is used as the iterative update operator to retrieve features from the stereo matching cost volume obtained above and iteratively update the disparity.

[0102] (4) Loss function: We use smoothing loss as our training loss function. We calculate the smoothing L1 loss between the predicted and true values ​​in the entire prediction sequence and use an exponentially weighted L1 distance similar to IGEV as the loss function.

[0103] First, let's introduce the feature extraction part of the front end:

[0104] Combination Figure 2 As shown, the feature extraction module consists of two parts: a context encoder for extracting multi-scale context features and a feature encoder for the left and right images.

[0105] The context encoder, based on the IGEV-Stereo (stereo matching network), consists of a series of residual blocks and downsampling layers, generating multi-scale contextual features at 1 / 4, 1 / 8, and 1 / 16 of the resolution of the 128-channel input image. These contextual features are used to initialize the hidden states of the update operator and are injected into the GRU during each iteration of the update operator.

[0106] In the feature encoder section, dilated convolutions can be used to extract features using multiple parallel convolutional layers with different sampling rates. These features are then further processed and fused to produce the final result. This approach combines semantic information from receptive fields of different sizes without losing resolution (without downsampling), improving the accuracy of feature extraction. To further enhance object pixel-level detail in tasks like stereo matching with dense pixel density, a variant of this dilated convolution scheme is implemented by combining kernel convolution normalization and adaptive bias. Specifically, it includes the following two steps:

[0107] (a) Adaptive bias:

[0108] Combination Figure 3 As shown, to address the issue of dilated convolution losing dense object features, and to obtain dense global contextual information to establish dependencies between pixels, we extend variable convolution to multi-scale pyramid pooling layers. We use learnable bias offsets to form an adaptive search window for multi-scale image feature extraction, achieving adaptive sampling of the input feature map without increasing memory or computational costs. This allows the network to dynamically adjust its receptive field based on the object's shape and orientation, thereby capturing more relevant and discriminative features. Figure 3 The diagram illustrates how the offset alters the formation of the feature extraction box.

[0109] Figure 3 middle, Figure 3 In the middle (a), the sampling search window for feature extraction with a normal inflation rate of 3 is shown. Figure 3 (b) shows the position that the sampling point should be offset after adding the bias offset; Figure 3 In the middle (c), the more accurate sampling search window is obtained after offsetting.

[0110] Typically, in feature extraction, any point in the feature map y... Relative to the initial image x The mapping relationship between points is as follows:

[0111] (1)

[0112] Apply bias to convolution Then, the mapping relationship becomes:

[0113] (2)

[0114] Combination Figure 4 As shown, firstly, four parallel adaptive convolutional layers with different dilation rates (0, 3, 4, and 5) are used to generate more refined and comprehensive multi-scale features. Then, the extracted feature maps containing features at different scales are fused. To save costs and without losing information, only three feature maps obtained from convolutional layers with different dilation rates are selected for fusion each time. This process is repeated three times, and finally, the results are concatenated using residuals to obtain the final feature map. The adaptive bias is also learned end-to-end during training to ensure that the network optimally adapts to the shape and scale of the target object, thereby generating more accurate and detailed feature maps. The structure is as follows: Figure 4 As shown.

[0115] Figure 4 middle, Figure 4 In the middle (a), the adaptive search window has a 3*3 kernel and an inflation rate of 0. Figure 4 In the middle (b), the adaptive search window has a 3*3 kernel and an inflation rate of 3. Figure 4 In the middle (c), the adaptive search window has a 3*3 kernel and an inflation rate of 4. Figure 4 In the middle (d), the adaptive search window has a 3*3 kernel and an inflation rate of 5. Figure 4 In (e), the convolutional layers a, b, c, b, c, d, a, b, d, and a, c, d are respectively taken, and the features of these three layers are fused into four feature maps. Then, a residual connection operation is performed to obtain the final feature map.

[0116] (b) Kernel convolution normalized feature fusion:

[0117] Extensive experiments revealed that in the final multi-scale feature fusion stage of feature extraction, the effectiveness of BatchNorm (batch normalization) is significantly reduced when environmental memory is limited and only small batches of data can be used for training. To address this, this solution replaces batch normalization with kernel normalization (KernelNorm). Kernel normalization first applies random dropout to the normalized units to obtain dropout units, and then normalizes the original normalized units by calculating the mean and variance of the dropout units. Hyperparameters include kernel size k, stride s, padding d, and dropout probability p.

[0118] (3);

[0119] (4).

[0120] in To exit the operation, U is the original normalized unit. For those leaving the organization, and These are the mean and variance for dropout units, respectively. It is a new normalized unit.

[0121] Applying kernel normalization to each convolutional layer in multi-scale feature extraction does not rely on the statistical information of the entire batch. This makes it focus more on local information rather than global information in the feature space, and also improves the dependence of multi-scale feature extraction on batch size, which is crucial for stereo matching tasks involving images of different scales and under different conditions. Furthermore, by normalizing the kernel, the model is less prone to overfitting to specific scales or features, thereby improving its generalization ability to unseen data.

[0122] (2) Part of the matching cost calculation refinement module based on the rotating cross calculation matching attention mechanism.

[0123] The matching cost calculation stage mainly includes three steps: constructing the initial cost volume, generating matching attention weights, and inputting the final cost volume into a GRU-based update operator for iterative updating of the disparity map.

[0124] The initial cost volume is typically calculated using group-correlation, which involves uniformly grouping the channels of the left and right feature matrices along the channel dimension at a disparity level using a vector inner product approach. Then, the inner product of each group is calculated. After the inner product of two four-dimensional matrices, the mean is calculated in the dimension with dim=2, resulting in a three-dimensional matrix. After traversing all disparity levels, the resulting matrices are split and concatenated to obtain the group-correlation volume. If there are a total of Channels, dividing them into If we consider a set of features, then the correlation between all feature groups g and all disparity levels d can be expressed as:

[0125] (5).

[0126] Combination Figure 5 In order to filter the initial relevance cost volume to emphasize useful information, suppress irrelevant information, and improve the accuracy and robustness of cost volume calculation in stereo matching, this invention proposes a new hybrid attention cost volume refinement method. The method applies cross-attention traversal to the initial 3D cost volume from both planar and spatial directions, thereby reducing computational complexity and memory usage while enhancing pixel-level representativeness.

[0127] Figure 5 In the middle, the left side shows the result after correlation calculation. D The 3D cost body, the middle part is for Parallax-level attention in planar computation and for The epipolar attention of the planar computation is finally refined into a cost volume after being processed by the MAR matching attention module. This cost volume will then be input into the GRU iterative update to further refine the disparity map.

[0128] First, at the parallax level ( ) to win A plane is traversed using a cross-attention traversal, that is, for any spatial location within it... According to Calculate the position of the central cross relative to its row or column. and Relevant weights ,but Includes The mapping information corresponds to any point in this cross-shaped positional relationship; then, repeating the above operation once more, for any other position in space... It can always be calculated that it corresponds to Relevant weights At this point, this plane, after attention refinement, becomes...

[0129] (6)

[0130] Then, calculate its arbitrary spatial position. The attention-related weights for each view difference level are determined based on the epipolar constraint principle, since binocular images are usually left and right eye images. ( ) to win On a plane, repeat the above operation to obtain...

[0131] (7)

[0132] Finally, the matching attention formula can be obtained as follows:

[0133] (8)

[0134] The final correlation can be expressed as:

[0135] (9)

[0136] The matching attention calculation method used in this invention enables the network to learn different features of the input data more effectively, while taking into account the interdependence between channel and global information. This effectively utilizes the valuable information from the stereo images and significantly reduces the difficulty for the network to find the correct matching point in the search space.

[0137] (3) GRU iterative update operator based on kernel convolution

[0138] While traditional GRUs are well-suited for processing sequential data, they have limitations in directly capturing spatial dependencies. This invention employs a kernel-convolution-based gated recurrent unit (GRU) as an iterative update operator within the main network framework. Features are retrieved from the stereo matching cost volume obtained earlier, and the disparity is iteratively updated using regression.

[0139] First, the matching cost volume is regressed to the initial disparity using the soft argmin function. Disparity is then updated through three GRU iterative update layers, each containing three kernel convolutions. The sigmoid function is used as the activation function, and relevance, dissimilarity, and contextual features are concatenated and injected into the GRU. The hidden state is then updated, and the new hidden state is used to predict the disparity update. During training, all stages of the disparity iterative update share the same weights, thus requiring no fine-tuning.

[0140] The input-to-hidden and hidden-to-hidden transformations in GRU utilize kernel convolutions to ensure spatial information is preserved and propagated effectively through recurrent connections. By integrating kernel convolutions into GRU, the network can process both spatial and temporal information simultaneously, enabling GRU to consider a broader context around each pixel, facilitating the fusion of local and global information, providing a more comprehensive understanding of the feature space, especially in challenging regions such as occlusion and homogeneous textures, and enhancing the network's ability to infer scene structures over larger areas.

[0141] (4) Loss function

[0142] Since smoothing loss is considered robust to disparity discontinuities and outliers, we use smoothing loss as our training loss function. We compute the smoothed L1 loss over the difference between predicted and true values ​​throughout the entire prediction sequence, using an exponentially weighted L1 distance similar to IGEV as the loss function for all predicted disparities. Given ground truth parallax The loss function is defined as

[0143] (10);

[0144] , (11).

[0145] This invention also provides a method for constructing a deep learning-based binocular stereo matching recursive network. In the feature extraction stage, an adaptive spatial pyramid pooling layer based on kernel-normalized convolutional layers is proposed to extract multi-scale features from the image, fully considering the spatial correlation between pixels, further improving the performance of the deep network, extracting rich contextual features, enabling the network to better handle overexposure, underexposure, and weak texture regions, and enabling the network to learn error correction capabilities. In the matching cost calculation stage, a novel attention method (MAR) is used in conjunction with regularized cost volume to generate a more accurate similarity measure. This effectively utilizes the effective information of the binocular images and greatly reduces the difficulty for the network to find the correct matching point in the search space. In the main network framework, a kernel-convolution-based gated recurrent unit (GRU) is used as the iterative update operator to retrieve features from the stereo matching cost volume and iteratively update the disparity. This enables GRU to consider a broader context around each pixel, facilitating the fusion of local and global information, providing a more comprehensive understanding of the feature space, especially in challenging regions such as occlusion and homogeneous textures. It enhances the network's ability to infer scene structures over larger areas, improves the pixel-level representation of images, and enhances the network's overall generalization ability.

[0146] Combination Figure 6 As shown, correspondingly, the present invention also provides a device for constructing a deep learning-based binocular stereo matching recursive network, comprising:

[0147] An adaptive multi-scale feature extraction module 601 includes a context encoder for extracting multi-scale context features and a feature encoder for the left and right images. The context encoder is composed of a series of residual blocks and downsampling layers based on the stereo matching network IGEV-Stereo. It is used to initialize the hidden state of the update operator and inject a gated recurrent unit (GRU) during each iteration of the update operator. The feature encoder uses multiple parallel convolutional layers with different sampling rates to extract features and combines kernel convolution normalization and adaptive bias to complete feature fusion.

[0148] The matching attention mechanism 602 constructs an initial cost body, generates matching attention weights on the initial cost body, and inputs the final stereo matching cost body into a GRU-based update operator for iterative update of the disparity map.

[0149] Iterative update operator 603 is used to retrieve features from the stereo matching cost volume by employing a kernel convolution-based GRU as an iterative update operator and iteratively update the disparity through regression.

[0150] Loss function 604 is used to calculate the smoothed L1 loss between the predicted and true values ​​in the entire prediction sequence using smoothed loss as the training loss function, and uses exponentially weighted L1 distance as the loss function.

[0151] This invention also provides a construction device for a deep learning-based binocular stereo matching recurrent network. In the feature extraction stage, an adaptive spatial pyramid pooling layer based on kernel-normalized convolutional layers is proposed to extract multi-scale features from the image, fully considering the spatial correlation between pixels, further improving the performance of the deep network, extracting rich contextual features, enabling the network to better handle overexposure, underexposure, and weak texture regions, and enabling the network to learn error correction capabilities. In the matching cost calculation stage, a novel attention method (MAR) is used in conjunction with regularized cost volume to generate a more accurate similarity measure. This effectively utilizes the effective information of the binocular images and greatly reduces the difficulty for the network to find the correct matching point in the search space. In the main network framework, a kernel-convolution-based gated recurrent unit (GRU) is used as an iterative update operator to retrieve features from the stereo matching cost volume and iteratively update the disparity. This allows the GRU to consider a wider range of contexts around each pixel, promoting the fusion of local and global information, providing a more comprehensive understanding of the feature space, especially in challenging regions such as occlusion and homogeneous textures, enhancing the network's reasoning ability for scene structures in larger areas.

[0152] Although embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of the present invention.

[0153] The specific embodiments of the present invention described above do not constitute a limitation on the scope of protection of the present invention. Any other corresponding changes and modifications made in accordance with the technical concept of the present invention should be included within the scope of protection of the claims of the present invention.

Claims

1. A method for constructing a deep learning-based binocular stereo matching recursive network, characterized in that, include: An adaptive multi-scale feature extraction module is constructed, which includes a context encoder for extracting multi-scale context features and a feature encoder for the left and right images. The context encoder is composed of a series of residual blocks and downsampling layers based on the stereo matching network IGEV-Stereo, used to initialize the hidden state of the update operator, and injects a gated recurrent unit (GRU) during each iteration of the update operator. The feature encoder uses multiple parallel convolutional layers with different sampling rates to extract features, and combines kernel convolution normalization and adaptive bias to complete feature fusion. A matching attention mechanism is constructed to build an initial cost body, generate matching attention weights on the initial cost body, and input the final stereo matching cost body into a GRU-based update operator for iterative update of the disparity map. The construction of the matching attention mechanism involves constructing an initial cost volume, generating matching attention weights on the initial cost volume, and inputting the final stereo matching cost volume into a GRU-based update operator for iterative disparity map updates. This includes: The initial cost volume is calculated using group correlation. Following the vector dot product approach, at one disparity level, the channels of the left and right feature matrices are evenly grouped along the channel dimension, and the vector dot product is calculated group by group. After the dot product of two four-dimensional matrices, the mean is calculated in the dimension with dim=2, resulting in a three-dimensional matrix. After traversing all disparity levels, the resulting matrices are split and concatenated to obtain the group correlation volume. If there are a total of Channels divide the matrices into If we consider a set of features, then the correlation between all feature groups g and all disparity levels d can be expressed as: （5）； The initial 3D cost volume is traversed using cross-attention from both planar and spatial directions, and correlation calculations are performed. D The 3D cost bodies are respectively for Parallax-level attention in planar computation and for The epipolar attention of planar computation, after being refined by the MAR matching attention module, is used to input the cost volume into the GRU iterative update to further refine the disparity map, where W is the width, H is the height, and D is the depth. The initial 3D cost volume is traversed using a cross-attention approach from both planar and spatial directions, and correlation calculations are performed. D The 3D cost bodies are respectively for Parallax-level attention in planar computation and for The epipolar attention in planar computation, after being refined by the MAR matching attention module, is used to input the cost volume into the GRU iterative update to further refine the disparity map, including: At parallax level ( ) to win A plane is traversed using a cross-attention traversal, that is, for any spatial location within it... According to Calculate the position of the central cross relative to its row or column. and Relevant weights ,but Includes The mapping information corresponds to any point in this cross-shaped positional relationship; then, repeating the above operation once more, for any other position in space... It can always be calculated that it corresponds to Relevant weights At this point, the plane, after attention refinement, becomes: （6）； Then calculate its arbitrary spatial position The attention-related weights for each view difference level are determined based on the epipolar constraint principle, since binocular images are usually left and right eye images. ( ) to win On a plane, repeat the above operation to obtain... （7）； The matching attention formula can be further obtained as follows: （8）； The final correlation can be expressed as: （9）； A kernel-based GRU is used as the iterative update operator to retrieve features from the stereo matching cost volume and update the disparity iteratively through regression. The smoothing loss is used as the training loss function to calculate the smoothed L1 loss between the predicted value and the true value in the entire prediction sequence, and the exponentially weighted L1 distance is used as the loss function.

2. The method for constructing a deep learning-based binocular stereo matching recursive network as described in claim 1, characterized in that, The feature fusion process, which combines kernel convolutional normalization and adaptive bias, includes: Adaptive bias steps: Variable convolution is extended to multi-scale pyramid pooling layers, and a learnable bias offset is used to form an adaptive search window for multi-scale extraction of image features. This achieves adaptive sampling of the input feature map without increasing memory and computational costs. Specifically, in feature extraction, any point in the feature map y... Relative to the initial image x The mapping relationship between points is as follows: （1）； Apply bias to convolution Then, the mapping relationship becomes: （2）； Kernel convolution normalization feature fusion steps: The normalized unit is randomly dropped to obtain the dropout unit. Then, the original normalized unit is normalized by calculating the mean and variance of the dropout unit. The hyperparameters include kernel size k, stride s, padding d, and dropout probability p. （3）； （4）； in, To exit the operation, U is the original normalized unit. For those leaving the organization, and These are the mean and variance for dropout units, respectively. It is a newly normalized unit.

3. The method for constructing a deep learning-based binocular stereo matching recursive network as described in claim 1, characterized in that, The method employs a kernel-based GRU as the iterative update operator to retrieve features from the stereo matching cost volume and iteratively update the disparity through regression, including: The matching cost volume is regressed to the initial disparity using the soft argmin function. The disparity is updated through three GRU iterative update layers. Each GRU iterative update layer contains three kernel convolutions. The activation function is the sigmoid function. The relevance, disparity and context features are concatenated and injected into the GRU. Update the hidden state and use the new hidden state to predict the parallax update.

4. The method for constructing a deep learning-based binocular stereo matching recursive network as described in claim 1, characterized in that, The method of using smoothing loss as the training loss function to calculate the smoothed L1 loss between the predicted and true values ​​in the entire prediction sequence, and using exponentially weighted L1 distance as the loss function, includes: Smoothing loss is used as the training loss function to calculate the smoothed L1 loss between the predicted and true values ​​throughout the entire prediction sequence, and exponentially weighted L1 distance is used as the loss function for all predicted disparities. Given ground truth parallax The loss function is defined as: （10）； ， （11）。 5. A device for constructing a deep learning-based binocular stereo matching recursive network, used to implement the method for constructing a deep learning-based binocular stereo matching recursive network as described in any one of claims 1-4, characterized in that, include: An adaptive multi-scale feature extraction module includes a context encoder for extracting multi-scale context features and a feature encoder for the left and right images. The context encoder is based on the IGEV-Stereo stereo matching network and consists of a series of residual blocks and downsampling layers. It is used to initialize the hidden state of the update operator and inject a gated recurrent unit (GRU) during each iteration of the update operator. The feature encoder uses multiple parallel convolutional layers with different sampling rates to extract features and combines kernel convolution normalization and adaptive bias to complete feature fusion. The matching attention mechanism constructs an initial cost body, generates matching attention weights on the initial cost body, and inputs the final stereo matching cost body into a GRU-based update operator for iterative update of the disparity map. An iterative update operator is used to retrieve features from the stereo matching cost volume by employing a kernel-based GRU as the iterative update operator and iteratively update the disparity through regression. The loss function is used as the training loss function, which calculates the smoothed L1 loss between the predicted value and the true value in the entire prediction sequence, and uses the exponentially weighted L1 distance as the loss function.

Images