Construction method and device of binocular stereo matching recursive network based on deep learning

Abstract

The invention relates to the technical field of computer vision, in particular to a method and a device for constructing a binocular stereo matching recursive network based on deep learning, in a feature extraction stage, a self-adaptive spatial pyramid pooling layer based on a kernel normalization convolution layer is provided, multi-scale feature extraction is performed on an image, and in a matching cost calculation stage, a self-adaptive spatial pyramid pooling layer based on a kernel normalization convolution layer is provided. A new attention approach is utilized in combination with a regularized cost volume to produce a more accurate similarity measure. And in the main network framework, a GRU (Gated Recirculation Unit) based on kernel convolution is adopted as an iterative update operator, features are retrieved from the stereo matching cost body, and parallax is updated through regression iteration. The GRU can consider the wider context around each pixel, the fusion of local and global information is promoted, the more comprehensive understanding of the feature space is provided, and the reasoning ability of the network for the scene structure of a larger area is enhanced.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and specifically provides a method and device for constructing a binocular stereo matching recursive network based on deep learning. Background Art

[0002] Binocular stereo matching has always been a hot topic in binocular vision research. The binocular camera takes the left and right viewpoint images of the same scene, and uses the stereo matching algorithm to obtain the disparity map, and then obtains the depth map. The application range of depth map is very wide. Since it can record the distance between the object in the scene and the camera, it has a wide range of applications in robot navigation, smart medical care, parameter detection of micro-operating systems, three-dimensional measurement and virtual reality. Traditional stereo matching algorithms are generally divided into four steps: calculating matching cost, cost aggregation, disparity calculation and disparity refinement. However, in the open world environment, traditional methods still cannot handle various situations such as image noise, uneven lighting, and large areas without texture. How to deal with images in actual scenes, especially outdoor images, also requires targeted consideration. In recent years, with the support of large synthetic datasets, stereo matching methods based on convolutional neural networks have brought the accuracy of disparity estimation to a new level. However, due to various real-world problems, how to achieve high accuracy and efficiency at the same time is crucial to practical applications. Depth estimation for some special structures (such as pathological areas such as thin objects and sharp edges) is still challenging.

[0003] In response to the above needs, there are currently many relevant solutions at home and abroad.

[0004] Chinese patent publication CN116245928A provides a three-dimensional reconstruction method based on binocular stereo matching, which combines binocular stereo matching algorithm, triangulation algorithm and surface texture mapping. Stereo calibration is performed after obtaining the internal and external parameters of the binocular camera by stereo calibration of the binocular camera, and then the image disparity map of the corresponding scene object is generated according to the optimized semi-global stereo matching algorithm. After obtaining the disparity information, the point cloud information of the object parallax conversion is triangulated according to the triangulation algorithm, and finally the three-dimensional surface reconstruction of the object is realized through texture mapping technology. The invention uses the traditional semi-global stereo matching algorithm. However, in the open world environment, the above traditional methods still cannot handle various situations such as image noise, uneven lighting, and large areas without texture well. How to deal with images in actual scenes, especially outdoor images, also requires targeted consideration.

[0005] Chinese patent publication CN117635989A proposes a binocular stereo matching method based on improved CFNet, using the optimized EfficientNetV2-M model as the feature extraction network to extract the multi-scale feature maps corresponding to the original left and right images; constructing and merging multi-scale cost volumes by group correlation and concatenation; adding guided cost volume excitation to the 3D convolution module to guide cost aggregation; using the cost self-reorganization strategy to redistribute the cost amount in the aggregated cost volume, and then using disparity regression to generate the initial disparity map; constructing a cascaded cost volume and combining it with the initial disparity map to refine the disparity in a coarse-to-fine manner, and finally obtaining a refined disparity map. The 2D convolutional neural network is used to extract features to form a cost body, which is then fed into the following cost aggregation and regularization module composed of 2D or 3D convolutional neural networks for processing. Compared with the iterative structure based on GRU, this method often ignores a large amount of high-frequency information, resulting in a relatively blurred disparity map and lack of details.

[0006] Chinese patent publication CN117808862A is similar to Chinese patent publication CN117635989A, and also proposes a binocular stereo matching method based on convolutional neural network, using deep separable convolution to reduce the number of binocular stereo matching parameters and increase the speed of calculation, and using attention mechanism to improve the ability of feature extraction and cost aggregation, and further optimize before the disparity value is regressed. Although the specific backend operations are different, the same problems exist.

[0007] Most current algorithms use 2D CNN to extract features and then form a cost volume, which is fed into the following cost aggregation and regularization module composed of 2D or 3D CNN. In this process, a large amount of high-frequency information is ignored, resulting in relatively blurry disparity maps and lack of details. Therefore, current methods perform poorly in recovering thin objects and sharp edges, and due to limited generalization capabilities, learning-based algorithms are often not as effective when applied to real-world scenarios as on specific datasets. Summary of the invention

[0008] In order to solve the above problems, the present invention provides a method and device for constructing a binocular stereo matching recursive network based on deep learning, which enhances the pixel-level representation capability of the image and improves the overall generalization capability of the network.

[0009] The present invention provides a method for constructing a binocular stereo matching recursive network based on deep learning, comprising: Constructing an adaptive multi-scale feature extraction module, the adaptive multi-scale feature extraction module includes a context encoder for extracting multi-scale context features and a feature encoder for left and right images, the context encoder is composed of a series of residual blocks and downsampling layers according to the stereo matching network IGEV-Stereo, and is used to initialize the hidden state of the update operator and inject a gated recurrent unit GRU during each iteration of the update operator, the feature encoder uses a plurality of parallel convolutional layers with different sampling rates to extract features, and completes feature fusion after combining kernel convolution normalization and adaptive bias; Constructing a matching attention mechanism, constructing an initial cost body, generating matching attention weights for the initial cost body, and inputting the resulting stereo matching cost body into a GRU-based update operator to iteratively update the disparity map; The kernel convolution-based GRU is used as the iterative update operator to retrieve features from the stereo matching cost volume and update the disparity through regression iteration; Using smooth loss as the training loss function, the smoothed L1 loss between the predicted value and the true value in the entire prediction sequence is calculated, and the exponentially weighted L1 distance is used as the loss function.

[0010] As a preferred solution, the feature fusion is completed after combining kernel convolution normalization and adaptive bias, including: Adaptive bias steps: The variable convolution is extended to the multi-scale pyramid pooling layer, and the learnable offset is used to form an adaptive search window to extract image features at multiple scales. The adaptive sampling of the input feature map is achieved without increasing the memory and computational costs. In the feature extraction, any point in the feature map y Relative to the initial image x The mapping relationship of the points is: (1); Applying bias to convolution After that, the mapping relationship becomes: (2); Kernel convolution normalization feature fusion steps: Apply random dropout to the normalized unit to get the dropout unit, and then normalize the original normalized unit by calculating the mean and variance of the dropout unit. The hyperparameters include kernel size k, stride s, padding d, and dropout probability p. (3); (4); in, is the exit operation, U is the original normalized unit, To exit the unit, and are the mean and variance of the dropout unit, is the new normalized unit.

[0011] As a preferred solution, the construction of the matching attention mechanism, the construction of the initial cost body, the generation of matching attention weights for the initial cost body, and the input of the finally obtained stereo matching cost body into the GRU-based update operator for iterative update of the disparity map include: The initial cost volume is calculated using group correlation. According to the vector inner product method, at a disparity level, the channels of the left and right feature matrices are evenly grouped along the channel dimension, and the vector inner product is calculated group by group. After the two four-dimensional matrices are inner-producted, the mean is calculated at the dimension of dim=2 to obtain a three-dimensional matrix. After traversing all disparity levels, the matrices are split and spliced ​​to obtain the group-corr volume. If there are Channels, dividing each matrix into group, then the correlation between all feature groups g and all disparity levels d can be expressed as: (5); The initial 3D cost volume is subjected to cross attention traversal from both the plane and space directions, and the correlation is calculated. D The 3D cost volume is for Parallax-level attention for planar computation and The epipolar level attention of the plane calculation is refined by the MAR matching attention module, and the cost body is input into the GRU iterative update to further refine the disparity map, where ‌W is the width, ‌H is the height, and ‌D is the depth.

[0012] As a preferred solution, the initial 3D cost volume is subjected to cross attention traversal from both the plane and space directions, and the correlation calculated D The 3D cost volume is for Parallax-level attention for planar computation and The epipolar level attention of the plane calculation is refined by the MAR matching attention module, and the cost volume is input into the GRU iterative update to further refine the disparity map, including: At parallax level ( ) to win plane, apply cross attention traversal, that is, for any spatial position , according to Calculate the position of the center cross relative to the same row or column and The relevant weight ,but Included Corresponding to the mutual mapping information of any point in this cross position relationship; then repeat the above operation once, then for any other position in the space , we can always calculate its corresponding The relevant weight At this point, this plane is refined by attention and is obtained as follows: (6); Then calculate its arbitrary position in space The attention-related weights at each parallax level are as follows: Since binocular images are usually left and right eye images, according to the epipolar constraint principle, ( ) to win Repeat the above operation to get (7); The matching attention formula can be further obtained as: (8); The final correlation can be expressed as: (9).

[0013] As a preferred solution, the kernel convolution-based GRU is used as an iterative update operator to retrieve features from the stereo matching cost volume and update the disparity through regression iteration, including: Use the soft argmin function to regress the matching cost volume to the initial disparity, and update the disparity through three GRU iterative update layers. Each GRU iterative update layer contains three layers of kernel convolution. The activation function uses the sigmoid function to connect the correlation, difference and context features and inject them into the GRU. Update the hidden state, using the new hidden state to predict the disparity update.

[0014] As a preferred solution, the smooth loss is used as the training loss function, the smooth L1 loss between the predicted value and the true value in the entire prediction sequence is calculated, and the exponentially weighted L1 distance is used as the loss function, including: Use smooth loss as the training loss function, calculate the smooth L1 loss between the predicted value and the true value in the entire prediction sequence, and use exponentially weighted L1 distance as the loss function, for all predicted disparities , given the ground truth disparity , the loss function is defined as (10); , (11).

[0015] On the other hand, the present invention also provides a device for constructing a binocular stereo matching recursive network based on deep learning, comprising: An adaptive multi-scale feature extraction module, wherein the adaptive multi-scale feature extraction module includes a context encoder for extracting multi-scale context features and a feature encoder for left and right images, wherein the context encoder is composed of a series of residual blocks and downsampling layers according to a stereo matching network IGEV-Stereo, and is used to initialize the hidden state of the update operator and inject a gated recurrent unit GRU during each iteration of the update operator, wherein the feature encoder extracts features using a plurality of parallel convolutional layers with different sampling rates, and completes feature fusion after combining kernel convolution normalization and adaptive bias; A matching attention mechanism, wherein the matching attention mechanism constructs an initial cost body, generates matching attention weights for the initial cost body, and inputs the resulting stereo matching cost body into a GRU-based update operator to iteratively update the disparity map; An iterative update operator, which uses a kernel convolution-based GRU as an iterative update operator to retrieve features from the stereo matching cost volume and iteratively update the disparity through regression; The loss function is used to use the smooth loss as the training loss function, calculate the smooth L1 loss between the predicted value and the true value in the entire prediction sequence, and use the exponentially weighted L1 distance as the loss function.

[0016] Compared with the prior art, the present invention can achieve the following beneficial effects: The present invention provides a method and device for constructing a binocular stereo matching recursive network based on deep learning. In the feature extraction stage, an adaptive spatial pyramid pooling layer based on a kernel normalized convolution layer is proposed to extract multi-scale features of the image, fully consider the spatial correlation between pixels, further improve the performance of the deep network, extract rich context features, enable the network to better cope with overexposure, underexposure and weak texture areas, and enable the network to learn error correction capabilities. In the matching cost calculation stage, a new attention method (MAR) is used to combine the regularized cost volume to generate a more accurate similarity measure. It not only effectively utilizes the effective information of the binocular image, but also greatly reduces the difficulty of the network to find the correct matching point in the search space. In the main network framework, a gated recurrent unit GRU based on kernel convolution is used as an iterative update operator to retrieve features from the stereo matching cost body and regress and iteratively update the disparity. GRU is enabled to consider a wider context around each pixel, promote the fusion of local and global information, provide a more comprehensive understanding of the feature space, especially in challenging areas such as occlusion and homogeneous texture, and enhance the network's reasoning ability for scene structures in larger areas. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flowchart of a method for constructing a binocular stereo matching recursive network based on deep learning provided according to an embodiment of the present invention.

[0018] Figure 2 It is a schematic diagram of the composition of a method for constructing a binocular stereo matching recursive network based on deep learning provided according to an embodiment of the present invention.

[0019] Figure 3 It is a schematic diagram of how the offset changes the formation of a feature extraction frame in a method for constructing a binocular stereo matching recursive network based on deep learning provided according to an embodiment of the present invention.

[0020] Figure 4 It is a schematic diagram of feature maps at different expansion rates in a method for constructing a binocular stereo matching recursive network based on deep learning provided according to an embodiment of the present invention.

[0021] Figure 5 It is a schematic diagram of applying cross attention traversal to the initial 3D cost volume from two directions, namely, plane and space, in a method for constructing a binocular stereo matching recursive network based on deep learning provided according to an embodiment of the present invention.

[0022] Figure 6 It is a structural block diagram of a device for constructing a binocular stereo matching recursive network based on deep learning provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0024] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0025] Combination Figure 1 As shown, the present invention provides a method for constructing a binocular stereo matching recursive network based on deep learning, comprising: S101. Construct an adaptive multi-scale feature extraction module, wherein the adaptive multi-scale feature extraction module includes a context encoder for extracting multi-scale context features and a feature encoder for left and right images, wherein the context encoder is composed of a series of residual blocks and downsampling layers according to the stereo matching network IGEV-Stereo, and is used to initialize the hidden state of the update operator and inject a gated recurrent unit (GRU) during each iteration of the update operator, wherein the feature encoder extracts features using multiple parallel convolutional layers with different sampling rates, and completes feature fusion after combining kernel convolution normalization and adaptive bias.

[0026] S102, constructing a matching attention mechanism, constructing an initial cost body, generating matching attention weights for the initial cost body, and inputting the final stereo matching cost body into a GRU-based update operator to iteratively update the disparity map.

[0027] S103, using the kernel convolution-based GRU as an iterative update operator, retrieving features from the stereo matching cost volume, and updating the disparity through regression iteration.

[0028] S104, using smooth loss as the training loss function, calculating the smooth L1 loss between the predicted value and the true value in the entire prediction sequence, and using the exponentially weighted L1 distance as the loss function.

[0029] It should be noted that the order of S101 to S104 can be adjusted as needed and is not specifically limited.

[0030] Furthermore, the feature fusion is completed after combining kernel convolution normalization and adaptive bias, including: Adaptive bias steps: The variable convolution is extended to the multi-scale pyramid pooling layer, and the learnable offset is used to form an adaptive search window to extract image features at multiple scales. The adaptive sampling of the input feature map is achieved without increasing the memory and computational costs. In the feature extraction, any point in the feature map y Relative to the initial image x The mapping relationship of the points is: (1); Applying bias to convolution After that, the mapping relationship becomes: (2); Kernel convolution normalization feature fusion steps: Apply random dropout to the normalized unit to get the dropout unit, and then normalize the original normalized unit by calculating the mean and variance of the dropout unit. The hyperparameters include kernel size k, stride s, padding d, and dropout probability p. (3); (4); in, is the exit operation, U is the original normalized unit, To exit the unit, and are the mean and variance of the dropout unit, is the new normalized unit.

[0031] Furthermore, the construction of the matching attention mechanism, the construction of the initial cost body, the generation of matching attention weights for the initial cost body, and the input of the finally obtained stereo matching cost body into the GRU-based update operator for iterative update of the disparity map include: The initial cost volume is calculated using group correlation. According to the vector inner product method, at a disparity level, the channels of the left and right feature matrices are evenly grouped along the channel dimension, and the vector inner product is calculated group by group. After the two four-dimensional matrices are inner-producted, the mean is calculated at the dimension of dim=2 to obtain a three-dimensional matrix. After traversing all disparity levels, the matrices are split and spliced ​​to obtain the group-corr volume. If there are Channels, dividing each matrix into group, then the correlation between all feature groups g and all disparity levels d can be expressed as: (5); The initial 3D cost volume is subjected to cross attention traversal from both the plane and space directions, and the correlation is calculated. D The 3D cost volume is for Parallax-level attention for planar computation and The epipolar level attention of the plane calculation is refined by the MAR matching attention module, and the cost body is input into the GRU iterative update to further refine the disparity map, where ‌W is the width, ‌H is the height, and ‌D is the depth.

[0032] Furthermore, the initial 3D cost volume is subjected to cross attention traversal from both the plane and space directions, and the correlation calculated D The 3D cost volume is for Parallax-level attention for planar computation and The epipolar level attention of the plane calculation is refined by the MAR matching attention module, and the cost volume is input into the GRU iterative update to further refine the disparity map, including: At parallax level ( ) to win plane, apply cross attention traversal, that is, for any spatial position , according to Calculate the position of the center cross relative to the same row or column and The relevant weight ,but Included Corresponding to the mutual mapping information of any point in this cross position relationship; then repeat the above operation once, then for any other position in the space , we can always calculate its corresponding The relevant weight At this point, this plane is refined by attention and is obtained as follows: (6); Then calculate its arbitrary position in space The attention-related weights at each parallax level are as follows: Since binocular images are usually left and right eye images, according to the epipolar constraint principle, ( ) to win Repeat the above operation to get (7); The matching attention formula can be further obtained as: (8); The final correlation can be expressed as: (9).

[0033] Furthermore, the kernel convolution-based GRU is used as an iterative update operator to retrieve features from the stereo matching cost volume and update the disparity through regression iteration, including: Use the soft argmin function to regress the matching cost volume to the initial disparity, and update the disparity through three GRU iterative update layers. Each GRU iterative update layer contains three layers of kernel convolution. The activation function uses the sigmoid function to connect the correlation, difference and context features and inject them into the GRU. Update the hidden state, using the new hidden state to predict the disparity update.

[0034] Furthermore, the smooth loss is used as the training loss function, the smooth L1 loss between the predicted value and the true value in the entire prediction sequence is calculated, and the exponentially weighted L1 distance is used as the loss function, including: Use smooth loss as the training loss function, calculate the smooth L1 loss between the predicted value and the true value in the entire prediction sequence, and use exponentially weighted L1 distance as the loss function, for all predicted disparities , given the ground truth disparity , the loss function is defined as (10); , (11).

[0035] Combination Figure 2 As shown, in order to facilitate a better understanding of the method for constructing a binocular stereo matching recursive network based on deep learning provided by the present invention, it is described below in conjunction with another embodiment.

[0036] The construction method of the binocular stereo matching recursive network based on deep learning mainly includes an adaptive multi-scale feature extraction module based on the kernel normalized convolution layer, a matching cost calculation module based on the matching attention mechanism of the rotation cross calculation, an iterative update operator based on the kernel convolution, and the final loss function part, specifically: (1) Feature extraction module: An adaptive multi-scale feature extraction module based on kernel normalized convolutional layers. The feature extractor consists of two parts: a context encoder for extracting multi-scale contextual features and a feature encoder for left and right images. The context encoder is based on IGEV-Stereo (stereo matching network) and consists of a series of residual blocks and downsampling layers. It is used to initialize the hidden state of the update operator and is injected into the GRU (gated recurrent unit) during each iteration of the update operator. The feature encoder part, inspired by the success of the spatial pyramid pooling method in the field of image segmentation, can use dilated convolution to extract features using multiple parallel convolutional layers with different sampling rates, and combines kernel convolution normalization and adaptive bias to implement a variant of this scheme, namely the KMP module.

[0037] (2) Matching attention mechanism: A matching cost calculation refinement module based on the rotating cross matching attention mechanism. The initial 3D cost volume is traversed from both the plane and space directions, which reduces the computational complexity and memory usage while enhancing the pixel-level representation capability.

[0038] (3) Iterative update operator: GRU iterative update operator based on kernel convolution. The gated recurrent unit GRU based on kernel convolution is used as the iterative update operator to retrieve features from the stereo matching cost volume obtained above and regress and iteratively update the disparity.

[0039] (4) Loss function: We use smooth loss as our training loss function. We calculate the smooth L1 loss between the predicted value and the true value in the entire prediction sequence, and use an exponentially weighted L1 distance similar to IGEV as the loss function.

[0040] First, let’s introduce the feature extraction part of the front end: Combination Figure 2 As shown, the feature extraction module consists of two parts, namely a context encoder used to extract multi-scale context features and a feature encoder applied to the left and right images.

[0041] The context encoder is based on IGEV-Stereo (stereo matching network), which consists of a series of residual blocks and downsampling layers to produce multi-scale context features at 1 / 4, 1 / 8 and 1 / 16 of the input image resolution with 128 channels. The context features are used to initialize the hidden state of the update operator and are injected into the GRU during each iteration of the update operator.

[0042] In the feature encoder part, you can use dilated convolution to extract features using multiple parallel convolution layers with different sampling rates, and further process and fuse them at the end to produce the final result. You can combine semantic information of receptive fields of different sizes without losing resolution (without downsampling) to improve the accuracy of feature extraction. In order to further improve the pixel-level details of objects for disparity estimation tasks with dense pixels such as stereo matching. We further combine kernel convolution normalization and adaptive bias to implement a variant of the dilated convolution scheme, which specifically includes the following two steps: (a) Adaptive bias: Combination Figure 3 As shown in the figure, in order to obtain dense global context information and establish dependencies between pixels, we extend the variable convolution to the multi-scale pyramid pooling layer to solve the problem that the dilated convolution will lose the dense features of the object. We use the learnable offset to form an adaptive search window to extract image features at multiple scales, and achieve adaptive sampling of the input feature map without increasing memory and computational costs. This allows the network to dynamically adjust the receptive field according to the shape and orientation of the object, thereby capturing more relevant and discriminative features. Figure 3 The figure shows how the offset changes the formation of the feature extraction box.

[0043] Figure 3 middle, Figure 3 (a) is the sampling search window for feature extraction with a normal dilation rate of 3; Figure 3 (b) is the position where the sampling point should be offset after adding the offset; Figure 3 (c) in the middle is a more accurate sampling search window obtained after the offset.

[0044] Usually in feature extraction, any point in the feature map y Relative to the initial image x The mapping relationship of the points is: (1) Applying bias to convolution After that, the mapping relationship becomes: (2) Combination Figure 4As shown in the figure, first, 4 layers of parallel adaptive convolution layers with different expansion rates (0, 3, 4, and 5, respectively) are used to generate more detailed and comprehensive multi-scale features. Then, the extracted feature maps containing features of different scales are fused. In order to save costs and not lose information, only three feature maps obtained by convolution layers with different expansion rates can be selected for fusion each time. This process is repeated three times, and finally the above results are residually cascaded to obtain the final feature map. Among them, the adaptive bias is also learned end-to-end during the training process to ensure that the network optimally adapts to the shape and scale of the target object, thereby generating more accurate and detailed feature maps. The structure is as follows Figure 4 shown.

[0045] Figure 4 middle, Figure 4 (a) is an adaptive search window with a convolution kernel of 3*3 and a dilation rate of 0; Figure 4 (b) is an adaptive search window with a convolution kernel of 3*3 and a dilation rate of 3; Figure 4 (c) in the middle is an adaptive search window with a convolution kernel of 3*3 and a dilation rate of 4; Figure 4 (d) in the middle is an adaptive search window with a convolution kernel of 3*3 and a dilation rate of 5; Figure 4 In (e), the above convolutional layers are taken from three layers a, b, c, three layers b, c, d, three layers a, b, d, and three layers a, c, d to fuse the features into four feature maps, and then the residual connection operation is performed to obtain the final feature map.

[0046] (b) Kernel convolution normalization feature fusion: Through a large number of experiments, it is found that in the final multi-scale feature fusion of the feature extraction stage, if the environment memory is limited and only small batches of data can be used for training, the effectiveness of BatchNorm (i.e. batch normalization) will be greatly reduced. In this regard, this solution uses kernel convolution normalization KernelNorm (i.e. kernel normalization) instead of batch normalization. Kernel normalization first applies random dropout to the normalized unit to obtain the dropout unit, and then normalizes the original normalized unit by calculating the mean and variance of the dropout unit. The hyperparameters include kernel size k, stride s, padding d, and dropout probability p.

[0047] (3); (4).

[0048] in is the exit operation, U is the original normalized unit, To exit the unit, and are the mean and variance of the dropout unit, is the new normalized unit.

[0049] After applying kernel normalization to each convolutional layer of multi-scale feature extraction, it does not rely on the statistical information of the entire batch, which makes it more focused on local information rather than global information in the feature space, and also improves the dependence of multi-scale feature extraction on batch size, which is crucial for stereo matching tasks involving images of different scales and conditions. In addition, by normalizing the kernel, the model is less likely to overfit to a specific scale or feature, thereby improving the generalization ability of unseen data.

[0050] (2) The part of the matching cost calculation refinement module based on the rotating cross calculation matching attention mechanism.

[0051] In the stage of calculating the matching cost, there are mainly three steps: construction of the initial cost volume, generation of matching attention weights, and input of the final cost volume into the GRU-based update operator for iterative update of the disparity map.

[0052] The initial cost volume is usually calculated using group correlation, that is, the channels of the left and right feature matrices are evenly grouped along the channel dimension at a disparity level in the manner of vector inner product. Then their inner products are calculated group by group. After the two four-dimensional matrices are inner-producted, the average is calculated at the dimension of dim=2 to obtain a three-dimensional matrix. After traversing all disparity levels, the matrices are split and spliced ​​to obtain the group-corr volume. If there are channels, dividing them into group, then the correlation between all feature groups g and all disparity levels d can be expressed as: (5).

[0053] Combination Figure 5 In order to filter the above-mentioned initial correlation cost volume to emphasize useful information, suppress irrelevant information, and improve the accuracy and robustness of cost volume calculation in stereo matching, the present invention proposes a new hybrid attention cost volume refinement method, which applies cross-attention traversal to the initial 3D cost volume from both plane and space directions, thereby reducing the computational complexity and memory usage while enhancing the pixel-level representation capability.

[0054] Figure 5 The left side is the correlation calculated D The 3D cost volume in the middle is for Parallax-level attention for planar computation and The polar-level attention of the plane calculation is finally the cost volume refined by the MAR matching attention module, which will be input into the GRU iterative update to further refine the disparity map.

[0055] First, at the parallax level ( ) to win plane, apply cross attention traversal, that is, for any spatial position , according to Calculate the position of the center cross relative to the same row or column and The relevant weight ,but Included Corresponding to the mutual mapping information of any point in this cross position relationship; then repeat the above operation once, then for any other position in the space , we can always calculate its corresponding The relevant weight At this time, this plane is refined by attention and obtained (6) Then, calculate its arbitrary position in space The attention-related weights at each parallax level are as follows: Since binocular images are usually left and right eye images, according to the epipolar constraint principle, ( ) to win Repeat the above operation to get (7) Finally, the matching attention formula is: (8) The final correlation can be expressed as: (9) The matching attention calculation method used in the present invention enables the network to learn different features of the input data more effectively, while taking into account the interdependence between channel and global information. It not only effectively utilizes the effective information of the binocular image, but also greatly reduces the difficulty of the network to find the correct matching point in the search space.

[0056] (3) GRU iterative update operator based on kernel convolution Traditional GRU is very suitable for processing sequential data, but has limitations in directly capturing spatial dependencies. In the present invention, a gated recurrent unit (GRU) based on kernel convolution is used as an iterative update operator in the main network framework to retrieve features from the stereo matching cost volume obtained above and regress and iteratively update the disparity.

[0057] First, the matching cost body is regressed to the initial disparity using the soft argmin function, and the disparity is updated through three GRU iterative update layers. Each GRU iterative update layer contains three layers of kernel convolution, and the activation function uses the sigmoid function to connect and inject the correlation, difference and context features into the GRU. Then the hidden state is updated, and the new hidden state is used to predict the disparity update. During training, all stages of the disparity iterative update share the same weights, so no fine-tuning is required.

[0058] The input-to-hidden and hidden-to-hidden transformations in GRU utilize kernel convolutions to ensure that spatial information is preserved and effectively propagated through recurrent connections. By integrating kernel convolutions into GRU, the network can process spatial and temporal information simultaneously, enabling GRU to consider a wider context around each pixel, promoting the fusion of local and global information, and providing a more comprehensive understanding of the feature space, especially in challenging areas such as occlusion and homogeneous textures, enhancing the network's ability to reason about scene structures in larger areas.

[0059] (4) Loss function Since smooth loss is believed to be robust to disparity discontinuities and outliers, we use smooth loss as our training loss function. The smooth L1 loss between the predicted and true values ​​in the entire prediction sequence is calculated, and an exponentially weighted L1 distance similar to IGEV is used as the loss function. , given the ground truth disparity , the loss function is defined as (10); , (11).

[0060] The present invention also provides a method for constructing a binocular stereo matching recursive network based on deep learning. In the feature extraction stage, an adaptive spatial pyramid pooling layer based on a kernel normalized convolution layer is proposed to extract multi-scale features of the image, fully consider the spatial correlation between pixels, further improve the performance of the deep network, extract rich contextual features, enable the network to better cope with overexposure, underexposure and weak texture areas, and enable the network to learn error correction capabilities. In the matching cost calculation stage, a new attention method (MAR) is used to combine the regularized cost volume to produce a more accurate similarity measure. It not only effectively utilizes the effective information of the binocular image, but also greatly reduces the difficulty of the network to find the correct matching point in the search space. In the main network framework, a gated recurrent unit GRU based on kernel convolution is used as an iterative update operator to retrieve features from the stereo matching cost body and regress and iteratively update the disparity. It enables GRU to consider a wider range of context around each pixel, promotes the fusion of local and global information, provides a more comprehensive understanding of the feature space, especially in challenging areas such as occlusion and homogeneous texture, enhances the network's ability to reason about scene structures in larger areas, enhances the pixel-level representation of images, and improves the overall generalization ability of the network.

[0061] Combination Figure 6 As shown, accordingly, the present invention also provides a device for constructing a binocular stereo matching recursive network based on deep learning, comprising: An adaptive multi-scale feature extraction module 601, wherein the adaptive multi-scale feature extraction module includes a context encoder for extracting multi-scale context features and a feature encoder for left and right images, wherein the context encoder is composed of a series of residual blocks and downsampling layers according to a stereo matching network IGEV-Stereo, and is used to initialize the hidden state of the update operator and inject a gated recurrent unit GRU during each iteration of the update operator, wherein the feature encoder extracts features using a plurality of parallel convolutional layers with different sampling rates, and completes feature fusion after combining kernel convolution normalization and adaptive bias; A matching attention mechanism 602, wherein the matching attention mechanism constructs an initial cost body, generates matching attention weights for the initial cost body, and inputs the resulting stereo matching cost body into a GRU-based update operator to iteratively update the disparity map; An iterative update operator 603, used to use a kernel convolution-based GRU as an iterative update operator, retrieve features from a stereo matching cost volume, and iteratively update the disparity through regression; The loss function 604 is used to use the smooth loss as the training loss function, calculate the smooth L1 loss between the predicted value and the true value in the entire prediction sequence, and use the exponentially weighted L1 distance as the loss function.

[0062] The present invention also provides a device for constructing a binocular stereo matching recursive network based on deep learning. In the feature extraction stage, an adaptive spatial pyramid pooling layer based on a kernel normalized convolution layer is proposed to extract multi-scale features of the image, fully consider the spatial correlation between pixels, further improve the performance of the deep network, extract rich contextual features, enable the network to better cope with overexposure, underexposure and weak texture areas, and enable the network to learn error correction capabilities. In the matching cost calculation stage, a new attention method (MAR) is used to combine the regularized cost volume to produce a more accurate similarity measure. It not only effectively utilizes the effective information of the binocular image, but also greatly reduces the difficulty of the network to find the correct matching point in the search space. In the main network framework, a gated recurrent unit GRU based on kernel convolution is used as an iterative update operator to retrieve features from the stereo matching cost body and regress and iteratively update the disparity. GRU is enabled to consider a wider context around each pixel, promote the fusion of local and global information, provide a more comprehensive understanding of the feature space, especially in challenging areas such as occlusion and homogeneous texture, and enhance the network's reasoning ability for scene structures in larger areas.

[0063] Although the embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. Those skilled in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.

[0064] The above specific implementations of the present invention do not constitute a limitation on the protection scope of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the protection scope of the claims of the present invention.

Claims

1. A method for constructing a binocular stereo matching recursive network based on deep learning, characterized in that: include: Constructing an adaptive multi-scale feature extraction module, the adaptive multi-scale feature extraction module includes a context encoder for extracting multi-scale context features and a feature encoder for left and right images, the context encoder is composed of a series of residual blocks and downsampling layers according to the stereo matching network IGEV-Stereo, and is used to initialize the hidden state of the update operator and inject a gated recurrent unit GRU during each iteration of the update operator, the feature encoder uses a plurality of parallel convolutional layers with different sampling rates to extract features, and completes feature fusion after combining kernel convolution normalization and adaptive bias; Constructing a matching attention mechanism, constructing an initial cost body, generating matching attention weights for the initial cost body, and inputting the resulting stereo matching cost body into a GRU-based update operator to iteratively update the disparity map; The kernel convolution-based GRU is used as the iterative update operator to retrieve features from the stereo matching cost volume and update the disparity through regression iteration; Using smooth loss as the training loss function, the smoothed L1 loss between the predicted value and the true value in the entire prediction sequence is calculated, and the exponentially weighted L1 distance is used as the loss function.

2. The method for constructing a binocular stereo matching recursive network based on deep learning as claimed in claim 1, characterized in that: The feature fusion is completed after combining kernel convolution normalization and adaptive bias, including: Adaptive bias steps: The variable convolution is extended to the multi-scale pyramid pooling layer, and the learnable offset is used to form an adaptive search window to extract image features at multiple scales. The adaptive sampling of the input feature map is achieved without increasing the memory and computational costs. In the feature extraction, any point in the feature map y Relative to the initial image x The mapping relationship of the points is: （1）； Applying bias to convolution After that, the mapping relationship becomes: （2）； Kernel convolution normalization feature fusion steps: Apply random dropout to the normalized unit to get the dropout unit, and then normalize the original normalized unit by calculating the mean and variance of the dropout unit. The hyperparameters include kernel size k, stride s, padding d, and dropout probability p. （3）； （4）； in, is the exit operation, U is the original normalized unit, To exit the unit, and are the mean and variance of the dropout unit, is the new normalized unit.

3. The method for constructing a binocular stereo matching recursive network based on deep learning as claimed in claim 1, characterized in that: The construction of the matching attention mechanism, the construction of the initial cost body, the generation of matching attention weights for the initial cost body, and the input of the finally obtained stereo matching cost body into the GRU-based update operator for iterative update of the disparity map include: The initial cost volume is calculated using group correlation. According to the vector inner product method, at a disparity level, the channels of the left and right feature matrices are evenly grouped along the channel dimension, and the vector inner product is calculated group by group. After the two four-dimensional matrices are inner-producted, the mean is calculated at the dimension of dim=2 to obtain a three-dimensional matrix. After traversing all disparity levels, the matrices are split and spliced ​​to obtain the group-corr volume. If there are Channels, dividing each matrix into group, then the correlation between all feature groups g and all disparity levels d can be expressed as: （5）； The initial 3D cost volume is subjected to cross attention traversal from both the plane and space directions, and the correlation is calculated. D The 3D cost volume is for Parallax-level attention for planar computation and The epipolar level attention of the plane calculation is refined by the MAR matching attention module, and the cost body is input into the GRU iterative update to further refine the disparity map, where W is the width, H is the height, and D is the depth.

4. The method for constructing a binocular stereo matching recursive network based on deep learning as claimed in claim 1, characterized in that: The initial 3D cost volume is subjected to cross-attention traversal from both the plane and space directions, and the correlation is calculated. D The 3D cost volume is for Parallax-level attention for planar computation and The epipolar level attention of the plane calculation is refined by the MAR matching attention module, and the cost volume is input into the GRU iterative update to further refine the disparity map, including: At parallax level ( ) to win plane, apply cross attention traversal, that is, for any spatial position , according to Calculate the position of the center cross relative to the same row or column and The relevant weight ,but Included Corresponding to the mutual mapping information of any point in this cross position relationship; then repeat the above operation once, then for any other position in the space , we can always calculate its corresponding The relevant weight At this point, this plane is refined by attention and is obtained as follows: （6）； Then calculate its arbitrary position in space The attention-related weights at each parallax level are as follows: Since binocular images are usually left and right eye images, according to the epipolar constraint principle, ( ) to win Repeat the above operation to get （7）； The matching attention formula can be further obtained as: （8）； The final correlation can be expressed as: （9）。 5. The method for constructing a binocular stereo matching recursive network based on deep learning as claimed in claim 1, characterized in that: The kernel convolution-based GRU is used as an iterative update operator to retrieve features from the stereo matching cost volume and update the disparity through regression iteration, including: Use the soft argmin function to regress the matching cost volume to the initial disparity, and update the disparity through three GRU iterative update layers. Each GRU iterative update layer contains three layers of kernel convolution. The activation function uses the sigmoid function to connect the correlation, difference and context features and inject them into the GRU. Update the hidden state, using the new hidden state to predict the disparity update.

6. The method for constructing a binocular stereo matching recursive network based on deep learning as claimed in claim 1, characterized in that: The smooth loss is used as the training loss function, the smooth L1 loss between the predicted value and the true value in the entire prediction sequence is calculated, and the exponentially weighted L1 distance is used as the loss function, including: Use smooth loss as the training loss function, calculate the smooth L1 loss between the predicted value and the true value in the entire prediction sequence, and use exponentially weighted L1 distance as the loss function, for all predicted disparities , given the ground truth disparity , the loss function is defined as: （10）； ， （11）。 7. A device for constructing a binocular stereo matching recursive network based on deep learning, characterized in that: include: An adaptive multi-scale feature extraction module, wherein the adaptive multi-scale feature extraction module includes a context encoder for extracting multi-scale context features and a feature encoder for left and right images, wherein the context encoder is composed of a series of residual blocks and downsampling layers according to a stereo matching network IGEV-Stereo, and is used to initialize the hidden state of the update operator and inject a gated recurrent unit GRU during each iteration of the update operator, wherein the feature encoder extracts features using a plurality of parallel convolutional layers with different sampling rates, and completes feature fusion after combining kernel convolution normalization and adaptive bias; A matching attention mechanism is used to construct an initial cost body, generate matching attention weights for the initial cost body, and input the resulting stereo matching cost body into a GRU-based update operator to iteratively update the disparity map. An iterative update operator, which uses a kernel convolution-based GRU as an iterative update operator to retrieve features from the stereo matching cost volume and iteratively update the disparity through regression; The loss function is used to use the smooth loss as the training loss function, calculate the smooth L1 loss between the predicted value and the true value in the entire prediction sequence, and use the exponentially weighted L1 distance as the loss function.

Images