Real-time stereo matching method for autonomous driving depth estimation based on pure vision

Abstract

The application discloses a kind of real-time stereo matching methods for realizing automatic driving depth estimation based on pure vision, comprising: obtaining stereo image pair and carrying out stereo correction;Utilize the pre-trained stereo matching model to process, first by feature extraction module, extract multiscale image features and context features, then utilize global disparity initialization module to construct global cost body, obtain initial disparity map by weight activated global cost aggregation and confidence upsampling, then utilize local disparity refinement module, based on initial disparity map and image feature, construct local cost body combined with neighborhood similarity feature, after local cost aggregation and disparity update, upsample to obtain final disparity map;Finally, the final disparity map is transformed into depth map, and depth estimation is completed.The application improves the depth estimation accuracy in complex automatic driving scene while ensuring real-time by utilizing two main modules of global disparity initialization and local disparity refinement.

Description

Technical Field

[0001] This invention relates to the technical fields of computer vision and autonomous driving, and in particular to a real-time stereo matching method for achieving depth estimation in autonomous driving based on pure vision. Background Technology

[0002] Three-dimensional environmental perception is the foundation for autonomous driving systems to achieve autonomous decision-making and safe interaction. The accuracy, real-time performance, and robustness of depth information acquisition directly determine the intelligence level and safety boundaries of autonomous driving systems. Among numerous depth perception technologies, stereo vision has become a highly promising core solution due to its comprehensive advantages over LiDAR, including lower cost, lower power consumption, smaller size, ability to provide texture information, and the ability to recover absolute physical scale compared to monocular depth estimation. The core purpose of stereo vision is to simulate human binocular vision using stereo cameras, recovering the three-dimensional geometric information of a scene from two two-dimensional images from different perspectives. Stereo matching is the key algorithm for achieving this goal: its task is to find the corresponding point in the right image for each pixel in the left image and calculate the disparity, which directly determines the depth value obtained from triangulation. After decades of development, stereo matching has evolved from traditional methods using handcrafted features to data-driven deep learning methods, significantly improving accuracy and robustness.

[0003] However, most existing stereo matching algorithms require constructing a four-dimensional cost volume and introducing 3D convolution for cost aggregation, resulting in bulky models and slow inference speeds, making it difficult to meet the real-time requirements of autonomous driving systems. To achieve real-time stereo matching, some methods construct a cost volume with sparse features and combine it with a lightweight cost aggregation module. While sparse cost volumes can reduce computational complexity, sparse features weaken the model's expressive power, thus affecting the accuracy of the results. Other methods first calculate a disparity map with extremely low resolution and then use a coarse-to-fine strategy to gradually correct the disparity map. Although multiple disparity refinements can restore disparity resolution and improve disparity accuracy, this usually requires multiple upsampling operations. Inappropriate upsampling algorithms can lead to unreasonable disparity allocation, thereby reducing the efficiency of the refinement process. This results in existing real-time stereo matching methods struggling to balance accuracy and speed in complex autonomous driving scenarios, exhibiting significant depth estimation errors in areas with weak textures, repetitive textures, or occlusions. Summary of the Invention

[0004] The purpose of this invention is to provide a real-time stereo matching method for autonomous driving depth estimation based on pure vision. By improving the feature representation capability of sparse cost volume and optimizing the parallax refinement process, it achieves fast inference while maintaining high accuracy.

[0005] To achieve the above objectives, the technical solution provided by this invention is: a real-time stereo matching method for autonomous driving depth estimation based on pure vision. This method utilizes a stereo matching model to achieve depth estimation for autonomous driving. The stereo matching model includes a feature extraction module, a global disparity initialization module, and a local disparity refinement module. The feature extraction module is used to extract image features and contextual features. The global disparity initialization module includes global cost volume construction, weighted activation global cost aggregation, and a confidence upsampling algorithm to calculate an initial disparity map. The local disparity refinement module includes local cost volume construction that fuses neighborhood similarity features and local cost aggregation based on 2D convolution to further refine the initial disparity map.

[0006] The specific implementation of this method includes the following steps:

[0007] S1: Use the calibrated stereo camera to acquire the stereo image pair to be tested in the autonomous driving scenario, and use the camera's calibration parameters to perform stereo correction on the stereo image pair.

[0008] S2: The stereo image after stereo correction is processed by the input pre-trained stereo matching model as follows:

[0009] First, the feature extraction module is used to extract features from the stereo image pair to obtain image feature maps and context feature maps at different scales;

[0010] Then, in the global disparity initialization module, the image feature map is used to construct a global cost body, and then global cost aggregation is performed on the global cost body. At the same time, the image feature map is used to calculate attention weights to activate the cost body in the global cost aggregation process. After global cost aggregation, a preliminary disparity map is obtained. Then, the context feature map is used to calculate upsampling weights, and the preliminary disparity map is upsampled using a confidence upsampling algorithm to obtain an initial disparity map.

[0011] Finally, in the local disparity refinement module, local costs are calculated based on the initial disparity map and the image feature map, and neighborhood similarity features calculated from the initial disparity map are fused to obtain a local cost volume. Then, a local cost aggregation module based on 2D convolution is used to perform local cost aggregation on the local cost volume. After local cost aggregation, a disparity residual map is obtained. The disparity residual map is used to update the initial disparity map to obtain an updated initial disparity map. The updated initial disparity map is then upsampled to obtain the final disparity map.

[0012] S3: By performing a linear transformation on the final disparity map using the calibration parameters of the stereo camera, the depth can be output, thereby achieving depth estimation for autonomous driving scenarios.

[0013] Furthermore, the feature extraction module performs the following operations:

[0014] A lightweight convolutional neural network is used as the backbone to extract image feature maps at different scales from stereo image pairs. , Where i represents the downsampling factor relative to the original image, l represents the left image in the stereo image pair, and r represents the right image in the stereo image pair; simultaneously, a small feature extractor composed of several 2D convolutional blocks is used to extract the context feature map of the stereo image pair. , , where c represents the context feature.

[0015] Furthermore, the global parallax initialization module performs the following operations:

[0016] Global cost volume construction: The feature map of the left image in the stereo image pair is downsampled by 4 times relative to the original image. The image feature map on the right is downsampled by 4 times compared to the original image. Divide the data into G groups along the channel dimension, calculate the L1 distance of features between groups, and construct a four-dimensional global cost volume. Its element calculation is expressed as:

[0017] ;

[0018] In the formula, x and y represent the image feature maps, respectively. The x and y coordinates of a point in the matrix, d is the candidate disparity, and g is the feature group index;

[0019] Weight-activated global cost aggregation: The global cost volume is aggregated using a 3D U-Net-like network consisting of 3D convolutions and skip connections. Global cost aggregation: During the downsampling and upsampling of the global cost volume, image feature maps are used before sampling and after skip connections. Calculate attention weights for the cost body in the global cost aggregation process. Gated activation enhances the response in the relevant region, mathematically represented as follows:

[0020] ;

[0021] ;

[0022] In the formula, This represents the cost volume after activation, where the subscript i indicates the downsampling factor relative to the original image, corresponding to the size of the image feature map; Conv1 represents the 2D convolutional layer used to calculate attention weights; This represents the Sigmoid function, which maps values ​​to the interval between 0 and 1. Represents the Hadamard product; This indicates a connection operation along the feature dimension, reflecting the jump connection between the cost body obtained from the upsampling process and the cost body obtained from the downsampling process; the superscript dwn of the cost body indicates the cost body of the downsampling process in the global cost aggregation process;

[0023] The cost body is obtained after global cost aggregation. For the cost body A preliminary disparity map is obtained by performing a soft-argmax regression using a Top-k strategy. ;

[0024] Confidence upsampling algorithm: using a multilayer perceptron to utilize the cost body Calculate the preliminary disparity map disparity confidence plot Then, a convolutional neural network is used to utilize the contextual feature map of the left image in the stereo image pair, which is downsampled by 2 relative to the original image. Predictive upsampling weight map Finally, the disparity confidence map is used. and the upsampling weight map The preliminary disparity map Upsampling yields the initial disparity map Mathematically, this can be expressed as:

[0025] ;

[0026] In the formula, express The disparity confidence level express right The sampling weights, x and y represent the initial disparity maps respectively. The x and y coordinates of a point in the equation. This indicates a floor operation, and a, b, c, and d are used to traverse the disparity sampling window.

[0027] Furthermore, the local parallax refinement module performs the following operations:

[0028] Local cost volume construction: based on the initial disparity map To guide this, we utilize the image feature map of the left image in the stereo image pair, which is downsampled by 2 times relative to the original image. The image feature map on the right is downsampled by 2 times compared to the original image. For each pixel, a local cost feature is constructed within its disparity neighborhood; an initial disparity map is extracted using a convolutional module. The neighborhood similarity features are used to determine the local cost features, and these features are combined with the local cost features to form a three-dimensional local cost volume. Mathematically, this can be expressed as:

[0029] ;

[0030] ;

[0031] In the formula, x and y represent the image feature maps, respectively. and initial disparity map The x and y coordinates of a point in the matrix, z represents the candidate inspection index, and g is the feature group index; O(z) represents the local candidate disparity value, and O(z) represents the z-th candidate disparity offset value. Represents the local cost feature in four dimensions; Conv3 represents the convolutional module used to extract neighborhood similarity features; Flat represents dimensionality reduction along the candidate disparity dimension; This indicates a join operation along the feature dimension;

[0032] Local cost aggregation: using a lightweight 2D convolutional module to aggregate the local cost volume. Aggregation is performed to obtain the aggregated cost body. Based on the cost body Predict the disparity offset and update the disparity, mathematically expressed as:

[0033] ;

[0034] In the formula, Represents the initial disparity map The probability score of the disparity offset of the pixel with x-axis and y-axis is O(z), where z and w are the indices of the candidate disparity offset set;

[0035] Obtain the updated initial disparity map Then, upsample it to the original image resolution to obtain the final disparity map.

[0036] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0037] 1. This invention uses a global disparity initialization module to quickly calculate the initial disparity and a local disparity refinement module to efficiently improve the disparity accuracy. The two main modules have a clear division of labor and an efficient structure, which significantly improves the matching accuracy in autonomous driving scenarios while ensuring real-time performance.

[0038] 2. In the global disparity initialization module, this invention designs a weighted activation global cost aggregation method. By aligning the feature spaces of cost volumes at different scales through weight activation, the expressive power of sparse feature cost volumes is improved, thereby indirectly improving the accuracy of the initial disparity map.

[0039] 3. In the global disparity initialization module, this invention designs a confidence upsampling algorithm, which introduces disparity confidence into the upsampling algorithm. By reducing the propagation of unreliable disparities during upsampling, a more accurate initial disparity map can be obtained.

[0040] 4. In the local disparity refinement module, this invention designs a local cost volume that combines neighborhood similarity features. By introducing neighborhood similarity features, it guides local cost aggregation to focus on the disparity similarity of the neighborhood, so as to make up for the lack of geometric constraints when using 2D convolution for cost aggregation compared with 3D convolution, thereby improving speed while reducing accuracy loss. Attached Figure Description

[0041] Figure 1 This is a diagram showing the overall architecture of the stereo matching model of the method of the present invention. Detailed Implementation

[0042] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.

[0043] This embodiment discloses a real-time stereo matching method for depth estimation in autonomous driving based on pure vision. This method utilizes a stereo matching model to achieve depth estimation for autonomous driving, such as... Figure 1 As shown, the stereo matching model includes a feature extraction module, a global disparity initialization module, and a local disparity refinement module. The feature extraction module includes a lightweight convolutional neural network and a small feature extractor for extracting image features and contextual features. The global disparity initialization module includes global cost volume construction, global cost aggregation with weight activation, and a confidence upsampling algorithm for calculating the initial disparity map. The local disparity refinement module includes local cost volume construction that fuses neighborhood similarity features and local cost aggregation based on 2D convolution for further refining the initial disparity map.

[0044] The specific implementation of this method includes the following steps:

[0045] 1) Acquire the test stereo image pair of the autonomous driving scene using the calibrated stereo camera, and perform stereo correction on the stereo image pair using the camera's calibration parameters.

[0046] 2) The stereo image after stereo correction is input to the pre-trained stereo matching model and processed as follows:

[0047] 2.1) The feature extraction module is used to extract features from the stereo image pairs to obtain image feature maps and context feature maps at different scales. The specific steps are as follows:

[0048] Using MobileNetV2 as the network backbone, image feature maps of different scales are extracted from stereo image pairs. , Where i represents the downsampling factor relative to the original image, l represents the left image in the stereo image pair, and r represents the right image in the stereo image pair; simultaneously, a small feature extractor consisting of 7 3×3 2D convolutional blocks is used to extract the context feature map of the stereo image pair. , , c represents the contextual features, where one 3×3 convolution is used to enhance the feature channels, four 3×3 dilated convolutions with residual connections and dilation rates of 1, 2, 1 and 1 are used to increase the receptive field, and two 3×3 convolutions are used to reduce the feature channels and output the probability scores of all candidate disparities.

[0049] 2.2) Obtain the initial disparity map using the global disparity initialization module. The specific steps are as follows:

[0050] Global cost volume construction: The feature map of the left image in the stereo image pair is downsampled by 4 times relative to the original image. The image feature map on the right is downsampled by 4 times compared to the original image. Divide the data into G groups along the channel dimension, calculate the L1 distance of features between groups, and construct a four-dimensional global cost volume. Its element calculation is expressed as:

[0051] ;

[0052] In the formula, x and y represent the image feature maps, respectively. The x and y coordinates of a point in the matrix, d is the candidate disparity, and g is the feature group index;

[0053] Weight-activated global cost aggregation: The global cost volume is aggregated using a 3D U-Net-like network consisting of 3D convolutions and skip connections. Global cost aggregation: During the downsampling and upsampling of the global cost volume, image feature maps are used before sampling and after skip connections. Calculate attention weights for the cost body in the global cost aggregation process. Gated activation enhances the response in the relevant region, mathematically represented as follows:

[0054] ;

[0055] ;

[0056] In the formula, This represents the cost volume after activation, where the subscript i indicates the downsampling factor relative to the original image, corresponding to the size of the image feature map; Conv1 represents the 2D convolutional layer used to calculate attention weights; This represents the Sigmoid function, which maps values ​​to the interval between 0 and 1. Represents the Hadamard product; This indicates a connection operation along the feature dimension, reflecting the jump connection between the cost body obtained from the upsampling process and the cost body obtained from the downsampling process; the superscript dwn of the cost body indicates the cost body of the downsampling process in the global cost aggregation process;

[0057] The cost body is obtained after global cost aggregation. For the cost body A preliminary disparity map is obtained by performing a soft-argmax regression using a Top-k strategy. ;

[0058] Confidence upsampling algorithm: using a multilayer perceptron to utilize the cost body Calculate the preliminary disparity map disparity confidence plot Then, a convolutional neural network is used to utilize the contextual feature map of the left image in the stereo image pair, which is downsampled by 2 relative to the original image. Predictive upsampling weight map Finally, the disparity confidence map is used. and the upsampling weight map The preliminary disparity map Upsampling yields the initial disparity map Mathematically, this can be expressed as:

[0059] ;

[0060] In the formula, express The disparity confidence level express right The sampling weights, x and y represent the initial disparity maps respectively. The x and y coordinates of a point in the equation. This indicates a floor operation, and a, b, c, and d are used to traverse the disparity sampling window.

[0061] 2.3) Calculate the final disparity map using the local disparity refinement module. The specific steps are as follows:

[0062] Local cost volume construction: based on the initial disparity map To guide this, we utilize the image feature map of the left image in the stereo image pair, which is downsampled by 2 times relative to the original image. The image feature map on the right is downsampled by 2 times compared to the original image. For each pixel, a local cost feature is constructed within its disparity neighborhood; an initial disparity map is extracted using a convolutional module. The neighborhood similarity features are used to determine the local cost features, and these features are combined with the local cost features to form a three-dimensional local cost volume. Mathematically, this can be expressed as:

[0063] ;

[0064] ;

[0065] In the formula, x and y represent the image feature maps, respectively. and initial disparity map The x and y coordinates of a point in the matrix, z represents the candidate inspection index, and g is the feature group index; O(z) represents the local candidate disparity value, and O(z) represents the z-th candidate disparity offset value. Represents the local cost feature in four dimensions; Conv3 represents the convolutional module used to extract neighborhood similarity features; Flat represents dimensionality reduction along the candidate disparity dimension; This indicates a join operation along the feature dimension;

[0066] Local cost aggregation: using a lightweight 2D convolutional module to aggregate the local cost volume. Aggregation is performed to obtain the aggregated cost body. Based on the cost body Predict the disparity offset and update the disparity, mathematically expressed as:

[0067] ;

[0068] In the formula, Represents the initial disparity map The probability score of the disparity offset of the pixel with x-axis and y-axis is O(z), where z and w are the indices of the candidate disparity offset set;

[0069] Obtain the updated initial disparity map Then, upsample it to the original image resolution to obtain the final disparity map.

[0070] 3) By performing a linear transformation on the final disparity map using the calibration parameters of the stereo camera, the depth can be output, thereby realizing depth estimation for autonomous driving scenarios.

[0071] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A real-time stereo matching method for automatic driving depth estimation based on pure vision, characterized in that, This method utilizes a stereo matching model to achieve depth estimation for autonomous driving. The stereo matching model includes a feature extraction module, a global disparity initialization module, and a local disparity refinement module. The feature extraction module is used to extract image features and contextual features. The global disparity initialization module includes global cost volume construction, weighted activation global cost aggregation, and a confidence upsampling algorithm to calculate the initial disparity map. The local disparity refinement module includes local cost volume construction that fuses neighborhood similarity features and 2D convolution-based local cost aggregation to further refine the initial disparity map. The specific implementation of this method includes the following steps: S1: Use the calibrated stereo camera to acquire the stereo image pair to be tested in the autonomous driving scenario, and use the camera's calibration parameters to perform stereo correction on the stereo image pair. S2: The stereo image after stereo correction is processed by the input pre-trained stereo matching model as follows: First, the feature extraction module is used to extract features from the stereo image pair to obtain image feature maps and context feature maps at different scales; Then, in the global disparity initialization module, the image feature map is used to construct a global cost body, and then global cost aggregation is performed on the global cost body. At the same time, the image feature map is used to calculate attention weights to activate the cost body in the global cost aggregation process. After global cost aggregation, a preliminary disparity map is obtained. Then, the context feature map is used to calculate upsampling weights, and the preliminary disparity map is upsampled using a confidence upsampling algorithm to obtain an initial disparity map. Finally, in the local disparity refinement module, local costs are calculated based on the initial disparity map and the image feature map, and neighborhood similarity features calculated from the initial disparity map are fused to obtain a local cost volume. Then, a local cost aggregation module based on 2D convolution is used to perform local cost aggregation on the local cost volume. After local cost aggregation, a disparity residual map is obtained. The disparity residual map is used to update the initial disparity map to obtain an updated initial disparity map. The updated initial disparity map is then upsampled to obtain the final disparity map. S3: By performing a linear transformation on the final disparity map using the calibration parameters of the stereo camera, the depth can be output, thereby achieving depth estimation for autonomous driving scenarios.

2. The method of claim 1, wherein, The feature extraction module performs the following operations: Using a lightweight convolutional neural network as the backbone, different scale image feature maps of the stereo image pair are extracted 、 , wherein i represents the down-sampling rate relative to the original image, l represents the left image in the stereo image pair, and r represents the right image in the stereo image pair; meanwhile, a small feature extractor composed of several 2D convolution blocks is used to extract the context feature map of the stereo image pair 、 , wherein c represents the context feature.

3. The real-time stereo matching method for autonomous driving depth estimation based on pure vision as described in claim 2, characterized in that, The global disparity initialization module performs the following operations: Global cost volume construction: The feature map of the left image in the stereo image pair is downsampled by 4 times relative to the original image. The image feature map on the right is downsampled by 4 times compared to the original image. Divide the data into G groups along the channel dimension, calculate the L1 distance of features between groups, and construct a four-dimensional global cost volume. Its element calculation is expressed as: ； In the formula, x and y represent the image feature maps, respectively. The x and y coordinates of a point in the matrix, d is the candidate disparity, and g is the feature group index; Weight-activated global cost aggregation: The global cost volume is aggregated using a 3D U-Net-like network consisting of 3D convolutions and skip connections. Global cost aggregation: During the downsampling and upsampling of the global cost volume, image feature maps are used before sampling and after skip connections. Calculate attention weights for the cost body in the global cost aggregation process. Gated activation enhances the response in the relevant region, mathematically represented as follows: ； ； In the formula, This represents the cost volume after activation, where the subscript i indicates the downsampling factor relative to the original image, corresponding to the size of the image feature map; Conv1 represents the 2D convolutional layer used to calculate attention weights; This represents the Sigmoid function, which maps values ​​to the interval between 0 and 1. Represents the Hadamard product; This indicates a connection operation along the feature dimension, reflecting the jump connection between the cost body obtained from the upsampling process and the cost body obtained from the downsampling process; the superscript dwn of the cost body indicates the cost body of the downsampling process in the global cost aggregation process; The cost body is obtained after global cost aggregation. For the cost body A preliminary disparity map is obtained by performing a soft-argmax regression using a Top-k strategy. ; Confidence upsampling algorithm: using a multilayer perceptron to utilize the cost body Calculate the preliminary disparity map disparity confidence plot Then, a convolutional neural network is used to utilize the contextual feature map of the left image in the stereo image pair, which is downsampled by 2 relative to the original image. Predictive upsampling weight map Finally, the disparity confidence map is used. and the upsampling weight map The preliminary disparity map Upsampling yields the initial disparity map Mathematically, this can be expressed as: ； In the formula, express The disparity confidence level express right The sampling weights, x and y represent the initial disparity maps respectively. The x and y coordinates of a point in the equation. This indicates a floor operation, and a, b, c, and d are used to traverse the disparity sampling window.

4. The real-time stereo matching method for autonomous driving depth estimation based on pure vision as described in claim 3, characterized in that, The local parallax refinement module performs the following operations: Local cost volume construction: based on the initial disparity map To guide this, we utilize the image feature map of the left image in the stereo image pair, which is downsampled by 2 times relative to the original image. The image feature map on the right is downsampled by 2 times compared to the original image. For each pixel, a local cost feature is constructed within its disparity neighborhood; an initial disparity map is extracted using a convolutional module. The neighborhood similarity features are used to determine the local cost features, and these features are combined with the local cost features to form a three-dimensional local cost volume. Mathematically, this can be expressed as: ； ； In the formula, x and y represent the image feature maps, respectively. and initial disparity map The x and y coordinates of a point in the matrix, z represents the candidate inspection index, and g is the feature group index; O(z) represents the local candidate disparity value, and O(z) represents the z-th candidate disparity offset value. Represents the local cost feature in four dimensions; Conv3 represents the convolutional module used to extract neighborhood similarity features; Flat represents dimensionality reduction along the candidate disparity dimension; This indicates a join operation along the feature dimension; Local cost aggregation: using a lightweight 2D convolutional module to aggregate the local cost volume. Aggregation is performed to obtain the aggregated cost body. Based on the cost body Predict the disparity offset and update the disparity, mathematically expressed as: ； In the formula, Represents the initial disparity map The probability score of the disparity offset of the pixel with x-axis and y-axis is O(z), where z and w are the indices of the candidate disparity offset set; Obtain the updated initial disparity map Then, upsample it to the original image resolution to obtain the final disparity map.

Images