Autonomous driving stereo matching method based on volume fusion and disparity space attention

Abstract

The application provides an automatic driving stereo matching method based on volume fusion and disparity space attention, and has the characteristics that the method comprises the following steps: step A: using two weight-shared convolutional neural networks to extract features of two images of the stereo matching image pair, and the extracted features are denoted as and respectively; step B: using the features and to construct an initial correlation volume and a series volume; step C: sending the constructed initial correlation volume into a cost aggregation module composed of a 3D convolution and a 3D hourglass convolution for processing; step D: sending the constructed initial cost volume into a disparity space attention module for processing, so as to obtain weights in the disparity dimension and the space dimension; and step E: sending the attention cost volume into a cost aggregation module composed of a 3D convolution and a 3D hour glass convolution for processing, and obtaining a prediction map result.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a stereo matching method for autonomous driving based on volume fusion and parallax spatial attention. Background Technology

[0002] In many computer vision applications, stereo matching is a key fundamental task. Stereo matching is the process of finding the correct matching feature points from different viewpoints of a pair of corrected stereo images and calculating the disparity of a pair of matching feature points to obtain depth information. Stereo matching has a wide range of applications in autonomous driving (such as ranging, navigation, etc.), intelligent control systems, motion segmentation [1]. With the widespread application of convolutional neural networks in stereo matching, many excellent models have emerged and achieved advanced performance.

[0003] Classical deep learning-based stereo matching models typically include four steps: cost volume construction, cost aggregation, difference calculation, and difference regression [2]. Cost volume quantifies the similarity between a pair of feature points and plays a key role in stereo matching models. The cost volume with rich content information and sufficient relevant information directly affects the performance of stereo matching models. Therefore, most learning-based stereo matching models improve performance by exploring different cost volume construction methods. DispNetC [3] obtains sufficient relevant information by calculating the relevant volume of feature points in the image pair. Relevant volume is a type of cost volume. However, it loses a lot of content information. On the other hand, PSMNet [4] and GC-Net [5] construct concatenated volumes by concatenating feature points in the image pair, thereby providing rich content information. Concatenated volume is another type of cost volume. However, they ignore the relevant information between feature points in the image pair, which requires the use of a lot of 3D convolutions for cost aggregation to obtain relevant information.

[0004] In order to utilize both content information and related information simultaneously, GWCNet[6] employs grouped correlation techniques to construct a correlation volume, thereby obtaining a superior similarity metric. Subsequently, the correlation volume and the concatenated volume are merged to form the cost volume. However, due to the different feature distributions of the correlation volume and the concatenated volume, the model fails to fully utilize the information contained in each of these two volumes. To address this issue, ACVNet[7] adopts a novel cost volume construction strategy, namely, using the correlation volume to generate weights. These weights can suppress redundant information and enhance the matching related information. However, this model uses two volumes to simultaneously guide the network training of the correlation volume module, so the network needs to consider the role of both content information and related information. This often leads to confusion problems and affects the accuracy of the final matching. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a stereo matching method for autonomous driving based on volume fusion and disparity spatial attention, which can effectively reduce the confusion problem caused by volume fusion during the stereo matching model process. The stereo matching method proposed in this invention can be applied to multiple fields such as autonomous driving, 3D reconstruction, machine intelligent control, and motion segmentation.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: an autonomous driving stereo matching method based on volume fusion and disparity spatial attention, comprising the following steps:

[0007] Step A: Given a stereo matching image pair consisting of two images, use two convolutional neural networks with shared weights to extract features from the two images in this stereo matching image pair. The extracted features are denoted as f. l and f r ;

[0008] Step B: Utilize feature f l and f r Construct the initial related volume C correlation and series volume C concat ;

[0009] Step C: In the volume fusion module, the constructed initial related volume C correlation The data is fed into a cost aggregation module consisting of 3D convolutions and 3D hourglass convolutions for processing, yielding a predicted map of the relevant volume. A smoothed L1 loss, calculated using the predicted disparity map and the actual ground disparity, guides the training of the relevant volume module network to obtain the best-performing relevant volume. Then, the network parameters of the relevant volume module are frozen to ensure that the best-performing relevant volume is obtained each time. Finally, this best-performing relevant volume is compressed and fused with the initial concatenated volume C using a weighted method. concat Obtain the initial cost volume C cost ;

[0010] Step D: Construct the initial cost volume C cost The data is fed into a disparity spatial attention module for processing to obtain weights for the disparity and spatial dimensions. These weights are then used to guide the stereo matching network to better learn the initial cost volume C. cost Features that provide more reliable attention cost volume C attention Improve stereo matching accuracy;

[0011] Step E: Adjust the attention cost volume C attentionThe data is fed into a cost aggregation module consisting of 3D convolution and 3D hourglass convolution for processing, and the prediction map result is obtained. The smoothed L1 loss calculated by using the prediction disparity map result and the real ground disparity result is used to guide the training of the network, and finally a stereo matching model with the best performance is obtained.

[0012] In a preferred embodiment, the initial relevant volume C in step B correlation and series volume C concat for;

[0013]

[0014] Where · represents each channel dimension, d represents each disparity dimension, (x, y) represents the pixel position, and (i, j) represents the correlation calculation range;

[0015] C concat (·, d, x, y) = concat{f l (x, y), f r (xd, y)}, #(Formula 2)

[0016] Where concat means to perform a series operation along the channel direction.

[0017] In a preferred embodiment, the attention cost volume in step D is C attention =M α (M β (C cost ))×C cost (Formula 3)

[0018] Among them, M α This represents the processing procedure of the parallax attention module, M. β This describes the processing procedure of the spatial attention module.

[0019] Compared with existing technologies, this invention has the following advantages: This invention proposes a stereo matching model based on volume fusion and disparity spatial attention. In the volume fusion module, during network training, a strategy of freezing all relevant network parameters effectively reduces the impact of confusion. Furthermore, in the disparity spatial attention module, we first adaptively calculate the weights of the disparity and spatial dimensions. These weights are then used to guide network training, thereby obtaining the attention cost for stereo matching, further reducing confusion. Attached Figure Description

[0020] Figure 1 A flowchart of a preferred embodiment of the present invention;

[0021] Figure 2This is a framework diagram of the parallax spatial attention module according to a preferred embodiment of the present invention;

[0022] Figure 3 This is a diagram showing the effect of a preferred embodiment of the present invention on the KITTI dataset;

[0023] Figure 4 The image shows the effect of a preferred embodiment of the present invention on the Scene Flow dataset. Detailed Implementation

[0024] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0025] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains.

[0026] It should be noted that the terminology used herein is for the purpose of describing particular implementations only and is not intended to limit the exemplary implementations according to this application; as used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise; furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0027] The autonomous driving stereo matching method based on volume fusion and disparity spatial attention described in this invention includes the following steps:

[0028] Step A. Given a stereo matching image pair consisting of two images, use two convolutional neural networks with shared weights to extract features from the two images in this stereo matching image pair. The extracted features are denoted as f. l and f r This step corresponds to the convolutional neural network part in the flowchart.

[0029] Step B. Utilize feature f l and f r Construct the initial related volume C correlation and series volume C concat .

[0030]

[0031] Where · represents each channel dimension, d represents each disparity dimension, (x, y) represents the pixel position, and (i, j) represents the correlation calculation range. This step corresponds to the correlation volume module in the flowchart.

[0032] C concat(·, d, x, y) = concat{f l (x, y), f r (xd, y)}, #(Formula 2)

[0033] Where c0ncat represents a series operation along the channel direction, this step corresponds to the series volume module part in the flowchart.

[0034] Step C. In the volume fusion module, the constructed initial related volume C correlation The data is fed into a cost aggregation module consisting of 3D convolutions and 3D hourglass convolutions for processing, yielding a predicted map of the relevant volume. A smoothed L1 loss, calculated using the predicted disparity map and the ground disparity, guides the training of the relevant volume module network to obtain the best-performing relevant volume. This step corresponds to the cost aggregation module 1 in the flowchart. Then, we freeze the network parameters of the relevant volume module to ensure that the best-performing relevant volume is obtained each time. Finally, we compress this best-performing relevant volume and fuse it with the initial concatenated volume C using a weighted approach. concat Obtain the initial cost volume C cost This step corresponds to the volume fusion module in the flowchart.

[0035] Step D. Construct the initial cost volume C. cost The data is fed into a disparity spatial attention module for processing to obtain weights for the disparity and spatial dimensions. These weights are then used to guide the stereo matching network to better learn the initial cost volume C. cost Features that provide more reliable attention cost volume C attention This improves the accuracy of stereo matching.

[0036] C attention =M α (M β (C cost ))×C cost (Formula 3)

[0037] Among them, M α This represents the processing procedure of the parallax attention module, M. β This indicates the processing procedure of the spatial attention module. This step corresponds to the parallax spatial attention module section in the flowchart.

[0038] Step E. Calculate the attention cost volume C attentionThe data is fed into a cost aggregation module consisting of 3D convolutions and 3D hourglass convolutions for processing, and the predicted map result is obtained. The smoothed L1 loss calculated using the predicted disparity map result and the real ground disparity result is used to guide the training of the network, and finally a stereo matching model with the best performance is obtained. This step corresponds to the cost aggregation module 2 in the flowchart.

[0039] Figure 3 The first row represents one of the four image pairs in the KITTI dataset, while the second and third rows represent the results obtained by PSMNet and the present invention (i.e., VFDSNet) on the corresponding images, respectively.

[0040] Figure 4 From left to right, the images are: the left image of a pair of images from the Scene Flow dataset, the true depth map provided by the Scene Flow dataset, and the result obtained by this invention (i.e., VFDSNet).

Claims

1. A stereo matching method for autonomous driving based on volume fusion and parallax spatial attention, characterized in that, Includes the following steps: Step A: Given a stereo matching image pair consisting of two images, use two convolutional neural networks with shared weights to extract features from the two images in this stereo matching image pair. The extracted features are denoted as follows: and ; Step B: Utilizing Features and Construct the initial related volume and series volume ; Step C: In the volume fusion module, the constructed initial relevant volumes... The data is fed into a cost aggregation module consisting of 3D convolutions and 3D hourglass convolutions for processing, yielding a predicted map of the relevant volume. A smoothed L1 loss, calculated using the predicted disparity map and the actual ground disparity, guides the training of the relevant volume module network to obtain the best-performing relevant volume. Then, the network parameters of the relevant volume module are frozen to ensure that the best-performing relevant volume is obtained each time. Finally, this best-performing relevant volume is compressed and the initial concatenated volumes are fused using a weighted method. Obtain the initial cost volume ; Step D: Construct the initial cost volume The data is fed into a disparity spatial attention module for processing, yielding weights for the disparity and spatial dimensions. These weights are then used to guide the stereo matching network in better learning the initial cost volume. Features that provide more reliable attention cost volume Improve stereo matching accuracy; Step E: Volume of attention cost The data is fed into a cost aggregation module consisting of 3D convolution and 3D hourglass convolution for processing, and the prediction map result is obtained. The smoothed L1 loss calculated by the prediction disparity map result and the real ground disparity result is used to guide the training of the network, and finally a stereo matching model with the best performance is obtained. The initial related volume in step B and series volume for; in Represents each channel dimension. Representing each disparity dimension, Indicates the position of a pixel. Indicates the range of correlation calculation; in This indicates a series operation along the channel direction; The attention cost volume in step D is... in, This describes the processing procedure of the parallax attention module. This describes the processing procedure of the spatial attention module.

Images