A method for uncertainty estimation of binocular matching based on evidence-based deep learning

Abstract

The present invention discloses a binocular matching uncertainty estimation method based on evidence deep learning, which extracts and aggregates features through a convolutional neural network to obtain a matching cost body and three uncertainty bodies, and calculates four hyperparameters γ, ν, α and β of the evidence distribution under the guidance of the matching cost; finally, the binocular matching disparity value, accidental uncertainty and cognitive uncertainty are calculated through the four hyperparameters. The present invention well reflects the difficulty of binocular matching, improves the performance of uncertainty estimation, and can give higher cognitive uncertainty when facing out-of-distribution data.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to a binocular matching uncertainty estimation method based on evidence deep learning. Background Art

[0002] Dense depth maps provide detailed scene information for applications such as 3D reconstruction, visual SLAM, and autonomous driving. Active sensing devices, such as depth cameras and lidars, face problems such as expensive imaging devices, limited sensing range, or sparse output, and often find it difficult to obtain dense depth maps that meet the requirements. Using a binocular camera to capture images, a dense disparity map can be obtained through binocular matching, and then the depth of the scene can be reconstructed in combination with the camera imaging model. This is a more convenient and faster method compared to active detection devices.

[0003] With the rapid development of deep learning technology, many binocular matching models have been proposed, such as PSM-Net, GA-Net and AA-Net, which can achieve high accuracy on public data sets. However, in practical applications, especially in scenarios with high safety requirements, in addition to obtaining high-precision depth maps, the uncertainty of the output acquired depth maps is also very important. For example, for the key obstacle avoidance task in autonomous driving, it not only requires accurate depth information to complete the obstacle avoidance operation, but also needs to know the credibility of the predicted depth information. Uncertainty can be divided into accidental uncertainty and cognitive uncertainty according to its source. In stereo matching, most methods (A quantitative evaluation of confidence measures for stereo vision. IEEE Transactions on Pattern Analysis and Machine intelligence, 2012) and (Quantitative evaluation of confidence measures in a machine learning world. IEEE International Conference on Computer Vision, 2017) only obtain confidence through manual rules to model accidental uncertainty, but cognitive uncertainty is often ignored. A few methods (What uncertainties do we need in Bayesian deep learning for computer vision. Advances in Neural Information Processing Systems, 2017) and (Sampling-free epistemic uncertainty estimation using approximated variance propagation. IEEE International Conference on Computer Vision, 2019) take into account aleatoric uncertainty and epistemic uncertainty, but they set priorities on network weights, which results in high computational sampling costs for estimating epistemic uncertainty during reasoning. The evidence method considers learning as a process of obtaining evidence. This method first places the prior directly on the likelihood function to form a high-order evidence distribution; then, evidence that conforms to this distribution is added to the training sample; finally, by learning the parameters of the estimated evidence distribution, aleatoric uncertainty and epistemic uncertainty can be obtained without sampling, greatly reducing the consumption of computing resources.

[0004] The present invention proposes an end-to-end binocular matching uncertainty estimation method based on the theory of evidence deep learning. When estimating the uncertainty value, the method estimates an uncertainty value for each candidate disparity value, and then uses the classification probability distribution of binocular matching as a guide to perform weighted averaging. The uncertainty obtained in this way fully reflects the uncertainty of the matching cost curve. Even if the output is the same disparity value, the uncertainty is different due to the different cost curves for calculating the final disparity. Further, two loss function terms are proposed to constrain the uncertainty estimation of areas without true disparity annotations, alleviating the dependence that the uncertainty estimation loss function can only be calculated in areas with true disparity annotations. The first loss term minimizes the evidence of the wrongly predicted area, thereby increasing the uncertainty of this part. The second loss function imposes smoothness constraints on the uncertainty value of the disparity smooth area, and the constraints at the disparity-annotated pixels can be transmitted to the surrounding pixels. Through the two loss function terms, pixels without true disparity value annotations can also be used to train the uncertainty estimation network, thereby improving the performance of uncertainty estimation. Summary of the invention

[0005] The technology of the present invention solves the problem: Overcoming the shortcomings of the prior art, a binocular matching uncertainty estimation method based on evidence deep learning is provided. The method takes a binocular image as input, uses four branches to respectively estimate the four parameters of the normal inverse gamma distribution, and simultaneously calculates the disparity value, accidental uncertainty and cognitive uncertainty through the parameters, thereby improving the accuracy of binocular matching and the performance of uncertainty estimation.

[0006] The technical solution adopted by the present invention to solve its technical problem is a binocular matching uncertainty estimation method based on evidence deep learning. The evidence deep learning method regards learning as a process of evidence collection, and constructs a higher-order evidence distribution (Evidential Distribution) by modeling the likelihood function. The training set samples conform to this distribution by adding evidence. Compared with other methods of uncertainty estimation through multiple sampling, by learning the parameters of the evidence distribution, accidental uncertainty and epistemic uncertainty can be directly obtained through the formula using the parameters of the evidence distribution without the need for multiple sampling to calculate. Sampling-based methods require a lot of computing resources, so the present invention chooses the end-to-end approach of evidence deep learning to estimate uncertainty.

[0007] The specific steps include:

[0008] (1) Input the left image and the right image, and use the twin network with shared parameters to extract the features of the left image and the right image respectively; through a series of two-dimensional convolution and pooling operations, obtain the feature maps of the left image and the right image;

[0009] (2) The feature maps of the left and right images are concatenated together, and the feature maps are cost aggregated using a three-dimensional convolution operation to obtain a matching cost volume and an uncertainty volume; the matching cost volume and the uncertainty volume are obtained after cost aggregation of the features, and the matching cost volume is used to calculate the matching similarity at different disparities, thereby obtaining the parameter γ; the uncertainty volume is used to calculate the uncertainty parameters at different disparities, and under the guidance of the above matching similarity, the parameters ν, α and β are calculated;

[0010] (3) Based on the matching cost body, a neural network is trained based on evidence deep learning. The evidence deep learning method regards learning as a process of evidence collection and selects the conjugate distribution of the Gaussian distribution, namely the normal inverse gamma distribution, as the evidence distribution. Then, based on the evidence distribution, a higher-order evidence distribution is modeled on the likelihood function parameters as the posterior distribution. Finally, the hyperparameters γ, ν, α, and β of the evidence distribution are calculated. These parameters are the parameters of the normal inverse gamma distribution and are used to calculate accidental uncertainty and epistemic uncertainty.

[0011] (4) The binocular disparity is estimated through the trained neural network, and the aleatoric uncertainty and epistemic uncertainty are calculated through a single forward pass operation.

[0012] Furthermore, step (2) is specifically implemented as follows:

[0013] Obtain the matching cost volume to calculate a classification probability vector for each pixel position. Each element in the classification probability vector corresponds to a possible disparity value, indicating the probability that the matching result is the disparity value; the uncertain volume has three dimensions w×h×S max , w and h are the width and height of the feature map respectively, S max is the maximum value of the disparity range; the obtained matching cost volume calculates a classification probability vector for each pixel position in the left eye image. The possible disparity value refers to the range between the minimum disparity value and the maximum disparity value determined by the internal and external parameters of the camera, and discretizes it into a finite number of values;

[0014] Repeat S for each channel of the feature map of the left image max The 3D features are constructed by using the feature map of the left image as a reference and different displacement values, i.e., 1 pixel, 2 pixels, ..., S max Pixel-shift each channel of the right image feature map and then stack it into a three-dimensional feature;

[0015] Repeatedly stack the left image features and the right image features translated by different values ​​in a cross manner to form new features;

[0016] The stacked features are processed using a three-dimensional convolution kernel, and different matching cost volumes and uncertainty volumes are obtained through four branches.

[0017] Furthermore, in step (3), the formula for the posterior distribution is:

[0018] q(μ,σ 2 )=p(μ,σ 2 |d1,d2,…,d N )

[0019] Where p represents the maximum likelihood, d i represents the estimated disparity value of the i-th pixel in the matching cost volume, specifically (d1, d2, ..., d N )~N(μ,σ 2 ), the mean μ follows a Gaussian distribution and is expressed as μ~N(γ,σ 2 v -1 ), variance σ 2 It follows the inverse gamma distribution represented by σ 2 ~Γ -1 (α, β), Γ -1 (·) is the inverse gamma equation, N represents Gaussian distribution, υ>0,α>1,β>0.

[0020] Furthermore, in step (4), the neural network is trained by minimizing the loss function, which consists of the evidence learning loss and two regularization loss terms, and the formula is:

[0021]

[0022] Among them, w represents the neural network model parameters, Learn a loss function for the evidence; and are two regularization loss terms; pred ,λ smooth and λ sup is the balancing factor; Used to maximize model evidence with the support of training samples, It aims to estimate smaller evidence parameters v and α in the region with wrong disparity estimation, thus obtaining larger uncertainty values. The smoothness of the uncertainty parameters is constrained in the disparity smooth region, so that the constraint on uncertainty is propagated from pixels with true disparity annotations to pixels without true disparity annotations, thus improving the constraint on uncertainty estimation.

[0023] Furthermore, in step (4), aleatoric uncertainty and epistemic uncertainty are calculated by the following formula:

[0024] E[μ]=γ

[0025]

[0026]

[0027] Among them, μ is the mean, σ 2 is the variance, γ, υ, α, β are the hyperparameters required for the calculation learned by the network, E[μ], E[σ 2 ] and Var[μ] represent parallax, aleatory uncertainty, and epistemic uncertainty, respectively.

[0028] The beneficial effects of the present invention are as follows:

[0029] The advantages of the present invention compared with the prior art are:

[0030] (1) The present invention provides a binocular matching uncertainty estimation method based on evidence deep learning. By using the matching classification probability as a guide to estimate the uncertainty, the difficulty of matching can be well reflected. During the network training process, the prediction error prior and the disparity consistency prior are used to constrain the uncertainty value at pixels without real disparity annotations, thereby improving the performance of uncertainty estimation.

[0031] (2) The present invention proposes two loss function terms to constrain the training of the uncertainty estimation network. By using the prediction error prior and the disparity consistency prior, the uncertainty values ​​at pixels without true disparity annotations can also be constrained, thereby improving the performance of uncertainty estimation.

[0032] (3) A large number of experimental results show that the method of the present invention can improve the accuracy of binocular matching. The method of the present invention can not only give higher uncertainty in areas with large prediction errors, but also give higher cognitive uncertainty when facing out-of-distribution data. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] Figure 1 It is a flowchart of a binocular matching uncertainty estimation method based on evidence deep learning of the present invention;

[0034] Figure 2 Comparison chart of the visual results of binocular matching and uncertainty estimation. DETAILED DESCRIPTION

[0035] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments.

[0036] like Figure 1As shown, the present invention provides a binocular matching uncertainty estimation method based on evidence deep learning, taking autonomous driving as an example, the specific steps are as follows:

[0037] 1) When the vehicle is driving on the road, the left and right images are obtained through the onboard binocular camera, and the features of the left and right images are extracted respectively using a twin network with shared parameters. Specifically, a CNN network with the same weights is used to extract features from the left and right view images respectively, and multi-scale features are fused.

[0038] 2) Through a series of two-dimensional convolution and pooling operations, feature maps of the left and right images are obtained; specifically, common deep learning-based stereo matching algorithms: PSM-Net, GA-Net and AA-Net can be used as the basic network structure to obtain feature maps. PSM-Net is a CNN-based stereo matching method that includes a spatial pyramid pooling module and a 3D CNN. The spatial pyramid pooling module releases global context information to form a cost volume, and the 3D CNN learns to adjust the cost volume. GA-Net introduces a semi-global aggregation layer and a local guided aggregation layer to capture local and global cost dependencies. AA-Net replaces the commonly used 3D convolution with a sparse point-based intra-scale cost aggregation and cross-scale cost aggregation model, thereby improving the inference speed.

[0039] 3) Repeat S for each channel of the feature map of the left image max To construct the 3D features, we use the feature map of the left image as a reference and use different displacement values ​​(1 pixel, 2 pixels, ..., S max Pixels) are used to translate each channel of the right eye image feature map, and then stacked into a three-dimensional feature. The left eye image features and the right eye image features translated by different values ​​are repeatedly stacked together in a cross manner to form a new feature. The stacked features are processed using a three-dimensional convolution kernel, and different matching cost volumes and uncertainty volumes are obtained through four branches. The matching cost volume calculates a classification probability vector for each pixel position. Each element in the vector corresponds to a possible disparity value, indicating the probability that the matching result is the disparity value; the uncertainty volume has three dimensions w×h×S max , w and h are the width and height of the feature map respectively, S max is the maximum value of the disparity range. For each pixel position, S max The vector of dimension stores the uncertainty parameter values ​​under all possible disparity values.

[0040] 4) Obstacle avoidance is accomplished by calculating the credibility of the predicted information. First, the matching cost is used as a guide, and based on evidence deep learning, a neural network is trained to calculate the hyperparameters γ, ν, α, and β of the evidence distribution.

[0041] Specifically, the evidence deep learning method regards learning as a process of evidence collection, selects the conjugate distribution of the Gaussian distribution, that is, the normal inverse gamma distribution as the evidence distribution, and then models a higher-order evidence distribution on the likelihood function parameters as the posterior distribution based on the evidence distribution.

[0042] The formula for the posterior distribution is:

[0043] q(μ,σ 2 )=p(μ,σ 2 |d1,d2,…,d N )

[0044] Where p represents the maximum likelihood, d i represents the estimated disparity value of the i-th pixel in the matching cost volume, specifically (d1, d2, ..., d N )~N(μ,σ 2 ), the mean μ follows a Gaussian distribution and is expressed as μ~N(γ,σ 2 v -1 ), variance σ 2 It follows the inverse gamma distribution represented by σ 2 ~Γ -1 (α, β), Γ -1 (·) is the inverse gamma equation, N represents Gaussian distribution, υ>0,α>1,β>0.

[0045] Specifically, the neural network is trained by minimizing the loss function. The complete loss function consists of the evidence learning loss term and two regularization loss terms, and the formula is:

[0046]

[0047] Where w represents the neural network model parameters; Learn a loss function for the evidence; and are two regularization loss terms; pred ,λ smooth and λ sup is the balancing factor. Used to maximize model evidence with the support of training samples, We strive to give a larger uncertainty value in the area of ​​​​error disparity estimation. When the on-board camera is blurred, blocked, overexposed, or enters an unfamiliar scene, the binocular matching algorithm has a high probability of making prediction errors. By estimating smaller evidence parameters v and α, a larger uncertainty value is obtained. The above regularization term only constrains α and υ, but not β. In order to reduce the noise of the hyperparameters of the output normal inverse gamma distribution and make the β value estimation more accurate, it is necessary to constrain the output of the area without real disparity annotation by utilizing the information of the area with real disparity annotation. For the binocular matching task, the hyperparameter estimation can benefit from the information of nearby areas and is applicable to areas with continuous disparity. Therefore, a smooth regularization loss term is proposed This loss term constrains the hyperparameter values ​​α, β, and υ in the parallax smoothing region. The smoothness of the uncertainty parameters is constrained in the disparity smooth region, so that the constraint on uncertainty is propagated from pixels with true disparity annotations to pixels without true disparity annotations, thus improving the constraint on uncertainty estimation.

[0048] 5) Following the framework of deep evidence regression, a neural network is trained to infer the hyperparameters m = (γ, υ, α, β) of the NIG evidence distribution. The binocular disparity is estimated through the trained network, and the accidental uncertainty and cognitive uncertainty are calculated through a single forward pass operation. A high uncertainty is given to the depth map that may have a large deviation, and an alarm can be given in time to prevent the autonomous driving system from making inappropriate decisions due to incorrectly estimated depth values. Specifically, binocular disparity, accidental uncertainty, and cognitive uncertainty can be calculated by the following formula:

[0049] E[μ]=γ

[0050]

[0051]

[0052] Where μ is the mean, σ 2 is the variance, γ, υ, α, β are the hyperparameters required for the calculation learned by the network.

[0053] The present invention is applicable to most binocular matching methods. PSM-Net, GA-Net and AA-Net are selected as the basic models, and the source code is implemented using the PyTorch framework. During training, the binocular image pairs are randomly cropped and input into the network. The size of the cropped images is consistent with the basic model in the original paper. The network is trained using 4 NVIDIA 2080TI GPUs, with 1 sample per training. The network is first pre-trained on the Sceneflow dataset for 15 training iterations. The network is then fine-tuned on the KITTI dataset with 200 training iterations, where the initial learning rate is set to 0.001.

[0054] This experiment compares the results of two different network structures to estimate disparity and uncertainty, as shown in Table 1. PSM-Net-re, GA-Net-re and AA-Net-re represent methods that directly convolve the output v, α and β from the cost aggregation module without using matching probability as a guide. PSM-Net-un, GA-Net-un and AA-Net-un represent the methods described in the present invention. In order to verify the effectiveness of the present invention, the average error is used as a metric to evaluate the average deviation of the estimated disparity from the ground truth. In addition, the error rate is calculated using the threshold of 3 recommended in the literature (Self-supervised multiscale adversarial regression network forstereo disparity estimation. IEEE Transactions on Cybernetics, 2020). In order to evaluate the quality of uncertainty, all pixels in the disparity map are first sorted according to the level of uncertainty elimination. Then, the pixels with the highest uncertainty are removed, and the disparity metric is calculated on the remaining pixels. Finally, the pixels are sorted according to the true error to obtain the best sparse map. Finally, two quantitative indicators are used: the area under the sparsification error (AUSE) and the area under the random gain (AURG). AUSE represents the difference between the estimated sparse graph and the optimal sparse graph, so the lower the better. AURG is obtained by subtracting the estimated sparse graph from the random sparse graph, so the higher the better.

[0055] Table 1 Comparison of estimation results

[0056]

[0057] Table 1 shows that PSM-Net-re, GA-Net-re and AA-Net-re perform poorly in uncertainty estimation compared to the present invention. In addition, they also reduce the accuracy of binocular matching because all pixels use the same convolution kernel without matching probability as a guide, and the evidence of different differences cannot be distinguished. By using matching probability as a guide, uncertainty can effectively reflect the difficulty of binocular matching. Therefore, the accidental uncertainty and epistemic uncertainty modeled by the present invention can improve the binocular matching performance.

[0058] Figure 2 The visualization results of binocular matching and uncertainty estimation are shown, (a) left eye image; (b) estimated disparity of PSM-Net-re; (c) estimated disparity of the present invention; (d) disparity error map; (e) accidental uncertainty; (f) cognitive uncertainty, where the brighter the part in (e) (f), the higher the uncertainty. Figure 2It can be seen that (b) PSM-Net-re has more brighter areas in the estimated disparity map, especially the sky part has a larger brightness difference compared with (c), and the estimated disparity is also larger. This is because some areas are difficult to match, such as the sky, object boundaries and small targets, which are assigned high uncertainty and are likely to get wrong predictions. The method described in the present invention performs well in these areas, and uncertainty modeling in binocular matching can be used to learn loss attenuation, thereby improving accuracy.

[0059] In some fields with high safety requirements, such as autonomous driving, in addition to obtaining the calculation results of the model, it is also necessary to evaluate the credibility of the results. Taking depth estimation based on binocular matching as an example, if it is known that the uncertainty of the depth value calculated at a certain position is high, an early warning can be given in time to avoid accidents. Uncertainty can be divided into accidental uncertainty and epistemic uncertainty according to its source. Accidental uncertainty describes the inherent noise in the data, which is an inevitable error and cannot be eliminated by increasing the number of samples. Epistemic uncertainty has nothing to do with a single data. It is the model's own estimate of the input data, which may be inaccurate due to insufficient training or insufficient training data set. It represents the uncertainty of the model parameters estimated by the training process itself.

[0060] Autonomous driving is just an example. The present invention can also be applied to other technical fields such as aerial surveying and mapping, virtual 3D reconstruction, and industrial precision detection, etc. However, the technologies used all belong to the field of computer vision technology.

[0061] Although the present invention has been described above in conjunction with the accompanying drawings, the present invention is not limited to the above-mentioned specific embodiments, which are merely illustrative rather than restrictive. Under the guidance of the present invention, ordinary technicians in this field can make many modifications without departing from the purpose of the present invention, all of which are within the protection of the present invention.

Claims

1. A binocular matching uncertainty estimation method based on evidence deep learning, characterized in that: Here are the steps: (1) Input the left and right images, and use the Siamese network with shared parameters to extract the features of the left and right images respectively; through a series of two-dimensional convolution and pooling operations, the feature maps of the left and right images are obtained; (2) The feature maps of the left and right images are concatenated together, and the feature maps are cost aggregated using a three-dimensional convolution operation to obtain a matching cost volume and an uncertainty volume; the matching cost volume and the uncertainty volume are obtained after cost aggregation of the features, and the matching cost volume is used to calculate the matching similarity at different disparities, thereby obtaining the parameters The uncertainty body is used to calculate the uncertainty parameters at different disparities, and under the guidance of the above matching similarity, the parameters are calculated. , and ; (3) Based on the matching cost body, a neural network is trained based on evidence deep learning. The evidence deep learning method regards learning as a process of evidence collection and selects the conjugate distribution of the Gaussian distribution, namely the normal inverse gamma distribution, as the evidence distribution. Then, based on the evidence distribution, a higher-order evidence distribution is modeled on the likelihood function parameters as the posterior distribution. Finally, the hyperparameters of the evidence distribution are calculated. , , and , these parameters are those of the normal inverse gamma distribution and are used to account for aleatoric uncertainty and epistemic uncertainty; (4) Estimating binocular disparity through a trained neural network and calculating aleatoric uncertainty and epistemic uncertainty through a single forward pass operation; In step (4), the neural network is trained by minimizing the loss function, which consists of the evidence learning loss and two regularization loss terms, and the formula is: ， in, represents the neural network model parameters, Learn a loss function for the evidence; and are two regularization loss terms; , and is the balancing factor; Used to maximize model evidence with the support of training samples, Focus on estimating smaller evidence parameters in regions with incorrect disparity estimates and , resulting in a larger uncertainty value, The smoothness of the uncertainty parameters is constrained in the disparity smooth region, so that the constraint on uncertainty is propagated from pixels with true disparity annotations to pixels without true disparity annotations, thus improving the constraint on uncertainty estimation.

2. The binocular matching uncertainty estimation method based on evidence deep learning according to claim 1 is characterized in that: The specific implementation of step (2) is: (21) Obtain the matching cost body. For each pixel position, a classification probability vector is calculated. Each element in the classification probability vector corresponds to a possible disparity value, indicating the probability that the matching result is the disparity value. The uncertainty body has three dimensions: , and are the width and height of the feature map, respectively. is the maximum value of the disparity range; the obtained matching cost volume calculates a classification probability vector for each pixel position in the left eye image. The possible disparity value refers to the range between the minimum disparity value and the maximum disparity value determined by the internal and external parameters of the camera, and discretizes it into a finite number of values; (22) Repeat each channel of the feature map of the left image To construct the 3D features, we use the feature map of the left image as a reference and use different displacement values, i.e. 1 pixel, 2 pixels, ... Pixel-shift each channel of the right image feature map and then stack it into a three-dimensional feature; (23) Repeatedly stacking the left image features and the right image features translated by different values ​​in a cross manner to form a new feature; (24) The stacked features are processed using a three-dimensional convolution kernel, and different matching cost volumes and uncertainty volumes are obtained through four branches.

3. The binocular matching uncertainty estimation method based on evidence deep learning according to claim 1, characterized in that: In step (3), the formula of the posterior distribution is: ， in represents the maximum likelihood, Indicates the first The estimated disparity value of pixels is , mean The Gaussian distribution is expressed as ,variance The inverse gamma distribution is expressed as , is the inverse gamma equation, N represents the Gaussian distribution, , , , .

4. The binocular matching uncertainty estimation method based on evidence deep learning according to claim 1, characterized in that: In step (4), aleatoric uncertainty and epistemic uncertainty are calculated by the following formula: ， ， ， in, is the mean, is the variance, The hyperparameters required for the computation learned by the network, , and They represent parallax, aleatory uncertainty, and epistemic uncertainty respectively.

Images