Stereo matching method based on unsupervised online learning

Abstract

The invention belongs to the technical field of stereoscopic vision, and discloses a stereoscopic matching method based on unsupervised online learning. The method is realized through a stereo matching module based on a multi-level cost body, a non-occlusion region confidence estimation module and a Mann-Kendall trend detection module. Compared with the prior art, the method has a better stereo matching effect. The model provided by the invention has the best result in both the initial stage and the later stage, and due to the good generalization of the model, the model can always keep a relatively accurate prediction result in a sheltered region. The model has a good prediction result in the occlusion area due to high generalization, and the tendency of parameter error update is relieved due to the fact that the self-adaptive adjustment strategy provided by the invention eliminates the influence of error loss of most of the occlusion areas, so that the network can maintain the optimal performance all the time.

Description

Technical Field

[0001] The present invention relates to the field of stereo vision technology, and in particular to a stereo matching method based on unsupervised online learning. Background Art

[0002] As one of the depth estimation methods, stereo matching has improved its performance with the development of computer graphics and deep learning. The current stereo matching methods mainly include traditional methods based on artificial feature matching and methods based on deep learning.

[0003] Traditional stereo matching methods extract pixel features from images based on artificially designed mathematical models. The basic process includes four steps: matching cost calculation, matching cost aggregation, disparity calculation, and disparity refinement. For example, in the matching cost calculation, the left and right image features are extracted based on the twin network, and the matching cost is calculated using the full convolution form; in the cost aggregation stage, a random forest classifier is used to detect highly reliable pixels as control points, and soft constraints are introduced into the matching body for optimization; in the disparity refinement step, after the left and right consistency detection, image processing techniques such as median filtering or bilateral filtering are traditionally used to post-process the disparity map. However, the artificially designed mathematical model considers limited feature possibilities, resulting in a decrease in the robustness of traditional methods in complex scenes.

[0004] Stereo matching methods based on deep learning use a large amount of training data, and perform deep nonlinear transformations on images through multiple function activations, pooling and other operations to drive the network to extract rich image features. According to the network structure, it can be roughly divided into two categories: 2D structure and 3D structure. The stereo matching model of the 2D structure is inspired by the U-Net model and is usually designed in the form of an encoder-decoder, using a correlation layer to calculate the similarity score between image features. Some work starts from learning residual information, combining the DispNetC model with a second sub-network, and determining the refinement of the initial disparity through a series of operations. The stereo matching model of the 3D structure explicitly encodes geometric information based on the encoder-decoder design. Some methods extract high-level features through two encoders with shared weights, construct a cost body and process it to obtain a disparity map.

[0005] In terms of domain adaptation strategies, some methods try to introduce pseudo labels with high confidence as supervisory information to guide network fine-tuning. Some methods use traditional stereo matching methods to obtain dense disparity, and then use confidence estimation methods to select disparity with high accuracy as pseudo labels. There are also methods that use the connection between disparity or optical flow and the input image itself to perform self-supervisory training on the network. For example, "Zhong Y, Li H, Dai Y. Open-World Stereo Video Matching with Deep RNN [C]. Computer Vision-ECCV 2018, 2018: 104-119" uses two CNN-LSTM modules to build a network that can be deployed without pre-training, and can learn and modify network parameters from stored past experience. It takes continuous binocular sequences as input and can basically converge after hundreds of frames of online learning. The proposed method requires more than 1 second per frame inference speed, requires a longer sequence input to achieve better results, and the performance stability on different data sets may be insufficient. "Tonioni A, Tosi F, Poggi M, et al. Real-Time Self-Adaptive Deep Stereo [C]. 2019 IEEE / CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019: 195-204" proposed a lightweight stereo matching network and an online learning strategy for modular fine-tuning networks. This strategy randomly selects a module in the network for update each time. After hundreds of iterations, its performance can be close to the strategy of full parameter update. While the proposed method improves the running (training and inference) speed of the model, it sacrifices the convergence speed of the network, that is, more iterations are required to adapt to the deployment scenario.

[0006] In terms of guided deep learning, some methods provide external cues to guide neural network adjustments, such as regulating the prediction results by enhancing or suppressing feature activations in the network, focusing a Gaussian function on each disparity value provided by sparse cues, and modulating features that have a strong relationship with the output according to the Gaussian function. Summary of the invention

[0007] The present invention aims to solve the domain shift problem faced by stereo matching networks during deployment, improve the generalization and accuracy of stereo matching networks, and reduce the computational cost and overfitting risk in the online learning process of the model.

[0008] A stereo matching method based on unsupervised online learning is implemented by a stereo matching module based on a multi-level cost volume, a non-occluded area confidence estimation module and a Mann-Kendall trend detection module; the left image and the right image are input into the stereo matching module to obtain the predicted disparity; the stereo matching module starts with a set of initial weights, and when new image data arrives, it predicts the corresponding disparity and chooses whether to update the stereo matching module parameters; the noisy reprojection loss is calculated based on the disparity predicted by the stereo matching module and the left image and the right image The non-occluded area confidence estimation module predicts the occluded area mask of the left view based on the Sobel edge information of the left image and the cost volume information output by the stereo matching module, which is used to filter the reprojection loss. The error in , forming the final self-supervised reprojection loss The self-supervised reprojection loss is recorded by the Mann-Kendall trend detection module, which statistically analyzes the trend of the self-supervised reprojection loss in recent frames to determine the convergence of the stereo matching module; the result of the Mann-Kendall trend detection module is used as a controller to determine whether the self-supervised reprojection loss needs to be used. The stereo matching module is adapted online, and switches autonomously between fine-tuning the stereo matching module and predicting disparity.

[0009] The training of the overall network of the non-occluded area confidence estimation module, the stereo matching module and the trend detection module is divided into two stages: online and offline; offline training is first performed to learn the parameters of the above three modules, and then online training is performed to learn the parameters of the stereo matching module; after the non-occluded area confidence estimation module is trained in the offline stage, its parameters will not be adjusted in the online adaptation stage.

[0010] The stereo matching module is a stereo matching module based on a multi-level cost body, which includes a feature extraction submodule and a disparity estimation submodule;

[0011] In the stereo matching module, the left and right images are input into the feature extraction submodule, and the multi-level feature F is output. 1 ~F 6 , the multi-level features are used to construct a multi-level cost body V 1 ~V 6 ;

[0012] Disparity Estimation Submodule D 2 ~D 6 Input corresponding cost volume V 2 ~V 6 , output 5 levels of coarse parallax d 2 [1 / 4]~d 6[1 / 64], where [1 / 4] means that the image resolution of this level of parallax is 1 / 4 of the input left image resolution; the previous level cost volume V i One of the inputs is the next level coarse disparity d i+1 , d i+1 By the corresponding level of cost body V i+1 After the disparity estimation submodule D i+1 Get; the disparity refinement submodule inputs the coarse disparity D 2 and the cost body V 1 , output the final disparity D; so the highest level cost volume V 1 The remaining coarse disparity and cost volume information are summarized as the input of the disparity refinement submodule, which is a coarse-to-fine disparity estimation method.

[0013] Before the stereo matching module is adapted online, it is trained offline using a synthetic dataset to obtain the initial weights. The supervisory information used in offline training is the true disparity value. The objective function is shown in formula (1). The disparity of each resolution will be scaled to the full resolution and then contribute to each other. The weight β is set to [0.005, 0.005, 0.01, 0.02, 0.08, 0.32].

[0014]

[0015] The feature extraction submodule is a domain normalization feature extraction submodule, which inputs the left image and the right image; firstly, the input image is subjected to standard normalization, and then a series of "convolution-convolution-domain normalization-Leaky ReLU activation function" structure is used to extract multi-level features F 1 ~F 6 As output; each level feature F i Contains the left features corresponding to the left and right images Right feature The convolution kernels of the two-dimensional convolution used in the domain normalization feature extraction submodule are all 3×3, and s1 and s2 represent the convolution step sizes of 1 and 2 respectively; the specific operation of domain normalization in the domain normalization feature extraction submodule is to perform L2 normalization on the features along the channel dimension of the convolution layer to extract features.

[0016] The construction of the multi-level cost body V is divided into two steps: the first step is to construct the cost body V 6 , by directly concatenating features in the channel dimension and get;

[0017]

[0018] The second step is to construct the cost volume V i , i=1,2,…5, first use the warp operation to transform the features Transformed to

[0019]

[0020] where x is the pixel coordinate, d i+1 is the disparity estimation submodule D i+1 Output coarse disparity; cost volume V i ,i=1,2,…5by and Channel dimension concatenation features and get;

[0021]

[0022] Where concat represents the feature concatenation operation.

[0023] The disparity estimation submodule is a disparity estimation submodule based on jump connection, which inputs the cost volume V and outputs the coarse disparity d i , i=2,3,…,6; the convolution kernel size of each convolution layer in the disparity estimation submodule based on skip connection is 3×3, the step size is 1, and the number of output feature channels are 128, 128, 96, 64, 32 and 1 respectively. These feature channel numbers are kept fixed in the disparity estimation modules at all levels; skip connection means that the input of each convolution layer is the output and input of the previous layer, that is, the input of the i-th convolution layer is composed of the output of the i-1th layer and the i-2th layer.

[0024] The disparity refinement submodule is a disparity refinement submodule based on dilated convolution, and the input cost volume V 1 and coarse parallax d 2 , and outputs the refined final disparity D; the disparity refinement submodule contains 5 layers of dilated convolutions, with expansion rates of 1, 2, 4, 1, 1 respectively, and the number of output channels of each dilated convolution is 64, 64, 32, 16, 1.

[0025] The non-occluded area confidence estimation module is divided into two inputs. The first input is the cost volume V 1 ; The first input is first downsampled to 1 / 4 the resolution of the input image, and then passes through three kernels to 3×3 The convolutional layer outputs rough confidence information; for the first two convolutional layers, domain normalization is performed and leaky ReLU is used as the activation function; the second input is the left image, the original image is first downsampled to the corresponding resolution, the edge information is obtained by mean filtering and sobel operator, it is cascaded with the rough confidence, and the edge area is refined by the hole convolution; the output of the non-occluded area confidence estimation module is the non-occluded area confidence

[0026] The objective function of the non-occluded area confidence estimation module is the following weighted cross entropy loss

[0027]

[0028] Among them, w is used to describe the importance of the non-occluded area in the loss. In order to balance the difference in the number of samples of the two types and increase the prediction tendency of the non-occluded area confidence estimation module for the non-occluded area, w is set to 0.3; y i is the true label of the confidence of the non-occluded area;

[0029] After obtaining the confidence output of the non-occluded area, it is converted into a mask map of the occluded area according to the probability threshold, which is used to completely filter the predicted occluded area; the probability threshold is set to 0.5, and the formula for calculating the mask of the occluded area is (4);

[0030]

[0031] In the online learning stage, the unequalized reprojection loss is first calculated according to formula (5); then the reprojection loss is filtered at the pixel level according to the mask to construct a benign loss for supervising network learning during online adaptation, as shown in formula (6);

[0032]

[0033] in, The left image is converted from the right image based on the disparity. SSIM represents the model that measures the structural differences between images. λ is set to 0.85.

[0034] In the offline learning stage, the non-occluded area confidence estimation module, stereo matching module and trend detection module are jointly trained, and the objective function is designed as formula (7);

[0035]

[0036] The online trend detection module based on Mann-Kendall theory detects the most recent frames of the time series during the online learning process. Perform statistical analysis to determine the operating status of the stereo matching module. For a loss sequence of given length n First calculate the statistic S:

[0037]

[0038] in, and represents the loss at time i, j, sgn(·) is the indicator function;

[0039]

[0040] The positive or negative value of the statistic S represents the upward or downward trend of the sequence; when n>8, the statistic S is approximately normally distributed, and its mean E(S) and variance Var(S) are shown in formula (10);

[0041] E(S)=0

[0042]

[0043] Among them, m is the number of knot groups in the sequence, and a knot group is a set of data with the same value in the sequence, and τ i is the number of elements in the i-th knot group; the statistic S is converted into a standard normal distribution according to the mean and variance to form a standard statistic Z;

[0044]

[0045] Among them, α is the statistical significance level, α = 0.05, that is, Z 1-α / 2 =1.96; set the sequence length n to 12; when |Z|>Z 1-α / 2 , indicating that the loss function There is a significant monotonic trend in the time series, and the positive or negative value of Z represents the upward or downward trend of the sequence. At this time, the trend detection module switches the operating state of the stereo matching module to "fine-tuning"; otherwise, it is considered that the sequence has no obvious trend, and the trend detection module switches the operating state of the stereo matching module to "inference".

[0046] Beneficial effects of the present invention: Compared with the prior art, the present invention has the following outstanding substantive features and significant improvements:

[0047] (1) When designing a stereo matching network, the present invention uses a multi-level cost body to introduce prior knowledge of stereo matching, integrates the geometric information of disparity and image pairs, and improves the feature representation capability compared to existing methods.

[0048] (2) The domain normalization feature extraction submodule of the present invention removes the influence of domain offset (such as color, style, brightness, etc.) and local contrast change at the image level, thereby improving the stability of feature extraction when facing different scene data.

[0049] (3) Compared with the prior art, the disparity estimation submodule based on skip connection of the present invention enhances the reusability of the extracted features, so that the features can fully transmit information and alleviate the gradient vanishing problem of the network during training; at the same time, it reduces the number of connections of dense connections and improves computational efficiency.

[0050] (4) The disparity refinement submodule of the present invention effectively improves the details lacking in the coarse disparity by expanding the receptive field in the shallow features, reduces the noise in the output disparity, and improves the quality of the final disparity map.

[0051] (5) Compared with the prior art, the non-occluded area confidence estimation module of the present invention increases the accuracy of the predicted confidence by combining the Sobel edge information of the input image.

[0052] (6) In summary, the present invention proposes a stereo matching network suitable for online fine-tuning and proposes a corresponding online fine-tuning strategy, which overcomes the serious dataset domain offset problem in the prior art, that is, it is difficult to be widely deployed after being trained on a synthetic dataset. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is the overall technical route of the present invention;

[0054] Figure 2 It is the overall structure of the stereo matching network;

[0055] Figure 3 It is the structure of feature extraction submodule;

[0056] Figure 4 The structure of the disparity estimation submodule;

[0057] Figure 5 Refine the structure of the submodule for disparity;

[0058] Figure 6 The structure of the non-occluded area confidence estimation module;

[0059] Figure 7 Comparison of disparity and error visualization during online adaptation; (a) is the left picture; (b) is the real disparity; (c) is the predicted occlusion map; (d) is the stereo matching effect of the prior art DispNet-WAD; (e) is the stereo matching effect of the prior art MADNet-AD; (f) is the stereo matching effect of the UONet of the present invention;

[0060] Figure 8 This is the online adaptive error curve of UONet of the present invention and other methods. DETAILED DESCRIPTION

[0061] The overall technical solution of the present invention is as follows Figure 1As shown in the figure, it mainly includes three parts: a lightweight and highly generalized stereo matching network based on a multi-level cost body, a non-occluded area confidence estimation network, and an online trend detection method based on the Mann-Kendall theory. The proposed model starts with a set of weights with good generalization. When new data arrives, it predicts the corresponding disparity and chooses whether to update the network parameters in a timely manner. When a new image arrives, first, the stereo matching network predicts the disparity as the output of the model, and calculates the reprojection loss based on the predicted disparity and the left and right images. At the same time, the non-occluded area confidence estimation network predicts the occluded area of ​​the left view based on the edge information of the left image and the relevant information of the shallow features, which is used to filter out errors in the reprojection loss and form a new self-supervised loss. Then, this loss will be recorded by the Mann-Kendall trend detection, and the loss change trend of the last few frames will be statistically analyzed to determine the convergence of the model. Finally, the result of the trend detection will be used as a controller to autonomously switch the operation mode of the model, that is, whether it is necessary to use the loss to perform online adaptation of the stereo matching network. 1. Stereo matching network based on multi-level cost body

[0062] The present invention constructs a lightweight and well-generalized stereo matching network. The lightweight requirement is to ensure that the model can maintain a high operating speed when the data is propagated bidirectionally before and after the online adaptation. The generalization requirement is to ensure that the model has relatively accurate initial parameters when crossing domains, and can perform a small amount of online adaptation starting from relatively more accurate results. The complete structure of the stereo matching network designed by the present invention is as follows: Figure 2 shown.

[0063] Coarse disparity used when constructing the cost volume (i represents resolution) is a positive or negative value automatically learned by the network, where positive and negative represent right and left disparity, respectively. Then, the features are warped using formula (1) to form new features. And according to its characteristics F with the original view l The cost body is constructed based on the correlation.

[0064]

[0065] Where x represents the pixel coordinate, l and r represent the left and right inputs of the network respectively. In order to solve the generalization problem, the present invention introduces the parallax constraint condition on this basis. First, the relu activation function is used to constrain the parallax Mapping to make it uniformly non-negative Then, formula (2) is used to warp the features of the right image and construct a cost volume for predicting the left disparity.

[0066]

[0067] If you need to swap the left and right input images to predict the right disparity, add a conditional branch and use And formula (1) to construct the cost body.

[0068] In order to enhance the universality of the extracted features and further strengthen the generalization of the stereo matching network, the present invention introduces a domain normalization operation on the features of the convolution output in the feature extraction module and uses a leaky relu activation function. In order to unify the consistency of data distribution, the left and right images are converted into a standard normal distribution according to the mean and standard deviation of each channel before being input into the stereo matching network.

[0069] In order to reduce the memory consumption of the computing unit, the present invention proposes the following Figure 4 The jump connection shown on the right. For the i-th convolution layer, its input consists of the i-1 and i-2 convolution outputs. This connection method greatly reduces memory consumption while enhancing the reuse and fluidity of features to a certain extent, improving the accuracy of prediction results.

[0070] In order to further improve the accuracy of the stereo matching network in disparity estimation of details, the present invention uses shallower features and correlation information as input and reconstructs a disparity refinement module consisting of five layers of dilated convolutions, such as Figure 5 First, the coarse parallax Bilinear upsampling to 1 / 2 resolution of input image Used to transform the shallow features of the right image Transform to the left, with the left image features Build correlation information together 1 The texture input to this refinement module is then constructed as F 1 l 、c 1 , and cascade, where coarse parallax It will also be used for the residual connection to output the refined final disparity.

[0071] Before the online adaptation, the stereo matching network is trained offline using a synthetic dataset to obtain good initial weights. The supervision information used in offline training is the true disparity value, and the objective function is shown in formula (3), which only calculates the predicted disparity. L between the true value d 1 The parallax of each resolution will be scaled to the full resolution and contribute separately, with the weight determined by β = [0.005, 0.005, 0.01, 0.02, 0.08, 0.32].

[0072]

[0073] Where N represents the number of pixels in the image.

[0074] 2. Non-occluded area confidence estimation network

[0075] The present invention designs a subnetwork for predicting occluded areas, namely, the confidence estimation network for non-occluded areas. In order to reduce memory usage and increase running speed, the resolution of the tensors operated by the network is 1 / 4 of the input image, and the network has two inputs. The specific structure is as follows: Figure 6 As shown. One of the inputs is the feature F on the left. 1 l And the transformed features The correlation between 1 , where the coarse disparity information used in the transformation is Since the coarse parallax has good accuracy, c 1 It can well reflect the information of the occluded area. This input will first be downsampled to 1 / 4 resolution of the input image, and then pass through three convolutional layers with a kernel of 3×3 to output rough confidence information. The first two layers of convolution are followed by domain normalization and leaky relu activation function to enhance the universality of features, and the third layer is only convolution. Since occlusion generally occurs near the edge of spatial objects, in order to enhance the accuracy of the prediction results, the second input of the network is designed to be sobel edge information. First, the original image is downsampled to the corresponding resolution, and then the edge information is obtained by mean filtering and sobel operator, and finally it is cascaded with the rough confidence information to guide the subsequent void convolution and residual connection to refine the edge area.

[0076] The present invention adopts supervised offline training for the network, and the data and labels come from a subset of FlyingThings3D. Since this task can be regarded as a pixel-level binary classification problem, the cross entropy loss is used as the objective function. In addition, the proportion of occluded areas in each image is relatively small, about 15%, so the objective function is modified to the weighted cross entropy loss shown in formula (4).

[0077]

[0078] Wherein w is used to describe the importance of positive samples (non-occluded areas) in the loss. In order to balance the difference in the number of samples of the two types and slightly increase the network's prediction tendency for non-occluded areas, the present invention sets w to 0.3.

[0079] After obtaining the confidence output of the non-occluded area Afterwards, it needs to be converted into a mask map of the occluded area according to the threshold value, so as to completely filter the predicted occluded area. The present invention sets the probability threshold value to 0.5, that is, generates a mask according to the formula.

[0080]

[0081] Then, the reprojection loss can be filtered pixel-wise based on this mask to construct a benign loss for supervised network learning during online adaptation, as shown in formula (7). The result of formula (6) is the unequalized reprojection loss.

[0082]

[0083] in The left image is converted from the right image based on the disparity. SSIM represents a model that measures the structural differences between images, and λ is set to 0.85.

[0084] After the non-occluded area confidence estimation network is trained in the offline stage, it will not be adjusted in the online adaptive stage. Therefore, it is not directly associated with the input image at the beginning of the design to avoid affecting its generalization. In the offline stage, the network can be trained jointly with the stereo matching network, and the predicted confidence information can further promote the learning of the stereo matching network. Therefore, the objective function of the entire network in the offline stage is designed as formula (8).

[0085]

[0086] 3. Mann-Kendall Trend Detection

[0087] When in online adaptation, assuming that the model has adapted to the current scene and the scene no longer changes significantly, if the model continues to learn, the response speed will be seriously reduced, but there will be no obvious improvement in network performance, and the network may fall into overfitting. Therefore, a strategy is needed to monitor the convergence of the model and autonomously switch the model's operating mode (adaptation or inference). The present invention can obtain a more accurate loss in the online process by filtering the reprojection loss with the predicted occlusion area mask. In the adaptive process, the most recent frames in the time series are Conduct statistical analysis and record The Mann-Kendall trend test is used to analyze the trend in the sequence, and the null hypothesis is that there is no obvious trend. Then for a loss sequence of a given length n First calculate the statistic S:

[0088]

[0089] in sgn(·) is the indicator function:

[0090]

[0091] The positive (negative) value of the statistic S represents the upward (downward) trend of the sequence. When n>8, the statistic S is approximately normally distributed, and its mean E(S) and variance Var(S) are shown in formula (11).

[0092]

[0093] Where m is the number of tied groups in the sequence, a tied group is a set of data with the same value in the sequence, and τ i is the number of elements in the i-th knot group. Then, the statistic S is transformed into a standard normal distribution according to the mean and variance to form a standard statistic Z.

[0094]

[0095] When |Z|>Z 1-α / 2 When , the null hypothesis of no obvious trend can be rejected, indicating that the sequence has a significant monotonic trend. Similarly, the positive (negative) value of Z represents the upward (downward) trend of the sequence. Where α is the statistical significance level, and in this invention, α=0.05, that is, Z 1-α / 2 =1.96. In addition, the sequence length n is set to 12 in the present invention.

[0096] Innovation in Stereo Matching Network Design

[0097] 1.1 Lightweight and highly generalizable stereo matching network

[0098] Multi-level cost body construction: The ReLU activation function is introduced to implement disparity constraints, prompting the network to learn in the direction of the constraints and improve generalization.

[0099] Feature extraction module: Add domain normalization operations to enhance the versatility of the extracted features, extract similar features from images in different domains, and strengthen the generalization ability of the network.

[0100] Disparity estimation module: Add inter-layer skip connections to enhance feature fluidity and reusability, and improve the accuracy of coarse disparity estimation.

[0101] Disparity refinement module: introduces shallow image features and left-right feature correlation information, focuses on details, and improves the performance of the network at details such as object edges.

[0102] 1.2 Overall performance balance: While maintaining the lightweight of the network, the generalization and accuracy of the network are improved. Experiments have shown that the performance on different data sets is better than some existing mainstream methods.

[0103] Innovation in unsupervised online learning strategies

[0104] 2.1 Non-occluded area confidence estimation network

[0105] Error loss filtering mechanism: To address the problem of incorrect calculation of online learning reprojection loss in occluded areas, this network is constructed to generate a mask map of the occluded area based on the confidence level, filter the error loss, provide accurate supervision information for network parameter updates, reduce the number of fine-tuning frames required for convergence, and improve the accuracy after fine-tuning.

[0106] 2.2 Trend Detection Module

[0107] Model adaptability judgment method: Based on the Mann-Kendall trend detection principle, the model and scene adaptability are detected according to the filtered reprojection loss. If there is no obvious trend in the loss, it is determined that the model is adapted to the scene and switched to inference mode. Otherwise, the network will continue to be fine-tuned to alleviate model overfitting and improve the overall running speed.

[0108] Online and offline experiments on the public datasets FlyingThings3D, KITTI stereo 2012 and 2015, DrivingStereo, and Middlebury demonstrate that the proposed method is superior to the existing techniques.

[0109] Compared with the prior art, the stereo matching network proposed in the present invention has a good balance between generalization and lightness. Table 1 shows the quantitative comparison results of the present invention and multiple prior arts. The network with the best generalization in the prior art is DSMNet, which can predict good results across domains even without fine-tuning, but the domain offset problem will still slightly reduce its accuracy. In addition, this method is too complicated, and the inference time of a single frame takes about 2 seconds, which is not suitable for additional parameter updates during deployment. The stereo matching network of the present invention is comparable to that of DSMNet, and its running time is only about 1% of that of DSMNet, which is more suitable for online adaptation during deployment. Compared with lightweight networks such as DispNetC and MADNet, the present invention has a great advantage in generalization and is comparable to them in running time, so it has better initial parameters for online adaptive deployment.

[0110] Table 1 Comparison of cross-domain generalization of stereo matching networks

[0111]

[0112]

[0113] from Figure 7The results of Ours-FAD also show that the model of the present invention has the best results both in the early and late stages, and due to its good generalization, it can maintain relatively accurate prediction results in occluded areas. From the analysis of the prior art MADNet-AD, it can be seen that since the network itself has poor generalization, during the parameter update process, the error is steadily decreasing, but in the later stage it stays at the stage where the parallax in the occluded area cannot be further optimized. The prior art DispNet-WAD has the ability to quickly adapt to scenes, but because its confidence results are only used to assign different weights to the reprojection loss, it does not actually remove the impact of the incorrectly calculated loss, so its performance after convergence still has a high error rate. Combined with Figure 7 It can be seen from the disparity and error effects of DispNet-WAD and MADNet-AD that they are ultimately unable to estimate reasonable disparity in occluded areas due to poor generalization, resulting in poor performance, especially the results in occluded areas. The model of the present invention has good prediction results in occluded areas due to its strong generalization, and the adaptive adjustment strategy proposed by the present invention eliminates the influence of most of the error losses in occluded areas, alleviates the tendency of parameter error updates, and enables the network to maintain optimal performance.

[0114] Compared with the prior art, the online adaptive strategy of the present invention is better. Figure 8 The performance changes of four different models and adaptive strategies on the same sequence are shown in Figure 1. The performance indicator used is D1. Figure 8 It can be seen that at the beginning of the adaptation, the stereo matching network of the present invention has good accuracy due to its high generalization. After about 300 frames of adaptation, the model of the present invention and the corresponding adaptive strategy "FAD-MK" have basically adapted to the scene, and the performance is significantly better than other models and adaptive schemes. The sudden increase in the curve is because the model encounters a new scene with large changes. Among them, the DispNet-L2AWAD scheme, due to its heavy reliance on training data during meta-training, even if the input is scaled to 1 / 2, its disparity range (about 0 to 100) is significantly larger than the test data (about 0 to 70), which makes its initial prediction results very poor (severely reduces generalization). However, the learn-to-adapt strategy enables it to adapt quickly, and a confidence network weighted by the reprojection loss is also constructed in the model, so that it can quickly adapt to the scene when using the L2AWAD strategy for online adaptation, and the effect of the method in this paper can be close to that in about 100 frames.

[0115] Compared with the prior art, the present invention is significantly superior to the prior art in terms of accuracy and speed. The average performance of multiple technologies is shown in Table 2. The improvement in speed is mainly due to the execution of the MK strategy. The Mann-Kendall trend detection method proposed in the present invention only adjusted the model parameters 465 times, and the rest of the time only predicted the disparity results, and this greatly reduced number of fine-tuning times hardly reduced the performance of online adaptation. Obviously, if this sequence has more data and the scene changes are not large, the average running speed of the present invention will be faster.

Claims

1. A stereo matching method based on unsupervised online learning, characterized in that: It is implemented through a stereo matching module based on a multi-level cost volume, a non-occluded area confidence estimation module and a Mann-Kendall trend detection module; the left and right images are input into the stereo matching module to obtain the predicted disparity; The stereo matching module starts with a set of initial weights. When new image data arrives, it predicts the corresponding disparity and chooses whether to update the stereo matching module parameters. The noisy reprojection loss is calculated based on the stereo matching module predicted disparity and the left and right images. The non-occluded area confidence estimation module predicts the occluded area mask of the left view based on the Sobel edge information of the left image and the cost volume information output by the stereo matching module, which is used to filter the reprojection loss. The error in , forming the final self-supervised reprojection loss The self-supervised reprojection loss is recorded by the Mann-Kendall trend detection module, which statistically analyzes the trend of the self-supervised reprojection loss in recent frames to determine the convergence of the stereo matching module; the result of the Mann-Kendall trend detection module is used as a controller to determine whether the self-supervised reprojection loss needs to be used. The stereo matching module is adapted online, and switches autonomously between fine-tuning the stereo matching module and predicting disparity. The training of the overall network of the non-occluded area confidence estimation module, stereo matching module and trend detection module is divided into two stages: online and offline. First, the parameters of the above three modules are learned through offline training, and then the parameters of the stereo matching module are learned through online training. After the non-occluded area confidence estimation module is trained in the offline stage, its parameters will not be adjusted in the online adaptation stage.

2. The stereo matching method based on unsupervised online learning according to claim 1, characterized in that: The stereo matching module is a stereo matching module based on a multi-level cost body, which includes a feature extraction submodule and a disparity estimation submodule; In the stereo matching module, the left image and the right image are input to the feature extraction submodule, and multi-level features F1 to F6 are output. The multi-level features are used to construct multi-level cost volumes V1 to V6. The disparity estimation submodules D2-D6 input the corresponding cost volumes V2-V6 and output five levels of coarse disparity d2[1 / 4]-d6[1 / 64], where [1 / 4] indicates that the image resolution of this level of disparity is 1 / 4 of the input left image resolution; the previous level cost volume V i One of the inputs is the next level coarse disparity d i+1 , d i+1 By the corresponding level of cost body V i+1 After the disparity estimation submodule D i+1 Obtain; The disparity refinement submodule inputs the coarse disparity D2 and the cost volume V1, and outputs the final disparity D; Before the stereo matching module is adapted online, it is trained offline using a synthetic dataset to obtain initial weights; The supervision information used in offline training is the true value of disparity. The objective function is shown in formula (1). The disparity of each resolution will be scaled to the full resolution and then contribute respectively. The weight β is set to [0.005, 0.005, 0.01, 0.02, 0.08, 0.32]; 3. The stereo matching method based on unsupervised online learning according to claim 2, characterized in that: The feature extraction submodule is a domain normalization feature extraction submodule, which inputs the left image and the right image; firstly, the input image is subjected to standard normalization, and then a series of "convolution-convolution-domain normalization-Leaky ReLU activation function" structures are used to extract multi-level features F1 to F6 as output; each level of feature F i Contains the left features corresponding to the left and right images Right feature The convolution kernels of the two-dimensional convolution used in the domain normalization feature extraction submodule are all 3×3, and s1 and s2 represent the convolution step sizes of 1 and 2 respectively; the specific operation of domain normalization in the domain normalization feature extraction submodule is to perform L2 normalization on the features along the channel dimension of the convolution layer to extract features.

4. The stereo matching method based on unsupervised online learning according to claim 2, characterized in that: The construction of the multi-level cost volume V is divided into two steps. The first step is to construct the cost volume V6 by directly concatenating the features in the channel dimension. and get; The second step is to construct the cost volume V i , i=1,2,…5, first use the warp operation to transform the features Transformed to where x is the pixel coordinate, d i+1 is the disparity estimation submodule D i+1 Output coarse disparity; cost volume V i ,i=1,2,…5by and Channel dimension concatenation features and get; Where concat represents the feature concatenation operation.

5. The stereo matching method based on unsupervised online learning according to claim 2, characterized in that: The disparity estimation submodule is a disparity estimation submodule based on jump connection, which inputs the cost volume V and outputs the coarse disparity d i , i=2,3,…,6; the convolution kernel size of each convolution layer in the disparity estimation submodule based on skip connection is 3×3, the step size is 1, and the number of output feature channels are 128, 128, 96, 64, 32 and 1 respectively. These feature channel numbers are kept fixed in the disparity estimation modules at all levels; skip connection means that the input of each convolution layer is the output and input of the previous layer, that is, the input of the i-th convolution layer is composed of the output of the i-1th layer and the i-2th layer.

6. The stereo matching method based on unsupervised online learning according to claim 2, characterized in that: The disparity refinement submodule is a disparity refinement submodule based on dilated convolution, which inputs a cost volume V1 and a coarse disparity d2, and outputs a refined final disparity D; the disparity refinement submodule includes 5 layers of dilated convolutions, and the expansion rates are 1, 2, 4, 1, 1 respectively, and the number of output channels of each dilated convolution is 64, 64, 32, 16, 1.

7. The stereo matching method based on unsupervised online learning according to any one of claims 2 to 6, characterized in that: The non-occluded area confidence estimation module is divided into two inputs. The first input is the cost volume V1. The first input is first downsampled to 1 / 4 resolution of the input image, and then passes through three convolutional layers with a kernel of 3×3 to output rough confidence information. Among them, the first two convolutional layers are subjected to domain normalization and leaky ReLU is used as the activation function. The second input is the left image. The original image is first downsampled to the corresponding resolution, and the edge information is obtained by mean filtering and sobel operator. It is cascaded with the rough confidence and input into the hole convolution to refine the edge area. The output of the non-occluded area confidence estimation module is the non-occluded area confidence. The objective function of the non-occluded area confidence estimation module is the following weighted cross entropy loss Among them, w is used to describe the importance of the non-occluded area in the loss. In order to balance the difference in the number of samples of the two types and increase the prediction tendency of the non-occluded area confidence estimation module for the non-occluded area, w is set to 0.3; y i is the true label of the confidence of the non-occluded area; After obtaining the confidence output of the non-occluded area, it is converted into a mask map of the occluded area according to the probability threshold, which is used to completely filter the predicted occluded area; the probability threshold is set to 0.5, and the occluded area mask is calculated as shown in formula (4); In the online learning stage, the unequalized reprojection loss is first calculated according to formula (5); then the reprojection loss is filtered at the pixel level according to the mask to construct a benign loss for supervising network learning during online adaptation, as shown in formula (6); in, The left image is converted from the right image based on the disparity. SSIM represents the model that measures the structural differences between images. λ is set to 0.

85. In the offline learning stage, the non-occluded area confidence estimation module, stereo matching module and trend detection module are jointly trained, and the objective function is designed as formula (7); 8. The stereo matching method based on unsupervised online learning according to any one of claims 2 to 6, characterized in that: The online trend detection module based on Mann-Kendall theory detects the most recent frames of the time series during the online learning process. Perform statistical analysis to determine the operating status of the stereo matching module. For a loss sequence of given length n First calculate the statistic S: in, and represents the loss at time i, j, sgn(·) is the indicator function; The positive or negative value of the statistic S represents the upward or downward trend of the sequence; when n>8, the statistic S is approximately normally distributed, and its mean E(S) and variance Var(S) As shown in formula (10); E(S)=0 Among them, m is the number of knot groups in the sequence, and a knot group is a set of data with the same value in the sequence, and τ i is the number of elements in the i-th knot group; the statistic S is converted into a standard normal distribution according to the mean and variance to form a standard statistic Z; Among them, α is the statistical significance level, α = 0.05, that is, Z 1-α / 2 =1.96; set the sequence length n to 12; when |Z|>Z 1-α / 2 , indicating that the loss function There is a significant monotonic trend in the time series, and the positive or negative value of Z represents the upward or downward trend of the sequence. At this time, the trend detection module switches the operation state of the stereo matching module to "fine-tuning"; otherwise, it is considered that the sequence has no obvious trend, and the trend detection module switches the operation state of the stereo matching module to "inference".

Images