An unsupervised binocular depth estimation method and system based on an occlusion decoupled network

Abstract

The application discloses a kind of unsupervised binocular depth estimation methods based on occlusion decoupling network, constructs binocular depth estimation model, including the matching module of occlusion decoupling, the occlusion perception module based on cost volume, the occlusion filling module of domain invariant;And the corresponding step-by-step training strategy of entire network.Matching module constructs a simplified binocular depth estimation network, to only regress non-occlusion area disparity, and in real scene by the way of self-supervision, occlusion perception module generates occlusion mask map using the cost volume aggregated in matching module, and occlusion filling module fills the disparity of occlusion area according to the disparity of non-occlusion area generated and the semantic information in camera left image.The application uses occlusion decoupling network, solves the problem of performance degradation of unsupervised pair occlusion area, suppresses the adverse effects of occlusion area on matching task, and improves the accuracy of prediction for occlusion and non-occlusion area.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to an unsupervised binocular depth estimation method and system based on an occlusion decoupling network. Background Art

[0002] Binocular depth estimation is a technique for recovering three-dimensional scene information from binocular images. It is widely used in numerous advanced vision tasks, including augmentation, autonomous driving, and 3D reconstruction. To obtain dense depth information for the entire image, occlusion filling is often considered a subsidiary task of binocular depth estimation to fill in ill-posed occlusion disparities. Traditional methods explicitly divide binocular depth estimation into four steps: matching cost calculation, cost aggregation, disparity regression, and disparity refinement.

[0003] With the development of convolutional neural networks, deep learning-based binocular depth estimation algorithms use end-to-end networks to directly regress full-image disparity, achieving significant improvements in accuracy and completeness. These algorithms rely on a large number of annotated ground truth values ​​for fine-tuning in real-world scenarios. The high cost of recording these ground truth values ​​hinders their widespread deployment and application in the real world. To mitigate this reliance on ground truth, existing algorithms construct self-supervised loss functions to potentially supervise the training of the entire model in real-world scenarios, alleviating the performance degradation caused by the lack of ground truth. However, these algorithms overlook the coupling between the matching and occlusion filling tasks within the model.

[0004] On the one hand, coupling causes the self-supervised loss function constructed using geometric prior knowledge to supervise the disparity of non-occluded regions while degrading the network's performance in occluded regions. Existing image reconstruction losses assume that every pixel in each image can be reconstructed from disparity in images from other viewpoints, but occluded regions in an image are not visible from other viewpoints, so this assumption does not hold for occluded regions. Existing disparity smoothing losses assume that the disparity of adjacent pixels should change smoothly, but disparity in occluded regions can jump, leading to incorrect disparity filling in occluded regions. On the other hand, coupling makes the matching task dependent on the occlusion filling task, preventing further improvement in prediction of non-occluded regions. When fine-tuned in a self-supervised manner in real scenes, because the network couples the processing of non-occluded and occluded regions during cost aggregation, the ambiguous predictions in occluded regions can potentially interfere with reliable predictions in non-occluded regions, making the matching task dependent on the occlusion filling task.

[0005] In summary, existing self-supervised binocular depth estimation methods have not adequately addressed the performance degradation caused by missing ground truth. Because their underlying networks couple matching and occlusion filling, they improve performance in unoccluded areas to some extent, but degrade the network's performance in occluded areas. Summary of the Invention

[0006] In order to solve the above problems, the present invention provides an unsupervised binocular depth estimation method based on an occlusion decoupling network, comprising the following steps:

[0007] S1. Obtain left and right image data, and divide the left and right image data into a synthetic scene data set and a real scene data set;

[0008] S2. Build a binocular depth estimation model, including: matching module, occlusion perception module, and occlusion filling module;

[0009] The input of the matching module is the left and right image data, and the output of the matching module is the aggregated cost volume;

[0010] The input of the occlusion perception module is the aggregated cost volume, and the output of the occlusion perception module is the occlusion mask map;

[0011] The first input of the occlusion filling module is the left image, the second input of the occlusion filling module is obtained by multiplying the occlusion mask map and the non-occlusion area disparity map, the non-occlusion area disparity map is obtained by performing disparity regression on the aggregated cost volume, and the output of the occlusion filling module is the disparity map;

[0012] S3, using the synthetic scene dataset and the modules of the binocular depth estimation model to perform distributed training to obtain a trained binocular depth estimation model;

[0013] Specifically:

[0014] S31, freezing the parameters of the occlusion perception module and the occlusion filling module, and using the synthetic scene dataset to train the matching module until convergence, to obtain a trained matching module;

[0015] S32, freezing the parameters of the occlusion filling module and the trained matching module, and using the output of the trained matching module to train the occlusion perception module until convergence, to obtain the trained occlusion perception module;

[0016] S33, freezing the parameters of the trained matching module and the trained occlusion perception module, performing disparity regression on the output of the trained matching module, and multiplying the result with the output of the trained occlusion perception module as the training input of the occlusion filling module, training the occlusion filling module until convergence, and obtaining the trained occlusion filling module;

[0017] S34, using a synthetic scene dataset to perform end-to-end training on the entire network consisting of the trained matching module, the trained occlusion perception module, and the trained occlusion filling module until convergence, to obtain an initially trained binocular depth estimation model;

[0018] S35, freezing the parameters of the occlusion perception module and the occlusion filling module of the initially trained binocular depth estimation model, and fine-tuning the matching module of the initially trained binocular depth estimation model using a real scene dataset in a self-supervised manner until convergence, thereby obtaining a trained binocular depth estimation model;

[0019] S4. Input the left and right images of the binocular camera to be detected into the trained binocular depth estimation model to obtain an estimated disparity map. Combined with the calibration parameters of the binocular camera and the binocular baseline, a depth map with the same size as the input image is obtained.

[0020] Furthermore, the synthetic scene dataset includes left and right images synthesized by computer-rendered 3D models and corresponding left and right dense disparity map true values; the real scene dataset is obtained by performing epipolar correction on the original left and right images captured by a binocular camera with intrinsic reference calibration.

[0021] Furthermore, the matching module includes a feature extraction unit, a cost volume formation unit, and a cost aggregation unit; the feature extraction unit extracts features of the input left and right images to obtain left and right feature maps; the cost volume formation unit forms an original cost volume by performing a piecewise dot product on the left and right feature maps, and the original cost volume is passed through the cost aggregation unit to obtain an aggregated cost volume;

[0022] The feature extraction unit extracts the corresponding 1 / 4 resolution feature map based on the block convolution;

[0023] The cost aggregation unit includes two 3D convolutions.

[0024] Furthermore, the occlusion perception module includes an attention body formation unit, a semantic information aggregation unit, and an occlusion regression unit;

[0025] The attention volume forming unit converts the input aggregated cost volume C(h,w,d) into an attention volume A(h,w,wd), where h,w,d represent the height, width, and depth of the aggregated cost volume, respectively. Each attention matrix is ​​added with additional rows and columns to represent the corresponding occlusion attention O. A , add additional rows and columns to each attention matrix in A to form an occlusion attention matrix O A Attention And use the Sinkhorn algorithm to focus on Perform normalization;

[0026] The semantic information aggregation unit is composed of five sub-units connected in sequence. Each sub-unit is composed of a two-dimensional convolution with a 3×3 kernel size and a ReLU activation layer. The normalized attention body and the left image are connected in series and then input into the semantic information aggregation unit. The occlusion mask image is obtained by the occlusion regression unit.

[0027] The occlusion regression unit includes a two-dimensional convolution with a kernel size of 3×3 and a Sigmoid activation layer.

[0028] Furthermore, the occlusion filling module includes an encoder and a decoder;

[0029] The encoder consists of six NAF convolution blocks, three 3×3 convolutions, six downsampling blocks, a ConCat function, and an edge extraction algorithm;

[0030] The masked disparity map is sequentially subjected to the first 3×3 convolution and the first NAF convolution block of the encoder to obtain a first output; the first output is sequentially subjected to the first downsampling and the second NAF convolution block of the encoder to obtain a second output; the second output is subjected to the second downsampling of the encoder to obtain a third output;

[0031] The left image of the camera goes through the encoder's edge extraction algorithm, the second 3×3 convolution, the third NAF convolution block, the third downsampling, the fourth NAF convolution block, and the fourth downsampling to obtain the fourth output;

[0032] The third and fourth outputs are concatenated through the ConCat function of the encoder, and then pass through the third 3×3 convolution and the fifth NAF convolution block of the encoder to obtain the fifth output.

[0033] The fifth output is sequentially processed by the fifth downsampling block and the sixth NAF convolution block of the encoder to obtain a sixth output; the sixth output is processed by the sixth downsampling block of the encoder, accumulated and downsampled to 1 / 16 of the original image, and then output to obtain a seventh output;

[0034] The decoder consists of a 3×3 convolution, a PixelShuffle upsampling, and a NAF convolution block;

[0035] The seventh output is sequentially subjected to 3×3 convolution and PixelShuffle upsampling by the decoder to obtain the eighth output;

[0036] The first output, the second output, the fifth output, and the sixth output are added to the eighth output through a jump connection to obtain a ninth output; the ninth output is passed through the NAF convolution block of the decoder to obtain a tenth output; the tenth output is added to the masked disparity map to obtain a filled disparity map.

[0037] The masked disparity map is obtained by multiplying the occlusion mask map and the non-occlusion area disparity map; the non-occlusion area disparity map is obtained by performing disparity regression on the aggregated cost volume.

[0038] Furthermore, in step S2, the loss function of the matching module is expressed as:

[0039]

[0040]

[0041]

[0042] in, Represents the difference between the predicted value and the true value of the non-occluded area, represents the SmoothL1 loss function, d noc Denotes the disparity of the predicted non-occluded area, d gt Indicates d noc The corresponding true disparity value, Represents the image reconstruction loss, I and D represent the input left image and the predicted disparity map respectively, α represents the weight coefficient, and x l Represents the RGB value of any pixel in the left image, Indicates the left image after remapping the right image according to the parallax, x r Represents the RGB value of any pixel in the right image, SSIM is the structural similarity calculation function, represents the parallax smoothing loss, represents the gradient along the x and y axes of the image, and ||||1 represents the 1-norm.

[0043] Furthermore, in step S2, the loss function of the occlusion perception module is expressed as:

[0044]

[0045] in, represents the cross entropy loss function, o represents the occlusion probability value of any pixel, O represents the predicted occlusion probability map, o gt The ground truth of the occlusion obtained from the left-right consistency check.

[0046] Furthermore, the edge extraction algorithm is expressed as follows:

[0047]

[0048] Where CIConv(x, y) represents the edge convolution at pixel x, y, x represents the horizontal coordinate value of the 2D coordinate of any pixel, y represents the vertical coordinate value of the 2D coordinate of any pixel, CI is an edge detector that is invariant to changes in illumination and color, σ is a parameter in the training process, ε represents a constant, and μ s is the sample mean, σ s is the sample standard deviation.

[0049] Furthermore, in step S2, the loss function of the occlusion filling module is expressed as follows:

[0050]

[0051] in, represents the loss function of the occlusion filling module, represents the SmoothL1 loss function, d occ represents the disparity of the occluded area, d gt Indicates d occ The corresponding true disparity value.

[0052] The present invention also proposes an unsupervised binocular depth estimation system based on an occlusion decoupling network, comprising:

[0053] A data acquisition module is used to acquire left and right image data and divide the left and right image data into a synthetic scene data set and a real scene data set;

[0054] Model building module, used to build a binocular depth estimation model, including: matching module, occlusion perception module, and occlusion filling module;

[0055] The input of the matching module is the left and right image data, and the output of the matching module is the aggregated cost volume;

[0056] The input of the occlusion perception module is the aggregated cost volume, and the output of the occlusion perception module is the occlusion mask map;

[0057] The first input of the occlusion filling module is the left image, the second input of the occlusion filling module is obtained by multiplying the occlusion mask map and the non-occlusion area disparity map, the non-occlusion area disparity map is obtained by performing disparity regression on the aggregated cost volume, and the output of the occlusion filling module is the disparity map;

[0058] A distribution training module is used to perform distribution training on the modules of the binocular depth estimation model using a synthetic scene dataset to obtain a trained binocular depth estimation model;

[0059] The specific distribution training is:

[0060] (1) Freeze the parameters of the occlusion perception module and the occlusion filling module, and use the synthetic scene dataset to train the matching module until convergence to obtain the trained matching module;

[0061] (2) Freeze the parameters of the occlusion filling module and the trained matching module, and use the output of the trained matching module to train the occlusion perception module until convergence, thereby obtaining the trained occlusion perception module;

[0062] (3) Freeze the parameters of the trained matching module and the trained occlusion perception module, use the output of the trained matching module to perform disparity regression, and then multiply it with the output of the trained occlusion perception module as the training input of the occlusion filling module. Train the occlusion filling module until convergence to obtain the trained occlusion filling module.

[0063] (4) Using a synthetic scene dataset, the overall network consisting of the trained matching module, the trained occlusion perception module, and the trained occlusion filling module is end-to-end trained until convergence, thereby obtaining a binocular depth estimation model after initial training;

[0064] (5) Freeze the parameters of the occlusion perception module and the occlusion filling module of the initially trained binocular depth estimation model, and use a self-supervised approach to fine-tune the matching module of the initially trained binocular depth estimation model using a real scene dataset until convergence, thereby obtaining a trained binocular depth estimation model;

[0065] The depth estimation module is used to input the left and right images of the binocular camera to be detected into the trained binocular depth estimation model to obtain the estimated disparity map. Combined with the calibration parameters of the binocular camera and the binocular baseline, a depth map with the same size as the input image is obtained.

[0066] The beneficial effects brought about by the technical solution provided by the present invention are:

[0067] The present invention proposes a self-supervised binocular depth estimation method based on an occlusion decoupling network. A binocular depth estimation model is constructed, including an occlusion-decoupled matching module, an occlusion perception module based on a cost volume, a domain-invariant occlusion filling module, and a step-by-step training strategy corresponding to the entire network. The matching module constructs a streamlined binocular depth estimation network to regress only the disparity of non-occluded areas and fine-tune the network through self-supervision in real scenes. The occlusion perception module uses the aggregated cost volume in the matching module to generate an occlusion mask map to suppress the ambiguity of the disparity map generated by the matching module in occluded areas, which is beneficial for subsequent occlusion filling. The occlusion filling module fills the disparity of the occluded areas based on the generated disparity of the non-occluded areas and the semantic information in the camera left image. Considering the front-to-back dependencies between the three modules, direct end-to-end training cannot prevent each module from playing its own role and may even fail to converge. Therefore, the present invention proposes a distributed training strategy. The distributed training strategy first enables the network models corresponding to the three modules to converge on their respective loss functions. It then uses end-to-end training to converge the entire network on a global loss function. Finally, the matching module is fine-tuned in real scenes using a self-supervised loss function to address the performance degradation caused by the domain difference between synthetic and real data. By explicitly decoupling occlusion filling from binocular depth estimation and building a matching network that focuses only on non-occluded areas and an occlusion perception and filling network that focuses only on occluded areas, the team achieves efficient self-supervised fine-tuning of non-occluded areas, suppresses the adverse effects of occluded areas, and robustly fills occluded areas. Compared with the existing technology, the advantages of the present invention are: (1) The present invention decouples from the network level instead of adapting the loss function to the occluded area, so that the loss function designed by geometric prior knowledge can more accurately constrain the network for fine-tuning, the network as a whole is more streamlined, and the reasoning is more efficient; (2) The present invention eliminates the adverse effects of occlusion on matching prediction, and the performance degradation of self-supervised fine-tuning on occlusion prediction is improved, and the accuracy of prediction for occluded and non-occluded areas is improved; (3) The present invention proposes a method for processing occluded areas, namely, occlusion perception based on cost volume and domain-invariant occlusion filling, which overcomes the problem that the existing network cannot predict the depth of the occluded area; (4) The present invention proposes a step-by-step training strategy combined with a hybrid weighted loss function to perform differentiated training for different tasks, improve the efficiency of network back propagation, and accelerate network convergence. BRIEF DESCRIPTION OF THE DRAWINGS

[0068] Figure 1 This is a structural diagram of the binocular depth estimation model implemented in the present invention;

[0069] Figure 2 is a structural diagram of an occlusion perception module in a binocular depth estimation model constructed according to an embodiment of the present invention;

[0070] Figure 3 is a structural diagram of an occlusion filling module in a binocular depth estimation model constructed according to an embodiment of the present invention;

[0071] Figure 4 is a flow chart of a distributed training strategy according to an embodiment of the present invention;

[0072] Figure 5 This is the visualized prediction result of the method of the present invention and other methods in the real scene CugDataset in the embodiment of the present invention. DETAILED DESCRIPTION

[0073] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0074] An unsupervised binocular depth estimation method based on an occlusion decoupling network in this embodiment includes the following steps:

[0075] S1. Obtain left and right image data and divide them into a synthetic scene dataset and a real scene dataset. The synthetic scene dataset contains left and right images synthesized by computer-rendered 3D models and the corresponding left and right dense disparity map ground truth values. The real scene dataset is obtained by performing epipolar correction on the original left and right images collected by a binocular camera with intrinsic reference calibration. The obtained left and right images also need to be normalized to normalize the pixel values ​​between -1 and 1.

[0076] S2. Build a binocular depth estimation model, refer to Figure 1 , Figure 1 This is a structural diagram of the present invention for implementing a binocular depth estimation model. It includes: a matching module, an occlusion perception module, and an occlusion filling module. The input of the matching module is the left and right image data, and the output of the matching module is the aggregated cost volume; the input of the occlusion perception module is the aggregated cost volume, and the output of the occlusion perception module is the occlusion mask map; the first input of the occlusion filling module is the left image, and the second input of the occlusion filling module is the multiplication of the occlusion mask map and the disparity map of the non-occluded area. The disparity map of the non-occluded area is obtained by performing disparity regression on the aggregated cost volume. The aggregated cost volume is upsampled to full resolution through bilinear interpolation, and all disparity information corresponding to each pixel is integrated by the soft argmin algorithm to finally obtain the disparity map of the non-occluded area. The output of the occlusion filling module is the disparity map.

[0077] (1) Constructing a matching module

[0078] The matching module includes a feature extraction unit, a cost volume formation unit, and a cost aggregation unit; the feature extraction unit extracts the features of the input left and right images, and extracts the corresponding 1 / 4 resolution feature maps based on block convolution to obtain left and right feature maps; the cost volume formation unit forms the original 4D cost volume through the segmented dot product method of the left and right feature maps, and the original cost volume is obtained by the cost aggregation unit to obtain the aggregated cost volume. The cost aggregation unit includes two 3D convolutions with a convolution kernel size of 3×3×3, which are used to aggregate the cost information of height, width and disparity dimensions.

[0079] Unlike the network structure that performs depth estimation on all areas, the matching module decouples the processing of occlusion from it, so the sub-network does not need to consider the processing of occluded areas when designing it.

[0080] Since the cost aggregation stage does not require a large number of convolutional layers to fill the missing costs of the occluded areas, the network has a relatively small number of parameters and is very efficient for predicting only the disparity of non-occluded areas, while suppressing the coupling between non-occluded and occluded areas caused by the aggregation stage.

[0081] (2) Constructing an occlusion perception module

[0082] The cost volume after cost aggregation contains the cost of each pixel at each disparity. Since there is no corresponding matching point for an occluded point, its cost distribution is significantly different from that of a non-occluded point. Therefore, in the embodiment of the present invention, the cost volume in the matching module is directly used as the input of the occlusion perception module to avoid additional encoding operations. Figure 2 , including the attention body formation unit, the semantic information aggregation unit, and the occlusion regression unit. The attention body formation unit transforms the input aggregated cost body C(h, w, d) into the attention body A(h, w, wd) through elementary transformation, where h, w, d represent the height, width, and depth of the aggregated cost body, respectively. Each attention matrix is ​​added with additional rows and columns to represent the corresponding occlusion attention O A The attention matrix is ​​a slice of the attention body in the H direction, which represents the attention (correlation) of the pixels on the epipolar line between the left image and the right image. Add additional rows and columns to each attention matrix in A to form an occlusion attention matrix O. A Attention And use the Sinkhorn algorithm to focus on Normalization is performed to make the difference in cost distribution between occluded and non-occluded points easier to distinguish.

[0083] The original occlusion probability is regressed by the following expression:

[0084]

[0085] where p represents any pixel position in the image, p' represents the pixel position of the potential matching point corresponding to p, represents the original occlusion probability corresponding to the pixel position p, represents the attention value of p and the potential matching point p', O A (p) represents the occlusion attention value at p.

[0086] In order to make the occlusion map more complete and accurate, a semantic information aggregation unit is used to aggregate the semantic information in the left image, the semantic information aggregation unit is composed of five sub-units connected in turn, each sub-unit is composed of a 3*3 kernel size two-dimensional convolution and a ReLU activation layer connected, the normalized attention volume and the left image are input into the semantic information aggregation unit after being concatenated, and C in the figure is a concatenate operation.

[0087] Finally, the occlusion mask graph is obtained through the occlusion regression unit, first, a 3*3 kernel size two-dimensional convolution and a Sigmoid activation layer are used to obtain the occlusion probability in the range of 0 to 1, and the binary occlusion mask graph can be converted from the occlusion probability graph by setting a threshold.

[0088] (3) constructing an occlusion filling module

[0089] In order to make the filled disparity more smooth, the semantic information of the original image needs to be introduced into the filling module, and at the same time, since the occlusion filling module cannot be fine-tuned in the real domain, domain-invariant semantic information needs to be extracted. Therefore, the left image of the input camera is first edge extracted, and then input into the network to guide the filling of the occlusion, and the structure diagram of the occlusion filling module in the binocular depth estimation model constructed by the embodiment of the application is as shown in Figure 3

[0090] The aggregated cost volume is subjected to disparity regression to obtain a non-occluded area disparity map, and the non-occluded area disparity map and the occlusion mask graph are multiplied to obtain a masked disparity map. The masked disparity map and the camera left image are used as the input of the occlusion filling module, and the camera left image is the camera left image in the input camera left and right images of the entire network model.

[0091] The occlusion filling module adopts a U-Net network structure, including an encoder and a decoder; the encoder is composed of six NAF convolution blocks, three 3*3 convolutions, six downsampling, a ConCat function and an edge extraction algorithm.

[0092] The masked disparity map is sequentially subjected to the first 3*3 convolution of the encoder, the first NAF convolution block to obtain the first output; the first output is sequentially subjected to the first downsampling of the encoder, the second NAF convolution block to obtain the second output; the second output is subjected to the second downsampling of the encoder to obtain the third output; ​

[0093] The left image of the camera goes through the encoder's edge extraction algorithm, the second 3×3 convolution, the third NAF convolution block, the third downsampling, the fourth NAF convolution block, and the fourth downsampling to obtain the fourth output;

[0094] The third and fourth outputs are concatenated through the ConCat function of the encoder, and then pass through the third 3×3 convolution and the fifth NAF convolution block of the encoder to obtain the fifth output.

[0095] The fifth output passes through the fifth downsampling of the encoder and the sixth NAF convolution block in sequence to obtain the sixth output; the sixth output passes through the sixth downsampling of the encoder, is cumulatively downsampled to 1 / 16 of the original image, and is output to obtain the seventh output.

[0096] The decoder consists of a 3×3 convolution, a PixelShuffle upsampling, and a NAF convolution block.

[0097] The seventh output is sequentially passed through the decoder by 3×3 convolution and PixelShuffle upsampling to obtain the eighth output;

[0098] The first, second, fifth, and sixth outputs are added to the eighth output via skip connections to produce the ninth output. The ninth output passes through the decoder's NAF convolution block to produce the tenth output. The tenth output is added to the masked disparity map to produce the padded disparity map. The final disparity output is the padded disparity map.

[0099] The NAF convolution block consists of two LayerNorm normalization layers, four 1×1 convolutions, one 3×3Dw convolution, two simplified gate operations, and one channel attention. The input passes through the first LayerNorm normalization layer, the first 1×1 convolution, the 3×3Dw convolution, the first simplified gate operation, the channel attention, and the second 1×1 convolution to obtain the first output of the NAF convolution block. The input of the NAF convolution block is added to the first output of the NAF convolution block through a jump connection to obtain the second output of the NAF convolution block. The second output of the NAF convolution block passes through the second LayerNorm normalization layer, the third 1×1 convolution, the second simplified gate operation, and the fourth 1×1 convolution to obtain the third output of the NAF convolution block. The second output of the NAF convolution block is added to the third output of the NAF convolution block through a jump connection to obtain the final output of the NAF convolution block.

[0100] The edge extraction algorithm is expressed as follows:

[0101]

[0102] Wherein, CIConv(x,y) represents the edge convolution at pixel (x,y), x represents the horizontal coordinate value of the 2D coordinate of the pixel, y represents the vertical coordinate value of the 2D coordinate of the pixel, CI is an edge detector that is invariant to changes in illumination and color (from: JM Geusebroek, R. van den Boomgaard, AWM Smeulders, and H. Geerts. "Color invariance," IEEE Transactions on Pattern Analysis and Machine Intelligence, 2001.), σ is a parameter in the training process, ε represents a constant, μ s is the sample mean, σ s is the sample standard deviation.

[0103] S3, using the synthetic scene dataset and the modules of the binocular depth estimation model to perform distributed training to obtain the trained binocular depth estimation model. Figure 4 , Figure 4 This is a flow chart of a distributed training strategy according to an embodiment of the present invention.

[0104] Specifically:

[0105] S31, freezing the parameters of the occlusion perception module and the occlusion filling module, and using the synthetic scene dataset to train the matching module until convergence, to obtain a trained matching module;

[0106] S32, freezing the parameters of the occlusion filling module and the trained matching module, and using the output of the trained matching module to train the occlusion perception module until convergence, to obtain the trained occlusion perception module;

[0107] S33, freezing the parameters of the trained matching module and the trained occlusion perception module, performing disparity regression on the output of the trained matching module, and multiplying the result with the output of the trained occlusion perception module as the training input of the occlusion filling module, training the occlusion filling module until convergence, and obtaining the trained occlusion filling module;

[0108] S34, using a synthetic scene dataset to perform end-to-end training on the entire network consisting of the trained matching module, the trained occlusion perception module, and the trained occlusion filling module until convergence, to obtain an initially trained binocular depth estimation model;

[0109] S35, freeze the parameters of the occlusion perception module and the occlusion filling module of the binocular depth estimation model after initial training, and use the self-supervision method to fine-tune the matching module of the binocular depth estimation model after initial training using the real scene data set until convergence, and obtain the trained binocular depth estimation model. In this embodiment, the weight parameters in the self-supervision are λ bce ,λ rc ,λ sm ,ω rc ,ω sm Set them to 1.0, 1.0, 1.0, 0.3, 0.03, 1.0, 0.18 respectively.

[0110] During the training process, the loss function of each module is expressed as follows:

[0111] (1) Loss function of matching module

[0112] In order to supervise the prediction of the matching module for non-occluded areas, Smooth is used on the synthetic scene dataset. L1 loss To measure the difference between the predicted value and the true value of the non-occluded area, a self-supervised loss without true value and image reconstruction loss are used on the real scene dataset. and parallax smoothing loss To fine-tune the pre-trained model. In order to prevent the matching network from overfitting to the synthetic dataset, a self-supervised loss function is also introduced into the training of the synthetic dataset. The above loss functions are expressed as:

[0113]

[0114]

[0115]

[0116] in, Represents the difference between the predicted value and the true value of the non-occluded area, represents the SmoothL1 loss function, d noc Denotes the disparity of the predicted non-occluded area, d gt Indicates d noc The corresponding true disparity value, Represents the image reconstruction loss, I and D represent the input left image and the predicted disparity map respectively, α represents the weight coefficient, and xl represents the RGB value of any pixel in the left image. It represents the left image after remapping the right image according to the parallax. W is the image remapping operation, x r Represents the RGB value of any pixel in the right image, SSIM is the structural similarity calculation function, represents the parallax smoothing loss, represents the gradient along the x and y axes of the image, and ||||1 represents the 1-norm.

[0117] (2) Loss function of occlusion perception module

[0118] In order to supervise the prediction of the occlusion map by the occlusion perception module, a binary cross entropy loss function is used on the synthetic dataset. It is defined as follows:

[0119]

[0120] in, represents the cross entropy loss function, o represents the occlusion probability value of any pixel, O represents the predicted occlusion probability map, o gt The ground truth of the occlusion obtained from the left-right consistency check.

[0121] (3) Loss function of the occlusion filling module

[0122] In order to improve the filling effect of the occlusion filling module, the SmoothL1 loss is used to supervise the disparity of the occluded area output on the synthetic dataset. The corresponding loss function is as follows:

[0123]

[0124] in, represents the loss function of the occlusion filling module, express Loss function, d occ represents the disparity of the occluded area, d gt Indicates d occ The corresponding true disparity value.

[0125] S4. Input the left and right images of the binocular camera to be detected into the trained binocular depth estimation model to obtain an estimated disparity map. Combined with the calibration parameters of the binocular camera and the binocular baseline, a depth map with the same size as the input image is obtained.

[0126] The present invention also proposes an unsupervised binocular depth estimation system based on an occlusion decoupling network, comprising:

[0127] A data acquisition module is used to acquire left and right image data and divide the left and right image data into a synthetic scene data set and a real scene data set;

[0128] Model building module, used to build a binocular depth estimation model, including: matching module, occlusion perception module, and occlusion filling module;

[0129] The input of the matching module is the left and right image data, and the output of the matching module is the aggregated cost volume;

[0130] The input of the occlusion perception module is the aggregated cost volume, and the output of the occlusion perception module is the occlusion mask map;

[0131] The first input of the occlusion filling module is the left image. The second input of the occlusion filling module is the multiplication of the occlusion mask map and the non-occlusion area disparity map. The non-occlusion area disparity map is obtained by performing disparity regression on the aggregated cost volume. The output of the occlusion filling module is the disparity map.

[0132] A distribution training module is used to perform distribution training on the modules of the binocular depth estimation model using a synthetic scene dataset to obtain a trained binocular depth estimation model;

[0133] The specific steps of distribution training are:

[0134] (1) Freeze the parameters of the occlusion perception module and the occlusion filling module, and use the synthetic scene dataset to train the matching module until convergence to obtain the trained matching module;

[0135] (2) Freeze the parameters of the occlusion filling module and the trained matching module, and use the output of the trained matching module to train the occlusion perception module until convergence, thereby obtaining the trained occlusion perception module;

[0136] (3) Freeze the parameters of the trained matching module and the trained occlusion perception module, use the output of the trained matching module to perform disparity regression, and then multiply it with the output of the trained occlusion perception module as the training input of the occlusion filling module. Train the occlusion filling module until convergence to obtain the trained occlusion filling module.

[0137] (4) Using a synthetic scene dataset, the overall network consisting of the trained matching module, the trained occlusion perception module, and the trained occlusion filling module is end-to-end trained until convergence, thereby obtaining a binocular depth estimation model after initial training;

[0138] (5) Freeze the parameters of the occlusion perception module and the occlusion filling module of the initially trained binocular depth estimation model, and use a self-supervised approach to fine-tune the matching module of the initially trained binocular depth estimation model using a real scene dataset until convergence, thereby obtaining a trained binocular depth estimation model;

[0139] The depth estimation module is used to input the left and right images of the binocular camera to be detected into the trained binocular depth estimation model to obtain the estimated disparity map. Combined with the calibration parameters of the binocular camera and the binocular baseline, a depth map with the same size as the input image is obtained.

[0140] Figure 5The paper demonstrates the visualization prediction results of the present invention, the unsupervised GWCNetGwc-Net (Group-wise correlation stereo network) and PASMNet (Parallax attention for unsupervised stereocorrespondence learning) in the real scene CugDataset.

[0141] The above description of the disclosed embodiments is intended to enable one skilled in the art to implement or use the present invention. Various modifications to these embodiments will be readily apparent to one skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention is not limited to the embodiments shown herein but is intended to conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. An unsupervised binocular depth estimation method based on occlusion decoupling network, characterized in that: The following steps are involved: S1. Obtain left and right image data, and divide the left and right image data into a synthetic scene data set and a real scene data set; S2. Build a binocular depth estimation model, including: matching module, occlusion perception module, and occlusion filling module; The input of the matching module is the left and right image data, and the output of the matching module is the aggregated cost volume; The input of the occlusion perception module is the aggregated cost volume, and the output of the occlusion perception module is the occlusion mask map; The first input of the occlusion filling module is the left image, the second input of the occlusion filling module is obtained by multiplying the occlusion mask map and the non-occlusion area disparity map, the non-occlusion area disparity map is obtained by performing disparity regression on the aggregated cost volume, and the output of the occlusion filling module is the disparity map; S3, using the synthetic scene dataset and the modules of the binocular depth estimation model to perform distributed training to obtain a trained binocular depth estimation model; Specifically: S31, freezing the parameters of the occlusion perception module and the occlusion filling module, and using the synthetic scene dataset to train the matching module until convergence, to obtain a trained matching module; S32, freezing the parameters of the occlusion filling module and the trained matching module, and using the output of the trained matching module to train the occlusion perception module until convergence, to obtain the trained occlusion perception module; S33, freezing the parameters of the trained matching module and the trained occlusion perception module, performing disparity regression on the output of the trained matching module, and multiplying the result with the output of the trained occlusion perception module as the training input of the occlusion filling module, training the occlusion filling module until convergence, and obtaining the trained occlusion filling module; S34, using a synthetic scene dataset to perform end-to-end training on the entire network consisting of the trained matching module, the trained occlusion perception module, and the trained occlusion filling module until convergence, to obtain an initially trained binocular depth estimation model; S35, freezing the parameters of the occlusion perception module and the occlusion filling module of the initially trained binocular depth estimation model, and fine-tuning the matching module of the initially trained binocular depth estimation model using a real scene dataset in a self-supervised manner until convergence, thereby obtaining a trained binocular depth estimation model; S4. Input the left and right images of the binocular camera to be detected into the trained binocular depth estimation model to obtain an estimated disparity map. Combined with the calibration parameters of the binocular camera and the binocular baseline, a depth map with the same size as the input image is obtained.

2. The unsupervised binocular depth estimation method based on occlusion decoupling network according to claim 1 is characterized in that The synthetic scene dataset includes left and right images synthesized by computer-rendered 3D models and the corresponding left and right dense disparity map true values; the real scene dataset is obtained by performing epipolar correction on the original left and right images collected by a binocular camera with intrinsic reference calibration.

3. The unsupervised binocular depth estimation method based on occlusion decoupling network according to claim 1 is characterized in that The matching module includes a feature extraction unit, a cost volume formation unit, and a cost aggregation unit; the feature extraction unit extracts features of the input left and right images to obtain left and right feature maps; the cost volume formation unit forms an original cost volume by performing a piecewise dot product on the left and right feature maps, and the original cost volume is passed through the cost aggregation unit to obtain an aggregated cost volume; The feature extraction unit extracts the corresponding 1 / 4 resolution feature map based on the block convolution; The cost aggregation unit includes two 3D convolutions.

4. The unsupervised binocular depth estimation method based on occlusion decoupling network according to claim 1 is characterized in that The occlusion perception module includes an attention body formation unit, a semantic information aggregation unit, and an occlusion regression unit; The attention volume forming unit converts the input aggregated cost volume C(h,w,d) into an attention volume A(h,w,wd), where h,w,d represent the height, width, and depth of the aggregated cost volume, respectively. Each attention matrix is ​​added with additional rows and columns to represent the corresponding occlusion attention O. A , add additional rows and columns to each attention matrix in A to form an occlusion attention matrix O A Attention And use the Sinkhorn algorithm to focus on Perform normalization; The semantic information aggregation unit is composed of five sub-units connected in sequence. Each sub-unit is composed of a two-dimensional convolution with a 3×3 kernel size and a ReLU activation layer. The normalized attention body and the left image are connected in series and then input into the semantic information aggregation unit. The occlusion mask image is obtained by the occlusion regression unit. The occlusion regression unit includes a two-dimensional convolution with a kernel size of 3×3 and a Sigmoid activation layer.

5. The unsupervised binocular depth estimation method based on occlusion decoupling network according to claim 1, characterized in that: The occlusion filling module includes an encoder and a decoder; The encoder consists of six NAF convolution blocks, three 3×3 convolutions, six downsampling blocks, a ConCat function, and an edge extraction algorithm; The masked disparity map is sequentially subjected to the first 3×3 convolution and the first NAF convolution block of the encoder to obtain a first output; the first output is sequentially subjected to the first downsampling and the second NAF convolution block of the encoder to obtain a second output; the second output is subjected to the second downsampling of the encoder to obtain a third output; The left image of the camera goes through the encoder's edge extraction algorithm, the second 3×3 convolution, the third NAF convolution block, the third downsampling, the fourth NAF convolution block, and the fourth downsampling to obtain the fourth output; The third and fourth outputs are concatenated through the ConCat function of the encoder, and then pass through the third 3×3 convolution and the fifth NAF convolution block of the encoder to obtain the fifth output. The fifth output is sequentially processed by the fifth downsampling block and the sixth NAF convolution block of the encoder to obtain a sixth output; the sixth output is processed by the sixth downsampling block of the encoder, accumulated and downsampled to 1 / 16 of the original image, and then output to obtain a seventh output; The decoder consists of a 3×3 convolution, a PixelShuffle upsampling, and a NAF convolution block; The seventh output is sequentially subjected to 3×3 convolution and PixelShuffle upsampling by the decoder to obtain the eighth output; The first output, the second output, the fifth output, and the sixth output are added to the eighth output via a skip connection to obtain a ninth output; the ninth output is passed through the NAF convolution block of the decoder to obtain a tenth output; the tenth output is added to the masked disparity map to obtain a padded disparity map; The masked disparity map is obtained by multiplying the occlusion mask map and the non-occlusion area disparity map; the non-occlusion area disparity map is obtained by performing disparity regression on the aggregated cost volume.

6. The unsupervised binocular depth estimation method based on occlusion decoupling network according to claim 1, characterized in that: In step S2, the loss function of the matching module is expressed as: in, Indicates the difference between the predicted value and the true value of the non-occluded area, smooth L1 () represents the SmoothL1 loss function, d noc Denotes the disparity of the predicted non-occluded area, d gt Indicates d noc The corresponding true disparity value, Represents the image reconstruction loss, I and D represent the input left image and the predicted disparity map respectively, α represents the weight coefficient, and x l Represents the RGB value of any pixel in the left image, Indicates the left image after remapping the right image according to the parallax, x r Represents the RGB value of any pixel in the right image, SSIM is the structural similarity calculation function, represents the parallax smoothing loss, represents the gradient along the x and y axes of the image, and ‖‖1 represents the 1-norm.

7. The unsupervised binocular depth estimation method based on occlusion decoupling network according to claim 1, characterized in that: In step S2, the loss function of the occlusion perception module is expressed as: in, represents the cross entropy loss function, o represents the occlusion probability value of any pixel, O represents the predicted occlusion probability map, o gt The ground truth of the occlusion obtained from the left-right consistency check.

8. The unsupervised binocular depth estimation method based on occlusion decoupling network according to claim 5, characterized in that: The edge extraction algorithm is expressed as follows: Among them, CIConv(x,y) represents the edge convolution at pixel x, y, x represents the horizontal coordinate value of the 2D coordinate of any pixel, y represents the vertical coordinate value of the 2D coordinate of any pixel, CI is an edge detector that is invariant to changes in illumination and color, σ is a parameter in the training process, ε represents a constant, μ s is the sample mean, σ s is the sample standard deviation.

9. The unsupervised binocular depth estimation method based on occlusion decoupling network according to claim 1, characterized in that: In step S2, the loss function of the occlusion filling module is expressed as: in, represents the loss function of the occlusion filling module, represents the SmoothL1 loss function, d occ represents the disparity of the occluded area, d gt Indicates d occ The corresponding true disparity value.

10. An unsupervised binocular depth estimation system based on occlusion decoupling network, characterized in that: include: A data acquisition module is used to acquire left and right image data and divide the left and right image data into a synthetic scene data set and a real scene data set; Model building module, used to build a binocular depth estimation model, including: matching module, occlusion perception module, and occlusion filling module; The input of the matching module is the left and right image data, and the output of the matching module is the aggregated cost volume; The input of the occlusion perception module is the aggregated cost volume, and the output of the occlusion perception module is the occlusion mask map; The first input of the occlusion filling module is the left image, the second input of the occlusion filling module is obtained by multiplying the occlusion mask map and the non-occlusion area disparity map, the non-occlusion area disparity map is obtained by performing disparity regression on the aggregated cost volume, and the output of the occlusion filling module is the disparity map; A distribution training module is used to perform distribution training on the modules of the binocular depth estimation model using a synthetic scene dataset to obtain a trained binocular depth estimation model; The specific distribution training is: (1) Freeze the parameters of the occlusion perception module and the occlusion filling module, and use the synthetic scene dataset to train the matching module until convergence to obtain the trained matching module; (2) Freeze the parameters of the occlusion filling module and the trained matching module, and use the output of the trained matching module to train the occlusion perception module until convergence, thereby obtaining the trained occlusion perception module; (3) Freeze the parameters of the trained matching module and the trained occlusion perception module, use the output of the trained matching module to perform disparity regression, and then multiply it with the output of the trained occlusion perception module as the training input of the occlusion filling module. Train the occlusion filling module until convergence to obtain the trained occlusion filling module. (4) Using a synthetic scene dataset, the overall network consisting of the trained matching module, the trained occlusion perception module, and the trained occlusion filling module is end-to-end trained until convergence, thereby obtaining a binocular depth estimation model after initial training; (5) Freeze the parameters of the occlusion perception module and the occlusion filling module of the initially trained binocular depth estimation model, and use a self-supervised approach to fine-tune the matching module of the initially trained binocular depth estimation model using a real scene dataset until convergence, thereby obtaining a trained binocular depth estimation model; The depth estimation module is used to input the left and right images of the binocular camera to be detected into the trained binocular depth estimation model to obtain the estimated disparity map. Combined with the calibration parameters of the binocular camera and the binocular baseline, a depth map with the same size as the input image is obtained.

Images