Disparity map inpainting method and system based on semi-supervised deep learning

Abstract

The present invention belongs to the technical field of visual image processing, and particularly relates to a disparity map repair method and system based on semi-supervised deep learning, which constructs a semi-supervised deep learning model for obtaining a disparity map between input view data and repairing a missing part of the disparity map, and a target loss function and sample data for model training, wherein corresponding sample data is constructed according to the target data distribution, and for the input view data, the model is used to perform feature extraction, pyramid pooling and convolution operations in sequence to obtain image features, and the matching cost of the view image features is used to perform disparity calculation, disparity regression and disparity value output; the semi-supervised deep learning model is pre-trained using the sample data and the target loss function; the weights obtained in the pre-training process are introduced into the semi-supervised deep learning model, and the semi-supervised deep learning model is used to fine-tune a preset number of labeled target data to achieve disparity map repair of the target data, so as to meet the application requirements of high-precision scenes.

Description

Technical Field

[0001] The present invention belongs to the technical field of visual image processing, and in particular relates to a disparity map repair method and system based on semi-supervised deep learning. Background Art

[0002] Dense matching is a core issue in the field of photogrammetry and computer vision. The disparity map generated by dense matching is combined with the internal and external orientation elements to obtain the distance information between the photographic point and the object. This technology has important applications in automated navigation, archaeology, medicine, and change detection. Traditional dense matching methods can be divided into local matching algorithms, global matching algorithms, and semi-global matching methods. Among them, local matching algorithms are usually faster, but the overall accuracy is lower. Common local matching algorithms include the sum of absolute grayscale differences (SAD), the sum of squared grayscale differences (SSD), gradient-based metrics, correlation coefficient matching, etc.; global matching methods use global optimization functions to reduce noise interference. Common methods include simulated annealing, dynamic programming, graph cut (GC), and confidence propagation; on the whole, local matching methods are more efficient, but the accuracy is limited, and global methods have higher overall accuracy, but the calculation time is longer. Semi-Global Matching (SGM) combines the advantages of global and local methods and has become the most popular method for dense matching in the past 10 years. Many commercial software are improved based on this matching method. However, the disparity map generated by traditional matching methods represented by SGBM is semi-dense, which is difficult to use in scenes that require dense disparity maps. Therefore, it is necessary to patch the missing areas of the disparity map. Traditional patching methods are based on the edges of the missing disparity areas, which requires extremely high edge accuracy. At the same time, the texture features of the original image are not considered during the patching process, so the patching accuracy is limited. Summary of the invention

[0003] To this end, the present invention provides a disparity map repair method and system based on semi-supervised deep learning, which repairs the disparity map generated by traditional matching methods represented by SGBM through a deep learning semi-supervised method to generate a dense disparity map, which can meet the needs of high-precision scene applications.

[0004] According to the design scheme provided by the present invention, a disparity map repair method based on semi-supervised deep learning is provided, which includes the following contents:

[0005] Constructing a semi-supervised deep learning model for obtaining a disparity map between input view data and repairing missing parts of the disparity map, as well as a target loss function and sample data for model training, wherein corresponding sample data is constructed according to the target data distribution, and for the input view data, the model is used to perform feature extraction, pyramid pooling and convolution operations in sequence to obtain image features, and the matching cost of the view image features is used to perform disparity calculation, disparity regression and disparity value output;

[0006] Pre-train the semi-supervised deep learning model using sample data and target loss function;

[0007] The weights obtained during the pre-training process are imported into a semi-supervised deep learning model, and the semi-supervised deep learning model is used to fine-tune a preset number of labeled target data to achieve disparity map inpainting of the target data.

[0008] As a disparity map inpainting method based on semi-supervised deep learning of the present invention, further, in the semi-supervised deep learning model, the input view data is first preprocessed, the left and right view data are cropped to a uniform size, and feature extraction, pyramid pooling and convolution layer convolution operations are performed on the preprocessed left and right view data in sequence based on weight sharing; and the disparity calculation is performed using the three-dimensional convolution of the PSMNetB network or the stacked hourglass method of the PSMNetS network.

[0009] As the disparity map inpainting method based on semi-supervised deep learning of the present invention, further, the target loss function adopts a smooth L1 loss function, wherein the smooth L1 loss function is expressed as x is the difference between the true value and the predicted value.

[0010] As a disparity map inpainting method based on semi-supervised deep learning of the present invention, further, for the target data, a semi-global block matching method is used to take the matching result as the target data label, and the semi-supervised deep learning model is used to fine-tune the label.

[0011] As a disparity map inpainting method based on semi-supervised deep learning of the present invention, further, in a semi-global block matching method, the minimum matching cost is obtained by sequentially calculating left and right view matching cost values, optimizing cost values ​​and calculating disparity values.

[0012] As a disparity map inpainting method based on semi-supervised deep learning of the present invention, further, in the cost value calculation, image features are extracted by preset feature descriptors, and the left and right view matching costs are calculated by two methods: direct calculation and filtering the view in the horizontal direction using the Sobel operator, and the matching costs calculated in the two methods are merged and block processed.

[0013] As a disparity map inpainting method based on semi-supervised deep learning of the present invention, further, in the cost value optimization and disparity value calculation, one-dimensional dynamic programming in multiple directions is used to aggregate the matching cost values ​​and select the disparity value that minimizes the matching cost as the final disparity of the corresponding pixel.

[0014] Furthermore, the present invention also provides a disparity map inpainting method based on semi-supervised deep learning, comprising: a model building module, a model training module and a target inpainting module, wherein:

[0015] A model construction module, for constructing a semi-supervised deep learning model for obtaining a disparity map between input view data and repairing missing parts of the disparity map, as well as a target loss function and sample data for model training, wherein corresponding sample data is constructed according to the target data distribution, and for the input view data, the model is used to perform feature extraction, pyramid pooling and convolution operations in sequence to obtain image features, and the matching cost of left and right view image features is used to perform disparity calculation, disparity regression and disparity value output;

[0016] Model training module, used to pre-train the semi-supervised deep learning model using sample data and target loss function;

[0017] The target inpainting module is used to use the weights obtained in the pre-training process to import into the semi-supervised deep learning model, and use the semi-supervised deep learning model to fine-tune a preset number of labeled target data to achieve disparity map inpainting of the target data.

[0018] Beneficial effects of the present invention:

[0019] The present invention uses the advantages of deep learning to extract features to repair missing areas in disparity maps, which are limited in accuracy and cannot meet the needs of high-precision scenes. It also uses transfer learning to accelerate convergence and further improve the repair accuracy. Experimental data show that the deep learning semi-supervised method in this case has obvious advantages and is of great use value. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a schematic diagram of the disparity map repair process based on semi-supervised deep learning in the embodiment;

[0021] Figure 2 This is a schematic diagram of the network structure of the semi-supervised deep learning model in the embodiment;

[0022] Figure 3 This is a schematic diagram of the pyramid pooling network structure in the embodiment;

[0023] Figure 4 It is a schematic diagram of the parallax construction process in the embodiment;

[0024] Figure 5 This is a schematic diagram of a common three-dimensional convolution module in the embodiment;

[0025] Figure 6 This is a schematic diagram of the process of the semi-global block matching method in the embodiment;

[0026] Figure 7 This is a schematic diagram of the classification of transfer learning in the embodiment;

[0027] Figure 8 Schematic diagram of SGBM weak texture and texture missing area in the embodiment;

[0028] Fig. 9 This is a schematic diagram of the loss of the SGBM method in the occluded area in the embodiment;

[0029] Fig.10 Schematic diagram of the visualization result of the repair method in the embodiment. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present invention clearer and more understandable, the present invention is further described in detail below in conjunction with the accompanying drawings and technical solutions.

[0031] The present invention provides a method for disparity map repair based on semi-supervised deep learning. Figure 1 As shown, it contains the following contents:

[0032] S101, constructing a semi-supervised deep learning model for obtaining a disparity map between input left and right view data and repairing missing parts of the disparity map, and a target loss function and sample data for model training, wherein corresponding sample data is constructed according to the target data distribution, and for the input left and right view data, the model is used to perform feature extraction, pyramid pooling and convolution operations in sequence to obtain image features, and the matching cost of the left and right view image features is used to perform disparity calculation, disparity regression and disparity value output;

[0033] S102, pre-training the semi-supervised deep learning model using sample data and target loss function;

[0034] S103, using the weights obtained in the pre-training process to import into the semi-supervised deep learning model, and using the semi-supervised deep learning model to fine-tune a preset number of labeled target data to implement disparity map repair of the target data.

[0035] As deep learning technology matures, its powerful feature extraction capabilities are unmatched by traditional methods. In this embodiment, the disparity map generated by traditional matching methods represented by SGBM is repaired by a deep learning semi-supervised method, which can generate a dense disparity map to meet the needs of high-precision scene applications.

[0036] Furthermore, in the semi-supervised deep learning model, the input left and right view data are first preprocessed and cropped to a uniform size. Based on weight sharing, feature extraction, pyramid pooling and convolution layer convolution operations are performed on the preprocessed left and right view data respectively; and the disparity is calculated using the three-dimensional convolution of the PSMNetB network or the stacked hourglass method of the PSMNetS network.

[0037] The disparity map generated by the SGBM method is semi-dense. In order to be suitable for scenes that require dense disparity maps, such as comparison with deep learning unsupervised methods and construction of digital surface models (DSM), the disparity map matched by the SGBM method needs to be repaired. Generally speaking, the disparity missing area is the occlusion area. These occlusion points usually appear at the junction of the background area and the foreground area. The foreground area is closer to the photographic point and has a larger disparity offset, which will occlude the background area. Therefore, the background area can be used to repair the missing area of ​​the disparity map. The specific assignment method is to find a non-missing point on the left and right of the missing point, and select a smaller disparity value to fill the missing point. The disparity map repaired by this method usually has horizontal "stripes", which can be eliminated by filtering. The semi-supervised training method is not a training method parallel to supervision, unsupervised and weak supervision. This training method belongs to a kind of incomplete supervision in weak supervision, that is, training is carried out under the condition that some data contain inaccurate labels to learn the mapping process from data to labels. In this embodiment, the PSMNetB network structure with higher training accuracy and faster training speed can be used. The loss function uses a smooth L1 loss function and inaccurate labels. PSMNetB is a version of the PSMNet network structure ablation experiment process. See Figure 2 As shown in the figure, the network structure mainly includes several main modules: feature extraction, pyramid pooling, matching cost construction, disparity calculation, and disparity soft regression. There is a difference between the PSMNetB version and the final network PSMNetS version in the disparity calculation part. PSMNetB uses three-dimensional convolution to implement disparity calculation, while PSMNetS uses stacked hourglass method to implement it.

[0038] The L1 loss function has a stable gradient, so there will be no gradient explosion problem. At the same time, this loss function has a relatively stable solution, but its disadvantages are also obvious. Its center point is not differentiable, which is not easy to solve. In addition, the stable gradient will cause the result to continue to oscillate near the solution. The L2 loss is smooth and continuous everywhere, which is easy to differentiate, but there may be a risk of gradient explosion. The Smooth L1 loss function combines the advantages of the L1 and L2 loss functions. While there is no map explosion problem, it has a relatively stable solution. Therefore, in this embodiment, the Smooth L1 loss function is used, which can be defined as follows:

[0039]

[0040] The image height, width and disparity dimension parameter settings in the network structure can be shown in the following table:

[0041]

[0042]

[0043]

[0044] The convolution parameter format in the layer setting is A×A, B, where A×A represents the convolution and size; B represents the number of convolution kernels.

[0045] Feature extraction includes coarse feature extraction (conv0_1, conv0_2, conv0_3) and fine feature extraction (convl_x, conv2_x conv3_x) of input data, where: The brackets in represent residual blocks, and C represents the number of residual blocks. The pyramid pooling process is as follows Figure 3 As shown in the figure, the result of fine feature extraction is pooled, convolved and upsampled, and the feature map is concatenated with the result of coarse feature extraction. Matching cost construction is an important mark to distinguish between general dense matching networks and special dense matching networks. By constructing a tensor of pairwise feature maps superimposed in the disparity dimension from zero to the maximum disparity maxdisp based on the baseline direction offset, it lays the foundation for subsequent disparity calculation and obtaining the final disparity. The specific construction process is as follows Figure 4 As shown in the figure, by keeping the left feature map unchanged, the right feature map is translated along the baseline direction according to a certain step length d, and then spliced ​​with the left feature map and superimposed into the cost matrix. The extra part after translation is cut off, and the missing part is filled with 0, finally forming a four-dimensional tensor (channel, disparity, height, width) of the matching cost. The matching cost construction increases the disparity dimension of the feature map, changing it from a three-dimensional tensor to a four-dimensional tensor. As the search range (maximum disparity maxdisp) increases, the required memory also increases exponentially.

[0046] The essence of disparity calculation is the process of calculating the constructed matching cost using three-dimensional convolution. This process can not only utilize the relevant information in the two-dimensional direction of the plane, but also utilize the correlation between the disparity dimension information. Therefore, the overall accuracy is higher than that of the dense matching general network. However, this process also has disadvantages. Compared with two-dimensional convolution, three-dimensional convolution requires more computation, which is one of the important reasons why this type of network has a slow computing speed. The disparity calculation is located after the matching cost is constructed. In terms of specific implementation, this module has two more representative calculation modules, namely the ordinary three-dimensional convolution module and the stacked hourglass module. In the embodiment of this case, an ordinary three-dimensional convolution method can be used, such as Figure 5As shown in the figure, the common 3D convolution module uses 6 convolution blocks to calculate the disparity. Each convolution block consists of two 3×3×3 3D convolution layers. A skip layer structure is set between the blocks to increase the fusion of multi-level information. The convolution module is followed by an upsampling and disparity soft regression module. After the disparity calculation is completed, the horizontal, vertical and depth resolutions are restored by upsampling, and the best disparity position is determined by disparity soft regression.

[0047] In a dense matching network based on the GCNet structure, after the disparity calculation module, within the range of 0-maxdisp, the disparity value of each position corresponds to its matching cost, which can be converted into the corresponding probability through the softmax function. The traditional method selects the position with the highest probability as the disparity value of the pixel, that is, selects the subscript of the maximum probability. This operation usually uses the argmax function, but because the process is not differentiable, it cannot be applied to deep learning end-to-end networks that require backpropagation. Therefore, GCNet uses a differentiable soft regression function, and its process can be described by the formula:

[0048]

[0049] Where maxdisp represents the maximum value of the disparity direction search during the matching process; d is any integer from 0 to maxdisp; c represents the matching cost output by the disparity calculation module; σ(·)d represents the probability of the disparity value being d, where σ(·) represents the softmax function, which can convert the matching cost into probability. The formula is:

[0050]

[0051] In the formula, c d is the matching cost of the disparity value d. The smaller the cost, the greater the corresponding probability. The two are opposite, so a negative sign is needed; e is a natural constant.

[0052] Finally, the classification problem is transformed into a regression problem through the parallax soft regression method, which greatly improves the matching accuracy.

[0053] Furthermore, for the target data, a semi-global block matching method is used to take the matching result as the target data label, and a semi-supervised deep learning model is used to fine-tune the label. Furthermore, in the semi-global block matching method, the minimum matching cost is obtained by sequentially calculating the left and right view matching cost values, optimizing the cost values, and calculating the disparity value. Furthermore, in the cost value calculation, the image features are extracted by a preset feature descriptor, and the left and right view matching costs are calculated by two methods: direct calculation and filtering the view in the horizontal direction using the Sobel operator. The matching costs calculated in the two methods are merged and block processed. Furthermore, in the cost value optimization and disparity value calculation, one-dimensional dynamic programming in multiple directions is used to aggregate the matching cost values ​​and select the disparity value that minimizes the matching cost as the final disparity of the corresponding pixel.

[0054] OpenCV uses the SGM method as the core to implement the Semi-Global Block Matching (SGBM) method and makes some changes to the original algorithm: first, the BT (Birchfield Tomasi) method is used to replace the matching cost calculation based on mutual information; second, some pre-processing and post-processing in the Block Matching (BM) algorithm are added; third, only 5 directions are used in dynamic programming. Figure 6 As shown in the figure, the SGBM algorithm can be divided into four stages: cost value calculation, cost value optimization, disparity value calculation, and disparity value post-processing. Among them, the cost value calculation mainly extracts image features by artificially designing feature descriptors, and then calculates the corresponding matching cost. The Sobel operator is an important processing method in the field of computer vision. It is mainly used to obtain the first-order gradient of digital images. Its common application and physical meaning is edge detection. The Sobel operator is a weighted difference of the grayscale values ​​of the four areas of up, down, left, and right of each pixel in the image, which reaches an extreme value at the edge to detect the edge; the operator contains two sets of 3x3 matrices, which are horizontal and vertical respectively. By performing a planar convolution with the image, the horizontal and vertical brightness difference approximations can be obtained respectively. If A represents the original image, Gx and Gy represent the images after horizontal and vertical edge detection respectively, the formula is as follows:

[0055]

[0056]

[0057] Therefore, in this embodiment, a horizontal G x Convolve the image. The cost of BT uses the sub-pixel grayscale information to calculate the absolute value of the pixel grayscale value difference.

[0058] Among them, I L and IR Represents the left and right images; x L , x represents the horizontal coordinate of the left IL; d represents the disparity value; x L +d, x+d means right I R The horizontal coordinate; y represents the vertical coordinate on the image. The full name of the SGM algorithm is Semi-Global Matching, which is essentially a cost aggregation algorithm, similar to the cost aggregation in the local stereo matching algorithm. In order to achieve the same effect of minimizing the global energy function as the global stereo matching algorithm, more or all pixels in the entire image are required to participate in the constraints of the current pixel. The SGM algorithm is similar to cost aggregation, while traditional cost aggregation is calculated in the neighborhood, such as summation or weighted average. In order to expand the scope without increasing the amount of calculation too much, the SGM method performs cost aggregation through one-dimensional dynamic programming, which can establish global constraints without constructing the global minimum energy function, avoiding performance degradation caused by complex calculations. Its one-dimensional cost aggregation is as follows:

[0059]

[0060] The cost aggregation value of a certain neighborhood direction on point P is divided into three items. The first is the matching cost of the current point; the second is min(the current disparity cost aggregation value of the pixel point pr in the current neighborhood direction, the cost aggregation value when the disparity difference of point pr is 1 + P1, the minimum cost aggregation value when the disparity difference of point pr is greater than 1 + P2); the last item is the minimum cost aggregation value when the disparity interpolation of point pr is greater than 1. The last item is a unified adjustment made to prevent the calculation result from being too large.

[0061] In the actual calculation process, the left and right images are calculated in two ways to calculate the matching cost respectively. One is to directly calculate the matching cost; the other is to use the Sobel operator to filter the horizontal direction of the original image and then calculate the BT cost. The two matching costs are then merged and block processed. The SGM method combines the advantages of the global matching method and the local matching method, greatly improving the matching efficiency while taking into account the accuracy. The idea of ​​SGM is to replace the two-dimensional dynamic programming with one-dimensional dynamic programming in multiple directions in the matching cost aggregation step to solve the minimum matching cost. The disparity calculation process calculates the final disparity based on the optimized cost value. The calculation of disparity includes multiple strategies, such as "winner-take-all" (WTA), weighted average, etc. The SGBM method uses the most common WTA method, that is, the disparity value that minimizes the matching cost is selected as the final disparity of the pixel. In the dense matching process, due to the influence of various factors, such as the occlusion and reflection factors of the data source itself, the factors of feature extraction description design, the cost optimization and disparity calculation strategy design and selection factors, etc., mismatching will occur. Therefore, most of the gross errors can be eliminated through confidence testing and left-right consistency detection. The confidence test verifies the correctness of the disparity value of the current pixel through the disparity value within the domain to ensure the global optimality of the pixel; the left-right consistency test verifies the correctness of the disparity through the disparity consistency of the same-name points in the left-right disparity map, which can effectively eliminate the occluded area. In addition, in the process of stereo matching, the disparity will be divided into multiple spatially discrete planes, but the real scene is continuous and gradually changing, so the surface disparity of the object can be made smoother through sub-pixel interpolation.

[0062] Transfer learning is a strategy to apply a trained model to other datasets. This method takes advantage of the generalization performance of the original model. Under the condition that the source dataset and the target dataset have similar distribution, transfer learning can produce better results and save a lot of computing time. In addition, transfer learning can effectively prevent the "overfitting" phenomenon caused by too few target data labels. Transfer learning is divided into two methods: direct transfer and model fine-tuning, depending on whether training is performed on the target dataset after transfer. See Figure 7 As shown in the figure, the former does not make any adjustments to the trained model and directly applies it to the new data set; the latter is to fine-tune the model on the new data set to make it more suitable for the data set. Model fine-tuning is divided into overall fine-tuning and partial layer fine-tuning. In this embodiment, direct migration and overall fine-tuning can be used in the experimental process.

[0063] Furthermore, based on the above method, an embodiment of the present invention also provides a disparity map inpainting method based on semi-supervised deep learning, comprising: a model building module, a model training module and a target inpainting module, wherein:

[0064] A model construction module, for constructing a semi-supervised deep learning model for obtaining a disparity map between input left and right view data and repairing missing parts of the disparity map, as well as a target loss function and sample data for model training, wherein corresponding sample data is constructed according to the target data distribution, and for the input left and right view data, the model is used to perform feature extraction, pyramid pooling and convolution operations in sequence to obtain image features, and the matching cost of the left and right view image features is used to perform disparity calculation, disparity regression and disparity value output;

[0065] Model training module, used to pre-train the semi-supervised deep learning model using sample data and target loss function;

[0066] The target inpainting module is used to use the weights obtained in the pre-training process to import into the semi-supervised deep learning model, and use the semi-supervised deep learning model to fine-tune a preset number of labeled target data to achieve disparity map inpainting of the target data.

[0067] In order to verify the effectiveness of this solution, the following is a further explanation based on the test data:

[0068] To ensure the effectiveness of the experiment and eliminate the errors that may be caused by random factors, the experiment was tested on multiple datasets. SceneFlow is a virtual scene dataset used to generate the pre-training model required for transfer learning; the two KITTI datasets are close-up driving scene datasets; Vaihingen and WHU are remote sensing datasets. Among them, the SceneFlow dataset is an important basis for pre-training of many classic networks before practical application, and plays a vital role in the field of dense matching. This dataset generates dense disparity maps by projecting three-dimensional modeling to two dimensions. The label accuracy is extremely high and the data volume is huge. It is suitable as a pre-training dataset for transfer learning models. The KITTI dataset is a real driving dataset composed of a large number of road scenes such as trees, cars, and houses. It includes two datasets, KITTI2012 and KITTI2015. The label data of the KITTI dataset is inversely calculated from the results of the vehicle-mounted laser radar ranging and is manually corrected. The label accuracy is high, but at the same time, due to the limitation of the laser radar resolution, the label disparity map is semi-dense. KITTI2012 contains 194 pairs of training images and 195 pairs of test images of 1226 pixels × 370 pixels, and KITTI2015 contains 200 pairs of training images and 200 pairs of test images of 1242 pixels × 375 pixels. In this case, the training sets of the two data sets can be selected for experiments. Vaihingen is an aerial photography data set taken in a relatively flat area. It contains 36 rural images with an image size of 9240 pixels × 14430 pixels. After sorting and cropping, 731 pairs of images with an image size of 955 pixels × 360 pixels are formed. There are fewer high-rise buildings, and the main rural features include houses, rivers, farmland, woods, etc. The label data is obtained by inverse calculation of DSM data combined with internal and external orientation elements. WHU data is a total of 1776 images of a certain area in Guizhou, China, with a size of 5376 pixels × 5376 pixels taken by drones. The main scenes include a small number of high-rise buildings, some factory areas, and natural features such as mountains and rivers. The label data is obtained by inverse calculation of DSM reconstructed by multiple cameras. After sorting and cropping, the dataset images were cropped to 768 pixels × 384 pixels, with a total of 10,979 pairs of images. Since the dataset is large, the single round of training takes too long, so this case can select 200 images from the training set for experiments.

[0069] The computer CPU used in the experiment is Xeon E5-2680, with a main frequency of 2.8GHz; the graphics card is GTX TITAN X, with a video memory size of 12G; the total memory is 64G. The experiment was conducted under the Windows 10 operating system, and the virtual environment was created through Anaconda. The programming language was Python, and the semi-supervised method was implemented based on the Pytorch deep learning experimental framework. The experiment mainly includes three parts: semi-dense disparity map generation, traditional method inpainting, and deep learning semi-supervised method inpainting.

[0070] In order to explore the missing disparity map in each data set, we first need to generate the disparity map of each data set through the SGBM method. In this case, SGBM is implemented using Opencv, which involves many parameters. The specific parameter values ​​and meanings are shown in Table 1.

[0071] Table 1 Parameter settings of SGBM method

[0072]

[0073] Traditional patching methods are relatively simple. In this case, the focus is on filtering methods. Gaussian filtering and median filtering are used to process horizontal stripes, and the convolution kernels include 3×3 and 5×5 sizes.

[0074] The semi-supervised deep learning method uses the SGBM matching results as labels for training. The network structure uses PSMNetB, and the loss function uses the smooth L1 loss function; the maximum disparity search range is 192 pixels; the batch parameter batchsize is set to 8. Considering that the dense matching network structure occupies a large amount of video memory during training, the process is implemented by gradient accumulation; the optimizer is Adam, β1=0.9, β2=0.999; the learning rate is 0.001; the maximum training round is 50 rounds. During the training process, in order to save video memory and increase randomness, the left and right images are cropped to 512 pixels × 256 pixels and then input into the network. The model used for transfer learning is the network obtained by 10 rounds of supervised training on the SceneFlow dataset; 50 rounds of training are performed on the KITTI2012, KITTI2015 and WHU datasets, and 10 rounds of training are performed on the Vaihingen dataset. Absolute endpoint error and p pixel error are used as evaluation indicators.

[0075] The absolute end point error (EPE) is defined as the average of the absolute values ​​of the differences between the effective pixels on the predicted disparity map and the true values ​​of the corresponding positions, which is defined as:

[0076]

[0077] in,

[0078]

[0079]

[0080] Where, mindisp and maxdisp are the lower and upper bounds of the disparity involved in the evaluation, respectively. In this case, they are set to 0 and 192, respectively; H and W represent the height and width of the input image pair, respectively; the subscripts i and j represent the row and column numbers of the image; the superscripts P and T represent the prediction and true value, respectively; D represents the disparity map; M represents the mask whose disparity value is between (mindisp, maxdisp), 1 represents valid, and 0 represents invalid; N represents the total number of valid points.

[0081] The p Point Error (pPE) can be described as the percentage of points whose absolute value of the difference between the predicted disparity and the true value is less than p pixels in the valid area to the total number of points involved in the test, which is defined as:

[0082]

[0083] In the formula, F sum represents the summation function, F sum (M) indicates the total points of participating in the test; F sum (M P ) indicates the number of points that meet the requirements in the disparity map; error point mask It can be obtained from formula 5:

[0084]

[0085] Where D diff Represents the predicted value.

[0086] The SGBM method was used to conduct experiments on four data sets. The indicators included disparity map density, EPE, and 1PE-3PE error. The specific results are shown in Table 2:

[0087] Table 2 Matching results of SGBM method

[0088]

[0089] As can be seen from the table, compared with the close-up driving dataset, the disparity map generated by the SGBM method on the remote sensing photography dataset has a higher density, reaching 92.82% and 97.85% respectively.

[0090] By observing the missing values ​​of the SGBM method on the four datasets, we can see that the missing disparity parts are mainly concentrated in relatively smooth areas such as weak texture and textureless areas, and in areas where there are no same-name points due to occlusion caused by the foreground and background.

[0091] from Figure 8It can be clearly seen in (a)(b) that weak texture areas and textureless areas such as the sky, concrete floor, and smooth surface of the car in (c)(d) are missing. Although there are also some weak texture and textureless areas in remote sensing images (e)(f), the proportion of this part is small, so there is a higher disparity map density. Theoretically, the occluded area cannot find the same name point, so this part should be missing. This is also confirmed by the actual matching results, such as Fig. 9 As shown in the figure, compared with the KITTI dataset, the remote sensing dataset has a smaller proportion of occluded areas due to the characteristics of aerial photography, so the disparity map density is higher.

[0092] After patching the SGBM matching results using traditional methods, the disparity map was filtered using a variety of filtering methods. The results are shown in Table 3.

[0093] Table 3 Results of each post-processing method

[0094]

[0095] Note: M represents mean filtering; G represents Gaussian filtering; the numbers represent the convolution kernel size

[0096] As can be seen from the table, no matter which filtering method is used, the accuracy of the disparity map after repair will decrease. The EPE and 3PE after repair are about 2-3 times that before repair, indicating that the disparity accuracy of the repaired area is much lower than that of the non-repaired area. In addition, a horizontal comparison of the two filters and different convolution kernel sizes shows that the median filter with a 5×5 convolution kernel has the best effect, so this result is used for comparison with subsequent experiments.

[0097] The deep learning semi-supervised method used direct training and transfer learning to perform semi-supervised method repair on four datasets. The results are shown in Table 4.

[0098] Table 4. Disparity map repair using deep learning method

[0099]

[0100] Note: PSMB indicates the adopted network structure; C semi Represents a semi-supervised method; pre represents pre-training.

[0101] It can be seen from the table that the deep learning semi-supervised method is much higher than the 5×5 median filter with the highest accuracy among the traditional patching methods, and the EPE and 3PE are reduced by 33.98% and 17.83% on average. Among them, the semi-supervised method based on transfer learning is more accurate than the semi-supervised method based on direct training, and the EPE and 3PE are reduced by 5.17% and 12.58% on average under the transfer learning condition.

[0102] Considering that transfer learning has the function of accelerating convergence, in order to verify the effect, the optimal round statistics of EPE and 3PE of the four data sets involved in the test are shown in Table 5.

[0103] Table 5 Convergence round unit: round

[0104]

[0105] As can be seen from the table, the pre-trained model formed after 10 rounds of SceneFlow virtual scene dataset can be effectively migrated to the real scene dataset, which not only has higher accuracy, but also can greatly improve the convergence speed and reduce the number of training rounds, which has strong practical significance.

[0106] To further analyze the inpainting effect, the label data and the results of each inpainting method are rendered with color patterns, where the colors change gradually. The dark area after red grayscale is the area with the largest disparity, and the light area after blue grayscale is the area with the smallest disparity. Since there is no light area after blue grayscale in this image, and the disparity value of the missing area in SGBM matching is -1, we can get Fig.10 (c) shows the missing areas of SGBM matching. Compared with the original image, it can be seen that the missing areas are concentrated at the edges of disparity discontinuities and image edges ( Fig.10 (c) right edge); Traditional inpainting methods rely on the lack of edge precision in disparity maps. Fig.10 (d) According to the experimental results, relatively accurate repair can be achieved for areas with high edge accuracy, but the repair effect is poor for areas where the edges themselves have mismatched. The deep learning semi-supervised method can effectively repair the missing areas, such as Fig.10 (e) and (f), further analysis revealed that the poorly repaired areas by the deep learning semi-supervised method were the result of mismatching by the SGBM method and had nothing to do with the deep learning method.

[0107] Unless otherwise specifically stated, the relative steps, numerical expressions and values ​​of the components and steps set forth in these embodiments do not limit the scope of the present invention.

[0108] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A disparity map inpainting method based on semi-supervised deep learning, characterized in that: Contains the following: A semi-supervised deep learning model for obtaining disparity maps between input view data and repairing missing parts of the disparity maps, as well as a target loss function and sample data for model training are constructed, wherein the corresponding sample data is constructed according to the target data distribution, and for the input view data, the model is used to perform feature extraction, pyramid pooling and convolution operations in sequence to obtain image features, and the matching cost of the view image features is used to perform disparity calculation, disparity regression and disparity value output; the target loss function uses a smooth L1 loss function, wherein the smooth L1 loss function is expressed as x is the difference between the true value and the predicted value; the feature extraction includes coarse feature extraction and fine feature extraction of the input data, and the convolution parameter form of the coarse feature extraction level setting is in the form of A×A, B, and the convolution parameter form of the fine feature extraction level setting is in the form of In the form of A×A, the convolution kernel size, B the number of convolution kernels, and C the number of residual blocks. Pre-train the semi-supervised deep learning model using sample data and target loss function; The weights obtained during the pre-training process are imported into a semi-supervised deep learning model, and for the target data, a semi-global block matching method is used to use the matching results as the target data labels, and the semi-supervised deep learning model is used to fine-tune a preset number of labeled target data to achieve disparity map repair of the target data. In the semi-global block matching method, the minimum matching cost is obtained by sequentially calculating the left and right view matching cost values, optimizing the cost values, and calculating the disparity value; in the cost value calculation, the image features are extracted by a preset feature descriptor, and the left and right view matching costs are calculated by two methods: direct calculation and filtering the view in the horizontal direction using the Sobel operator, and the matching costs calculated in the two methods are merged and block processed; in the cost value optimization and disparity value calculation, one-dimensional dynamic programming in multiple directions is used to aggregate the matching cost values ​​and select the disparity value that minimizes the matching cost as the final disparity of the corresponding pixel.

2. The disparity map inpainting method based on semi-supervised deep learning according to claim 1, characterized in that: In the semi-supervised deep learning model, the input view data is first preprocessed, and the left and right view data are cropped to a uniform size. Based on weight sharing, feature extraction, pyramid pooling and convolution layer convolution operations are performed on the preprocessed left and right view data respectively; and the disparity is calculated using the three-dimensional convolution of the PSMNetB network or the stacked hourglass method of the PSMNetS network.

3. A disparity map inpainting system based on semi-supervised deep learning, characterized in that: The method according to claim 1 is implemented, comprising: a model building module, a model training module and a target patching module, wherein: A model construction module, for constructing a semi-supervised deep learning model for obtaining a disparity map between input view data and repairing missing parts of the disparity map, as well as a target loss function and sample data for model training, wherein corresponding sample data is constructed according to the target data distribution, and for the input view data, the model is used to perform feature extraction, pyramid pooling and convolution operations in sequence to obtain image features, and the matching cost of left and right view image features is used to perform disparity calculation, disparity regression and disparity value output; Model training module, used to pre-train the semi-supervised deep learning model using sample data and target loss function; The target inpainting module is used to use the weights obtained in the pre-training process to import into the semi-supervised deep learning model, and use the semi-supervised deep learning model to fine-tune a preset number of labeled target data to achieve disparity map inpainting of the target data.

Images