A pipeline monocular video 3D reconstruction and depth prediction method and system

Abstract

The present invention discloses a pipeline monocular video 3D reconstruction and depth prediction method and system, which relates to the field of deep learning and image processing technology. The method includes, based on the COLMAP method, respectively extracting features and matching, incrementally reconstructing the image sequences of different pipeline scenes collected, obtaining the camera extrinsic parameter matrix corresponding to each image, and constructing a pipeline 3D reconstruction data set; using existing public data sets to train the Fast‑MVSNet network and the PatchMatchNet network respectively, and selecting the optimal network model from multiple trained network models to perform 3D reconstruction and depth prediction of actual pipeline scenes. The present invention constructs an overall framework from monocular video to 3D scene, which can effectively realize 3D reconstruction and depth prediction of drainage pipes, and intuitively display the internal structure information of the pipes.

Description

Technical Field

[0001] The present invention relates to the technical field of deep learning and image processing, and in particular to a method and system for three-dimensional reconstruction and depth prediction of monocular video of a drainage pipe. Background Art

[0002] Urban underground drainage pipes are relatively special due to their complex internal environment and unique geographical location, and are a high-frequency point for accidents. In urban drainage pipes, internal pipe inspection is mainly based on internal pipe videos obtained by pipe robots, but many cities still use two-dimensional management mode for such video data, making it difficult to grasp the overall condition and detailed characteristics of the pipes, lacking accurate positioning information for disease detection results, and unable to adapt to today's increasingly complex pipe networks. There are many drawbacks such as low data utilization and poor visualization effects, making it difficult to keep up with the pace of urban modernization, informatization, and intelligent development. Therefore, three-dimensional reconstruction of the collected monocular video is extremely important for pipeline management and maintenance.

[0003] The core idea of ​​3D reconstruction is to restore the 3D structure of the original scene from the 2D image acquired by the robot. Most of the current 3D reconstruction methods are based on large outdoor buildings or small objects, and there are few algorithms for 3D reconstruction of pipeline environments. The early traditional 3D reconstruction methods based on monocular vision mainly used structure from motion (SfM). However, due to the error accumulation caused by the continuous participation of images in the reconstruction process, 3D reconstruction is prone to drift problems. In addition, the selection of the initial image pair and the quality of the image will seriously interfere with the results of 3D reconstruction, resulting in low efficiency, sparse 3D point clouds, and difficulty in reconstructing weak texture areas of the pipeline. In recent years, research based on deep learning methods has also shown an upward trend. These methods mainly use general network architectures (e.g., MVSNet). Although the integrity and accuracy of reconstruction are better than traditional methods, they rely on a large amount of video memory and memory, and have high requirements for hardware devices. However, urban underground pipelines have characteristics such as repeated textures and few textures, and are in a dark or low-light environment. The details of the collected video images are blurred, which makes these algorithms poor in 3D reconstruction in drainage pipeline scenes, and the practical application of pipeline 3D reconstruction and depth prediction is difficult. Summary of the invention

[0004] The purpose of the present invention is to provide a pipeline monocular video 3D reconstruction and depth prediction method and system, which can greatly improve the accuracy of pipeline 3D reconstruction and depth prediction.

[0005] To achieve the above object, the present invention provides the following solutions:

[0006] In a first aspect, the present application provides a pipeline monocular video 3D reconstruction and depth prediction method, comprising:

[0007] The depth camera is calibrated using Zhang Zhengyou's calibration method to obtain the camera's internal parameters, and a pipeline robot equipped with the depth camera is used to collect monocular videos in different pipeline scenes;

[0008] For each pipeline scene picture sequence composed of the monocular video, the COLMAP method is respectively applied to perform feature extraction and matching and incremental reconstruction to obtain the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, and a pipeline 3D reconstruction dataset is constructed according to the camera intrinsic parameters, the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, the pipeline scene picture sequence and the corresponding real depth map;

[0009] Using public data sets to train the Fast-MVSNet network and the PatchMatchNet network respectively, a plurality of trained Fast-MVSNet network models and a plurality of trained PatchMatchNet network models are obtained; the public data sets include the DTU data set, the BlendedMVS data set and the ETH3D data set;

[0010] Inputting the pipeline 3D reconstruction data set into all the trained Fast-MVSNet network models and all the trained PatchMatchNet network models respectively, and evaluating all the trained network models to obtain the optimal network model; the trained network models include all the trained Fast-MVSNet network models and all the trained PatchMatchNet network models;

[0011] The optimal network model is used to perform three-dimensional reconstruction and depth prediction on the pipeline three-dimensional reconstruction data set to be processed.

[0012] In a second aspect, the present application provides a pipeline monocular video 3D reconstruction and depth prediction system, comprising:

[0013] The video acquisition module is used to calibrate the depth camera using the Zhang Zhengyou calibration method to obtain the camera internal parameters, and use the pipeline robot equipped with the depth camera to collect monocular videos in different pipeline scenes;

[0014] A pipeline 3D reconstruction dataset construction module is used to apply the COLMAP method to perform feature extraction and matching and incremental reconstruction on each pipeline scene picture sequence composed of the monocular video, obtain the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, and construct a pipeline 3D reconstruction dataset according to the camera intrinsic parameters, the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, the pipeline scene picture sequence and the corresponding real depth map;

[0015] A network training module, used to train the Fast-MVSNet network and the PatchMatchNet network respectively using public data sets to obtain multiple trained Fast-MVSNet network models and multiple trained PatchMatchNet network models; the public data sets include the DTU data set, the BlendedMVS data set, and the ETH3D data set;

[0016] A network testing and screening module, used for inputting the pipeline 3D reconstruction data set into all the trained Fast-MVSNet network models and all the trained PatchMatchNet network models respectively, and evaluating all the trained network models to obtain the optimal network model; the trained network models include all the trained Fast-MVSNet network models and all the trained PatchMatchNet network models;

[0017] The three-dimensional reconstruction and depth prediction module is used to perform three-dimensional reconstruction and depth prediction on the three-dimensional reconstruction data set of the processing pipeline using the optimal network model.

[0018] According to the specific embodiments provided by the present invention, the present invention discloses the following technical effects:

[0019] The present application provides a pipeline monocular video 3D reconstruction and depth prediction method and system, firstly using a depth camera to obtain a monocular video in an experimental pipeline scene, and constructing a pipeline 3D reconstruction data set; using a public data set to train the Fast-MVSNet network and the PatchMatchNet network respectively, and obtain multiple trained network models; then using the pipeline 3D reconstruction data set as a test set, testing each trained network model, so as to evaluate the optimal network model for application to the actual pipeline 3D reconstruction and depth prediction. The present invention constructs an overall framework from monocular video to 3D scene, including the application of the COLMAP method, the application of the Fast-MVSNet network and the PatchMatchNet network, which can effectively realize the 3D reconstruction and depth prediction of drainage pipes, intuitively display the internal structure information of the pipes, and can be used in engineering fields such as real-time monitoring of drainage pipe detection robots, detection and identification of underground pipeline diseases, and regular maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0021] Figure 1 A flow chart of a pipeline monocular video 3D reconstruction and depth prediction method provided in the first embodiment of the present invention;

[0022] Figure 2 A flowchart based on the COLMAP algorithm provided in Example 1 of the present invention;

[0023] Figure 3 A flowchart based on the Fast-MVSNet algorithm provided in Example 1 of the present invention;

[0024] Figure 4 A flowchart based on the PatchMatchNet algorithm provided in the first embodiment of the present invention;

[0025] Figure 5 This is a diagram of the outlier point filtering effect provided in Example 1 of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] The purpose of the present invention is to provide a pipeline monocular video 3D reconstruction and depth prediction method and system, and construct an overall framework from monocular video to 3D scene based on Fast-MVSNet network and PatchMatchNet network. The intrinsic parameters of the monocular video acquired by the pipeline robot are calibrated and fixed by this camera. The camera extrinsic parameters are estimated by COLMAP, and the camera extrinsic parameters of different pictures of different pipelines are different. The overall framework of monocular video 3D reconstruction includes COLMAP and Fast-MVSNet network and PatchMatchNet network, the former estimates the camera extrinsic parameters, and the latter predicts the camera depth map. The overall framework can complete the reconstruction of the pipeline 3D scene through a series of picture sequences when the camera extrinsic parameters are unknown. The optimal model has high efficiency and low video memory requirements, while improving the accuracy and integrity of depth prediction and 3D point cloud, and can obtain a denser pipeline 3D point cloud scene.

[0028] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments.

[0029] Embodiment 1

[0030] like Figure 1As shown, this embodiment provides a pipeline monocular video 3D reconstruction and depth prediction method, including:

[0031] S1: The depth camera is calibrated using Zhang Zhengyou's calibration method to obtain the camera's internal parameters, and a pipeline robot equipped with the depth camera is used to collect monocular videos in different pipeline scenes.

[0032] In the pinhole imaging model of the depth camera of this application, there is a linear relationship between the image coordinate points and the spatial coordinate points of the object. The transformation of a single two-dimensional image point to a three-dimensional space point depends on the solution of the camera's internal and external parameters, and its expression is:

[0033]

[0034] In formula (1), (X w ,Y w ,Z w ) represents the coordinates of the object in the world coordinate system, the parameter Z represents the Z-axis coordinates in the corresponding camera coordinate system, (y, v) represents the coordinates in the corresponding pixel coordinate system projected onto the two-dimensional image, and the parameter M 1 and [R, T] are the intrinsic parameter matrix and extrinsic parameter matrix of the camera respectively.

[0035] Make a 12×8 black and white checkerboard calibration plate, and use the depth camera to take 15 pictures of the calibration plate at different angles. Calibrate the depth camera according to Zhang Zhengyou's calibration method to obtain the internal parameter matrix M 1 and distortion parameter β, whose expressions are:

[0036]

[0037] β=(k 1 ,k 2 ,p 1 ,p 2 ,k 3 ) (3)

[0038] In formula (2) and formula (3), the parameter f x and f y are the focal lengths of the camera in the x and y directions, respectively, (c x ,c y ) is the optical center of the image, parameter k 1 , k 2 , k 3 is the radial distortion parameter, parameter p 1 、p 2 is the tangential distortion parameter.

[0039] S2: For each pipeline scene picture sequence composed of the monocular video, the COLMAP method is applied to perform feature extraction and matching as well as incremental reconstruction to obtain the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, and a pipeline 3D reconstruction dataset is constructed according to the camera intrinsic parameters, the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, the pipeline scene picture sequence and the real depth map.

[0040] Figure 2 This is a flow chart of the present invention based on the COLMAP algorithm, which is an optional implementation mode. The flow chart includes two stages, namely feature extraction and matching, and incremental reconstruction.

[0041] SIFT feature extraction and feature matching are performed on the continuous input images of the same pipeline scene image sequence. The feature set is used to measure the similarity to obtain the feature relationship between image pairs A and B, and the potential scene overlap is obtained. After the matching is completed, since there are usually many mismatched point pairs, the epipolar geometry relationship is used to refine the matching point pairs in each matching image pair to eliminate erroneous matches.

[0042] All images are sorted from large to small according to the number of matching points. The first image is selected from the one with the most matching points. If the number of matching points between the second image and the first image is greater than 100 and the translation vector between the two views is greater than 0.95, the two images are selected as the initialization image pair. In the image with unknown camera pose, the optimal frame is selected as the new image to be added to the three-dimensional pipeline scene according to the number of 2D points in the image and the 3D points in the constructed three-dimensional scene and their distribution on the image. The new image is oriented 2D-2D relative to the image in the constructed three-dimensional pipeline scene model to obtain the 2D-3D matching between the two, so as to calculate the camera pose of the new image using RANSAC-PNP, and generate new three-dimensional points in the map by triangulation. Since the image registration and triangulation processes are relatively independent, the uncertainty of the camera pose (camera extrinsic parameter) will affect the triangulated points. As the number of cameras increases, in order to eliminate this cumulative error, BA optimization is performed after completing image registration and triangulation, that is, the reprojection error is minimized by adjusting the camera pose and the position of the map point.

[0043] After completing the incremental reconstruction, a pipeline test dataset is constructed, which includes each pipeline RGB image in different image sequences and the corresponding depth map, camera internal and external parameters, depth information, and the best source view number. The quaternion Q(q w ,q x ,q y ,q z ) and the translation vector T(t x ,ty ,t z ), q w Belongs to the real part, indicating the rotation angle; q x ,q y ,q z It belongs to the imaginary part, indicating the direction of the rotation axis. The camera external parameters [R, T] are obtained by converting the quaternion into a rotation matrix and other operations. n And the depth information of each image, n represents the nth image; the camera extrinsic parameter expression is as follows:

[0044]

[0045] Considering the continuity of pipeline image sequences and the small angle change of adjacent frames, 10 best source views are selected for each 2D image in the image sequence according to the number of frames, that is, the closer the adjacent frames are to the 2D reference image, the higher the score.

[0046] Therefore, as an optional implementation, step S2 specifically includes:

[0047] S21: performing SIFT feature extraction and feature matching on each pipeline scene image sequence, and eliminating wrong matching point pairs in each matching image pair by using epipolar geometry.

[0048] S22: For each of the pipeline scene image sequences, all images are sorted according to the number of matching points, and the image with the largest number of matching points is taken as the starting point, and two images whose number of matching point pairs between the images is greater than a first preset value and whose translation vectors between the images are greater than a second preset value are selected as the initial image pair; and the camera extrinsic parameters of the first image in the initialized image pair are set to the unit matrix.

[0049] S23: Calculate the camera extrinsic parameters of the second image in the initial image pair based on the 2D-2D matching relationship of the initial image pair and the unit matrix using the epipolar geometry relationship. The 2D-2D matching relationship refers to the SIFT feature matching relationship obtained in step S21.

[0050] S24: Triangulate the initial image pair to generate initial pipeline scene three-dimensional points in the pipeline three-dimensional scene.

[0051] S25: In the currently unselected picture, the best picture is selected as the current newly added picture according to the number of points in the picture that have a corresponding relationship with the currently constructed three-dimensional points of the pipeline scene and the distribution of the corresponding two-dimensional points in the corresponding picture; the currently constructed three-dimensional points of the pipeline scene include the initial pipeline scene three-dimensional points and all the currently optimized three-dimensional points; the corresponding two-dimensional points refer to the two-dimensional points in the picture that have a corresponding relationship with the currently constructed three-dimensional points of the pipeline scene.

[0052] S26: According to the 2D-2D matching relationship between the newly added picture and the currently selected pictures, the 2D-3D matching relationship between the newly added picture and each target picture is obtained, and the camera extrinsic parameters of the newly added picture are calculated using the RANSAC-PNP method according to the 2D-3D matching relationship and the camera intrinsic parameters; the target picture is a picture among the currently selected pictures that has a 2D-2D matching relationship with the newly added picture.

[0053] S27: Triangulate the newly added image and the target image to generate newly added pipeline scene 3D points in the pipeline 3D scene; and perform BA optimization on the camera extrinsic parameters of each currently selected image and the currently constructed pipeline scene 3D points to obtain the optimized 3D points and optimized camera extrinsic parameters corresponding to each currently selected image.

[0054] S28: Return to execute step S25 "Select the best picture as the current newly added picture from the currently unselected pictures according to the number of points in the picture that have a corresponding relationship with the three-dimensional points of the currently constructed pipeline scene and the distribution of the corresponding two-dimensional points in the corresponding picture", until the newly added picture cannot be selected.

[0055] S29: Construct the pipeline 3D reconstruction dataset using the initial image pair and all the newly added images; the pipeline 3D reconstruction dataset includes all the selected images, the real depth map corresponding to each of the selected images, the camera internal and external parameters, the depth information of each pixel and the best source view number; the selected images include the initial image pair and all the newly added images; the real depth map corresponding to each of the selected images is obtained using the depth camera; the best source view number corresponding to each of the selected images is the frame numbers of multiple images that are closest to the frame number of the selected image.

[0056] S3: Use public data sets to train the Fast-MVSNet network and the PatchMatchNet network respectively, and obtain multiple trained Fast-MVSNet network models and multiple trained PatchMatchNet network models; the public data sets include the DTU data set, the BlendedMVS data set, and the ETH3D data set.

[0057] Wherein, step S3 includes the training of Fast-MVSNet network and PatchMatchNet network respectively.

[0058] The same view selection strategy as MVSNet is used, and the number of source views is set to 5. The datasets used in the training process are all public datasets, including the DTU dataset, the BlendedMVS dataset, and the ETH3D dataset. In the DTU dataset, 79 groups are selected as training sets and 18 groups are selected as validation sets for training. The fixed input sample size is H×W×D=512×640×48, and the depth range is [425,935]. Among them, H, W, and D are the height, width, and number of depth planes of the dataset images, respectively. In the BlendedMVS dataset, it is divided into 106 groups of training scenes and 7 groups of validation scenes. Since the depth range in this dataset is not an absolute distance in millimeters, but belongs to some unknown units, it is difficult to evaluate the absolute accuracy. Therefore, while fixing the input sample size to H×W×D=576×768×48, the depth range is approximately between 100 and 1000, and other coefficients are corrected accordingly. The ETH3D dataset is divided into a validation set and a training set, and the images and depth maps are further cropped to H×W×D=448×896×48. Since the ETH3D dataset only contains three depth parameters, the maximum depth D is calculated by formula (5). MAX Calculate and calibrate camera parameters.

[0059] D MAX =D MIN +D INTERVAL ×(max d -1) (5)

[0060] In formula (5), D MIN , D INTERVAL 、max d They represent the minimum depth, depth interval, and number of depth planes respectively.

[0061] Figure 3 The flowchart based on the Fast-MVSNet algorithm provided by the present invention is an optional implementation method. MVSNet is used as the basic network, and a framework from sparse to dense, from coarse to fine is adopted, which is divided into predicting sparse but high-resolution depth maps, densifying depth maps, and optimizing depth maps. Figure 3 parts.

[0062] In the Fast-MVSNet network, four rounds of pre-training are performed on the two modules of predicting sparse depth maps and depth map propagation, and then end-to-end training is performed on the entire model until the loss function converges. ref And the corresponding 5 source images with the highest scores Use 8-layer 2D CNN to extract 32-channel image features F ref , In the spatial domain, a 1 / 8W×1 / 8H×D×F sparse cost volume is constructed based on the obtained feature map and the internal and external parameters of the camera. E and H are the width and height of the input image, respectively, and a small number of depth planes D=48 is used, and F is the number of feature channels. The sparse cost volume is regularized by a multi-scale 3D CNN, and the depth map is regressed through a differentiable argmax to obtain a sparse depth map P at a lower cost. s . Sparse depth map P s Fill the holes with the nearest neighbor strategy and use a 10-layer convolutional network from reference image J ref Extract features to predict the weight α of each pixel, and choose joint bilateral upsampling to achieve the densification of the initial sparse depth map P d , whose expression is:

[0063]

[0064] In formula (6), β d 、p i , α d They are respectively the normalization term and the pixel point p d Neighborhood pixel points, pixel point p d The weight of .

[0065] A network structure similar to that of deep graph propagation is used to extract reference graph J ref With source view The deep feature representation of the image is obtained by iterating the Gauss-Newton layer twice, that is, processing the depth map with a size of 1 / 4 and 1 / 2 of the original image, so as to improve the accuracy of the depth map and obtain the final depth map.

[0066] Therefore, the training of Fast-MVSNet network is as follows: use the public data set to train Fast-MVSNet network and obtain multiple trained Fast-MVSNet network models, including:

[0067] S311: Determine training sample data sets from the DTU data set, the BlendedMVS data set and the ETH3D data set respectively; the training sample data sets include the DTU sample data set, the BlendedMVS sample data set and the ETH3D sample data set.

[0068] S312: Use the DTU sample data set, the BlendedMVS sample data set and the ETH3D sample data set to train the Fast-MVSNet network respectively, and obtain three trained Fast-MVSNet network models.

[0069] Therefore, as an optional implementation, step S312 specifically includes:

[0070] S3121: Using a 2D convolutional network to extract image features from each reference image and all corresponding source images in each of the training sample data sets, and obtaining a first reference feature map corresponding to each reference image and a first source feature map corresponding to each of the source images.

[0071] S3122: For each of the first reference feature maps, a sparse cost volume is constructed according to the first reference feature map and each corresponding source feature map and the corresponding internal and external parameters of the camera, and the sparse cost volume is regularized by a 3D convolutional network. The result of the regularization is subjected to depth map regression by differentiable argmax to obtain an initial depth map corresponding to each of the reference images.

[0072] S3123: Apply the nearest neighbor strategy to each of the initial depth maps to fill the holes, and obtain a filled depth map.

[0073] S3124: Apply a joint bilateral upsampling strategy to each of the compensated depth maps and the corresponding reference image obtained after the first convolutional network, to obtain a dense depth map corresponding to each of the reference images.

[0074] S3125: Applying a Gauss-Newton layer to process the dense depth map, the corresponding camera internal and external parameters, the corresponding reference image, and the corresponding source image to obtain an optimized depth map corresponding to each reference image.

[0075] The Fast-MVSNet network is trained for multiple rounds of steps S311 to S3125 until the loss function converges, completing the training of the Fast-MVSNet network and obtaining three trained Fast-MVSNet network models.

[0076] As an optional implementation, step S3125 “applying a Gauss-Newton layer to process the dense depth map, the corresponding camera internal and external parameters, the corresponding reference image, and the corresponding source image to obtain an optimized depth map corresponding to each reference image” specifically includes:

[0077] (1) For each of the reference images, the reference image of 1 / 4 original size and the corresponding source image of 1 / 4 original size are passed through a second convolutional network to obtain a third reference feature map and a corresponding second source feature map.

[0078] (2) Using a Gauss-Newton layer, the third reference feature map, the corresponding second source feature map, the corresponding dense depth map, and the corresponding camera internal and external parameters are processed to obtain a first incremental result, and an iterative depth map is obtained according to the first incremental result and the dense depth map.

[0079] (3) Passing the reference image of 1 / 2 original size and the corresponding source image of 1 / 2 original size through the second convolutional network to obtain a fourth reference feature map and a corresponding third source feature map.

[0080] (4) Using the Gauss-Newton layer, the fourth reference feature map, the corresponding third source feature map, the first iterative depth map and the corresponding internal and external parameters of the camera are processed to obtain a second incremental result, and a second iterative depth map, i.e., the optimized depth map, is obtained according to the second incremental result and the first iterative depth map.

[0081] The training of the PatchMatchNet network is as follows: using a public data set to train the PatchMatchNet network, and multiple trained PatchMatchNet network models include:

[0082] S321: Use the DTU sample data set, the BlendedMVS sample data set and the ETH3D sample data set to train the PatchMatchNet network respectively, and obtain three trained PatchMatchNet network models.

[0083] Figure 4 The flowchart based on the PatchMatchNet algorithm provided for the present invention is an optional implementation mode, and the flowchart includes three parts: feature extraction, PatchMatch module, and depth map optimization.

[0084] The PatchMatchNet algorithm uses the feature pyramid to extract each reference image J ref Corresponding source image features to obtain feature maps F with 1 / 8, 1 / 4, and 1 / 2 original sizes ref , In the process from the third layer PatchMatch module to the first layer PatchMatch module, the depth map is refined step by step from low resolution to high resolution. The number of iterations of each layer of PatchMatch module is (1, 2, 2). min ,d max ] is a reference figure J ref Each pixel p in m is initialized pdepth hypothesis values, the scale of the depth interval and the number of depth samples in each layer are (0.005, 0.0125, 0.025) and (8, 8, 16) respectively. Based on the obtained feature map and the internal and external parameters of the camera, the position of the pixel point p in each source image is obtained, and the matching cost of each depth hypothesis value in the pixel point p is calculated. With the help of the idea of ​​adaptive propagation, adaptive spatial cost aggregation is performed, and the number of neighborhoods is (9, 9, 9). The aggregated spatial cost is obtained, and the softmax activation function is used to convert the cost body into a probability, and the initial depth map is obtained by weighted sum. The depth map is upsampled and locally perturbed as the initial depth value of the next layer, and adaptive depth propagation is only performed in the first and second layer modules. The expansion parameters and the number of neighborhoods used in the 2D convolution are (6, 4, 2) and (0, 8, 16) respectively, and finally a depth map P of size W / 2×H / 2 is obtained. I . The initial depth map P I Normalize, pre-scale to the range of [0:1], sample and refine through the deep residual network, and restore the depth map to the original depth range to obtain the final depth map

[0085] Therefore, as an optional implementation, step S321 specifically includes:

[0086] (a): For each reference image in each training sample data set and all the corresponding source images, image features are extracted using a feature pyramid to obtain a 1 / 8 original-size reference feature map, a 1 / 4 original-size reference feature map, a 1 / 2 original-size reference feature map corresponding to each reference image and a 1 / 8 original-size source feature map, a 1 / 4 original-size source feature map, a 1 / 2 original-size source feature map corresponding to each source image.

[0087] (b): For each of the reference images, a plurality of depth assumption values ​​are set for each pixel in each of the reference images within a preset depth range to obtain a depth initialization result for each reference image.

[0088] (c): For each of the reference images, a homography transformation is performed according to the 1 / 8 original-size reference feature map, the 1 / 8 original-size source feature map, the depth initialization result and the corresponding camera internal and external parameters to obtain a result of a first homography transformation.

[0089] (d): For each of the reference images, a matching cost calculation is performed on each depth hypothesis value of each pixel in the reference image according to the result of the first homography transformation, the 1 / 8 original-size reference feature map and the 1 / 8 original-size source feature map to obtain a result of the first matching cost calculation.

[0090] (e): Perform adaptive spatial cost aggregation according to the result of the first matching cost calculation to obtain the result after the first cost aggregation, and perform depth map regression on the result after the first cost aggregation to obtain a 1 / 8 original size depth map corresponding to each of the reference images.

[0091] (f): Upsampling the 1 / 8 original-size depth map of each of the reference images to obtain a first upsampled depth map, performing local perturbation on the first upsampled depth map to obtain a first local perturbation result, performing adaptive depth propagation on the first local perturbation result to obtain a first adaptive depth propagation result.

[0092] (g): Perform a homography transformation based on the result of the first adaptive depth propagation, the 1 / 4 original-size reference feature map, the 1 / 4 original-size source feature map and the corresponding camera internal and external parameters to obtain a result of a second homography transformation.

[0093] (h): Based on the result of the second homography transformation, the 1 / 4 original-size reference feature map and the 1 / 4 original-size source feature map, a matching cost calculation is performed on each depth hypothesis value of each pixel in the reference image to obtain a result of the second matching cost calculation.

[0094] (i): performing adaptive spatial cost aggregation according to the result of the second matching cost calculation to obtain a result after the second cost aggregation, and performing depth map regression on the result after the second cost aggregation to obtain a 1 / 4 original size depth map corresponding to each of the reference images;

[0095] (j): Replace the first upsampled depth map with the current 1 / 4 original-size depth map, and return to execute step (f) of "performing local perturbation on the first upsampled depth map to obtain a first local perturbation result, performing adaptive depth propagation on the first local perturbation result to obtain a first adaptive depth propagation result", until the number of returned iterations reaches the first preset number of iterations.

[0096] (k): Perform upsampling on the 1 / 4 original-size depth map obtained in the last iteration to obtain a second upsampled depth map, perform local perturbation on the second upsampled depth map to obtain a second local perturbation result, perform adaptive depth propagation on the second local perturbation result to obtain a second adaptive depth propagation result.

[0097] (l): Perform a homography transformation based on the result of the second adaptive depth propagation, the 1 / 2 original-size reference feature map, the 1 / 2 original-size source feature map and the corresponding camera internal and external parameters to obtain a result of a third homography transformation.

[0098] (m): Based on the result of the third homography transformation, the 1 / 2 original-size reference feature map and the 1 / 2 original-size source feature map, a matching cost calculation is performed on each depth hypothesis value of each pixel point in the reference image to obtain a result of the third matching cost calculation.

[0099] (n): Perform adaptive spatial cost aggregation according to the result of the third matching cost calculation to obtain the result after the third cost aggregation, and perform depth map regression on the result after the third cost aggregation to obtain a 1 / 2 original size depth map corresponding to each of the reference images.

[0100] (o): Replace the second upsampled depth map with the current 1 / 2 original-size depth map, and return to execute step (k) of "performing local perturbation on the second upsampled depth map to obtain a second local perturbation result, and performing adaptive depth propagation on the second local perturbation result to obtain a second adaptive depth propagation result", until the number of returned iterations reaches the second preset number of iterations.

[0101] (p): The 1 / 2 original-size depth map obtained in the last iteration is upsampled and refined through the deep residual network to obtain the final depth map.

[0102] The PatchMatchNet network is trained for multiple rounds of steps (a) to (p) until the loss function converges, completing the training of the PatchMatchNet network and obtaining three trained PatchMatchNet network models.

[0103] S4: Inputting the pipeline 3D reconstruction data set into all the trained Fast-MVSNet network models and all the trained PatchMatchNet network models respectively, and evaluating all the trained network models to obtain the optimal network model; the trained network models include all the trained Fast-MVSNet network models and all the trained PatchMatchNet network models.

[0104] The trained 3D reconstruction network model (trained network model) is used to predict the depth map and confidence map of the self-built pipeline test data set, and the depth map is projected into the 3D space to obtain the final point cloud. At the same time, the method of outlier filtering and depth map fusion is used to filter outliers in the depth map using geometric consistency and photometric consistency to suppress the noise generated during the 3D point cloud fusion process.

[0105] Among them, photometric consistency measures the accuracy of pixel depth prediction. Compared with the phenomenon that the probability distribution of correctly matched pixels is concentrated on a peak, the probability distribution of incorrect pixels is relatively scattered. Based on this, the confidence of each pixel is calculated by the probability and depth estimation of the four neighborhoods, and the confidence map is used to measure the quality of depth estimation. If the confidence of the pixel is high, the depth estimation of the pixel is considered to be more accurate. Therefore, with the help of the confidence map obtained by the network model, points with probability values ​​lower than the photometric consistency threshold are regarded as outliers.

[0106] At the same time, geometric consistency measures the depth continuity between multiple views. Using the camera internal and external parameter transformation, the reference image pixel n ref The depth value d ref Projection to the neighborhood view n i Then, n i The depth value d i Reproject to reference image n' ref At the depth d' ref If the following formulas (7) and (8) are satisfied, by controlling the geometric consistency threshold V, it is guaranteed that pixel n ref The depth prediction value d at ref At least V views are consistent. In addition, the depth value d ref and depth value d i The average value of the pixel n ref The final depth value.

[0107] |n' ref -n ref |<1 (7)

[0108] |d' ref -d ref |÷d ref <0.01 (8)

[0109] like Figure 5 The point cloud visualization effect diagram is shown. Different types of outliers can be filtered by adjusting the relevant threshold parameters. As the geometric consistency threshold η and the photometric consistency threshold V increase, the requirements for depth prediction values ​​become more and more stringent, the reconstruction noise is effectively suppressed, and the accuracy of the point cloud results in three-dimensional space is improved, but at the same time, its integrity is also reduced. In order to balance the relationship between point cloud integrity and accuracy, η = 0.55 and V = 3 are selected as the thresholds for photometric consistency and geometric consistency.

[0110] A self-built pipeline test dataset (i.e., pipeline 3D reconstruction dataset) is used to verify the effect of the overall framework, realize 3D point cloud reconstruction of pipeline videos, and conduct objective evaluation.

[0111] The pipeline 3D reconstruction dataset is used to verify the overall framework effect. The pipeline depth maps predicted by different network models are objectively evaluated. The objective quality evaluation criteria used are: the average absolute error and absolute relative error of each group of pipelines, as shown in Table 1; the reconstructed pipeline 3D point clouds are objectively evaluated. The objective quality evaluation criteria used are: the average number of 3D point clouds reconstructed by each group of pipelines, as shown in Table 2.

[0112] Table 1 Objective evaluation table of depth prediction accuracy

[0113]

[0114] Table 2 Objective evaluation table of 3D reconstruction point cloud quantity

[0115]

[0116] The mean absolute error (MAE) represents the average value of the absolute difference between the predicted depth map and the true depth map. The smaller the value, the smaller the average deviation between the predicted depth value and the true depth value, and the higher the accuracy of the depth prediction. The calculation formula is as follows:

[0117]

[0118] In formula (9), N is the number of samples in the test set, represents the predicted depth map of a certain frame of the pipeline, P n Represents the true depth map.

[0119] Absolute Relative Error (AbsRel) represents the average value of the ratio of the absolute difference between the predicted depth value and the true depth value to the true depth. It is a measure of the accuracy of the predicted value. Its calculation formula is as follows:

[0120]

[0121] The data in Tables 1 and 2 show that the best model is the network model trained by the PatchMatchNet network on the DTU dataset. In the process of fusing the pipeline depth prediction map and the three-dimensional point cloud, it has good integrity and accuracy, and can achieve the maximum restoration of the pipeline scene with a small video memory.

[0122] Therefore, as an optional implementation, step S4 specifically includes:

[0123] S41: Using the pipeline 3D reconstruction dataset as a test dataset, and using the test dataset to test each of the trained network models respectively, to obtain a predicted depth map and a confidence map for each pipeline scene output by each of the trained network models.

[0124] S42: For each of the confidence maps, compare the confidence value corresponding to each pixel in the confidence map with the confidence threshold to obtain a comparison result.

[0125] S43: Using the camera internal and external parameters, each pixel n in each reference image ref The depth prediction value d ref Projected to the corresponding pixel point n of each source image i At, then pixel n i The depth value d i Reproject to pixel n' of the reference image ref At pixel n' ref The corresponding depth value is d' ref .

[0126] S44: Determine each pixel n in each of the reference images ref The depth prediction value d ref Whether the pixel projection condition is satisfied on at least V source images, a judgment result is obtained; the pixel projection condition is: |n' ref -n ref |<1 and |d' ref -d ref |÷d ref <0.01.

[0127] S45: for each predicted depth map, filter out abnormal pixels in the predicted depth map according to the comparison result and the judgment result, and for the pixels that pass the comparison, set the depth value d ref and depth value d i The average value of the pixel n ref The final depth value.

[0128] S46: Projecting the filtered predicted depth map corresponding to each trained network model into a three-dimensional space to obtain a three-dimensional reconstructed point cloud corresponding to each trained network model.

[0129] S47: Determine the optimal network model according to the number of three-dimensional reconstructed point clouds corresponding to each of the trained network models, the average absolute error and / or the absolute relative error between the predicted depth map and the true depth map.

[0130] S5: Using the optimal network model to perform three-dimensional reconstruction and depth prediction on the pipeline three-dimensional reconstruction data set to be processed.

[0131] In summary, the present invention adopts Zhang Zhengyou calibration method, SfM and other technologies to obtain the internal and external parameters of the depth camera, predicts the pipeline depth map and confidence map based on the Fast-MVSNet and PatchMatchNet network structures, uses photometric consistency and geometric consistency to suppress the noise in the point cloud fusion reconstruction process, and establishes an overall framework for 3D reconstruction of monocular video to pipeline point cloud, which is suitable for 3D reconstruction and depth prediction of monocular video of drainage pipes.

[0132] Embodiment 2

[0133] This embodiment provides a pipeline monocular video 3D reconstruction and depth prediction system, including:

[0134] The video acquisition module is used to calibrate the depth camera using the Zhang Zhengyou calibration method to obtain the camera internal parameters, and use the pipeline robot equipped with the depth camera to collect monocular videos in different pipeline scenes;

[0135] The pipeline 3D reconstruction dataset construction module is used to apply the COLMAP method to perform feature extraction and matching and incremental reconstruction on each pipeline scene picture sequence composed of the monocular video, obtain the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, and construct the pipeline 3D reconstruction dataset according to the camera intrinsic parameters, the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, the pipeline scene picture sequence and the corresponding real depth map.

[0136] The network training module is used to train the Fast-MVSNet network and the PatchMatchNet network respectively using public data sets to obtain multiple trained Fast-MVSNet network models and multiple trained PatchMatchNet network models; the public data sets include the DTU data set, the BlendedMVS data set and the ETH3D data set.

[0137] A network testing and screening module is used to input the pipeline 3D reconstruction data set into all the trained Fast-MVSNet network models and all the trained PatchMatchNet network models respectively, and evaluate all the trained network models to obtain the optimal network model; the trained network models include all the trained Fast-MVSNet network models and all the trained PatchMatchNet network models.

[0138] The three-dimensional reconstruction and depth prediction module is used to perform three-dimensional reconstruction and depth prediction on the three-dimensional reconstruction data set of the processing pipeline using the optimal network model.

[0139] Embodiment 3

[0140] This embodiment provides an electronic device, including a memory and a processor, wherein the memory is used to store a computer program, and the processor runs the computer program to enable the electronic device to execute a pipeline monocular video three-dimensional reconstruction and depth prediction method of embodiment 1.

[0141] Optionally, the above-mentioned electronic device may be a server.

[0142] In addition, an embodiment of the present invention further provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements a pipeline monocular video three-dimensional reconstruction and depth prediction method of embodiment 1.

[0143] Embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0144] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0145] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0146] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0147] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0148] The principles and implementation methods of the present invention are described in this article using specific examples. The description of the above embodiments is only used to help understand the method and core idea of ​​the present invention. At the same time, for those skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present invention.

Claims

1. A pipeline monocular video 3D reconstruction and depth prediction method, characterized in that: include: The depth camera is calibrated using Zhang Zhengyou's calibration method to obtain the camera's internal parameters, and a pipeline robot equipped with the depth camera is used to collect monocular videos in different pipeline scenes; For each pipeline scene picture sequence composed of the monocular video, the COLMAP method is respectively applied to perform feature extraction and matching and incremental reconstruction to obtain the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, and a pipeline 3D reconstruction dataset is constructed according to the camera intrinsic parameters, the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, the pipeline scene picture sequence and the corresponding real depth map; Using public data sets to train the Fast-MVSNet network and the PatchMatchNet network respectively, a plurality of trained Fast-MVSNet network models and a plurality of trained PatchMatchNet network models are obtained; the public data sets include the DTU data set, the BlendedMVS data set and the ETH3D data set; Inputting the pipeline 3D reconstruction data set into all the trained Fast-MVSNet network models and all the trained PatchMatchNet network models respectively, and evaluating all the trained network models to obtain the optimal network model; the trained network models include all the trained Fast-MVSNet network models and all the trained PatchMatchNet network models; Using the optimal network model to perform three-dimensional reconstruction and depth prediction on the pipeline three-dimensional reconstruction data set to be processed; Among them, for each pipeline scene picture sequence composed of the monocular video, the COLMAP method is respectively applied to perform feature extraction and matching and incremental reconstruction to obtain the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, and the pipeline 3D reconstruction data set is constructed according to the camera intrinsic parameters, the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, the pipeline scene picture sequence and the corresponding real depth map, specifically including: Perform SIFT feature extraction and feature matching on each pipeline scene image sequence, and use epipolar geometry to eliminate erroneous matching point pairs in each matching image pair; For each pipeline scene picture sequence, all pictures are sorted according to the number of matching points, and the picture with the largest number of matching points is taken as the starting point, and two pictures whose number of matching point pairs between the pictures is greater than a first preset value and whose translation vectors between the pictures are greater than a second preset value are selected as the initial picture pair; and the camera extrinsic parameter of the first picture in the initial picture pair is set to the unit matrix; Calculating the camera extrinsic parameter of the second image in the initial image pair according to the 2D-2D matching relationship of the initial image pair and the unit matrix using the epipolar geometry relationship; the 2D-2D matching relationship refers to the SIFT feature matching relationship; Performing triangulation processing on the initial image pair to generate initial pipeline scene three-dimensional points in the pipeline three-dimensional scene; In the currently unselected pictures, the best picture is selected as the current newly added picture according to the number of points in the picture that have a corresponding relationship with the currently constructed three-dimensional points of the pipeline scene and the distribution of the corresponding two-dimensional points in the picture to which they belong; the corresponding two-dimensional points refer to the two-dimensional points in the picture that have a corresponding relationship with the currently constructed three-dimensional points of the pipeline scene; According to the 2D-2D matching relationship between the newly added picture and each of the currently selected pictures, a 2D-3D matching relationship between the newly added picture and each target picture is obtained, and the camera extrinsic parameters of the newly added picture are calculated by using the RANSAC-PNP method according to the 2D-3D matching relationship and the camera intrinsic parameters; the target picture is a picture among the currently selected pictures that has a 2D-2D matching relationship with the newly added picture; Triangulate the newly added picture and the target picture to generate newly added pipeline scene 3D points in the pipeline 3D scene; and perform BA optimization on the camera extrinsics of each currently selected picture and the currently constructed pipeline scene 3D points to obtain optimized 3D points and optimized camera extrinsics corresponding to each currently selected picture; Return to the step of "selecting the best picture as the current newly added picture from the currently unselected pictures according to the number of points in the picture that have a corresponding relationship with the currently constructed three-dimensional points of the pipeline scene and the distribution of the corresponding two-dimensional points in the picture to which they belong", until the newly added picture cannot be selected; The pipeline 3D reconstruction dataset is constructed using the initial image pair and all the newly added images; the pipeline 3D reconstruction dataset includes all the selected images, the real depth map corresponding to each of the selected images, the camera internal and external parameters, the depth information of each pixel and the best source view number; the selected images include the initial image pair and all the newly added images; the real depth map corresponding to each of the selected images is obtained using the depth camera; the best source view number corresponding to each of the selected images is the frame number of multiple images that are closest to the frame number of the selected image.

2. The method for 3D reconstruction and depth prediction of pipeline monocular video according to claim 1, characterized in that: The Fast-MVSNet network is trained using public data sets, and multiple trained Fast-MVSNet network models are obtained, including: Determine a training sample data set from the DTU data set, the BlendedMVS data set and the ETH3D data set respectively; the training sample data set includes a DTU sample data set, a BlendedMVS sample data set and an ETH3D sample data set; The Fast-MVSNet network is trained respectively using the DTU sample data set, the BlendedMVS sample data set, and the ETH3D sample data set to obtain three trained Fast-MVSNet network models.

3. The pipeline monocular video 3D reconstruction and depth prediction method according to claim 2, characterized in that: The Fast-MVSNet network is trained using the DTU sample data set, the BlendedMVS sample data set, and the ETH3D sample data set, respectively, to obtain three trained Fast-MVSNet network models, specifically including: Extracting image features from each reference image and all corresponding source images in each of the training sample data sets using a 2D convolutional network to obtain a first reference feature map corresponding to each of the reference images and a first source feature map corresponding to each of the source images; For each of the first reference feature maps, a sparse cost volume is constructed according to the first reference feature map and each of the corresponding source feature maps and the corresponding camera internal and external parameters, and the sparse cost volume is regularized by a 3D convolutional network, and the result of the regularization is regressed by a differentiable argmax to obtain an initial depth map corresponding to each of the reference images; Applying a nearest neighbor strategy to fill holes on each of the initial depth maps to obtain a filled depth map; Applying a joint bilateral upsampling strategy to each of the compensated depth maps and the corresponding second reference feature map obtained after the first convolutional network of the reference image, so as to obtain a dense depth map corresponding to each of the reference images; Applying a Gauss-Newton layer to the dense depth map, the corresponding camera internal and external parameters, the corresponding reference image and the corresponding source image to obtain an optimized depth map corresponding to each reference image; When the loss function of the Fast-MVSNet network converges, the training of the Fast-MVSNet network is completed, and three trained Fast-MVSNet network models are obtained.

4. The method for 3D reconstruction and depth prediction of pipeline monocular video according to claim 3, characterized in that: Applying a Gauss-Newton layer to process the dense depth map, the corresponding camera internal and external parameters, the corresponding reference image, and the corresponding source image to obtain an optimized depth map corresponding to each reference image specifically includes: For each of the reference images, the reference image of 1 / 4 original size and the corresponding source image of 1 / 4 original size are subjected to a second convolutional network to obtain a third reference feature map and a corresponding second source feature map; Using a Gauss-Newton layer, the third reference feature map, the corresponding second source feature map, the corresponding dense depth map, and the corresponding camera internal and external parameters are processed to obtain a first incremental result, and an iterative depth map is obtained according to the first incremental result and the dense depth map; Passing the reference image of 1 / 2 original size and the corresponding source image of 1 / 2 original size through the second convolutional network to obtain a fourth reference feature map and a corresponding third source feature map; The fourth reference feature map, the corresponding third source feature map, the first iterative depth map and the corresponding camera internal and external parameters are processed using the Gauss-Newton layer to obtain a second incremental result, and a second iterative depth map, i.e., the optimized depth map, is obtained according to the second incremental result and the first iterative depth map.

5. The method for 3D reconstruction and depth prediction of pipeline monocular video according to claim 4, characterized in that: The PatchMatchNet network is trained using public datasets. Multiple trained PatchMatchNet network models include: The PatchMatchNet network is trained respectively using the DTU sample data set, the BlendedMVS sample data set, and the ETH3D sample data set to obtain three trained PatchMatchNet network models.

6. The pipeline monocular video 3D reconstruction and depth prediction method according to claim 5, characterized in that: The PatchMatchNet network is trained using the DTU sample data set, the BlendedMVS sample data set, and the ETH3D sample data set, respectively, to obtain three trained PatchMatchNet network models, specifically including: Extract image features from each reference image and all corresponding source images in each training sample data set using a feature pyramid to obtain a 1 / 8 original-size reference feature map, a 1 / 4 original-size reference feature map, and a 1 / 2 original-size reference feature map corresponding to each reference image and a 1 / 8 original-size source feature map, a 1 / 4 original-size source feature map, and a 1 / 2 original-size source feature map corresponding to each source image; For each of the reference images, a plurality of depth assumption values ​​are set for each pixel in each of the reference images within a preset depth range to obtain a depth initialization result for each reference image; For each of the reference images, a homography transformation is performed according to the 1 / 8 original-size reference feature map, the 1 / 8 original-size source feature map, the depth initialization result, and the corresponding camera internal and external parameters to obtain a result of a first homography transformation; For each of the reference images, performing matching cost calculation on each of the depth hypothesis values ​​of each pixel in the reference image according to the result of the first homography transformation, the 1 / 8 original-size reference feature map, and the 1 / 8 original-size source feature map, to obtain a result of the first matching cost calculation; Performing adaptive spatial cost aggregation according to the result of the first matching cost calculation to obtain a result after first cost aggregation, and performing depth map regression on the result after the first cost aggregation to obtain a 1 / 8 original size depth map corresponding to each of the reference images; Performing upsampling on the 1 / 8 original-size depth map of each of the reference images to obtain a first upsampled depth map, performing local perturbation on the first upsampled depth map to obtain a first local perturbation result, and performing adaptive depth propagation on the first local perturbation result to obtain a first adaptive depth propagation result; Performing a homography transformation according to the result of the first adaptive depth propagation, the 1 / 4 original-size reference feature map, the 1 / 4 original-size source feature map, and the corresponding camera internal and external parameters to obtain a result of a second homography transformation; According to the result of the second homography transformation, the 1 / 4 original-size reference feature map and the 1 / 4 original-size source feature map, performing matching cost calculation on each of the depth hypothesis values ​​of each pixel in the reference image to obtain a result of the second matching cost calculation; Performing adaptive spatial cost aggregation according to the result of the second matching cost calculation to obtain a result after the second cost aggregation, and performing depth map regression on the result after the second cost aggregation to obtain a 1 / 4 original size depth map corresponding to each of the reference images; Replace the first upsampled depth map with the current 1 / 4 original-size depth map, and return to the step of "performing local perturbation on the first upsampled depth map to obtain a first local perturbation result, and performing adaptive depth propagation on the first local perturbation result to obtain a first adaptive depth propagation result", until the number of returned iterations reaches a first preset number of iterations; Perform upsampling on the 1 / 4 original-size depth map obtained by the last iteration to obtain a second upsampled depth map, perform local perturbation on the second upsampled depth map to obtain a second locally perturbed result, perform adaptive depth propagation on the second locally perturbed result to obtain a second adaptive depth propagation result; Performing a homography transformation according to the result of the second adaptive depth propagation, the 1 / 2 original-size reference feature map, the 1 / 2 original-size source feature map, and the corresponding camera internal and external parameters to obtain a result of a third homography transformation; According to the result of the third homography transformation, the 1 / 2 original-size reference feature map and the 1 / 2 original-size source feature map, performing matching cost calculation on each of the depth hypothesis values ​​of each pixel in the reference image to obtain a result of the third matching cost calculation; Performing adaptive spatial cost aggregation according to the result of the third matching cost calculation to obtain a result after the third cost aggregation, and performing depth map regression on the result after the third cost aggregation to obtain a 1 / 2 original size depth map corresponding to each of the reference images; Replace the second upsampled depth map with the current 1 / 2 original-size depth map, and return to the step of "performing local perturbation on the second upsampled depth map to obtain a second local perturbation result, and performing adaptive depth propagation on the second local perturbation result to obtain a second adaptive depth propagation result", until the number of returned iterations reaches a second preset number of iterations; The 1 / 2 original size depth map obtained from the last iteration is passed through the deep residual network to obtain the final depth map; When the loss function of the PatchMatchNet network converges, the training of the PatchMatchNet network is completed, and three trained PatchMatchNet network models are obtained.

7. The pipeline monocular video 3D reconstruction and depth prediction method according to claim 3, characterized in that: The pipeline 3D reconstruction dataset is input into all the trained Fast-MVSNet network models and all the trained PatchMatchNet network models respectively, and all the trained network models are evaluated to obtain the optimal network model, specifically including: The pipeline 3D reconstruction dataset is used as a test dataset, and each of the trained network models is tested using the test dataset to obtain a predicted depth map and a confidence map in each pipeline scene output by each of the trained network models; For each of the confidence maps, comparing the confidence value corresponding to each pixel in the confidence map with the confidence threshold to obtain a comparison result; Using the camera internal and external parameters, each pixel n in each reference image is ref The depth prediction value d ref Projected to the corresponding pixel point n of each source image i At the pixel n i The depth value d i Reproject to the corresponding pixel n' of the reference image ref At pixel n' ref The corresponding depth value is d' ref ; Determine each pixel n in each of the reference images ref The depth prediction value d ref Whether the pixel projection condition is satisfied on at least v source images, a judgment result is obtained; the pixel projection condition is: |n' ref -n ref |<1 and |d' ref -d ref |÷d ref <0.01; For each of the predicted depth maps, filtering abnormal pixels in the predicted depth map according to the comparison result and the judgment result; Respectively projecting the filtered predicted depth map corresponding to each of the trained network models into a three-dimensional space to obtain a three-dimensional reconstructed point cloud corresponding to each of the trained network models; The optimal network model is determined according to the number of three-dimensional reconstructed point clouds corresponding to each of the trained network models, the average absolute error and / or the absolute relative error between the predicted depth map and the true depth map.

8. A pipeline monocular video 3D reconstruction and depth prediction system, characterized in that: include: The video acquisition module is used to calibrate the depth camera using the Zhang Zhengyou calibration method to obtain the camera internal parameters, and use the pipeline robot equipped with the depth camera to collect monocular videos in different pipeline scenes; A pipeline 3D reconstruction dataset construction module is used to apply the COLMAP method to perform feature extraction and matching and incremental reconstruction on each pipeline scene picture sequence composed of the monocular video, obtain the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, and construct a pipeline 3D reconstruction dataset according to the camera intrinsic parameters, the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, the pipeline scene picture sequence and the corresponding real depth map; Among them, for each pipeline scene picture sequence composed of the monocular video, the COLMAP method is respectively applied to perform feature extraction and matching and incremental reconstruction to obtain the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, and the pipeline 3D reconstruction data set is constructed according to the camera intrinsic parameters, the camera extrinsic parameters corresponding to the pictures in each pipeline scene picture sequence, the pipeline scene picture sequence and the corresponding real depth map, specifically including: Perform SIFT feature extraction and feature matching on each pipeline scene image sequence, and use epipolar geometry to eliminate erroneous matching point pairs in each matching image pair; For each pipeline scene picture sequence, all pictures are sorted according to the number of matching points, and the picture with the largest number of matching points is taken as the starting point, and two pictures whose number of matching point pairs between the pictures is greater than a first preset value and whose translation vectors between the pictures are greater than a second preset value are selected as the initial picture pair; and the camera extrinsic parameter of the first picture in the initial picture pair is set to the unit matrix; Calculating the camera extrinsic parameter of the second image in the initial image pair according to the 2D-2D matching relationship of the initial image pair and the unit matrix using the epipolar geometry relationship; the 2D-2D matching relationship refers to the SIFT feature matching relationship; Performing triangulation processing on the initial image pair to generate initial pipeline scene three-dimensional points in the pipeline three-dimensional scene; In the currently unselected pictures, the best picture is selected as the current newly added picture according to the number of points in the picture that have a corresponding relationship with the currently constructed three-dimensional points of the pipeline scene and the distribution of the corresponding two-dimensional points in the picture to which they belong; the corresponding two-dimensional points refer to the two-dimensional points in the picture that have a corresponding relationship with the currently constructed three-dimensional points of the pipeline scene; According to the 2D-2D matching relationship between the newly added picture and each of the currently selected pictures, a 2D-3D matching relationship between the newly added picture and each target picture is obtained, and the camera extrinsic parameters of the newly added picture are calculated by using the RANSAC-PNP method according to the 2D-3D matching relationship and the camera intrinsic parameters; the target picture is a picture among the currently selected pictures that has a 2D-2D matching relationship with the newly added picture; Triangulate the newly added picture and the target picture to generate newly added pipeline scene 3D points in the pipeline 3D scene; and perform BA optimization on the camera extrinsics of each currently selected picture and the currently constructed pipeline scene 3D points to obtain optimized 3D points and optimized camera extrinsics corresponding to each currently selected picture; Return to the step of "selecting the best picture as the current newly added picture from the currently unselected pictures according to the number of points in the picture that have a corresponding relationship with the currently constructed three-dimensional points of the pipeline scene and the distribution of the corresponding two-dimensional points in the picture to which they belong", until the newly added picture cannot be selected; The pipeline 3D reconstruction dataset is constructed using the initial image pair and all the newly added images; the pipeline 3D reconstruction dataset includes all the selected images, the real depth map corresponding to each of the selected images, the camera internal and external parameters, the depth information of each pixel and the best source view number; the selected images include the initial image pair and all the newly added images; the real depth map corresponding to each of the selected images is obtained by using the depth camera; the best source view number corresponding to each of the selected images is the frame number of multiple images that are closest to the frame number of the selected image; A network training module, used to train the Fast-MVSNet network and the PatchMatchNet network respectively using public data sets to obtain multiple trained Fast-MVSNet network models and multiple trained PatchMatchNet network models; the public data sets include the DTU data set, the BlendedMVS data set, and the ETH3D data set; A network testing and screening module, used for inputting the pipeline 3D reconstruction data set into all the trained Fast-MVSNet network models and all the trained PatchMatchNet network models respectively, and evaluating all the trained network models to obtain the optimal network model; the trained network models include all the trained Fast-MVSNet network models and all the trained PatchMatchNet network models; The three-dimensional reconstruction and depth prediction module is used to perform three-dimensional reconstruction and depth prediction on the three-dimensional reconstruction data set of the processing pipeline using the optimal network model.

Images