Deep-sea mining non-metal pipeline monocular image three-dimensional sparse reconstruction method and device

Abstract

The invention relates to a deep-sea mining nonmetal pipeline monocular image three-dimensional sparse reconstruction method and device. The method comprises the following steps that a robot carrying a monocular camera is used for obtaining internal pictures of a to-be-detected nonmetal pipeline to form a data set; performing feature point detection and matching on the data set by using a feature point matching network model based on deep learning; carrying out aggregation and redistribution on the obtained feature points; and based on the aggregated feature points, performing three-dimensional sparse reconstruction by using an improved incremental reconstruction algorithm to obtain internal structure information of the non-metal pipeline to be detected. The method can be widely applied to the technical field of nonmetal pipeline detection.

Description

Technical Field

[0001] The invention relates to a method and device for three-dimensional sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining, belonging to the technical field of non-metallic pipeline detection. Background Art

[0002] As the "artery" of the deep-sea mining system, the non-metallic flexible mixed pipeline is the most critical transportation equipment in the hydraulic hose lifting mining system. It can not only effectively prevent the corrosion problem that metal pipelines are susceptible to in deep-sea environments, but also has the advantages of easy installation, low maintenance cost and long service life. Although non-metallic pipelines have many advantages, some problems are gradually exposed during use, such as leakage and breakage. Therefore, it is very important to continuously monitor the status and evaluate the performance of these long-term non-metallic pipelines.

[0003] In recent years, with the rapid development of computer vision technology, researchers are increasingly using 3D reconstruction technology for pipeline modeling and inspection. This method can reconstruct a complete pipeline model by simply taking images of the inner wall of the pipeline through visual equipment, overcoming the problems of high cost, poor versatility and flexibility of dedicated sensors in traditional non-destructive testing technology, and providing a new way for pipeline defect assessment. According to the number of cameras, 3D reconstruction technology can be divided into three types: monocular, binocular and multi-camera. Monocular reconstruction refers to using a camera to take multiple photos to build a 3D model, establishing the relationship between features and spatial structure by analyzing the feature changes in the image sequence, and reconstructing the 3D shape of the object. Compared with binocular or multi-camera, monocular reconstruction is cheaper, but the reconstruction process is more complicated.

[0004] Current research focuses on the reconstruction of metal pipes. Since deep-sea non-metal pipes generally lack texture and are in a dark or low-light environment, the details of the video images captured by the monocular camera are blurred. Therefore, the technology of using a monocular camera to reconstruct the inner surface of non-metal pipes has not been widely studied. Existing technologies have poor results when applied to deep-sea non-metal pipes, and even cannot be reconstructed. Summary of the invention

[0005] In view of the above problems, the purpose of the present invention is to provide a method and device for three-dimensional sparse reconstruction of monocular images of deep-sea mining non-metallic pipelines, which can realize three-dimensional sparse reconstruction of deep-sea non-metallic pipelines and intuitively display the structural information inside the pipelines.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining, comprising the following steps:

[0008] A robot equipped with a monocular camera is used to obtain internal images of non-metallic pipes to be inspected to form a data set;

[0009] Use a deep learning-based feature point matching network model to detect and match feature points in the dataset;

[0010] Aggregate and redistribute the obtained feature points;

[0011] Based on the aggregated feature points, the improved incremental reconstruction algorithm is used to perform three-dimensional sparse reconstruction to obtain the internal structure information of the non-metallic pipe to be inspected.

[0012] Furthermore, the data set is formed by obtaining internal images of the non-metallic pipe to be inspected by using a robot equipped with a monocular camera, including:

[0013] Determine the robot structure;

[0014] The robot comprises a camera protection cover, a camera compartment, a plurality of odometer wheels, a base and support wheels; the lower part of the base is fixedly connected to a bottom plate through a flange, and three support wheels are arranged at the lower part of the base plate to ensure that the center line of the robot is in line with the center line of the pipeline; the upper part of the base is connected to the camera compartment through another flange, and a spring is arranged between the camera compartment and the other flange; the camera protection cover is arranged on the top of the camera compartment to protect the monocular camera placed in the camera compartment; the odometer wheels are arranged outside the camera compartment at intervals, and the odometer wheels have a built-in magnetic encoder to measure the rotation angle of the odometer wheels; a circuit system is placed inside the base to receive the pulse signal sent by the magnetic encoder and calculate the robot displacement data according to the odometer wheel parameters, and control the monocular camera to take pictures of the inside of the pipeline according to the displacement data;

[0015] A monocular camera is mounted on the robot, and the monocular camera is calibrated using Zhang's calibration method to obtain the camera's intrinsic parameters;

[0016] The robot is placed inside the non-metallic pipe to be inspected and controlled to move forward at a preset speed. At the same time, a monocular camera is used to take pictures of the inside of the non-metallic pipe to be inspected to form a data set.

[0017] Furthermore, the robot is equipped with a monocular camera, and the monocular camera is calibrated using Zhang's calibration method to obtain the camera's internal parameters, including:

[0018] Fix the preset black and white checkerboard calibration paper on the table;

[0019] Use the monocular camera on the robot to take multiple images of the calibration paper from different angles and positions, ensuring that the checkerboard pattern in each image is in the field of view. The posture and position of the monocular camera should be varied enough to cover different viewing angles.

[0020] Based on the image information of the calibration paper, Zhang’s calibration method is used to calculate the intrinsic parameters of the monocular camera.

[0021] Furthermore, the robot is placed inside the non-metallic pipe to be inspected, and the robot is controlled to move forward at a preset speed, and a monocular camera is used to take pictures of the inside of the non-metallic pipe to be inspected to form a data set, including:

[0022] Place the robot equipped with a monocular camera at the entrance of the non-metallic pipe to be inspected and clear the data of the odometer wheel;

[0023] The robot is controlled to move forward in the non-metallic pipe to be inspected. At the same time, the circuit system in the base obtains the displacement information of the robot in the non-metallic pipe to be inspected through the odometer, and controls the monocular camera to take pictures of the inside of the non-metallic pipe to be inspected according to the displacement information, and saves the pictures;

[0024] The robot stops after reaching the end point, uses the host computer to read all the pictures, and creates a data set of pictures of the inside of the non-metallic pipe to be inspected in the order in which the pictures were taken.

[0025] Furthermore, the feature point detection and matching of the data set using the feature point matching network model based on deep learning includes:

[0026] Use the feature pyramid network to extract features of different scales from the input image to obtain feature maps of 1 / 2 scale and 1 / 8 scale;

[0027] Based on the 1 / 8 scale feature map, a coarse-grained matching model is used to obtain a rough matching area;

[0028] Based on the 1 / 2 scale feature map and the rough matching area, a fine-grained matching model is used to obtain the precise position of the matching feature points.

[0029] Furthermore, the feature pyramid network is used to extract features of different scales from the input image to obtain a feature map of 1 / 2 scale and a feature map of 1 / 8 scale, including:

[0030] Perform feature extraction on the two input images, obtain downsampled feature maps of 1 / 2 scale and 1 / 4 scale respectively, and output feature maps of 1 / 8 scale;

[0031] The bilinear interpolation method is used to upsample the downsampled feature maps of different scales obtained in the feature extraction stage, and upsampled feature maps of 1 / 4 scale and 1 / 2 scale are obtained in turn;

[0032] The 1 / 2 scale up-sampled feature map is fused with the 1 / 2 scale down-sampled feature map to obtain the final 1 / 2 scale feature map.

[0033] Further, the feature map based on the 1 / 8 scale is processed using a coarse-grained matching model to obtain a rough matching area, including the following steps:

[0034] Perform dimension transformation on the coarse-level features of the two images at a scale of 1 / 8 to obtain the score matrix S;

[0035] Apply the softmax operator to the two dimensions of the score matrix respectively, convert the score matrix into probability, and obtain the confidence matrix P c ;

[0036] Based on the confidence matrix P c , for confidence values ​​below the threshold θ c The matches are filtered out, and the remaining matches are subjected to the mutual neighbor standard to obtain a rough matching area

[0037] Furthermore, the method of using a fine-grained matching model to process the 1 / 2 scale feature map and the rough matching area to obtain the precise position of the matching feature point includes the following steps:

[0038] Will roughly match Positioning on fine-level feature maps and In the local neighborhood

[0039] Crop two sets of local features of size w×w and For local neighborhood Perform Feature-Transformer transformation to obtain two local feature maps and

[0040] Use similarity calculation and softmax function to get the heat map of the matching probability between two feature maps;

[0041] The DSNT algorithm is used to calculate the exact location of the matching points from the heat map.

[0042] Furthermore, the three-dimensional sparse reconstruction is performed using the improved incremental reconstruction algorithm to obtain the internal structure information of the non-metallic pipe to be detected, including the following steps:

[0043] ① Select two pictures, use the PnP algorithm to obtain the initial camera pose, triangulate the image according to the initial camera pose to obtain 3D points as the initial point cloud model and use local BA optimization;

[0044] ② Select the image with the most matching points in the point cloud model, use the PnP algorithm to obtain the camera pose of the new image, and triangulate it according to the camera pose to obtain the new 3D point;

[0045] ③ Use the improved RANSAC algorithm to detect whether the point cloud model is cylindrical. If a cylinder is detected, a cylindrical constraint item is added to the local RA to optimize the 3D point position.

[0046] ④ Repeat steps ② to ③ until the feature points of all images are added to the point cloud model.

[0047] In a second aspect, the present invention provides a device for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining, comprising:

[0048] A data acquisition module is used to use a robot equipped with a monocular camera to acquire internal images of the non-metallic pipe to be inspected to form a data set;

[0049] The feature point matching module is used to detect and match feature points of the data set using a feature point matching network model based on deep learning;

[0050] Aggregation and redistribution module, used to aggregate and redistribute the obtained feature points;

[0051] The three-dimensional coefficient reconstruction module is used to perform three-dimensional sparse reconstruction using an improved incremental reconstruction algorithm to obtain the internal structure information of the non-metallic pipe to be inspected.

[0052] The present invention adopts the above technical solution, which has the following advantages:

[0053] 1. The present invention first uses a feature point matching model based on deep learning to extract and match the feature points of non-metallic pipes. Subsequently, an improved incremental reconstruction technology is used to generate a three-dimensional sparse point cloud to accurately present the geometric shape and structural information of deep-sea non-metallic flexible pipes. The present invention helps to monitor the deformation, damage and wear of flexible pipes in complex deep-sea environments, thereby providing timely warning and maintenance to prevent pipeline failure. Through the three-dimensional reconstruction of non-metallic flexible pipes, environmental pollution or mining accidents caused by pipeline damage or design defects can be effectively avoided, thereby improving the safety of deep-sea mining.

[0054] 2. The robot of the present invention only needs to be equipped with a monocular camera, and the subsequent reconstruction process is mainly implemented by the algorithm, without the need to use expensive dedicated sensors.

[0055] 3. The present invention designs a feature point matching model based on deep learning. The model proposes a feature point extraction and matching network based on deep learning with reference to human behavior patterns, including two matching stages from coarse to fine and a heavy parameter network design for expanding the receptive field, which solves the problem of difficulty in feature matching of textureless images.

[0056] 4. The present invention proposes an improved incremental reconstruction algorithm, which can reduce the radius error rate and the distortion of 3D points by improving the RANSAC algorithm and adding a circular constraint term to BA optimization.

[0057] Therefore, the present invention can be widely applied to the field of non-metallic pipeline detection technology, especially the field of deep-sea mining technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Various other advantages and benefits will become apparent to those of ordinary skill in the art by reading the detailed description of the preferred embodiments below. The accompanying drawings are only for the purpose of illustrating the preferred embodiments and are not to be considered as limiting the present invention. Throughout the accompanying drawings, the same reference numerals are used to represent the same components. In the accompanying drawings:

[0059] Figure 1 This is a flow chart of a method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining provided by an embodiment of the present invention;

[0060] Figure 2 is a structural diagram of a robot provided by an embodiment of the present invention;

[0061] Figure 3 is a flow chart of a feature point matching model based on deep learning provided by an embodiment of the present invention;

[0062] Figure 4 is a flow chart of a feature pyramid network provided by an embodiment of the present invention;

[0063] Figure 5 is a structural diagram of an encoder of a Feature-Transformer module provided in an embodiment of the present invention;

[0064] Figure 6 is a flow chart of an improved incremental reconstruction algorithm provided by an embodiment of the present invention;

[0065] Figure 7 is a flow chart of an improved RANSAC algorithm provided by an embodiment of the present invention;

[0066] Figure 8 This is a cone point defect reconstruction effect diagram provided by an embodiment of the present invention;

[0067] Fig. 9 This is a diagram showing the effect of slot defect reconstruction provided by an embodiment of the present invention;

[0068] Fig.10 This is a rendering of a long defect reconstruction provided by an embodiment of the present invention;

[0069] The reference numerals in the figures are as follows:

[0070] 1. Camera protection cover; 2. Camera compartment; 3. Mileage wheel; 4. Base; 5. Support wheel. DETAILED DESCRIPTION

[0071] In order to make the purpose, technical solution and advantages of the embodiment of the present invention clearer, the technical solution of the embodiment of the present invention will be clearly and completely described below in conjunction with the drawings of the embodiment of the present invention. Obviously, the described embodiment is a part of the embodiment of the present invention, not all of the embodiments. Based on the described embodiment of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.

[0072] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0073] In some embodiments of the present invention, a method for 3D sparse reconstruction of monocular images of non-metallic pipelines for deep-sea mining is provided. The method solves the problem of difficulty in feature matching of textureless images based on feature point extraction and matching network models through deep learning. At the same time, the RANSAC algorithm and BA algorithm in the incremental reconstruction method are improved, which can effectively realize 3D sparse reconstruction of deep-sea non-metallic flexible pipes and intuitively display the structural information inside the pipelines.

[0074] Correspondingly, in some other embodiments of the present invention, a device for 3D sparse reconstruction of monocular images of non-metallic pipelines for deep-sea mining is provided.

[0075] Example 1

[0076] like Figure 1 As shown, this embodiment provides a method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining, which includes the following steps:

[0077] S1. Using a robot equipped with a monocular camera to obtain internal images of non-metallic pipes to be inspected to form a data set;

[0078] S2, use the deep learning-based feature point matching network model to detect and match feature points of the data set;

[0079] S3, aggregate and redistribute the obtained feature points;

[0080] S4. Based on the aggregated feature points, the improved incremental reconstruction algorithm is used to perform three-dimensional sparse reconstruction to obtain the internal structure information of the non-metallic pipe to be inspected.

[0081] Furthermore, in the above step S1, a robot equipped with a monocular camera is used to obtain internal images of the non-metallic pipe to be inspected to form a data set, which includes the following steps:

[0082] S1.1. Determine the robot structure;

[0083] S1.2, mount a monocular camera on the robot, and calibrate the monocular camera using Zhang’s calibration method to obtain the camera’s internal parameters;

[0084] S1.3. Place the robot inside the non-metallic pipe to be inspected and control the robot to move forward at a preset speed. At the same time, use a monocular camera to take pictures of the inside of the non-metallic pipe to be inspected to form a data set.

[0085] Furthermore, in the above step S1.1, if Figure 2 As shown, the robot used in this embodiment includes: a camera protection cover 1, a camera compartment 2, a mileage wheel 3, a base 4 and a support wheel 5. Among them, the lower part of the base 4 is fixedly connected to a bottom plate through a flange, and three support wheels 5 are arranged at the lower part of the bottom plate to ensure that the center line of the robot is in line with the center line of the pipeline, and no data will be measured; the upper part of the base 4 is connected to the camera compartment 2 through another flange, and a spring is arranged between the camera compartment 2 and the flange; the camera protection cover 1 is arranged on the top of the camera compartment 2 to protect the monocular camera placed in the camera compartment 2; the mileage wheel 3 is arranged outside the camera compartment 2, and the built-in magnetic encoder is used to measure the rotation angle of the mileage wheel 3; the inside of the base 4 is placed with a circuit system with stm32f4 as the core, which is used to receive the pulse signal sent by the magnetic encoder and calculate the robot displacement data according to the mileage wheel parameters, and then control the monocular camera to take pictures of the inside of the pipeline according to the displacement data.

[0086] Furthermore, in the above step S1.2, the monocular camera is calibrated using Zhang's calibration method to obtain the camera intrinsic parameters, including the following steps:

[0087] S1.2.1. Fix the preset black and white checkerboard calibration paper on the table;

[0088] S1.2.2. Use the monocular camera on the robot to take multiple images of the calibration paper from different angles and positions, ensuring that the checkerboard pattern in each image is in the field of view. The posture and position of the monocular camera should vary enough during shooting to cover different viewing angles.

[0089] S1.2.3. Based on the image information of the calibration paper, use Zhang’s calibration method to calculate the intrinsic parameters of the monocular camera.

[0090] In this embodiment, the internal parameters of the monocular camera include the internal parameter matrix M1 and the distortion coefficient β, which are respectively expressed as:

[0091]

[0092] β=(k1,k2,k3,p1,p2) (2)

[0093] In the formula, f x and f y are the focal lengths in the x and y directions, c x and c y are the principal point coordinates of the camera, (k1, k2, k3) are the radial distortion coefficients, and (p1, p2) are the tangential distortion coefficients.

[0094] These parameters are indispensable basic information in the subsequent 3D reconstruction process, ensuring the accuracy of the 3D reconstruction results and ultimately achieving high-quality 3D reconstruction.

[0095] Furthermore, in the above step S1.3, the robot is placed inside the non-metallic pipe to be inspected, and the robot is controlled to move forward at a preset speed, and a monocular camera is used to take pictures of the inside of the non-metallic pipe to be inspected to form a data set, including the following steps:

[0096] S1.3.1. Place the robot equipped with a monocular camera at the entrance of the non-metallic pipe to be inspected, and clear the data of the odometer wheel 3;

[0097] S1.3.2, control the robot to move forward in the non-metallic pipe to be inspected, and at the same time, the circuit system in the base body 4 obtains the displacement information of the robot in the non-metallic pipe to be inspected through the mileage wheel 3, and controls the monocular camera to take pictures of the inside of the non-metallic pipe to be inspected according to the displacement information, and saves the pictures;

[0098] S1.3.3. The robot stops after reaching the end point, uses the host computer to read all the images, and creates a data set of images of the inside of the non-metallic pipe to be inspected in the order in which the images were taken.

[0099] Furthermore, in the above step S2, the feature matching method can be divided into two types: based on feature detectors and without detectors. The feature point matching network model designed by the present invention is a detector-free method that does not rely on the detection of feature points. Drawing on the strategy adopted by the human brain in matching similar patterns, the present invention proposes a progressive matching mode from coarse to fine, which uses a large core convolutional neural network with reparameterizable characteristics to generate feature vectors for all areas of the picture at a coarse granularity, and traverses all possible matching situations of the two images, and directly generates matching point pairs after further refinement, and the refined positions are used as feature point positions.

[0100] like Figure 3 As shown, the feature point matching network model based on deep learning constructed in this embodiment specifically includes three parts: a feature pyramid network, a coarse-grained matching model and a fine-grained matching model. Specifically, it includes the following steps:

[0101] S2.1. Use the feature pyramid network to extract features of different scales from the input image, and obtain feature maps of 1 / 2 scale, 1 / 4 scale and 1 / 8 scale, where the feature map of 1 / 4 scale is used as an intermediate variable.

[0102] S2.2, based on the 1 / 8 scale feature map, use the coarse-grained matching model to obtain a rough matching area;

[0103] S2.3. Based on the 1 / 2 scale feature map and the rough matching area, a fine-grained matching model is used to obtain the precise position of the matching feature points.

[0104] Furthermore, in the above step S2.1, if Figure 4 As shown in the figure, when the feature pyramid network is used to extract features of different scales from the input image, it is mainly divided into three stages: feature extraction stage, feature amplification stage, and horizontal connection feature fusion stage. This structure aims to achieve effective extraction, amplification and integration of image features to achieve the matching task of textureless images.

[0105] Specifically, the steps include:

[0106] S2.1.1. Perform feature extraction on the two input images, obtain downsampled feature maps of 1 / 2 scale and 1 / 4 scale respectively, and output a feature map of 1 / 8 scale. The downsampled feature map is further used in subsequent steps.

[0107] S2.1.2, using bilinear interpolation method to upsample the downsampled feature maps of different scales obtained in the feature extraction stage, and obtain upsampled feature maps of 1 / 4 scale and 1 / 2 scale respectively;

[0108] S2.1.3. Fuse the 1 / 2 scale upsampled feature map with the 1 / 2 scale downsampled feature map to obtain the final 1 / 2 scale feature map.

[0109] Furthermore, in the above step S2.1.1, the feature extraction stage is set to three feature extraction modules according to the needs, and the feature extraction modules are connected by a downsampling module. Among them, the downsampling module is composed of a 3×3 convolution layer with a step size of 2 to reduce the size of the feature map and double the number of channels. The feature extraction stage is composed of multiple large-core convolution modules, each of which is composed of a re-parameterized dilated convolution block, a channel attention module, a spatial attention module, a feedforward neural network (FFN) layer, and a batch normalization (BN) layer to enhance the representation ability of the network. Each large-core convolution module uses a residual connection to promote the back propagation of gradients and avoid the gradient disappearance problem in deep network training.

[0110] Specifically, the introduction is as follows:

[0111] ① Re-parameterized dilated convolutional blocks

[0112] Re-parameterized dilated convolution According to the homogeneity principle, the batch normalization (BN) layer can be merged into the convolution layer. For the jth channel of the output feature map, the convolution and BN operations are written as follows (assuming that the convolution kernel bias is 0):

[0113]

[0114] Among them, the subscripts of all symbols represent the jth channel; I and F are the input and convolutional layers; μ j and σ j are the mean and standard deviation of the jth channel of the feature map I*F; r j and β j is the standard deviation and mean learned by the BN layer. Combining formula (3) and homogeneity, the j-th convolution kernel F of the convolution layer F can be j The reparameterization is as follows:

[0115]

[0116] And the convolution kernel F j The corresponding bias is reparameterized as:

[0117]

[0118] The large-size convolution kernel used in the present invention uses a non-expanded small kernel and multiple expanded small kernels as parallel branches, all of which are designed for depthwise separable convolution, and their output results are superimposed. Its hyperparameters include the size of the large-size convolution kernel K, the size of the parallel convolution layer k, and the expansion rate r.

[0119] ②Channel Attention Module

[0120] In the channel attention module, first, the average pooling and maximum pooling operations are used to integrate the spatial information of the feature map to form two different spatial context descriptors: and These two descriptors correspond to the features after average pooling and maximum pooling, respectively. Then, these two descriptors are input into a common neural network, which is used to generate the channel attention map M c ∈R c×1×1 This neural network is a multi-layer perceptron (MLP) with a single hidden layer. To optimize parameter usage, the activation dimension of the hidden layer is set to M c ∈R c / r×1×1 , where r represents the reduction rate. Finally, the output feature vectors of the two spatial context descriptors are fused through a bitwise addition operation. Channel attention can be expressed as formula (6).

[0121]

[0122] Among them, σ is the sigmoid function, W1 and W0 are the weights of MLP, which are shared by the two inputs of average pooling features and maximum pooling features.

[0123] ③Spatial attention module

[0124] The spatial attention module uses overlapping circular pooling instead of the traditional pooling structure. Since the pooling area is circular, the pooling area cannot completely cover the pooling window. Therefore, when calculating the maximum pooling or average pooling, the element value is weighted according to the ratio of the coverage area to the element area. The improved spatial attention can be expressed by formula (7):

[0125]

[0126] Among them, σ is the sigmoid function, f 7×7 represents a convolution operation with a convolution kernel size of 7×7, Cir_AvgPool(·) and Cir_MaxPool(·) represent circular average pooling and circular maximum pooling operations respectively, and [;] represents the Concat splicing operation.

[0127] Furthermore, in the above step S2.1.2, each point P in the upsampled feature map corresponds to a position half of the original position in the original image, and then the four feature vectors {Q 11 ,Q 12 ,Q 21 ,Q 22}. Then the eigenvector value of point P is:

[0128]

[0129] in:

[0130]

[0131] In the formula, x1 and x2 represent Q 11 and Q 12 The horizontal axis, y1 and y2 represent Q 11 and Q 21 The vertical coordinate, x p and p Represent the horizontal and vertical coordinates of point P respectively.

[0132] The upsampled feature map is added to the corresponding feature map of the horizontal connection to integrate feature information at different levels. To eliminate the aliasing effect that may be introduced by upsampling, the network further processes the fused feature map through a 3×3 convolutional layer. This process is repeated until the feature map size is restored to the original input size.

[0133] Furthermore, in the above step S2.1.3, in the feature fusion stage, the network integrates the upsampled feature map with the downsampled feature map of the same scale.

[0134] This process is completed by direct addition to fuse feature information from different levels. To ensure the consistency of the number of channels during the fusion process, the network adjusts the number of channels through a 1×1 convolutional layer to meet the needs of subsequent processing.

[0135] Furthermore, in the above step S2.2, based on the 1 / 8 scale feature map, a coarse-grained matching model is used to process and obtain a rough matching area, including the following steps:

[0136] S2.2.1. Perform dimension transformation on the coarse-level features of the two images at a scale of 1 / 8 to obtain the score matrix S.

[0137] After extracting multi-level features in the feature pyramid network, use and To represent the coarse-level features of the two images at 1 / 8 scale, use and To represent the fine-level features of the two images at 1 / 2 scale.

[0138] feature and The dimension of is (C, w / 8, h / 8), where C is the number of channels, and w and h are the width and height of the original image. To facilitate calculation, the last two dimensions of the feature are first merged into one dimension and converted into a vector sequence with a dimension of C and a length of w×h / 64. and The correlation is expressed by vector inner product to obtain the score matrix S. The specific formula is as follows:

[0139]

[0140] in, and They represent the converted vector sequence respectively, <,> represent the inner product, and τ is a hyperparameter used to adjust the numerical range of the score matrix.

[0141] S2.2.2. Apply the softmax operator to the two dimensions of the score matrix respectively, convert the score matrix into probability, and obtain the confidence matrix P c , expressed by the following formula:

[0142] P c (i, j) = softmax(S(i,)) j softmax(S(·, j)) i (11)

[0143] S2.2.3. Based on the confidence matrix P c , for confidence values ​​below the threshold θ c The matches are filtered out and the mutual neighbor standard is applied to the remaining matches to obtain a rough matching area.

[0144] Implement the mutual nearest neighbor standard, that is, the feature in Figure A The feature with the highest matching probability is in Figure B. On the contrary, the feature in Figure B The feature with the highest matching probability is also in Figure A. but and A pair of matches is considered to satisfy only the mutual nearest neighbor criterion, denoted by It is expressed as formula (12). This method improves the accuracy and robustness of matching by ensuring that the matching is bidirectional.

[0145]

[0146] Furthermore, in the above step S2.3, based on the 1 / 2 scale feature map and the rough matching area, the fine-grained matching model is used to process and obtain the precise position of the matching feature point, including the following steps:

[0147] S2.3.1. Roughly match Positioning on fine-level feature maps and In the local neighborhood

[0148] S2.3.2. Crop two sets of local features of size w×w and For local neighborhood Perform Feature-Transformer transformation to obtain two local feature maps and

[0149] S2.3.3, use similarity calculation and softmax function to obtain the heat map of the matching probability between the two feature maps;

[0150] S2.3.4. Use the DSNT algorithm to calculate the exact location of the matching point from the heat map.

[0151] Furthermore, the above step S2.3.2 specifically includes the following steps:

[0152] ① Before feature transformation, two sets of local features of size w×w are first transformed. and Perform position encoding.

[0153] The attention mechanism of Transformer can extract the contextual information of features, but its structure is not sensitive to the sequence position order, and it cannot capture the spatial position information when applied to images. Therefore, position encoding is required before transforming local features. This embodiment adopts the 2D extended version of the absolute sine and cosine position encoding in DETR, and the position encoding vector is given by the following formula:

[0154]

[0155] in, d is the number of channels to which position encoding features need to be added, i is the index of the feature channel, is the position encoding vector. In this way, the position encoding gives each element unique position information in a sine and cosine manner, thereby incorporating the position information into the input of the model. This is crucial for producing accurate matches in textureless areas.

[0156] ②The Feature-Transformer module used performs feature transformation on the feature map that incorporates position information.

[0157] The original Transformer is composed of an encoder and a decoder. The Feature-Transformer module used in this embodiment is composed of multiple cascaded encoders. Each encoder structure is as follows: Figure 5 As shown. First, the encoder combines the input position encoding vector sequence with the weight matrix W q , W k , W v Multiply them to get the Q (Query), K (Key) and V (Value) matrices; secondly, input the Q, K and V matrices into the multi-head attention mechanism, extract the context information, and then pass the outputs of multiple heads through a linear layer and then normalize them; thirdly, concatenate the output of the normalization layer with the input of the encoder and pass it through a feedforward neural network for information fusion; finally, add the original input and the output of the feedforward neural network as the final result.

[0158] According to the different input features, the attention layer is divided into two types: self-attention layer and cross-attention layer. The input feature f processed by the self-attention layer i and f j From the same feature map, or The input features processed by the cross attention layer come from two different feature maps. and Self-attention layers and cross-attention layers are interlaced to form a deep network structure.

[0159] Furthermore, in the above step S2.3.3, two local feature maps are obtained after Feature-Transformer transformation: and Respectively and The feature map is calculated using formula (14): The center vector and feature map The similarity s of all vectors in .

[0160]

[0161] The similarity s is converted into a probability distribution through the softmax function and represented in the form of a two-dimensional image to obtain a heat map H(j) representing the matching probability.

[0162]

[0163] Among them, C is the number of feature map channels.

[0164] Furthermore, in the above step S2.3.4, the DSNT (Differentiable Spatial to Numerical Transform) algorithm is used to calculate the sub-pixel coordinates of each matching point in the image B from the heat map H. DSNT first treats the heat map as a probability distribution and then calculates its spatial expectation to obtain the position with sub-pixel accuracy.

[0165]

[0166] At the same time, the variance of the coordinates is calculated to estimate the uncertainty of the match for subsequent calculations:

[0167]

[0168] Furthermore, when training the deep learning-based feature point matching network model constructed in step S2, its loss function must be determined first.

[0169] In the present invention, the final loss is composed of coarse-grade loss and fine-grade loss:

[0170]

[0171] Since the coarse-level matching is applied to features that are 1 / 8 the size of the original image, and each feature represents a pixel grid in the original image, there may be one-to-many matches. This makes it difficult to determine the true value of the coarse-level matching. Therefore, we take the mutual nearest neighbors between the center positions of the 1 / 8 grid of the input image as the approximate true value. Specifically, take the center position of the 1 / 8 grid in the left image, project it to the same scale as the depth map, and index its depth. According to its depth value and the camera intrinsic parameters, calculate its three-dimensional coordinates in the camera coordinate system; according to the camera pose, transform it to the camera coordinate system of the second image, and finally reproject it into the pixel coordinate system. Among them, it is necessary to check whether the transformed point is within the image boundary and the depth consistency error is less than 0.2, and take the nearest neighbors of the points that meet the conditions as matching candidates. The same process is repeated from the right image to the left image. The two sets of nearest neighbor matches obtained are also filtered out according to the mutual nearest neighbor criterion to obtain the final true value. The true coarse-level match is recorded as Compute focal loss.

[0172] For positive samples, The position of , its loss is:

[0173]

[0174] For negative samples, The position of , its loss is:

[0175]

[0176] Among them, α is the coefficient for balancing the weights of positive and negative samples, and γ is the adjustment factor used to reduce the weights of easily classifiable samples. The final loss is:

[0177]

[0178] Here, |M pos | and |M neg | are the numbers of positive and negative samples respectively, which are used to standardize the loss. In this way, the loss of each part is averaged by their respective sample numbers, which helps to balance the influence of positive and negative samples, especially when their numbers are unbalanced.

[0179] In the fine-grained loss, for each query point the total variance of the heatmap is calculated to represent its uncertainty. The reciprocal of the variance is used to weight the 2-norm of the error, aiming to optimize the positions with lower uncertainty. After taking the average, the final weighted loss function is obtained:

[0180]

[0181] Among them, is the predicted point corresponding to the query point , is the point obtained by reprojection of according to the camera pose. is the total variance of the heatmap. M f represents the set of fine-grained matching points.

[0182] Furthermore, in the above step S3, since the present invention is a detector-free matching method and the expected value is calculated according to Equation (14) when refining the positions of feature points, it cannot guarantee the generation of consistent matching points between pictures, resulting in fragmentation of the feature trajectory. Therefore, it is necessary to aggregate and redistribute the feature points before reconstruction.

[0183] Specifically, assume there is a set of image collections {I1, I2, … I N}. After feature point extraction and matching, the matching points of each image and all other images are obtained. For any two pictures I n and I m , the matching result can be expressed as a series of matching pairs M nm = {(k ni , k mj )}, where k ni is the i-th feature point in image I n , and k mj is the j-th feature point in image I m . Each feature point can be further expressed as two-dimensional coordinates (x ni,y ni ). For each image I n , initialize the empty aggregation feature point set K n ={}. Quantize the feature points: Divide the image into multiple units of size Δx×Δy. Each unit is uniquely identified by an index pair (m,n), where m and n represent the index of the grid in the x and y directions, respectively. According to the position of the feature points, assign them to each unit. The quantization function q is expressed as:

[0184]

[0185] For the quantized feature point q(k ni ), if in the aggregate feature point set K n If there is no same quantization position in ni ) to K n If the new feature point k ni Assigned to the existing feature point k′ ni quantized positions, then according to their scores (matching probabilities), k′ ni The position of is fine-tuned. This is achieved through weighted averaging:

[0186]

[0187] Among them, w ni and w′ ni They are k ni and k′ ni And update the existing score by adding the new matching score:

[0188] w′ ni =w′ ni +w ni (25)

[0189] After processing the matching of all image pairs, the optimized aggregate feature point set K of each image is obtained. n .

[0190] Based on the clustered feature point set K n Redistribute the feature point matching results.

[0191] In this embodiment, for each matching pair (k ai ,k bj )∈M ab , use the nearest neighbor search (NN search) to find its position in the aggregated feature point set (k′ ai ,k′ bj ) and recorded as the new matching pair under the aggregated feature point set.

[0192] Through the two steps of aggregation and redistribution, the feature points are screened and optimized, the possibility of mismatching is reduced, and the robustness of the feature point position is enhanced. At the same time, this process extracts consistent matching information from multiple images, increases the trajectory length of the feature points, and makes subsequent reconstruction more stable.

[0193] Furthermore, in the above step S4, if Figure 6 As shown, the improved incremental reconstruction algorithm is used to perform three-dimensional sparse reconstruction to obtain the internal structure information of the non-metallic pipe to be detected, including the following steps:

[0194] S4.1 selects two images, uses the PnP algorithm to obtain the initial camera pose, triangulates the image according to the initial camera pose to obtain three-dimensional points as the initial point cloud model and uses local BA optimization;

[0195] S4.2, select the image with the most matching points in the point cloud model, use the PnP algorithm to obtain the camera pose of the new image, and triangulate it according to the camera pose to obtain new 3D points;

[0196] S4.3. Use the improved RANSAC algorithm to detect whether the point cloud model is cylindrical. If a cylinder is detected, add a cylindrical constraint term to the local RA to optimize the 3D point position;

[0197] S4.4. Repeat steps S4.2 to S4.3 until the feature points of all images are added to the point cloud model.

[0198] The PnP algorithm and the image triangulation algorithm adopt the techniques known to those skilled in the art, which will not be elaborated in the present invention.

[0199] Furthermore, in the above step S4.3, after adding a new three-dimensional point to the point cloud model, the improved RANSAC algorithm is used to estimate the cylinder model parameter C. In order to estimate the model more robustly, when estimating the distance from the point cloud to the model, the angle difference between the normal direction of the point and the axis direction of the cylinder is used for weighting.

[0200] First, for each point q in the point cloud i , search for the K nearest neighboring points, and obtain the point cloud set P = {p1, p2, ..., p n}, where each The neighborhood center

[0201]

[0202] Decentralize P by subtracting the neighborhood center from each element in P:

[0203]

[0204] Calculate the covariance matrix C:

[0205]

[0206] Calculate the eigenvalues ​​λ of the covariance matrix i and the eigenvector v i (i=1,2,3):

[0207] Cv i =λ i v i (29)

[0208] Sort by eigenvalue size. Assuming λ1≥λ2≥λ3, the eigenvector corresponding to λ1 is point q. i Normal vector. Ideally, the distance from a point on the cylindrical surface to the cylindrical axis is the radius r, and the normal vector of the point is perpendicular to the cylindrical axis. Therefore, the distance from the point to the estimated cylindrical model calculated using improved RANSAC can include these two items:

[0209] D total (p i , C)=|rD(p i ,C)|+W|n i ·d| (30)

[0210] Where r is the radius of the cylinder, D(p i ,C) is point p i Distance to the cylinder axis, n i is the normal vector of the point, d is the axis direction vector of the cylinder, and W is the weight coefficient, which is used to balance the distance from the point to the cylinder axis and the perpendicularity between the point normal and the axis normal. In practical applications, we do not estimate the cylinder at the beginning of reconstruction, but after reconstructing a certain number of 3D points, because a small number of sample points may lead to large estimation errors. Once the cylinder is detected, the cylinder constraint can be added in the bundle adjustment:

[0211] E(X, P, K, C) = E rep (X,P,K)+αE cyl (X,C) (31)

[0212] Among them, the first term E rep (X, P, K) is the reprojection error term, and the second term E cyl (X, C) Cylinder constraint term, X, P, K, C represent 3D point set, image pose, camera intrinsics, and detected cylinder parameters respectively. α is the weight controlling the two error terms. The cylinder constraint term calculates the distance from the 3D point to the cylinder surface:

[0213]

[0214] Where d(X j ,C) is a 3D point X j Distance function to the cylinder.

[0215] Figure 7 This is the flowchart of the improved RANSAC algorithm used. First, select the minimum required data set S min To estimate the model parameters. Assume that the current optimal solution is found in a certain iteration. In order to further improve the accuracy of the solution, a local optimization strategy is introduced on this basis, that is, a local iteration number k is set. In this process, an additional sample point p is selected from the internal point set I and added to S min An extended dataset S is formed exp Use S exp Recalculate the model parameters and evaluate the number of inliers of the new model. If the number of inliers of the extended model exceeds that of the current optimal solution, accept the new model as the optimal solution and update the current optimal state; if not, reject the sample point and select another point from I to try. This process is repeated within k local iterations to try to find a more accurate model solution by adding sample points.

[0216] After the above steps, the present invention finally obtains a three-dimensional sparse point cloud image inside the pipeline, which can be used for subsequent dense reconstruction and texture mapping. Figure 8 , Fig. 9 and Fig.10 It is a three-dimensional sparse reconstructed point cloud image of the inner wall of the pipeline obtained by the method of the present invention. Figure 8 This is the reconstruction effect of cone point defect. Fig. 9 This is the reconstruction effect diagram of the slot defect. Fig.10 This is the reconstruction effect of the long strip defect.

[0217] Example 2

[0218] The above-mentioned embodiment 1 provides a method for 3D sparse reconstruction of monocular images of non-metallic pipelines for deep-sea mining. Correspondingly, this embodiment provides a device for 3D sparse reconstruction of monocular images of non-metallic pipelines for deep-sea mining. The device provided in this embodiment can implement the method for 3D sparse reconstruction of monocular images of non-metallic pipelines for deep-sea mining in embodiment 1, and the device can be implemented by software, hardware, or a combination of software and hardware. For example, the device may include integrated or separate functional modules or functional units to execute the corresponding steps in each method of embodiment 1. Since the device of this embodiment is basically similar to the method embodiment, the process described in this embodiment is relatively simple, and the relevant parts can refer to the partial description of embodiment 1. The embodiment of the device provided in this embodiment is only schematic.

[0219] This embodiment provides a device for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining, comprising:

[0220] A data acquisition module is used to use a robot equipped with a monocular camera to acquire internal images of the non-metallic pipe to be inspected to form a data set;

[0221] The feature point matching module is used to detect and match feature points of the data set using a feature point matching network model based on deep learning;

[0222] Aggregation and redistribution module, used to aggregate and redistribute the obtained feature points;

[0223] The three-dimensional coefficient reconstruction module is used to perform three-dimensional sparse reconstruction using an improved incremental reconstruction algorithm to obtain the internal structure information of the non-metallic pipe to be inspected.

[0224] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A method for 3D sparse reconstruction of monocular images of non-metallic pipelines in deep-sea mining, characterized in that: The following steps are involved: A robot equipped with a monocular camera is used to obtain internal images of non-metallic pipes to be inspected to form a data set; Use a deep learning-based feature point matching network model to detect and match feature points in the dataset; Aggregate and redistribute the obtained feature points; Based on the aggregated feature points, the improved incremental reconstruction algorithm is used to perform three-dimensional sparse reconstruction to obtain the internal structure information of the non-metallic pipe to be inspected.

2. The method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining as claimed in claim 1, characterized in that: The data set is formed by using a robot equipped with a monocular camera to obtain internal images of the non-metallic pipe to be inspected, including: Determine the robot structure; The robot comprises a camera protection cover, a camera compartment, a plurality of odometer wheels, a base and support wheels; the lower part of the base is fixedly connected to a bottom plate through a flange, and three support wheels are arranged at the lower part of the base plate to ensure that the center line of the robot is in line with the center line of the pipeline; the upper part of the base is connected to the camera compartment through another flange, and a spring is arranged between the camera compartment and the other flange; the camera protection cover is arranged on the top of the camera compartment to protect the monocular camera placed in the camera compartment; the odometer wheels are arranged outside the camera compartment at intervals, and the odometer wheels have a built-in magnetic encoder to measure the rotation angle of the odometer wheels; a circuit system is placed inside the base to receive the pulse signal sent by the magnetic encoder and calculate the robot displacement data according to the odometer wheel parameters, and control the monocular camera to take pictures of the inside of the pipeline according to the displacement data; A monocular camera is mounted on the robot, and the monocular camera is calibrated using Zhang's calibration method to obtain the camera's intrinsic parameters; The robot is placed inside the non-metallic pipe to be inspected and controlled to move forward at a preset speed. At the same time, a monocular camera is used to take pictures of the inside of the non-metallic pipe to be inspected to form a data set.

3. The method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining as claimed in claim 2, characterized in that: The monocular camera is mounted on the robot, and the monocular camera is calibrated by using Zhang's calibration method to obtain the camera internal parameters, including: Fix the preset black and white checkerboard calibration paper on the table; Use the monocular camera on the robot to take multiple images of the calibration paper from different angles and positions, ensuring that the checkerboard pattern in each image is in the field of view. The posture and position of the monocular camera should be varied enough to cover different viewing angles. Based on the image information of the calibration paper, Zhang’s calibration method is used to calculate the intrinsic parameters of the monocular camera.

4. The method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining as claimed in claim 2, characterized in that: The robot is placed inside the non-metallic pipe to be inspected, and the robot is controlled to move forward at a preset speed. At the same time, a monocular camera is used to take pictures of the inside of the non-metallic pipe to be inspected to form a data set, including: Place the robot equipped with a monocular camera at the entrance of the non-metallic pipe to be inspected and clear the data of the odometer wheel; The robot is controlled to move forward in the non-metallic pipe to be inspected. At the same time, the circuit system in the base obtains the displacement information of the robot in the non-metallic pipe to be inspected through the odometer, and controls the monocular camera to take pictures of the inside of the non-metallic pipe to be inspected according to the displacement information, and saves the pictures; The robot stops after reaching the end point, uses the host computer to read all the pictures, and creates a data set of pictures of the inside of the non-metallic pipe to be inspected in the order in which the pictures were taken.

5. The method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining as claimed in claim 1, characterized in that: The method of using a deep learning-based feature point matching network model to detect and match feature points on a data set includes: Use the feature pyramid network to extract features of different scales from the input image to obtain feature maps of 1 / 2 scale and 1 / 8 scale; Based on the 1 / 8 scale feature map, a coarse-grained matching model is used to obtain a rough matching area; Based on the 1 / 2 scale feature map and the rough matching area, a fine-grained matching model is used to obtain the precise position of the matching feature points.

6. The method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining as claimed in claim 5, characterized in that: The feature pyramid network is used to extract features of different scales from the input image to obtain a 1 / 2 scale feature map and a 1 / 8 scale feature map, including: Perform feature extraction on the two input images, obtain downsampled feature maps of 1 / 2 scale and 1 / 4 scale respectively, and output feature maps of 1 / 8 scale; The bilinear interpolation method is used to upsample the downsampled feature maps of different scales obtained in the feature extraction stage, and upsampled feature maps of 1 / 4 scale and 1 / 2 scale are obtained in turn; The 1 / 2 scale up-sampled feature map is fused with the 1 / 2 scale down-sampled feature map to obtain the final 1 / 2 scale feature map.

7. The method for 3D sparse reconstruction of a non-metallic pipeline in deep-sea mining according to claim 5, characterized in that: The method of obtaining a rough matching area based on the 1 / 8 scale feature map using a coarse-grained matching model includes the following steps: Perform dimension transformation on the coarse-level features of the two images at a scale of 1 / 8 to obtain the score matrix S; Apply the softmax operator to the two dimensions of the score matrix respectively, convert the score matrix into probability, and obtain the confidence matrix P c ; Based on the confidence matrix P c , for confidence values ​​below the threshold θ c The matches are filtered out, and the remaining matches are subjected to the mutual neighbor standard to obtain a rough matching area 8. The method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining as claimed in claim 5, characterized in that: The method of obtaining the precise position of the matching feature point based on the 1 / 2 scale feature map and the rough matching area using a fine-grained matching model includes the following steps: Will roughly match Positioning on fine-level feature maps and In the local neighborhood Crop two sets of local features of size w×w and For local neighborhood Perform Feature-Transformer transformation to obtain two local feature maps and Use similarity calculation and softmax function to get the heat map of the matching probability between two feature maps; The DSNT algorithm is used to calculate the exact location of the matching points from the heat map.

9. The method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining as claimed in claim 1, characterized in that: The method of using the improved incremental reconstruction algorithm to perform three-dimensional sparse reconstruction to obtain the internal structure information of the non-metallic pipe to be detected includes the following steps: ① Select two pictures, use the PnP algorithm to obtain the initial camera pose, triangulate the image according to the initial camera pose to obtain 3D points as the initial point cloud model and use local BA optimization; ② Select the image with the most matching points in the point cloud model, use the PnP algorithm to obtain the camera pose of the new image, and triangulate it according to the camera pose to obtain the new 3D point; ③ Use the improved RANSAC algorithm to detect whether the point cloud model is cylindrical. If a cylinder is detected, a cylindrical constraint item is added to the local RA to optimize the 3D point position. ④ Repeat steps ② to ③ until the feature points of all images are added to the point cloud model.

10. A 3D sparse reconstruction device for monocular images of non-metallic pipelines in deep-sea mining, characterized in that: include: A data acquisition module is used to use a robot equipped with a monocular camera to acquire internal images of the non-metallic pipe to be inspected to form a data set; The feature point matching module is used to detect and match feature points of the data set using a feature point matching network model based on deep learning; Aggregation and redistribution module, used to aggregate and redistribute the obtained feature points; The three-dimensional coefficient reconstruction module is used to perform three-dimensional sparse reconstruction using an improved incremental reconstruction algorithm to obtain the internal structure information of the non-metallic pipe to be inspected.

Images