A method and device for 3D sparse reconstruction of monocular images of non-metallic pipelines in deep-sea mining

Abstract

The present invention relates to a method and apparatus for 3D sparse reconstruction of monocular images of non-metallic pipelines in deep-sea mining. The method comprises the following steps: using a robot equipped with a monocular camera to acquire internal images of the non-metallic pipeline to be inspected to form a dataset; using a deep learning-based feature point matching network model to detect and match feature points in the dataset; aggregating and redistributing the obtained feature points; and, based on the aggregated feature points, using an improved incremental reconstruction algorithm to perform 3D sparse reconstruction to obtain internal structural information of the non-metallic pipeline to be inspected. The present invention can be widely applied in the field of non-metallic pipeline detection technology.

Description

Technical Field

[0001] The present invention relates to a method and device for three-dimensional sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining, belonging to the technical field of non-metallic pipeline detection. Background Art

[0002] Non-metallic flexible mixed transport pipelines, the "arteries" of deep-sea mining systems, are crucial conveying equipment in hydraulic hose lifting systems. They not only effectively prevent the corrosion that metal pipelines are susceptible to in deep-sea environments, but also offer advantages such as easy installation, low maintenance costs, and a long service life. Despite their numerous advantages, non-metallic pipelines have gradually become susceptible to failures such as leakage and breakage during use. Therefore, continuous condition monitoring and performance evaluation of these long-term non-metallic pipelines are crucial.

[0003] In recent years, with the rapid development of computer vision technology, researchers have increasingly used 3D reconstruction techniques for pipeline modeling and inspection. This method simply uses a visual device to capture images of the pipeline's inner wall to reconstruct a complete pipeline model. This overcomes the high cost, poor versatility, and inflexibility of specialized sensors in traditional nondestructive testing techniques, providing a new approach for pipeline defect assessment. Based on the number of cameras, 3D reconstruction technology can be divided into three types: monocular, binocular, and multi-camera. Monocular reconstruction involves using a single camera to capture multiple images to construct a 3D model. By analyzing feature changes in the image sequence, the relationship between features and spatial structure is established, and the 3D shape of the object is reconstructed. Compared with binocular or multi-camera systems, monocular reconstruction, while less expensive, is more complex.

[0004] Current research focuses primarily on the reconstruction of metallic pipelines. Because deep-sea non-metallic pipelines generally lack texture and operate in dark or low-light environments, the details of video images captured by monocular cameras are blurred. Therefore, the use of monocular cameras to reconstruct the inner surface of non-metallic pipelines has not been widely studied. Existing technologies have shown poor results, or even failure, when applied to deep-sea non-metallic pipelines. Summary of the Invention

[0005] In response to the above problems, the purpose of the present invention is to provide a method and equipment for three-dimensional sparse reconstruction of monocular images of deep-sea mining non-metallic pipelines, which can realize three-dimensional sparse reconstruction of deep-sea non-metallic pipelines and intuitively display the structural information inside the pipelines.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining, comprising the following steps:

[0008] A robot equipped with a monocular camera is used to obtain internal images of non-metallic pipes to be inspected to form a dataset.

[0009] Use a deep learning-based feature point matching network model to detect and match feature points in the dataset;

[0010] Aggregate and redistribute the obtained feature points;

[0011] Based on the aggregated feature points, the improved incremental reconstruction algorithm is used to perform three-dimensional sparse reconstruction to obtain the internal structure information of the non-metallic pipe to be inspected.

[0012] Furthermore, the method of using a robot equipped with a monocular camera to obtain internal images of the non-metallic pipe to be inspected to form a data set includes:

[0013] Determine the robot structure;

[0014] Among them, the robot includes a camera protection cover, a camera compartment, several odometer wheels, a base and support wheels; the lower part of the base is fixedly connected to a base plate through a flange, and three support wheels are provided at the lower part of the base plate to ensure that the center line of the robot is collinear with the center line of the pipeline; the upper part of the base is connected to the camera compartment through another flange, and a spring is provided between the camera compartment and the other flange; the camera protection cover is provided on the top of the camera compartment to protect the monocular camera placed in the camera compartment; the odometer wheels are arranged at intervals outside the camera compartment, and the odometer wheels have a built-in magnetic encoder for measuring the rotation angle of the odometer wheels; a circuit system is placed inside the base to receive the pulse signal sent by the magnetic encoder and calculate the robot displacement data according to the odometer wheel parameters, and control the monocular camera to take pictures of the inside of the pipeline according to the displacement data;

[0015] A monocular camera is mounted on the robot and calibrated using Zhang's calibration method to obtain the camera's intrinsic parameters.

[0016] The robot is placed inside the non-metallic pipe to be inspected and controlled to move forward at a preset speed. At the same time, a monocular camera is used to capture images of the interior of the non-metallic pipe to be inspected to form a data set.

[0017] Furthermore, the robot is equipped with a monocular camera, and the monocular camera is calibrated using Zhang's calibration method to obtain the camera's intrinsic parameters, including:

[0018] Fix the preset black and white checkerboard calibration paper on the table;

[0019] Use the robot's monocular camera to capture multiple images of the calibration paper from different angles and positions, ensuring that the checkerboard pattern is in view in each image. The monocular camera's pose and position should vary enough during the capture to cover different viewpoints.

[0020] Based on the image information of the obtained calibration paper, the Zhang calibration method is used to calculate the intrinsic parameters of the monocular camera.

[0021] Furthermore, the robot is placed inside the non-metallic pipe to be inspected, and the robot is controlled to move forward at a preset speed. At the same time, a monocular camera is used to capture images of the interior of the non-metallic pipe to be inspected to form a data set, including:

[0022] Place the robot equipped with a monocular camera at the entrance of the non-metallic pipe to be inspected and reset the odometer data.

[0023] The robot is controlled to move forward in the non-metallic pipe to be inspected. At the same time, the circuit system in the base obtains the displacement information of the robot in the non-metallic pipe to be inspected through the odometer. Based on the displacement information, the monocular camera is controlled to take pictures of the inside of the non-metallic pipe to be inspected and the pictures are saved.

[0024] The robot stops after reaching the end point, uses the host computer to read all the pictures, and creates a data set of pictures of the interior of the non-metallic pipe to be inspected according to the order in which the pictures were taken.

[0025] Furthermore, the feature point detection and matching of the data set using a feature point matching network model based on deep learning includes:

[0026] Use the feature pyramid network to extract features of different scales from the input image, and obtain feature maps of 1 / 2 scale and 1 / 8 scale;

[0027] Based on the 1 / 8 scale feature map, a coarse-grained matching model is used to obtain a rough matching area;

[0028] Based on the 1 / 2 scale feature map and the rough matching area, a fine-grained matching model is used to obtain the precise position of the matching feature points.

[0029] Furthermore, the feature pyramid network is used to extract features of different scales from the input image to obtain a 1 / 2 scale feature map and a 1 / 8 scale feature map, including:

[0030] Perform feature extraction on the two input images, obtain downsampled feature maps of 1 / 2 scale and 1 / 4 scale in turn, and output a feature map of 1 / 8 scale;

[0031] The bilinear interpolation method is used to upsample the downsampled feature maps of different scales obtained in the feature extraction stage, and the upsampled feature maps of 1 / 4 scale and 1 / 2 scale are obtained in turn;

[0032] The 1 / 2 scale upsampled feature map is fused with the 1 / 2 scale downsampled feature map to obtain the final 1 / 2 scale feature map.

[0033] Furthermore, the 1 / 8 scale feature map is processed using a coarse-grained matching model to obtain a rough matching area, including the following steps:

[0034] Perform dimension transformation on the coarse-level features of the two images at 1 / 8 scale to obtain the score matrix S;

[0035] Apply the softmax operator to the two dimensions of the score matrix respectively, convert the score matrix into probability, and obtain the confidence matrix P c ;

[0036] Based on the confidence matrix P c , for confidence values ​​below the threshold θ c The matches are filtered out and the mutual neighbor standard is applied to the remaining matches to obtain a rough matching area.

[0037] Furthermore, the method of using a fine-grained matching model to process the 1 / 2 scale feature map and the rough matching area to obtain the precise position of the matching feature point includes the following steps:

[0038] will roughly match Positioning on fine-level feature maps and In the local neighborhood

[0039] Crop two sets of local features of size w×w and For local neighborhood Perform Feature-Transformer transformation to obtain two local feature maps and

[0040] Use similarity calculation and softmax function to obtain the heat map of the matching probability between the two feature maps;

[0041] The DSNT algorithm is used to calculate the exact location of the matching points from the heat map.

[0042] Furthermore, the improved incremental reconstruction algorithm is used to perform three-dimensional sparse reconstruction to obtain internal structure information of the non-metallic pipe to be inspected, including the following steps:

[0043] ① Select two pictures, use the PnP algorithm to obtain the initial camera pose, triangulate the image according to the initial camera pose to obtain 3D points as the initial point cloud model and use local BA optimization;

[0044] ② Select the image with the most matching points in the point cloud model, use the PnP algorithm to obtain the camera pose of the new image, and triangulate it according to the camera pose to obtain the new 3D points;

[0045] ③ Use the improved RANSAC algorithm to detect whether the point cloud model is cylindrical. If a cylinder is detected, a cylindrical constraint term is added to the local RA to optimize the 3D point position.

[0046] ④ Repeat steps ② to ③ until all feature points of the images are added to the point cloud model.

[0047] In a second aspect, the present invention provides a device for 3D sparse reconstruction of monocular images of non-metallic pipelines in deep-sea mining, comprising:

[0048] The data acquisition module is used to use a robot equipped with a monocular camera to obtain internal images of the non-metallic pipe to be inspected to form a data set;

[0049] Feature point matching module, which is used to detect and match feature points of the dataset using a feature point matching network model based on deep learning;

[0050] Aggregation and redistribution module, used to aggregate and redistribute the obtained feature points;

[0051] The three-dimensional coefficient reconstruction module is used to perform three-dimensional sparse reconstruction using the improved incremental reconstruction algorithm to obtain the internal structure information of the non-metallic pipe to be inspected.

[0052] The present invention has the following advantages due to the adoption of the above technical solution:

[0053] 1. This invention first utilizes a deep learning-based feature point matching model to extract and match the feature points of non-metallic pipes. Subsequently, an improved incremental reconstruction technique is employed to generate a three-dimensional sparse point cloud that accurately represents the geometry and structural information of deep-sea non-metallic flexible pipes. This invention facilitates monitoring deformation, damage, and wear of flexible pipes in complex deep-sea environments, enabling timely early warning and maintenance to prevent pipeline failure. Through three-dimensional reconstruction of non-metallic flexible pipes, environmental pollution and mining accidents caused by pipeline damage or design flaws can be effectively avoided, thereby improving the safety of deep-sea mining.

[0054] 2. The robot of the present invention only needs to be equipped with a monocular camera, and the subsequent reconstruction process is mainly achieved by the algorithm, without the need for expensive dedicated sensors.

[0055] 3. The present invention designs a feature point matching model based on deep learning. This model proposes a feature point extraction and matching network based on deep learning with reference to human behavior patterns. It includes two matching stages from coarse to fine and a heavy parameter network design that expands the receptive field, solving the problem of difficult feature matching of textureless images.

[0056] 4. The present invention proposes an improved incremental reconstruction algorithm, which can reduce the radius error rate and the distortion of 3D points by improving the RANSAC algorithm and adding a circular constraint term to the BA optimization.

[0057] Therefore, the present invention can be widely applied to the field of non-metallic pipeline detection technology, especially the field of deep-sea mining technology. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Various other advantages and benefits will become apparent to those skilled in the art upon reading the detailed description of the preferred embodiment below. The accompanying drawings are for illustration purposes only and are not to be considered as limiting the present invention. Throughout the drawings, the same reference numerals are used to denote the same components. In the drawings:

[0059] Figure 1 This is a flow chart of a method for sparse 3D reconstruction of monocular images of non-metallic pipelines for deep-sea mining provided by an embodiment of the present invention;

[0060] Figure 2 is a structural diagram of a robot provided by an embodiment of the present invention;

[0061] Figure 3 This is a flow chart of a feature point matching model based on deep learning provided by an embodiment of the present invention;

[0062] Figure 4 is a flow chart of a feature pyramid network provided by an embodiment of the present invention;

[0063] Figure 5 is a diagram of the encoder structure of the Feature-Transformer module provided by an embodiment of the present invention;

[0064] Figure 6 is a flow chart of an improved incremental reconstruction algorithm provided by an embodiment of the present invention;

[0065] Figure 7 is a flow chart of an improved RANSAC algorithm provided by an embodiment of the present invention;

[0066] Figure 8 This is a diagram showing the reconstruction effect of a cone point defect provided by an embodiment of the present invention;

[0067] Figure 9 This is a rendering of the slot defect reconstruction provided by an embodiment of the present invention;

[0068] Figure 10 This is a rendering of a strip defect reconstruction provided by an embodiment of the present invention;

[0069] The reference numerals in the figures are as follows:

[0070] 1. Camera protection cover; 2. Camera compartment; 3. Odometer wheel; 4. Base; 5. Support wheel. DETAILED DESCRIPTION

[0071] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the described embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field are within the scope of protection of the present invention.

[0072] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is intended to include the plural form. In addition, it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or combinations thereof.

[0073] In some embodiments of the present invention, a method for sparse three-dimensional reconstruction of monocular images of non-metallic pipelines for deep-sea mining is provided. This method, based on feature point extraction and matching network models using deep learning, solves the problem of difficulty in feature matching of textureless images. At the same time, it improves the RANSAC algorithm and BA algorithm in the incremental reconstruction method, effectively realizing sparse three-dimensional reconstruction of deep-sea non-metallic flexible pipes and intuitively displaying the structural information inside the pipes.

[0074] Correspondingly, in some other embodiments of the present invention, a device for sparse three-dimensional reconstruction of monocular images of non-metallic pipelines in deep-sea mining is provided.

[0075] Example 1

[0076] like Figure 1 As shown, this embodiment provides a method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining, which includes the following steps:

[0077] S1. Use a robot equipped with a monocular camera to obtain internal images of non-metallic pipes to be inspected to form a dataset.

[0078] S2. Use a deep learning-based feature point matching network model to detect and match feature points in the dataset;

[0079] S3, aggregate and redistribute the obtained feature points;

[0080] S4. Based on the aggregated feature points, the improved incremental reconstruction algorithm is used to perform three-dimensional sparse reconstruction to obtain the internal structure information of the non-metallic pipe to be inspected.

[0081] Furthermore, in the above step S1, a robot equipped with a monocular camera is used to obtain internal images of the non-metallic pipe to be inspected to form a data set, which includes the following steps:

[0082] S1.1. Determine the robot structure;

[0083] S1.2. Mount a monocular camera on the robot and calibrate it using Zhang's calibration method to obtain the camera's intrinsic parameters.

[0084] S1.3. Place the robot inside the non-metallic pipe to be inspected and control the robot to move forward at a preset speed. At the same time, use a monocular camera to capture images of the interior of the non-metallic pipe to be inspected to form a dataset.

[0085] Furthermore, in the above step S1.1, if Figure 2 As shown, the robot used in this embodiment includes: a camera protective cover 1, a camera compartment 2, an odometry wheel 3, a base 4, and support wheels 5. The base 4 is fixedly connected to a base plate via a flange at the bottom, and three support wheels 5 are provided below the base plate to ensure that the robot's centerline is collinear with the pipeline's centerline and that no data is measured. The base 4 is connected to the camera compartment 2 at the top via another flange, and a spring is provided between the camera compartment 2 and the flange. The camera protective cover 1 is provided on top of the camera compartment 2 to protect the monocular camera placed in the camera compartment 2. The odometry wheel 3 is provided outside the camera compartment 2 and has a built-in magnetic encoder for measuring the angle of rotation of the odometry wheel 3. The base 4 houses a circuit system based on an STM32F4, which receives pulse signals from the magnetic encoder and calculates the robot's displacement data based on the odometry wheel parameters. This displacement data is then used to control the monocular camera to capture images of the pipeline interior.

[0086] Furthermore, in the above step S1.2, the monocular camera is calibrated using Zhang's calibration method to obtain the camera intrinsic parameters, including the following steps:

[0087] S1.2.1. Fix the preset black and white checkerboard calibration paper on the table;

[0088] S1.2.2. Use the robot's monocular camera to capture multiple images of the calibration paper from different angles and positions, ensuring that the checkerboard pattern is in view in each image. The monocular camera's pose and position should vary sufficiently during the capture to cover different viewpoints.

[0089] S1.2.3. Based on the image information of the obtained calibration paper, use Zhang's calibration method to calculate the intrinsic parameters of the monocular camera.

[0090] In this embodiment, the internal parameters of the monocular camera include the internal parameter matrix M1 and the distortion coefficient β, which are respectively expressed as:

[0091]

[0092] β=(k1,k2,k3,p1,p2) (2)

[0093] Where, f x and f y are the focal lengths in the x and y directions, c x and c y are the principal point coordinates of the camera, (k1, k2, k3) are the radial distortion coefficients, and (p1, p2) are the tangential distortion coefficients.

[0094] These parameters are indispensable basic information in the subsequent 3D reconstruction process, ensuring the accuracy of the 3D reconstruction results and ultimately achieving high-quality 3D reconstruction.

[0095] Furthermore, in the above step S1.3, the robot is placed inside the non-metallic pipe to be inspected, and the robot is controlled to move forward at a preset speed. At the same time, a monocular camera is used to capture images of the interior of the non-metallic pipe to be inspected to form a data set, which includes the following steps:

[0096] S1.3.1. Place the robot equipped with a monocular camera at the entrance of the non-metallic pipe to be inspected and reset the data of odometer wheel 3.

[0097] S1.3.2. Control the robot to advance in the non-metallic pipe to be inspected. Simultaneously, the circuit system within the base body 4 obtains displacement information of the robot in the non-metallic pipe to be inspected via the odometer wheel 3. Based on the displacement information, the circuit system controls the monocular camera to capture an image of the interior of the non-metallic pipe to be inspected, and saves the image.

[0098] S1.3.3. The robot stops after reaching the end point. The host computer reads all the images and creates a data set of images of the interior of the non-metallic pipe to be inspected in the order in which the images were taken.

[0099] Furthermore, in the above step S2, the feature matching method can be divided into two types: based on feature detectors and without detectors. The feature point matching network model designed by the present invention is a detector-free method that does not rely on the detection of feature points. Drawing on the strategy adopted by the human brain in matching similar patterns, the present invention proposes a progressive matching mode from coarse to fine, which uses a large core convolutional neural network with reparameterizable characteristics to generate feature vectors for all areas of the image at a coarse granularity, and traverses all possible matching situations of the two images. After further refinement, matching point pairs are directly generated, and the refined positions are used as feature point positions.

[0100] like Figure 3 As shown, the feature point matching network model based on deep learning constructed in this embodiment specifically includes three parts: a feature pyramid network, a coarse-grained matching model, and a fine-grained matching model. Specifically, it includes the following steps:

[0101] S2.1. Use the feature pyramid network to extract features of the input image at different scales, obtaining feature maps at 1 / 2 scale, 1 / 4 scale, and 1 / 8 scale. The feature map at 1 / 4 scale is used as an intermediate variable.

[0102] S2.2, based on the 1 / 8 scale feature map, use the coarse-grained matching model to obtain a rough matching area;

[0103] S2.3. Based on the 1 / 2 scale feature map and the rough matching area, a fine-grained matching model is used to obtain the precise location of the matching feature points.

[0104] Furthermore, in the above step S2.1, if Figure 4 As shown in Figure 1, when using a feature pyramid network to extract features at different scales from an input image, the process is divided into three main stages: feature extraction, feature amplification, and feature fusion using horizontal connections. This structure aims to effectively extract, amplify, and integrate image features to achieve matching tasks on textureless images.

[0105] Specifically, the method includes the following steps:

[0106] S2.1.1. Perform feature extraction on the two input images, obtain downsampled feature maps at 1 / 2 scale and 1 / 4 scale, and output a feature map at 1 / 8 scale. The downsampled feature maps are used in subsequent steps.

[0107] S2.1.2. Use bilinear interpolation to upsample the downsampled feature maps of different scales obtained in the feature extraction stage to obtain upsampled feature maps of 1 / 4 scale and 1 / 2 scale respectively;

[0108] S2.1.3. Fuse the 1 / 2 scale upsampled feature map with the 1 / 2 scale downsampled feature map to obtain the final 1 / 2 scale feature map.

[0109] Furthermore, in the above step S2.1.1, the feature extraction stage is set to three feature extraction modules according to the needs, and the feature extraction modules are connected by a downsampling module. Among them, the downsampling module is composed of a 3×3 convolution layer with a stride of 2 to reduce the size of the feature map while doubling the number of channels. The feature extraction stage is composed of multiple large-core convolution modules, each of which is composed of a re-parameterized dilated convolution block, a channel attention module, a spatial attention module, a feedforward neural network (FFN) layer and a batch normalization (BN) layer to enhance the representation ability of the network. Each large-core convolution module adopts a residual connection to promote the back propagation of the gradient and avoid the gradient disappearance problem in deep network training.

[0110] Specifically, the introduction is as follows:

[0111] ① Reparameterized dilated convolution block

[0112] Reparameterized dilated convolution can incorporate the batch normalization (BN) layer into the convolution layer based on the homogeneity principle. For the j-th channel of the output feature map, the convolution and BN operations can be expressed as follows (assuming that the convolution kernel bias is 0):

[0113]

[0114] Among them, the subscripts of all symbols represent the jth channel; I and F are the input and convolution layers; μ j and σ j are the mean and standard deviation of the jth channel of the feature map I*F; r j and β j is the standard deviation and mean learned by the BN layer. Combining formula (3) and homogeneity, the j-th convolution kernel F of the convolution layer F can be j The reparameterization is as follows:

[0115]

[0116] And the convolution kernel F j The corresponding bias is reparameterized as:

[0117]

[0118] The large-scale convolution kernel used in this invention uses a non-expanded small kernel and multiple expanded small kernels as parallel branches, all designed for depthwise separable convolution, and their outputs are superimposed. Its hyperparameters include the size of the large-scale convolution kernel K, the size of the parallel convolution layer k, and the expansion rate r.

[0119] ②Channel Attention Module

[0120] In the channel attention module, first, the average pooling and maximum pooling operations are used to integrate the spatial information of the feature map to form two different spatial context descriptors: and These two descriptors correspond to the features after average pooling and maximum pooling, respectively. Then, these two descriptors are input into a common neural network, which is used to generate the channel attention map M c ∈R c×1×1 This neural network is a multi-layer perceptron (MLP) with a single hidden layer. To optimize parameter usage, the activation dimension of the hidden layer is set to M c ∈R c / r×1×1 , where r represents the reduction rate. Finally, the output feature vectors of the two spatial context descriptors are fused through a bitwise addition operation. Channel attention can be expressed as Equation (6).

[0121]

[0122] Among them, σ is the sigmoid function, W1 and W0 are the weights of MLP, which are shared by the two inputs of average pooling features and maximum pooling features.

[0123] ③Spatial attention module

[0124] The spatial attention module uses overlapping circular pooling instead of the traditional pooling structure. Since the pooling area is circular, the pooling area cannot completely cover the pooling window. Therefore, when calculating the maximum pooling or average pooling, the element value is weighted according to the ratio of the coverage area to the element area. The improved spatial attention can be expressed by formula (7):

[0125]

[0126] Where σ is the sigmoid function, f 7×7 represents a convolution operation with a convolution kernel size of 7×7, Cir_AvgPool(·) and Cir_MaxPool(·) represent circular average pooling and circular maximum pooling operations respectively, and [;] represents the Concat splicing operation.

[0127] Furthermore, in the above step S2.1.2, each point P in the upsampled feature map corresponds to a position half of the original position in the original image, and then finds its four nearest feature vectors {Q 11 ,Q 12 ,Q 21 ,Q 22}. Then the eigenvector value of point P is:

[0128]

[0129] in:

[0130]

[0131] Where x1 and x2 represent Q 11 and Q 12 The horizontal axis, y1 and y2 represent Q 11 and Q 21 The vertical coordinate, x p and y p Represent the horizontal and vertical coordinates of point P respectively.

[0132] The upsampled feature map is added to the corresponding feature map from the horizontal connection to integrate feature information from different levels. To eliminate aliasing effects that may be introduced by upsampling, the network further processes the fused feature map through a 3×3 convolutional layer. This process is repeated until the feature map size is restored to the original input size.

[0133] Furthermore, in the above step S2.1.3, in the feature fusion stage, the network integrates the upsampled feature map with the downsampled feature map of the same scale.

[0134] This process is completed by direct addition to fuse feature information from different levels. To ensure the consistency of the number of channels during the fusion process, the network adjusts the number of channels through a 1×1 convolutional layer to meet the needs of subsequent processing.

[0135] Furthermore, in the above step S2.2, based on the 1 / 8 scale feature map, a coarse-grained matching model is used to obtain a rough matching area, including the following steps:

[0136] S2.2.1. Perform dimension transformation on the coarse-level features of the two images at the 1 / 8 scale to obtain the score matrix S.

[0137] After extracting multi-level features from the feature pyramid network, use and To represent the coarse-level features of the two images at 1 / 8 scale, use and to represent the fine-level features of the two images at 1 / 2 scale.

[0138] feature and The dimension is (C, w / 8, h / 8), where C is the number of channels, and w and h are the width and height of the original image. To facilitate calculation, the last two dimensions of the feature are first merged into one dimension and converted into a vector sequence with a dimension of C and a length of w×h / 64. and The correlation is expressed by vector inner product to obtain the score matrix S. The specific formula is as follows:

[0139]

[0140] in, and They represent the converted vector sequences, <,> represent the inner product, and τ is a hyperparameter used to adjust the numerical range of the score matrix.

[0141] S2.2.2. Apply the softmax operator to the two dimensions of the score matrix respectively, convert the score matrix into probability, and obtain the confidence matrix P c , expressed by the following formula:

[0142] P c (i, j) = softmax(S(i,)) j softmax(S(·, j)) i (11)

[0143] S2.2.3. Based on the confidence matrix P c , for confidence values ​​below the threshold θ c The matches are filtered out and the mutual neighbor standard is applied to the remaining matches to obtain a rough matching area.

[0144] Implement the mutual nearest neighbor standard, that is, the features in Figure A The feature with the highest matching probability is in Figure B On the contrary, the features in Figure B The feature with the highest matching probability is also in Figure A but and A pair of matches is considered to satisfy only the mutual nearest neighbor criteria, denoted as This method improves the accuracy and robustness of matching by ensuring that the matching is bidirectional.

[0145]

[0146] Furthermore, in the above step S2.3, based on the 1 / 2 scale feature map and the coarse matching area, the fine-grained matching model is used to obtain the precise position of the matching feature point, including the following steps:

[0147] S2.3.1. Roughly match Positioning on fine-level feature maps and In the local neighborhood

[0148] S2.3.2. Crop two sets of local features of size w×w and For local neighborhood Perform Feature-Transformer transformation to obtain two local feature maps and

[0149] S2.3.3. Use similarity calculation and softmax function to obtain the heat map of the matching probability between the two feature maps;

[0150] S2.3.4. Use the DSNT algorithm to calculate the exact location of the matching point from the heat map.

[0151] Furthermore, the above step S2.3.2 specifically includes the following steps:

[0152] ① Before feature transformation, two sets of local features of size w×w are first transformed. and Perform position encoding.

[0153] The Transformer's attention mechanism can extract contextual information about features, but its structure is insensitive to the order of sequence positions. When applied to images, it cannot capture spatial position information. Therefore, position encoding must be performed before transforming local features. This embodiment uses a 2D extended version of the absolute sine and cosine position encoding in DETR. The position encoding vector is given by the following formula:

[0154]

[0155] in, d is the number of channels to which position encoding features need to be added, i is the index of the feature channel, is the position encoding vector. In this way, the position encoding gives each element unique position information in a sine and cosine manner, thereby incorporating position information into the model input. This is crucial for producing accurate matches in textureless areas.

[0156] ②The Feature-Transformer module used performs feature transformation on the feature map that integrates the position information.

[0157] The original Transformer is composed of an encoder and a decoder. The Feature-Transformer module used in this embodiment is composed of multiple cascaded encoders. Each encoder structure is as follows: Figure 5 As shown. First, the encoder combines the input position encoding vector sequence with the weight matrix W q , W k , W v Multiply them to obtain the Q (Query), K (Key) and V (Value) matrices; secondly, input the Q, K and V matrices into the multi-head attention mechanism, extract the context information, and then pass the output of multiple heads through a linear layer and then normalize them; thirdly, concatenate the output of the normalization layer with the input of the encoder and pass it through a feedforward neural network for information fusion; finally, add the original input and the output of the feedforward neural network as the final result.

[0158] According to the different input features, the attention layer is divided into two types: self-attention layer and cross-attention layer. The input feature f processed by the self-attention layer i and f j From the same feature map, or The input features processed by the cross attention layer come from two different feature maps. and The self-attention layer and the cross-attention layer are interlaced to form a deep network structure.

[0159] Furthermore, in the above step S2.3.3, two local feature maps are obtained after Feature-Transformer transformation and Respectively and The feature map is calculated using formula (14) The center vector and feature maps The similarity s of all vectors in .

[0160]

[0161] The similarity s is converted into a probability distribution through the softmax function and expressed in the form of a two-dimensional image to obtain a heat map H(j) representing the matching probability.

[0162]

[0163] Where C is the number of feature map channels.

[0164] Furthermore, in step S2.3.4 above, the DSNT (Differentiable Spatial to Numerical Transform) algorithm is used to calculate the sub-pixel coordinates of each matching point in image B from the heat map H. DSNT first treats the heat map as a probability distribution and then calculates its spatial expectation to obtain the sub-pixel precision position.

[0165]

[0166] At the same time, the variance of the coordinates is calculated to estimate the uncertainty of the match for subsequent calculations:

[0167]

[0168] Furthermore, when performing model training on the deep learning-based feature point matching network model constructed in step S2, its loss function must first be determined.

[0169] In the present invention, the final loss consists of coarse loss and fine loss:

[0170]

[0171] Since coarse-level matching is applied to features that are 1 / 8 the size of the original image, each feature represents a pixel grid in the original image, so there may be one-to-many matches. This makes it difficult to determine the true value of the coarse-level matching. So we take the mutual nearest neighbors between the center positions of the 1 / 8 grid of the input image as the approximate true value. Specifically, take the center position of the 1 / 8 grid in the left image, project it to the same scale as the depth map, and index its depth. According to its depth value and the camera intrinsic parameters, calculate its three-dimensional coordinates in the camera coordinate system; according to the camera pose, convert it to the camera coordinate system of the second image, and finally reproject it into the pixel coordinate system. Among them, it is necessary to check whether the transformed point is within the image boundary and the depth consistency error is less than 0.2, and take the nearest neighbors of the points that meet the conditions as matching candidates. The same process is repeated from the right image to the left image. The two sets of nearest neighbor matches obtained are also filtered out according to the mutual nearest neighbor criterion to obtain the final true value. The true coarse-level match is recorded as Compute focal loss.

[0172] For positive samples, The position of , its loss is:

[0173]

[0174] For negative samples, The position of , its loss is:

[0175]

[0176] Among them, α is the coefficient for balancing the weights of positive and negative samples, and γ is the adjustment factor used to reduce the weights of easily classified samples. The final loss is:

[0177]

[0178] Here, |M pos | and |M neg | are the numbers of positive and negative samples respectively, which are used to standardize the loss. In this way, the loss of each part is averaged by their respective sample numbers, which helps to balance the influence of positive and negative samples, especially when their numbers are unbalanced.

[0179] In the fine-level loss, for each query point The total variance of the heatmap is calculated to represent its uncertainty. The reciprocal of the variance is used to weight the 2-norm of the error, aiming to optimize the positions with lower uncertainty. After averaging, the final weighted loss function is obtained:

[0180]

[0181] Among them, is the predicted point corresponding to the query point , is the point obtained by reprojection of according to the camera pose. is the total variance of the heatmap. M f represents the set of fine-grained matching points.

[0182] Furthermore, in step S3 above, since the present invention is a detector-free matching method and the expected value is calculated according to Equation (14) when refining the positions of feature points, it cannot guarantee the generation of consistent matching points between pictures, resulting in fragmentation of the feature trajectory. Therefore, the feature points need to be aggregated and redistributed before reconstruction.

[0183] Specifically, assume there is a set of image collections {I1, I2, … I N}. After feature point extraction and matching, the matching points of each image with all other images are obtained. For any two pictures I n and I m , the matching result can be expressed as a series of matching pairs M nm ={(k ni , k mj )}, where k ni is the i-th feature point in image I n , and k mj is the j-th feature point in image I m . Each feature point can be further expressed as two-dimensional coordinates (x ni,y ni ). For each image I n , initialize the empty aggregation feature point set K n ={}. Quantize the feature points: Divide the image into multiple cells of size Δx×Δy. Each cell is uniquely identified by an index pair (m,n), where m and n represent the grid index in the x and y directions, respectively. Based on the location of the feature points, assign them to each cell. The quantization function q is expressed as:

[0184]

[0185] For the quantized feature point q(k ni ), if in the aggregate feature point set K n If there is no same quantization position in ni ) to K n If the new feature point k ni Assigned to the existing feature point k′ ni The quantized position of k′ is calculated based on their scores (matching probability). ni Fine-tune the position of . This is achieved through weighted averaging:

[0186]

[0187] Among them, w ni and w′ ni They are k ni and k′ ni And update the existing score by adding the new matching score:

[0188] w′ ni =w′ ni +w ni (25)

[0189] After processing the matching of all image pairs, the optimized aggregate feature point set K of each image is obtained. n .

[0190] Based on the aggregated feature point set K n Redistribute the feature point matching results.

[0191] In this embodiment, for each matching pair (k ai ,k bj )∈M ab , use the nearest neighbor search (NN search) to find its position in the aggregated feature point set (k′ ai ,k′ bj ) and recorded as the new matching pair under the aggregated feature point set.

[0192] Through these two steps of aggregation and redistribution, we achieve the screening and optimization of feature points, reducing the possibility of mismatching and enhancing the robustness of feature point positions. Furthermore, this process extracts consistent matching information from multiple images, increasing the length of feature point trajectories and making subsequent reconstruction more stable.

[0193] Furthermore, in the above step S4, if Figure 6 As shown, the improved incremental reconstruction algorithm is used to perform 3D sparse reconstruction to obtain the internal structure information of the non-metallic pipe to be inspected, including the following steps:

[0194] S4.1 Select two images, use the PnP algorithm to obtain the initial camera pose, triangulate the images according to the initial camera pose to obtain 3D points as the initial point cloud model and use local BA optimization;

[0195] S4.2. Select the image with the most matching points in the point cloud model, use the PnP algorithm to obtain the camera pose of the new image, and triangulate it according to the camera pose to obtain new 3D points;

[0196] S4.3. Use the improved RANSAC algorithm to detect whether the point cloud model is cylindrical. If a cylinder is detected, a cylindrical constraint term is added to the local RA to optimize the 3D point position.

[0197] S4.4. Repeat steps S4.2 to S4.3 until the feature points of all images are added to the point cloud model.

[0198] The PnP algorithm and the image triangulation algorithm adopt technologies well known to those skilled in the art, which will not be elaborated in detail in the present invention.

[0199] Furthermore, in step S4.3 above, after adding a new 3D point to the point cloud model, the improved RANSAC algorithm is used to estimate the cylinder model parameters C. To achieve a more robust estimation model, the angle difference between the point normal direction and the cylinder axis direction is weighted when estimating the distance from the point cloud to the model.

[0200] First, for each point q in the point cloud i , search for the K nearest neighboring points and obtain the point cloud set P in the neighborhood = {p1,p2,…,p n}, where each The neighborhood center

[0201]

[0202] Decentralize P by subtracting the neighborhood center from each element in P:

[0203]

[0204] Calculate the covariance matrix C:

[0205]

[0206] Calculate the eigenvalues ​​λ of the covariance matrix i and the eigenvector v i (i=1,2,3):

[0207] Cv i =λ i v i (29)

[0208] Sort by eigenvalue size. Assuming λ1≥λ2≥λ3, the eigenvector corresponding to λ1 is point q. i Normal vector. Ideally, the distance from a point on the cylindrical surface to the cylindrical axis is radius r, and the normal vector of the point is perpendicular to the cylindrical axis. Therefore, the distance from the point to the estimated cylindrical model calculated using improved RANSAC can include these two items:

[0209] D total (p i , C)=|rD(p i ,C)|+W|n i ·d| (30)

[0210] Where r is the radius of the cylinder, D(p i ,C) is point p i Distance to the cylinder axis, n i is the point's normal vector, d is the cylinder's axis direction vector, and W is a weight coefficient used to balance the distance from the point to the cylinder axis and the perpendicularity of the point normal to the axis normal. In practice, cylinder estimation is not performed at the beginning of reconstruction, but after a certain number of 3D points have been reconstructed. This is because a small number of sample points can lead to large estimation errors. Once the cylinder is detected, a cylinder constraint can be added to the bundle adjustment:

[0211] E(X, P, K, C) = E rep (X,P,K)+αE cyl (X, C) (31)

[0212] Among them, the first item E rep (X, P, K) is the reprojection error term, and the second term E cyl (X, C) Cylindrical constraint term, where X, P, K, and C represent the 3D point set, image pose, camera intrinsics, and detected cylinder parameters, respectively. α is the weight that controls the two error terms. The cylindrical constraint term calculates the distance from the 3D point to the cylindrical surface:

[0213]

[0214] Where d(X j ,C) is a 3D point X j Distance function to the cylindrical surface.

[0215] Figure 7 This is the flow chart of the improved RANSAC algorithm used. First, select the minimum required data set S min To estimate the model parameters. Assume that the current optimal solution is found in a certain iteration. In order to further improve the accuracy of the solution, a local optimization strategy is introduced on this basis, that is, a local iteration number k is set. During this process, an additional sample point p is selected from the internal point set I and added to S min An extended dataset S is formed exp Use S exp Recalculate the model parameters and evaluate the number of inliers in the new model. If the number of inliers in the expanded model exceeds that of the current optimal solution, accept the new model as the optimal solution and update the current optimal state. If not, reject the sample point and try another one from I. This process is repeated for k local iterations, attempting to find a more accurate model solution by adding sample points.

[0216] After the above steps, the present invention finally obtains a three-dimensional sparse point cloud image inside the pipeline, which can be used for subsequent dense reconstruction and texture mapping. Figure 8 、 Figure 9 and Figure 10 The three-dimensional sparse reconstructed point cloud image of the inner wall of the pipeline is obtained by the method of the present invention. Figure 8 This is the reconstruction effect diagram of the cone point defect. Figure 9 This is the reconstruction effect diagram of the slot defect. Figure 10 This is the reconstruction effect of the long strip defect.

[0217] Example 2

[0218] The above-mentioned embodiment 1 provides a method for 3D sparse reconstruction of monocular images of non-metallic pipelines for deep-sea mining. Correspondingly, this embodiment provides a device for 3D sparse reconstruction of monocular images of non-metallic pipelines for deep-sea mining. The device provided in this embodiment can implement the method for 3D sparse reconstruction of monocular images of non-metallic pipelines for deep-sea mining in embodiment 1. The device can be implemented by software, hardware, or a combination of software and hardware. For example, the device can include integrated or separate functional modules or functional units to perform the corresponding steps in each method of embodiment 1. Since the device of this embodiment is basically similar to the method embodiment, the process described in this embodiment is relatively simple. For relevant matters, please refer to the partial description of embodiment 1. The embodiment of the device provided in this embodiment is merely illustrative.

[0219] This embodiment provides a device for 3D sparse reconstruction of monocular images of non-metallic pipelines for deep-sea mining, comprising:

[0220] The data acquisition module is used to use a robot equipped with a monocular camera to obtain internal images of the non-metallic pipe to be inspected to form a data set;

[0221] Feature point matching module, which is used to detect and match feature points of the dataset using a feature point matching network model based on deep learning;

[0222] Aggregation and redistribution module, used to aggregate and redistribute the obtained feature points;

[0223] The three-dimensional coefficient reconstruction module is used to perform three-dimensional sparse reconstruction using the improved incremental reconstruction algorithm to obtain the internal structure information of the non-metallic pipe to be inspected.

[0224] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A method for 3D sparse reconstruction of monocular images of non-metallic pipelines in deep-sea mining, characterized by: The following steps are involved: A robot equipped with a monocular camera is used to obtain internal images of non-metallic pipes to be inspected to form a dataset. Use a deep learning-based feature point matching network model to detect and match feature points in the dataset; Aggregate and redistribute the obtained feature points; Based on the aggregated feature points, an improved incremental reconstruction algorithm is used to perform 3D sparse reconstruction to obtain the internal structure information of the non-metallic pipe to be inspected. The method of using a deep learning-based feature point matching network model to detect and match feature points on a data set includes: Use the feature pyramid network to extract features of different scales from the input image, and obtain feature maps of 1 / 2 scale and 1 / 8 scale; Based on the 1 / 8 scale feature map, a coarse-grained matching model is used to obtain a rough matching area; Based on the 1 / 2 scale feature map and the rough matching area, a fine-grained matching model is used to obtain the precise location of the matching feature points. The method of performing three-dimensional sparse reconstruction using the improved incremental reconstruction algorithm to obtain internal structural information of the non-metallic pipe to be inspected includes the following steps: ① Select two pictures, use the PnP algorithm to obtain the initial camera pose, triangulate the image according to the initial camera pose to obtain 3D points as the initial point cloud model and use local BA optimization; ② Select the image with the most matching points in the point cloud model, use the PnP algorithm to obtain the camera pose of the new image, and triangulate it according to the camera pose to obtain the new 3D points; ③ Use the improved RANSAC algorithm to detect whether the point cloud model is cylindrical. If a cylinder is detected, a cylindrical constraint term is added to the local BA to optimize the 3D point position. ④ Repeat steps ② to ③ until all feature points of the images are added to the point cloud model.

2. The method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining according to claim 1, characterized in that: The data set is formed by using a robot equipped with a monocular camera to obtain internal images of the non-metallic pipe to be inspected, including: Determine the robot structure; Among them, the robot includes a camera protection cover, a camera compartment, several odometer wheels, a base and support wheels; the lower part of the base is fixedly connected to a base plate through a flange, and three support wheels are provided at the lower part of the base plate to ensure that the center line of the robot is collinear with the center line of the pipeline; the upper part of the base is connected to the camera compartment through another flange, and a spring is provided between the camera compartment and the other flange; the camera protection cover is provided on the top of the camera compartment to protect the monocular camera placed in the camera compartment; the odometer wheels are arranged at intervals outside the camera compartment, and the odometer wheels have a built-in magnetic encoder for measuring the rotation angle of the odometer wheels; a circuit system is placed inside the base to receive the pulse signal sent by the magnetic encoder and calculate the robot displacement data according to the odometer wheel parameters, and control the monocular camera to take pictures of the inside of the pipeline according to the displacement data; A monocular camera is mounted on the robot and calibrated using Zhang's calibration method to obtain the camera's intrinsic parameters. The robot is placed inside the non-metallic pipe to be inspected and controlled to move forward at a preset speed. At the same time, a monocular camera is used to capture images of the interior of the non-metallic pipe to be inspected to form a data set.

3. The method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining according to claim 2, characterized in that: The robot is equipped with a monocular camera, and the monocular camera is calibrated using the Zhang calibration method to obtain the camera's intrinsic parameters, including: Fix the preset black and white checkerboard calibration paper on the table; Use the robot's monocular camera to capture multiple images of the calibration paper from different angles and positions, ensuring that the checkerboard pattern is in view in each image. The monocular camera's pose and position should vary enough during the capture to cover different viewpoints. Based on the image information of the obtained calibration paper, the Zhang calibration method is used to calculate the intrinsic parameters of the monocular camera.

4. The method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining according to claim 2, characterized in that: The robot is placed inside the non-metallic pipe to be inspected and controlled to move forward at a preset speed. A monocular camera is used to capture images of the interior of the non-metallic pipe to be inspected to form a data set, including: Place the robot equipped with a monocular camera at the entrance of the non-metallic pipe to be inspected and reset the odometer data. The robot is controlled to move forward in the non-metallic pipe to be inspected. At the same time, the circuit system in the base obtains the displacement information of the robot in the non-metallic pipe to be inspected through the odometer. Based on the displacement information, the monocular camera is controlled to take pictures of the inside of the non-metallic pipe to be inspected and the pictures are saved. The robot stops after reaching the end point, uses the host computer to read all the pictures, and creates a data set of pictures of the interior of the non-metallic pipe to be inspected according to the order in which the pictures were taken.

5. The method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining according to claim 1, characterized in that: The feature pyramid network is used to extract features of different scales from the input image to obtain a 1 / 2 scale feature map and a 1 / 8 scale feature map, including: Perform feature extraction on the two input images, obtain downsampled feature maps of 1 / 2 scale and 1 / 4 scale in turn, and output a feature map of 1 / 8 scale; The bilinear interpolation method is used to upsample the downsampled feature maps of different scales obtained in the feature extraction stage, and the upsampled feature maps of 1 / 4 scale and 1 / 2 scale are obtained in turn; The 1 / 2 scale upsampled feature map is fused with the 1 / 2 scale downsampled feature map to obtain the final 1 / 2 scale feature map.

6. The method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining according to claim 1, characterized in that: The method of obtaining a rough matching area based on the 1 / 8 scale feature map using a coarse-grained matching model includes the following steps: Perform dimension conversion on the coarse-level features of the two images at 1 / 8 scale to obtain the score matrix ; Apply the two dimensions of the score matrix separately Operator, converts the score matrix into probability and obtains the confidence matrix ; Based on the confidence matrix , for confidence levels below the threshold The matches are filtered out and the mutual neighbor standard is applied to the remaining matches to obtain a rough matching area. .

7. The method for 3D sparse reconstruction of a monocular image of a non-metallic pipeline for deep-sea mining according to claim 6, characterized in that: The method of obtaining the precise position of the matching feature points using a fine-grained matching model based on the 1 / 2 scale feature map and the coarse matching area includes the following steps: Roughly match the area Positioning on fine-level feature maps and In the local neighborhood ; Cut the two groups to size Local features and For local neighborhood Perform Feature-Transformer transformation to obtain two local feature maps and ; Using similarity calculation and The function obtains a heat map of the matching probability between two feature maps; The DSNT algorithm is used to calculate the exact location of the matching points from the heat map.

8. A 3D sparse reconstruction device for monocular images of non-metallic pipelines in deep-sea mining, characterized by: include: The data acquisition module is used to use a robot equipped with a monocular camera to obtain internal images of the non-metallic pipe to be inspected to form a data set; Feature point matching module, which is used to detect and match feature points of the dataset using a feature point matching network model based on deep learning; Aggregation and redistribution module, used to aggregate and redistribute the obtained feature points; The 3D coefficient reconstruction module is used to perform 3D sparse reconstruction based on the aggregated feature points using an improved incremental reconstruction algorithm to obtain the internal structure information of the non-metallic pipe to be inspected; The method of using a deep learning-based feature point matching network model to detect and match feature points on a data set includes: Use the feature pyramid network to extract features of different scales from the input image, and obtain feature maps of 1 / 2 scale and 1 / 8 scale; Based on the 1 / 8 scale feature map, a coarse-grained matching model is used to obtain a rough matching area; Based on the 1 / 2 scale feature map and the rough matching area, a fine-grained matching model is used to obtain the precise location of the matching feature points. The method of performing three-dimensional sparse reconstruction using the improved incremental reconstruction algorithm to obtain internal structural information of the non-metallic pipe to be inspected includes the following steps: ① Select two pictures, use the PnP algorithm to obtain the initial camera pose, triangulate the image according to the initial camera pose to obtain 3D points as the initial point cloud model and use local BA optimization; ② Select the image with the most matching points in the point cloud model, use the PnP algorithm to obtain the camera pose of the new image, and triangulate it according to the camera pose to obtain the new 3D points; ③ Use the improved RANSAC algorithm to detect whether the point cloud model is cylindrical. If a cylinder is detected, a cylindrical constraint term is added to the local BA to optimize the 3D point position. ④ Repeat steps ② to ③ until all feature points of the images are added to the point cloud model.

Images