Method for Detecting Foreign Objects in Pipeline Based on Curvature Mutation of 3D Point Cloud

Abstract

The present invention relates to a method for detecting foreign objects in pipelines based on the sudden change of the curvature of three-dimensional point clouds. First, the internal and external parameter matrices are obtained through camera parameter calibration. Secondly, according to the characteristics of the change of the internal cross-section of the pipeline, the pipeline to be detected is classified and segmented. Visual servo control is used to align the depth camera with the inner surface of each section of the pipeline, and the original point cloud of the inner surface of this section of the pipeline is collected. Further, the original point cloud is preprocessed by a filtering and noise reduction method, and a voxelized point cloud optimization method is used to compress the smoothed point cloud after preprocessing. Again, the dynamic curvature threshold segmentation method is used to extract the foreign object point data in the compressed point cloud to obtain a foreign object point set, and the region growing method is used for clustering and segmentation of the foreign objects to obtain the foreign object point cloud of each foreign object. Furthermore, the centroid coordinates of each group of foreign object point clouds are calculated to determine the position of each foreign object. This method does not rely on the prior information of foreign objects and can achieve the efficient detection, accurate identification and precise positioning of different foreign objects in various pipelines.

Description

Technical Field

[0001] The present invention relates to the field of foreign object detection, and particularly to a pipeline foreign object detection method based on the curvature mutation of three-dimensional point clouds. Background Art

[0002] Pipeline foreign objects pose numerous challenges such as diverse types, uncertain positions, and complex feature models. However, the method of relying on manual inspection takes a long time and it is difficult to detect hidden foreign objects. The present invention solves the problems of long required time and low accuracy in manual detection by introducing the method of three-dimensional point cloud curvature mutation, and expands the methods and technical means for pipeline foreign object detection.

[0003] Compared with the prior art, the present invention has the following advantages: 1) By analyzing the spatial features of point cloud data, the recognition ability of foreign objects is improved; 2) Through the point cloud data compression technology, the computational amount of data processing is reduced, and the detection efficiency is improved; 3) The application of the dynamic threshold method and the region growing method enhances the extraction and segmentation ability of foreign object point clouds; 4) This method is applicable to various foreign object types and complex environments.

[0004] Currently, the common pipeline foreign object detection method is the method based on visual object detection. This method requires a large amount of cumbersome training and the arrangement of professional-level equipment, and there are limitations on foreign object types in the application field. For example, the Chinese invention patent with the publication number CN114898079A requires the arrangement of detection equipment and a server, and the determination and training of foreign object types. Summary of the Invention

[0005] The summary of the invention mainly aims at the foreign object detection requirements of pipelines, and proposes a pipeline foreign object detection method based on the curvature mutation of three-dimensional point clouds, aiming to overcome the above problems and limitations existing in the prior art.

[0006] The present invention is realized through the following technical solutions:

[0007] A pipeline foreign object detection method based on the curvature mutation of three-dimensional point clouds according to the present invention is characterized by including the steps:

[0008] Step 1: Camera parameter calibration: Install the depth camera in the normal direction at the end of the inspection robotic arm, and use the Zhang Zhengyou calibration method to calculate the internal parameter matrix, distortion parameters, and external parameter matrix of the depth camera;

[0009] Step 2: Pipeline cross-section classification: According to the cross-section change characteristics of the theoretical pipeline model, apply the end cross-section classification method to classify and segment the theoretical pipeline model along the axis direction, and perform camera pose alignment and point cloud acquisition processing on each type and each segment of the pipeline respectively;

[0010] Step 3: Camera pose alignment: Use the visual servo control method to align the end of the inspection robotic arm with the central axis of each section of the pipeline, align the Z-axis of the depth camera coordinate system with the normal direction of the surface to be measured of this section of the pipeline, and ensure that the X-axis of the depth camera coordinate system is aligned with the axis direction of this section of the pipeline;

[0011] Step 4: Acquisition of raw point cloud: Capture the depth image of the surface to be measured of this section of the pipeline through the depth camera and convert it into raw point cloud data;

[0012] Step 5: Preprocessing of raw point cloud: Use the filtering and noise reduction method to preprocess the raw point cloud to obtain a smooth point cloud with a data volume of N;

[0013] Step 6: Compression of smooth point cloud: Compress the preprocessed smooth point cloud, and use the voxelized point cloud optimization method to reduce the data volume while retaining the main shape features to obtain a compressed point cloud;

[0014] Step 7: Extraction of foreign object point set: For the compressed point cloud, use the dynamic curvature threshold segmentation method to extract the foreign object point data on the inner surface of the pipeline to obtain a foreign object point set;

[0015] Step 8: Clustering segmentation of foreign object point cloud: For the foreign object point set, use the region growing method to perform clustering segmentation of foreign objects to obtain the foreign object point cloud of each foreign object;

[0016] Step 9: Estimation of foreign object position: For each foreign object point cloud, use the centroid method in the camera coordinate system to calculate the camera coordinates of the centroid of the foreign object, and obtain its foreign object position in the world coordinates through the coordinate transformation method.

[0017] Preferably, in step 2, the end cross-section classification method includes the following steps:

[0018] Step 2.1: Interception of pipeline cross-section: In the pipeline axis direction, intercept the pipeline successively with planes perpendicular to the pipeline axis at a step size of t c to obtain a set of pipeline cross-sections;

[0019] Step 2.2: Classification of pipeline segments: Select consecutive pipeline cross-sections that are adjacent in position and have the same cross-sectional shape in sequence as a section of type I pipeline, and all type I pipelines form a type I pipeline set; Select consecutive pipeline cross-sections that are adjacent in position and have a cross-sectional shape that changes according to a fixed rule in sequence as a section of type II pipeline, and all type II pipelines form a type II pipeline set;

[0020] Step 2.3: Processing of pipeline segments: Extract each type I pipeline and type II pipeline in sequence along the pipeline axis direction, and perform camera pose alignment and point cloud acquisition processing respectively.

[0021] Preferably, in step 5, the filtering and noise reduction method includes direct filtering and Gaussian filtering:

[0022] The direct filtering is to remove noise data points through a distance threshold h; for a polygonal cross-section pipeline, dist where L1 is the normal distance of the cross-section to be detected in the polygonal cross-section opposite to the Z-axis of the depth camera coordinate system, and M4 is the change threshold of the normal distance; for a circular pipeline, where d1 is the diameter of the circular pipeline and M5 is the change threshold of the diameter; for a non-circular curved surface pipeline, where d2 is the average normal distance of the neighborhood of the non-circular curved surface opposite to the Z-axis of the depth camera coordinate system, and M6 is the change threshold of the average normal distance; assuming that the coordinate of the data point collected by the depth camera is p =(x i ,y i ,z i ), when the normal distance h i between the data point and the depth camera i is greater than the threshold h dist , then the data point is considered a noise point and removed;

[0023] The Gaussian filtering includes the following steps:

[0024] Step 5.1: Divide the XOZ cross-section: Along the Y-axis direction of the depth camera coordinate system, divide the XOZ cross-section in sequence with a step size of t p ;

[0025] Step 5.2: Axial data sorting: Select all data points in the same XOZ cross-section, and sort the data points in sequence in the pipeline axis direction, i.e., the X-axis direction of the depth camera coordinate system, to generate a sorted point cloud;

[0026] Step 5.3: Neighborhood point division: For the i-th data point in the sorted point cloud, find neighborhood points with a quantity not exceeding U in the neighborhood δ x , and calculate the filtering weight G σ (j) corresponding to the j-th neighborhood point according to the axial distance between the j-th neighborhood point and this point using a Gaussian function:

[0027]

[0028] where σ is the weight coefficient of the Gaussian function;

[0029] Step 5.4: Gaussian smoothing: Perform weighted average on the z j coordinates of all neighborhood points as the z i coordinate of the i-th data point in the sorted point cloud:

[0030] z i =∑ U z j G σ (j), U:|x j-x i |≤δ x (2).

[0031] Preferably, the voxelized point cloud optimization method in step 6 includes the following steps:

[0032] Step 6.1: Parameter initialization: According to the intrinsic parameters of the depth camera, set the maximum horizontal field of view angle T row and the maximum vertical field of view angle T col , and initialize the angular resolution t angle , the upper threshold M of the point cloud density h , the lower threshold M of the point cloud density d , and the angular resolution step t s ;

[0033] Step 6.2: Voxel grid division: Calculate the number of rows n angle计 and the number of columns n row of the voxel grid division according to the current angular resolution t col :

[0034] n row =T row / t angle (3)

[0035] n col =T col / t angle (4);

[0036] Step 6.3: Point cloud - voxel network mapping: For each data point p i =(x i ,y i ,z i ) in the smoothed point cloud, calculate the position of this point in the voxel grid according to the current angular resolution t angle :

[0037] angle row =tan -1 (x i / z i ) (5)

[0038] angle col =tan -1 (y i / z i ) (6)

[0039] row=(n row / 2)+(angle row / t angle ) (7)

[0040] col=(ncol / 2)+(angle col / t angle ) (8)

[0041] where angle row and angle col are the row resolution angle and the column resolution angle respectively, and row and col are the row number and column number of this point in the voxel grid;

[0042] Step 6.4: Average point cloud density iteration: Calculate the average point cloud density ρ = N / m of the point cloud - voxel grid according to the current angular resolution t angle , where N is the data volume of the smoothed point cloud and m is the number of non - empty voxel grids; if the average point cloud density is greater than the upper threshold M h of the point cloud density, then reduce the angular resolution t angle by one angular resolution step t s , and return to Step 6.2; if the average point cloud density is less than the lower threshold M d of the point cloud density, then increase the angular resolution t angle by one angular resolution step t s , and return to Step 6.2; otherwise, enter Step 6.5;

[0043] Step 6.5: Point cloud data compression: Divide all data points into each voxel grid, calculate the center point of all data points in each voxel grid, and use this center point as the data point of this voxel grid after data compression:

[0044]

[0045] where P(x, y, z) is the center point of each voxel grid after point cloud compression, and p i (x i , y i , z i ) are all data points in each voxel grid before point cloud compression.

[0046] Preferably, the dynamic curvature threshold segmentation method in Step 7 includes the following steps:

[0047] Step 7.1: Compressed point cloud sorting: Sort the compressed point cloud in the direction of the pipeline axis, i.e., the X - axis direction of the depth camera coordinate system: Determine the layer number of the voxel grid where the data point is located according to the Y - axis coordinate of the data point, project the compressed point cloud in this layer of voxel grid along the pipeline axis direction onto the XOZ cross - section at the corresponding position on the Y - axis, and sort all data points in each XOZ cross - section according to the X - axis coordinate;

[0048] Step 7.2: Z - axis curvature value calculation: For each XOZ cross - section, calculate the data point P iCurvature value Z caused by the change in Z-axis height cur (i):

[0049]

[0050] Z cur (i) = abs(diffz i × 10) (11)

[0051] where z i is the Z-axis coordinate value of the current point P i and z n is the Z-axis coordinate value of the neighboring point of P i and diffz i is the curvature dissimilation value;

[0052] Step 7.3: Initialization of segmentation threshold: Set the threshold error ΔT as the termination condition for dynamic threshold iteration, and calculate the average curvature of all points in the compressed point cloud as the initial threshold T0:

[0053]

[0054] Step 7.4: Classification of point cloud with dynamic threshold: Divide the compressed point cloud into two subsets G1 and G2 according to the current threshold T0. G1 contains all data points P j with curvature less than or equal to T0, and the number of data points is n1. G2 contains all data points P k with curvature greater than T0, and the number of data points is n2. Calculate the average curvature μ1 of all data points in subset G1 and the average curvature μ2 of all data points in subset G2 respectively:

[0055]

[0056]

[0057] Step 7.5: Update of segmentation threshold: Calculate the updated value T1 of the segmentation threshold according to the average curvatures μ1 and μ2 of all data points in subsets G1 and G2:

[0058]

[0059] Step 7.6: Dynamic threshold iteration: If the difference between the current threshold T0 and the updated threshold value T1 is less than the set threshold error ΔT, go to Step 7.7; otherwise, update T0 with the value of T1 and return to Step 7.4;

[0060] Step 7.7: Threshold segmentation of foreign object point cloud: Use the current threshold T0 as the segmentation threshold h cur and perform threshold segmentation on all data points in the compressed point cloud, and for the curvature value Z cur (i) exceeding the segmentation threshold hcur The data points are regarded as foreign object points, and all foreign object points are used as the foreign object point set.

[0061] Preferably, the region growing method in the foreign object point cloud clustering and segmentation in step 8 includes the following steps:

[0062] Step 8.1: Seed point selection: Determine whether there are unvisited foreign object points in the foreign object point set. If so, randomly select an unvisited foreign object point from the foreign object point set as the seed point of the new foreign object; if there are no unvisited foreign object points, the method ends.

[0063] Step 8.2: Seed point neighborhood search: Search whether there are other unvisited foreign object points within the neighborhood range of the current seed point. If so, add them to the foreign object point cloud of the foreign object to which the current seed point belongs, mark them as visited, and continue to perform iterative search with the current neighborhood point as the new seed point.

[0064] Step 8.3: Neighborhood search termination: For the foreign object point cloud to which the current belongs, if there are no other unvisited foreign object points in the neighborhood of each seed point, the current foreign object point cloud has been clustered and segmented, and the foreign object point cloud of the current cluster is obtained, and return to step 8.1.

[0065] Preferably, the foreign object position estimation in step 9 includes the following content:

[0066] The centroid method includes: For each foreign object point cloud, calculate the centroid coordinates according to the coordinate positions of all data points of the foreign object in the camera coordinate system as the camera coordinate position of the foreign object:

[0067]

[0068] Among them, P(x, y, z) is the centroid coordinate of the foreign object point cloud, and p i (x i , y i , z i ) are the camera coordinates of all data points in each foreign object point cloud;

[0069] The coordinate transformation method includes: Using the external parameter matrix of the depth camera, transform the position of each foreign object from the camera coordinate system to the world coordinate system to obtain the world coordinate position of each foreign object:

[0070]

[0071] Among them, R is the rotation transformation matrix and T is the translation transformation matrix.

[0072] Advantages of the present invention:

[0073] The method for detecting foreign objects in pipelines based on the curvature mutation of 3D point clouds improves the stability and accuracy of the foreign object detection system by optimizing the pipeline feature model, point cloud acquisition and processing, and the process of obtaining the position of foreign objects, and has broad market application prospects and technical promotion value. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] The present invention will be further described in detail below with reference to the accompanying drawings:

[0075] Figure 1 is a flowchart of the method for detecting foreign objects in pipelines based on the curvature mutation of 3D point clouds according to the present invention;

[0076] Figure 2 is a schematic diagram of the voxelized point cloud optimization method according to the present invention;

[0077] Figure 3 is a schematic diagram of the region growing method according to the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0078] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0079] Figure 1 is a flowchart of the method for detecting foreign objects in pipelines based on the curvature mutation of 3D point clouds according to the present invention. It is characterized by including camera parameter calibration, pipeline cross-section classification, camera pose alignment, original point cloud acquisition, original point cloud preprocessing, smooth point cloud compression, foreign object point set extraction, foreign object point cloud clustering and segmentation, and foreign object position estimation.

[0080] The method for detecting foreign objects in pipelines based on the curvature mutation of 3D point clouds specifically includes the following steps:

[0081] Step 1: Camera parameter calibration: Install the depth camera in the normal direction at the end of the inspection robotic arm, and use the Zhang Zhengyou calibration method to calculate the internal parameter matrix, distortion parameters, and external parameter matrix of the depth camera;

[0082] Step 2: Pipeline cross-section classification: According to the cross-section change characteristics of the theoretical pipeline model, apply the end cross-section classification method to classify and segment the theoretical pipeline model along the axis direction, and perform camera pose alignment and point cloud acquisition and processing on each class and each segment of the pipeline respectively;

[0083] Step 3: Camera pose alignment: Use the visual servo control method to align the end of the inspection robotic arm with the central axis of each section of the pipeline, align the Z-axis of the depth camera coordinate system with the normal direction of the surface to be measured of this section of the pipeline, and ensure that the X-axis of the depth camera coordinate system is aligned with the axis direction of this section of the pipeline;

[0084] Step 4: Acquisition of original point cloud: Capture the depth image of the surface to be measured of this section of the pipeline through the depth camera and convert it into original point cloud data;

[0085] Step 5: Preprocessing of the original point cloud: Use the filtering and noise reduction method to preprocess the original point cloud to obtain a smooth point cloud with a data volume of N;

[0086] Step 6: Compression of the smooth point cloud: Compress the preprocessed smooth point cloud, and use the voxelized point cloud optimization method to reduce the data volume while retaining the main shape features to obtain a compressed point cloud;

[0087] Step 7: Extraction of the foreign object point set: For the compressed point cloud, use the dynamic curvature threshold segmentation method to extract the foreign object point data on the inner surface of the pipeline to obtain a foreign object point set;

[0088] Step 8: Clustering segmentation of the foreign object point cloud: For the foreign object point set, use the region growing method to perform clustering segmentation of the foreign objects to obtain the foreign object point cloud of each foreign object;

[0089] Step 9: Estimation of the foreign object position: For each foreign object point cloud, use the centroid method to calculate the camera coordinates of the centroid of the foreign object in the camera coordinate system, and obtain its foreign object position in the world coordinates through the coordinate transformation method.

[0090] Calibrate the parameters of the camera. Install the depth camera in the normal direction at the end of the inspection robotic arm, and use the Zhang Zhengyou calibration method to calculate the internal parameter matrix, distortion parameters, and external parameter matrix of the depth camera.

[0091] Perform sectional classification on the pipeline: According to the characteristics of the change in the inner cross-section of the theoretical pipeline model, apply the end cross-section classification method to classify and segment the theoretical pipeline model along the axis direction, and perform camera pose alignment and point cloud acquisition processing on each type and each section of the pipeline respectively. The end cross-section classification method includes the following steps:

[0092] 1) Pipeline cross-section interception: In the pipeline axis direction, intercept the pipeline successively with planes perpendicular to the pipeline axis at a step size of t c to obtain a set of pipeline cross-sections;

[0093] 2) Pipeline sectional classification: Select consecutive pipeline cross-sections that are adjacent in position and have the same cross-section shape in sequence as a section of type I pipeline, and all type I pipelines form a type I pipeline set; Select consecutive pipeline cross-sections that are adjacent in position and have a cross-section shape that changes according to a fixed rule in sequence as a section of type II pipeline, and all type II pipelines form a type II pipeline set;

[0094] 3) Pipeline segmentation processing: Extract each section of type-I pipeline and type-II pipeline in sequence along the pipeline axis direction, and perform camera pose alignment and point cloud acquisition processing respectively.

[0095] Align the pose of the camera. Adopt the visual servo control method to align the end of the inspection robotic arm with the central axis of the pipeline, align the Z-axis of the depth camera coordinate system with the normal direction of the surface to be measured of this section of the pipeline, and ensure that the X-axis direction of the depth camera coordinate system is aligned with the pipeline axis direction.

[0096] Collect the original point cloud of the pipeline. Capture the depth image of the inner surface of the pipeline through the depth camera and convert it into the original point cloud data.

[0097] Preprocess the original point cloud by using the filtering and noise reduction method to obtain a smooth point cloud with a data volume of N. The filtering and noise reduction method includes direct filtering and Gaussian filtering to eliminate noise points.

[0098] The direct filtering is to remove the noise data points through the distance threshold h dist ; for the pipeline with a polygonal cross-section, where L1 is the normal distance of the cross-section to be detected in the polygon cross-section corresponding to the Z-axis of the depth camera coordinate system, and M4 is the variation threshold of the normal distance; for the circular pipeline, where d1 is the diameter of the circular pipeline, and M5 is the diameter variation threshold; for the non-circular curved surface pipeline, where d2 is the average normal distance of the neighborhood of the non-circular curved surface corresponding to the Z-axis of the depth camera coordinate system, and M6 is the variation threshold of the average normal distance; assume that the coordinate of the data point collected by the depth camera is p i =(x i , y i , z i ), when the normal distance h i between the data point and the depth camera is greater than the threshold h dist , then this data point is considered a noise point and is removed;

[0099] The Gaussian filtering includes the following steps:

[0100] 1) Divide the XOZ section: Along the Y-axis direction of the depth camera coordinate system, divide the XOZ section in sequence at the step size t p ;

[0101] 2) Axial data sorting: Select all the data points in the same XOZ section, and sort the data points in sequence in the pipeline axis direction, that is, the X-axis direction of the depth camera coordinate system, to generate the sorted point cloud;

[0102] 3) Neighborhood point division: For the i-th data point in the sorted point cloud, in the neighborhood δ xFind neighborhood points with a quantity not exceeding U. According to the axial distance between the j-th neighborhood point and this point, use the Gaussian function to calculate the filtering weight G corresponding to the j-th neighborhood point σ (j):

[0103]

[0104] where σ is the weight coefficient of the Gaussian function;

[0105] 4) Gaussian smoothing: Perform weighted averaging on the z j coordinates of all neighborhood points as the z i coordinate of the i-th data point in the sorted point cloud:

[0106] z i = ∑ U z j G σ (j), U: |x j - x i | ≤ δ x (2).

[0107] Compress the preprocessed smoothed point cloud. Use the voxelized point cloud optimization method to reduce the data volume while retaining the main shape features to obtain the compressed point cloud. As Figure 2 shown, the voxelized point cloud optimization method includes the following steps:

[0108] 1) Parameter initialization: According to the intrinsic parameters of the depth camera, set the maximum horizontal field of view angle T row 、the maximum vertical field of view angle T col , and initialize the angular resolution t angle , the upper threshold M h of the point cloud density, the lower threshold M d of the point cloud density, and the angular resolution step t s ;

[0109] 2) Voxel grid division: Calculate the number of rows n angle and the number of columns n row of the voxel grid division according to the current angular resolution t col :

[0110] n row = T row / t angle (3)

[0111] n col = T col / t angle (4);

[0112] 3) Point cloud - voxel network mapping: For each data point p i = (xi , y i , z i ), according to the current angular resolution t angle Calculate the position of this point in the voxel grid:

[0113] angle row = tan -1 (x i / z i ) (5)

[0114] angle col = tan -1 (y i / z i ) (6)

[0115] row = (n row / 2)+(angle row / t angle ) (7)

[0116] col = (n col / 2)+(angle col / t angle ) (8)

[0117] where angle row and angle col are the row resolution angle and column resolution angle respectively, and row and col are the row number and column number of this point in the voxel grid;

[0118] 4) Average point cloud density iteration: According to the current angular resolution t angle Calculate the average point cloud density ρ = N / m of the point cloud - voxel grid, where N is the data volume of the smoothed point cloud and m is the number of non - empty voxel grids; if the average point cloud density is greater than the upper threshold M h of the point cloud density, then reduce the angular resolution t angle by one angular resolution step t s , and return to step 2); if the average point cloud density is less than the lower threshold M d of the point cloud density, then increase the angular resolution t angle by one angular resolution step t s , and return to step 2); otherwise, go to step 5);

[0119] 5) Point cloud data compression: Divide all data points into each voxel grid, calculate the center point of all data points in each voxel grid, and use this center point as the data point of this voxel grid after data compression:

[0120]

[0121] Among them, P(x, y, z) is the center point of each voxel grid after point cloud compression, and p i (x i , y i , z i ) are all data points in each voxel grid before point cloud compression.

[0122] For the compressed point cloud, the dynamic curvature threshold segmentation method is used to extract the foreign object point data in the pipeline, and a foreign object point set is obtained. The dynamic curvature threshold segmentation method includes the following steps:

[0123] 1) Sorting of compressed point cloud: Sort the compressed point cloud along the pipeline axis direction, i.e., the X-axis direction of the depth camera coordinate system: Determine the number of voxel grid layers where the data point is located according to the Y-axis coordinate of the data point, project the compressed point cloud in this layer of voxel grid along the pipeline axis direction onto the XOZ cross-section at the corresponding position of the Y-axis, and sort all data points in each XOZ cross-section according to the X-axis coordinate;

[0124] 2) Calculation of Z-axis curvature value: For each XOZ cross-section, calculate the curvature value Z i caused by the change in Z-axis height of the data point P cur (i):

[0125]

[0126] Z cur (i) = abs(diffz i × 10) (11)

[0127] where z i is the Z-axis coordinate value of the current point P i , z n is the Z-axis coordinate value of the neighboring point of P i , and diffz i is the curvature alienation value;

[0128] 3) Initialization of segmentation threshold: Set the threshold error ΔT as the termination condition for dynamic threshold iteration, and calculate the average curvature of all points in the compressed point cloud as the initial threshold T0:

[0129]

[0130] 4) Classification of dynamic threshold point cloud: Divide the compressed point cloud into two subsets G1 and G2 according to the current threshold T0. G1 contains all data points P j whose curvature is less than or equal to T0, and the number of data points is n1. G2 contains all data points P k whose curvature is greater than T0, and the number of data points is n2; Calculate the average curvature μ1 of all data points in the G1 subset and the average curvature μ2 of all data points in the G2 subset respectively:

[0131]

[0132]

[0133] 5) Split threshold update: Calculate the updated value T1 of the split threshold according to the average curvatures μ1 and μ2 of all data points in the two subsets G1 and G2:

[0134]

[0135] 6) Dynamic threshold iteration: If the difference between the current threshold T0 and the threshold update value T1 is less than the set threshold error ΔT, go to step 7); otherwise, update T0 with the value of T1 and return to step 4);

[0136] 7) Foreign object point cloud threshold segmentation: Use the current threshold T0 as the segmentation threshold h cur and perform threshold segmentation on all data points in the compressed point cloud. For the curvature value Z cur (i) The data points exceeding the segmentation threshold h cur are regarded as foreign objects, and all foreign objects are used as the foreign object set.

[0137] For the foreign object set, use the region growing method to perform clustering segmentation of foreign objects to obtain the foreign object point cloud of each foreign object; as Figure 3 shown, the region growing method includes the following steps:

[0138] 1) Seed point selection: Determine whether there are unvisited foreign object points in the foreign object set. If so, randomly select an unvisited foreign object point from the foreign object set as the seed point of the new foreign object; if there are no unvisited foreign object points, the method ends;

[0139] 2) Seed point neighborhood search: Search whether there are other unvisited foreign object points within the neighborhood range of the current seed point. If so, add them to the foreign object point cloud of the foreign object to which the current seed point belongs, mark them as visited, and continue to perform iterative search with the current neighborhood point as the new seed point;

[0140] 3) Neighborhood search termination: For the foreign object point cloud to which the current belongs, if there are no other unvisited foreign object points in the neighborhood of each seed point, the current foreign object point cloud has completed clustering segmentation, obtain the foreign object point cloud of the current clustering, and return to step 1).

[0141] For each foreign object point cloud, use the centroid method to calculate the centroid position of each foreign object point cloud in the camera coordinate system, and obtain its foreign object position in the world coordinate through the coordinate transformation method. It includes the following content:

[0142] The centroid method includes: for each foreign object point cloud, calculating the centroid coordinates based on the coordinate positions of all data points of the foreign object in the camera coordinate system as the camera coordinate position of the foreign object:

[0143]

[0144] where P(x, y, z) is the centroid coordinate of the foreign object point cloud, and p i (x i , y i , z i ) is the camera coordinate of all data points in each foreign object point cloud;

[0145] The coordinate transformation method includes: using the external parameter matrix of the depth camera to transform the position of each foreign object from the camera coordinate system to the world coordinate system to obtain the world coordinate position of each foreign object:

[0146]

[0147] where R is the rotation transformation matrix and T is the translation transformation matrix.

[0148] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, several improvements can be made without departing from the principle of the present invention, and these improvements should also be regarded as the protection scope of the present invention.

Claims

1. A pipeline foreign object detection method based on the curvature mutation of 3D point cloud, characterized in that, Including the steps: Step 1: Camera parameter calibration: Install the depth camera in the normal direction at the end of the inspection robotic arm, and use Zhang Zhengyou calibration method to calculate the internal parameter matrix, distortion parameters, and external parameter matrix of the depth camera; Step 2: Pipeline cross-section classification: According to the cross-section change characteristics in the theoretical pipeline model, apply the end cross-section classification method to classify and segment the theoretical pipeline model along the axis direction. For each type and segment of the pipeline, perform camera pose alignment and point cloud acquisition processing; Step 3: Camera pose alignment: Use the visual servo control method to align the end of the inspection robotic arm with the central axis of each pipeline segment, align the Z-axis of the depth camera coordinate system with the normal direction of the surface to be measured of this pipeline segment, and ensure that the X-axis of the depth camera coordinate system is aligned with the axis direction of this pipeline segment; Step 4: Original point cloud acquisition: Capture the depth image of the surface to be measured of this pipeline segment through the depth camera and convert it into original point cloud data; Step 5: Original point cloud preprocessing: Use the filtering and noise reduction method to preprocess the original point cloud to obtain a smooth point cloud with a data volume of N; Step 6: Smooth point cloud compression: Compress the preprocessed smooth point cloud. Use the voxelized point cloud optimization method to reduce the data volume while retaining the main shape features to obtain a compressed point cloud; Step 7: Foreign object point set extraction: For the compressed point cloud, use the dynamic curvature threshold segmentation method to extract the foreign object point data on the inner surface of the pipeline to obtain a foreign object point set; Step 8: Foreign object point cloud clustering segmentation: For the foreign object point set, use the region growing method to perform clustering segmentation of the foreign objects to obtain the foreign object point cloud of each foreign object; Step 9: Foreign object position estimation: For each foreign object point cloud, use the centroid method to calculate the camera coordinates of the centroid of the foreign object in the camera coordinate system, and obtain its foreign object position in the world coordinate system through the coordinate transformation method; The dynamic curvature threshold segmentation method includes the following steps: Step 7.1: Compressed point cloud sorting: Sort the compressed point cloud along the pipeline axis direction, that is, the X-axis direction of the depth camera coordinate system: Determine the number of voxel grid layers where the data point is located according to the Y-axis coordinate of the data point, project the compressed point cloud in this layer of voxel grid along the pipeline axis direction into the XOZ cross-section at the corresponding position of the Y-axis, and sort all data points in each XOZ cross-section according to the X-axis coordinate; Step 7.2: Z-axis curvature value calculation: For each XOZ cross-section, calculate the data point P in the X-axis direction i The curvature value Z caused by the change in Z-axis height cur (i): Z cur (i) = abs(diffz i × 10)(2) where z i is the Z-axis coordinate value of the current point P i , z n is the Z-axis coordinate value of the neighboring point of P i , and diffz i is the curvature deviation value; Step 7.3: Initialization of segmentation threshold: Set the threshold error ΔT as the termination condition of the dynamic threshold iteration, and calculate the average curvature of all points in the compressed point cloud as the initial threshold T0: Step 7.4: Dynamic threshold point cloud classification: Divide the compressed point cloud into two subsets G1 and G2 according to the current threshold T0. G1 contains all data points P with curvature less than or equal to T0 j , and the number of data points is n1. G2 contains all data points P with curvature greater than T0 k , and the number of data points is n2. Calculate the average curvature μ1 of all data points in subset G1 and the average curvature μ2 of all data points in subset G2 respectively: Step 7.5: Update of segmentation threshold: Calculate the updated value T1 of the segmentation threshold according to the average curvatures μ1 and μ2 of all data points in the two subsets G1 and G2; Step 7.6: Dynamic threshold iteration: If the difference between the current threshold T0 and the threshold updated value T1 is less than the set threshold error ΔT, then enter Step 7.7; otherwise, update T0 with the value of T1 and return to Step 7.4; Step 7.7: Threshold segmentation of foreign object point cloud: Use the current threshold T0 as the segmentation threshold h cur , and perform threshold segmentation on all data points in the compressed point cloud. For the curvature value Z cur (i) The data points exceeding the segmentation threshold h cur are regarded as foreign object points, and all foreign object points are used as the foreign object point set.

2. The pipeline foreign object detection method based on the sudden change of three-dimensional point cloud curvature according to claim 1, characterized in that The end cross-section classification method includes the following steps: Step 2.1: Pipe cross-section interception: In the pipe axis direction, intercept the pipe successively with planes perpendicular to the pipe axis at a step length of t to obtain a set of pipe cross-sections; c ​ Step 2.2: Pipeline segmentation and classification: Sequentially select adjacent and continuous pipeline cross-sections with the same cross-sectional shape as one section of type-I pipeline, and all type-I pipelines form a set of type-I pipelines; sequentially select adjacent and continuous pipeline cross-sections with cross-sectional shapes changing according to a fixed rule as one section of type-II pipeline, and all type-II pipelines form a set of type-II pipelines. Step 2.3: Pipeline segment processing: Sequentially extract each section of type-I pipeline and type-II pipeline along the pipeline axis direction, and perform camera pose alignment and point cloud acquisition processing respectively.

3. A pipeline foreign object detection method based on the sudden change of three-dimensional point cloud curvature according to claim 1, characterized in that, The filtering and noise reduction method includes direct filtering and Gaussian filtering: The direct filtering is to remove noise data points through a distance threshold h dist ; for a polygonal cross-section pipeline, where L1 is the normal distance of the cross-section to be detected in the polygon cross-section corresponding to the Z-axis of the depth camera coordinate system, and M4 is the change threshold of the normal distance; for a circular pipeline, where d1 is the diameter of the circular pipeline and M5 is the change threshold of the diameter; for a non-circular curved surface pipeline, where d2 is the average normal distance of the neighborhood of the non-circular curved surface corresponding to the Z-axis of the depth camera coordinate system, and M6 is the change threshold of the average normal distance; assuming that the coordinate of the data point collected by the depth camera is p i =(x i , y i , z i ), when the normal distance h i between the data point and the depth camera is greater than the threshold h dist , then this data point is considered a noise point and is removed; The Gaussian filtering includes the following steps: Step 5.1: Divide the XOZ section: Along the Y-axis direction of the depth camera coordinate system, divide the XOZ section in sequence with a step size of t p in sequence; Step 5.2: Axial data sorting: Select all data points in the same XOZ cross-section, and sequentially sort the data points along the pipeline axis direction, i.e., the X-axis direction of the depth camera coordinate system, to generate sorted point clouds. Step 5.3: Neighborhood point division: For the \(i\)-th data point in the sorted point cloud, find neighborhood points with a quantity not exceeding \(U\) in the neighborhood \(\delta\) x and calculate the filtering weight \(G\) σ (j) corresponding to the \(j\)-th neighborhood point using a Gaussian function based on the axial distance between the \(j\)-th neighborhood point and this point: where σ is the weight coefficient of the Gaussian function; Step 5.4: Gaussian smoothing: Perform weighted average on the z j coordinates of all neighborhood points, and use it as the z i coordinate of the i-th data point in the sorted point cloud: z i = ∑ U z j G σ (j), U: |x j - x i | ≤ δ x (8).

4. A method for detecting foreign objects in a pipeline based on the sudden change of curvature of a three-dimensional point cloud according to claim 1, characterized in that The voxelized point cloud optimization method includes the following steps: Step 6.1: Parameter initialization: Set the maximum horizontal field of view angle T row , the maximum vertical field of view angle T col , and initialize the angular resolution t angle , the upper threshold M of the point cloud density h , the lower threshold M of the point cloud density d , the angular resolution step t s ; Step 6.2: Voxel grid division: According to the current angular resolution t angle Calculate the number of rows n of the voxel grid division row and the number of columns n col : n row = T row / t angle (9) n col = T col / t angle (10); Step 6.3: Point cloud - voxel network mapping: For each data point p in the smoothed point cloud i =(x i , y i , z i ), calculate the position of this point in the voxel grid according to the current angular resolution t angle : angle row = tan -1 (x i / z i ) (11) angle col = tan -1 (y i / z i ) (12) row=(n row / 2)+(angle row / t angle ) (13) col=(n col / 2)+(angle col / t angle ) (14) where angle row and angle col are the row resolution angle and the column resolution angle respectively, and row and col are the row number and column number of this point in the voxel grid; Step 6.4: Average point cloud density iteration: According to the current angular resolution t angle Calculate the average point cloud density ρ = N / m of the point cloud-voxel grid, where N is the data volume of the smoothed point cloud and m is the number of non-empty voxel grids; if the average point cloud density is greater than the upper threshold M of the point cloud density h , then reduce the angular resolution t angle by one angular resolution step t s , and return to Step 6.2; if the average point cloud density is less than the lower threshold M of the point cloud density d , then increase the angular resolution t angle by one angular resolution step t s , and return to Step 6.2; otherwise, proceed to Step 6.5; Step 6.5: Point cloud data compression: Divide all data points into each voxel grid, calculate the center point of all data points in each voxel grid, and use this center point as the data point of this voxel grid after data compression. Among them, P(x, y, z) is the center point of each voxel grid after point cloud compression, and p i (x i , y i , z i ) are all data points in each voxel grid before point cloud compression.

5. A method for detecting foreign objects in pipelines based on curvature mutation of 3D point clouds according to claim 1, wherein The region growing method includes the following steps: Step 8.1: Seed point selection: Determine whether there are unvisited foreign object points in the foreign object point set. If so, randomly select an unvisited foreign object point from the foreign object point set as the seed point of the new foreign object; if there are no unvisited foreign object points, the method ends. Step 8.2: Seed point neighborhood search: Search whether there are other unvisited foreign object points within the neighborhood range of the current seed point. If so, add them to the foreign object point cloud of the foreign object to which the current seed point belongs, mark them as visited, and continue to perform iterative search with the current neighborhood point as the new seed point. Step 8.3: Neighborhood search termination: For the foreign object point cloud to which the current belongs, if there are no other unvisited foreign object points in the neighborhood of each seed point, the current foreign object point cloud has been clustered and segmented, and the foreign object point cloud of the current cluster is obtained, then return to Step 8.

1.

6. A pipeline foreign object detection method based on three-dimensional point cloud curvature mutation according to claim 1, characterized in that: The centroid method includes: For each foreign object point cloud, calculate the centroid coordinates according to the coordinate positions of all data points of the foreign object in the camera coordinate system as the camera coordinate position of the foreign object. Among them, P(x, y, z) is the centroid coordinate of the foreign object point cloud, and p i (x i , y i , z i ) is the camera coordinate of all data points in each foreign object point cloud; The coordinate transformation method includes: Using the external parameter matrix of the depth camera, transform the position of each foreign object from the camera coordinate system to the world coordinate system to obtain the world coordinate position of each foreign object. where R is the rotation transformation matrix and T is the translation transformation matrix.

Images