Multi-mode underground pipeline target detection system

Abstract

The invention relates to the technical field of underground pipeline target detection, and discloses a multi-mode underground pipeline target detection system, which comprises the following steps of denoising radar point cloud based on a point cloud region segmentation method; performing radar point cloud clustering based on an improved DBSCAN method; and multi-channel array construction based on radar and camera data fusion. According to the method, the three-dimensional point cloud is divided into a flat region and a non-flat region, and bilateral filtering and statistical filtering are adopted for processing the geometric differences of different regions respectively, so that the balance between noise removal and geometric feature reservation is realized; secondly, a dynamic neighborhood half-sum density weighting function is introduced, interference of low-density noise points on clustering is reduced, radar point cloud noise is effectively filtered out, and a target area to be detected is positioned; and finally, the radar physical features and the camera RGB image are spliced into a multi-channel array, so that heterogeneous feature fusion of the radar and the camera is realized, and the accuracy and robustness of the target detection system in a dark environment are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of underground pipeline target detection, and particularly relates to a multi-modal underground pipeline target detection system. BACKGROUND

[0002] Millimeter wave radar target detection refers to a technology of using a radar sensor working in a millimeter wave frequency band (30GHz~300GHz) to detect the distance, speed, direction, etc. of a target by transmitting and receiving millimeter wave signals, and combining signal processing and algorithms to identify the existence of the target.

[0003] Camera target detection refers to a technology of using an optical camera to collect images or video streams, and using computer vision algorithms (such as convolutional neural networks CNN) to identify target objects in the images and output their categories, positions (bounding boxes), etc.

[0004] Common point cloud denoising methods include statistical filtering and bilateral filtering. Statistical filtering is a method of determining an outlier threshold by calculating the mean and standard deviation based on the distance statistics of the neighborhood, and then removing noise. Bilateral filtering is a method of constructing a weight function combining spatial distance and feature difference, and implementing denoising and edge preservation by weighted average of neighborhood points.

[0005] DBSCAN is a density-based clustering algorithm suitable for discovering clusters of sample points with connected density in space. The core idea is that in the sample space, regions with high density belong to the same cluster, while regions with low density (noise points or boundary points) are considered abnormal or do not belong to any cluster.

[0006] Existing sensor fusion methods for millimeter wave radar and cameras usually use two sensors to independently detect targets and output prediction results (such as bounding boxes, categories), and then fuse them into final results through post-processing (such as data association, Kalman filtering).

[0007] Existing single sensor target detection schemes have obvious limitations in the working environment of underground cable pipelines. The pure camera monitoring scheme is significantly affected by environmental lighting, and in low light environments, the image signal-to-noise ratio is low, the target edge is blurred, and the detection accuracy is reduced. Using only millimeter wave radar can only obtain the distance, speed, direction and reflection intensity of the target, and cannot distinguish the target category. If two sensors are combined for target detection tasks, data type heterogeneity will cause fusion difficulties, and the process of independently processing two sensor data and then fusing them increases the processing time, which cannot meet the real-time requirements of pipeline foreign object detection.

[0008] Target detection using millimeter wave radar also needs to filter and cluster the noise of the generated point cloud. The commonly used point cloud filtering algorithms are bilateral filtering and statistical filtering. Bilateral filtering is weak in feature difference when processing point cloud areas with small curvature values and approximate point distribution, the weight depends on the spatial distance, which causes excessive smoothing, and it is difficult to remove outliers. When processing areas with geometric features such as edges, corners and cusps, the fixed neighborhood and threshold are easy to misdelete feature points, and cannot distinguish between noise and feature points. The traditional DBSCAN point cloud clustering algorithm fixes the threshold to judge the core point, and does not consider the local density difference of the sample; the traditional DBSCAN cannot adapt to the multi-scale characteristics of data distribution.

[0009] Therefore, there is a need to further solve the above problems, and the applicant proposes a multi-modal underground pipeline target detection system. SUMMARY

[0010] In view of the deficiencies of the prior art, the present application provides a multi-modal underground pipeline target detection system, which solves the defects that the conventional point cloud noise reduction method cannot consider the noise reduction and feature preservation requirements of different curvature regions, and is easy to cause feature loss or excessive smoothing; the main defect of the traditional DBSCAN clustering algorithm is the strong dependence on parameters, which needs to be manually set and is difficult to adapt to the problem of uneven density of point cloud data distribution, and the existing pure visual algorithm has low target detection accuracy in the pipeline environment.

[0011] To achieve the above purpose, the present application provides the following technical scheme: A multi-modal underground pipeline target detection system, the system comprising the following steps: S1: denoising the radar point cloud based on a point cloud region segmentation method; S2: radar point cloud clustering based on an improved DBSCAN method; S3: multi-channel array construction based on radar and camera data fusion; Step S1 specifically includes: S11: obtaining a radar point cloud data set For a point in the point cloud , search for the 30 points closest to the point in space, and use these points to form the neighborhood point set of the point , calculate the neighborhood centroid and construct the covariance matrix C, the formula of the covariance matrix C is as follows: In the formula, is the point in the neighborhood except the centroid; S12: singular value decomposition is performed on the matrix to obtain , 、 Three singular values, wherein is the smallest singular value, defining the point curvature , the average curvature of the entire point cloud is calculated as a threshold , if , the point is initially classified as a flat region; S13: based on the distance calculation point neighborhood point weight , and construct a weighted covariance matrix : In the formula represent the distance attenuation parameter, control the weight of the distance change speed; singular value decomposition is performed on the matrix , and the eigenvector corresponding to the minimum singular value is taken as the normal vector , then the angle between the normal vector of point and its neighborhood point is calculated , if the angle is less than the set threshold, it is confirmed that the point belongs to the flat region; otherwise, the point is classified into the non-flat region; S14: the curvature of the flat region is small, and the noise is mostly an outlier, and statistical filtering can accurately denoise by dynamic neighborhood and curvature weighting; the non-flat region contains edges and other features, so that bilateral filtering combines spatial distance and normal difference, which can denoise while preserving geometric details, and the two adapt to different regional characteristics and noise distribution.

[0012] As a further scheme of the application: step S2 specifically comprises: S21: setting the initial neighborhood radius of clustering , the minimum number of neighborhood points MinPts, the weight adjustment parameter , the scale adjustment parameter , calculating the distance of the MinPts-1th nearest neighbor point of the point in the data set , and taking as the new neighborhood radius, counting the number of points in the new neighborhood of the point and the average number of neighborhood points of all points , , , calculating the field radius weight of each point in the point cloud data set, the formula is as follows: Arrange all the weight values of the sample points from small to large, and take the weight value at the 75% position as the core point determination threshold ; S22: Traversing the point cloud dataset All sample points in , if the point It is judged as a core point and the dynamic neighborhood radius of the core point is calculated : Get of All points in the neighborhood , and Join cluster , loop processing Each point in ,like Then Join cluster , otherwise it is regarded as a potential noise point.

[0013] As a further solution of the present invention: step S2 specifically further includes: S23: After the initial clustering is completed, the points that are not classified into any cluster are temporarily defined as noise points. , check whether there is a cluster The core point is of In the neighborhood, if there is and The weight of the field radius , then Divided into clusters .

[0014] As a further solution of the present invention: step S3 specifically includes: S31: In order to accurately map the radar point cloud data, distance, speed, and intensity to the pixel locations of the visual image, it is first necessary to convert the radar point cloud from the radar coordinate system to the image coordinate system to achieve spatial alignment; Define the radar installation location as the origin of the radar coordinate system , radar coordinate system The axis is parallel to the ground and the direction of travel is vertical, Parallel to the ground and pointing straight ahead of the radar, Vertical and and Pointing straight up, define the camera's optical center as the origin of the camera's world coordinate system , the coordinate axis direction is the same as the radar coordinate system, and the upper left corner of the image is defined as the origin of the image coordinate system , image coordinate system Pointing in the width direction of the image, Pointing to the image height direction, the optical center of the camera is defined as the origin of the camera coordinate system , the camera coordinate system axis, The axis is in the same direction as the image coordinate system, The axis points directly in front of the camera. S32: First, convert the radar coordinate system to the camera world coordinate system, the formula is as follows: wherein , , is the distance between the origin of the radar coordinate system and the camera world coordinate system, then convert the camera world coordinate into the three-dimensional coordinate system inside the camera, the conversion matrix is as follows: wherein, is the installation pitch angle of the camera, H is the installation height of the camera in the world coordinate system, finally project the three-dimensional coordinates of the camera to the two-dimensional image plane to get the pixel coordinates , the formula is as follows: wherein, and are the focal lengths of the camera x-axis and y-axis, and are the pixel coordinates of the image center.

[0015] As a further scheme of the application: step S3 further comprises: S33: In order to convert the continuous physical signals collected by the radar into standardized image features suitable for deep learning model processing, and realize cross-modal fusion with the RGB image of the camera, the information collected by the radar needs to be processed, set the distance D, the speed V, and the intensity I as the three channels of the original data of the millimeter wave radar after conversion, the specific conversion formula is as follows: wherein, d represents the relative distance between the millimeter wave radar and the target, the maximum measurement distance of the millimeter wave radar is set to 30m; v represents the motion speed of the millimeter wave radar, the maximum speed of the radar is 33.3m / s, SNR represents the ratio of the radar echo signal power to the noise power, reflecting the signal quality; S34: Concatenate the three channels of the RGB image and the D, V, and I channels of the radar image in sequence to form a multi-channel input array, and input the concatenated array into the target detection deep learning model.

[0016] Compared with the prior art, the present invention has the following beneficial effects: 1. This method divides the 3D point cloud into flat and non-flat areas based on curvature and normal information. Bilateral filtering and statistical filtering are used to process the geometric differences of different areas respectively to achieve a balance between noise removal and geometric feature preservation.

[0017] 2. A density weight function is introduced into the conventional DBSCAN clustering algorithm to dynamically assign weights to each sample point, so that points in high-density areas contribute more to clustering, and the noise weight of points in low-density areas is reduced. In addition, a dynamic neighborhood radius is introduced to dynamically adjust the diffusion range according to the density weight and local structure of the current point, thereby improving the accuracy of clustering.

[0018] 3. This method combines the radar physical features and the camera RGB image into a multi-channel array after aligning the spatial coordinates, thus realizing the heterogeneous feature fusion of radar and camera, and improving the accuracy and robustness of the target detection system in dark environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a flow chart of radar point cloud processing of the system of the present invention; Figure 2 This is one of the effect diagrams of foreign body recognition in pipelines of the system of the present invention; Figure 3 This is the second effect diagram of the foreign body recognition in the pipeline of the system of the present invention; Figure 4 This is the average accuracy curve of target detection of foreign objects in pipelines using existing pure image information; Figure 5 The figure is a graph showing the average accuracy of target detection of foreign objects in pipelines using the system of the present invention. DETAILED DESCRIPTION

[0020] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0021] Example See also Figure 1 The present invention provides a multi-modal underground pipeline target detection system, which includes the following steps: Step 1: Denoising radar point cloud based on point cloud region segmentation method Get radar point cloud dataset , for a point in the point cloud , search for the 30 points closest to the point in space, and use these points to form the point Neighborhood point set , calculate the neighborhood centroid And construct the covariance matrix C. The formula of the covariance matrix C is as follows: in the formula is the point in the neighborhood except the centroid; Pair Matrix Perform singular value decomposition and get 、 、 Three singular values, among which is the smallest singular value, defining the curvature of the point , calculate the average curvature of the entire point cloud as the threshold ,like , then the point is initially classified as a flat area; Calculate points based on distance Neighborhood weight , and construct the weighted covariance matrix : in the formula Represents the distance attenuation parameter, which controls how fast the weight changes with distance; Pair Matrix Perform singular value decomposition and take the eigenvector corresponding to the minimum singular value as the normal vector , then calculate the point and its neighboring points The normal vector angle If the angle is less than the set threshold, the point is confirmed to belong to the flat area; otherwise, the point is classified into the non-flat area; Flat areas have small curvature and noise is mostly outliers. Statistical filtering can be used to accurately remove noise through dynamic neighborhood and curvature weighting. Non-flat areas contain features such as edges, so bilateral filtering combines spatial distance and normal difference to remove noise while retaining geometric details. The two are adapted to different regional characteristics and noise distribution.

[0022] Step 2: Radar point cloud clustering based on improved DBSCAN method Set the initial neighborhood radius for clustering , minimum number of neighborhood points MinPts, weight adjustment parameter , scale adjustment parameter , calculate the data set midpoint The distance of the MinPts-1th neighbor point , and As the new neighborhood radius, the statistical points The number of points in the new neighborhood And the average number of neighborhood points of all points globally , Calculate the radius weight of each point in the point cloud dataset using the following formula: Arrange the weight values ​​of all sample points from small to large, and take the weight value at the 75% position as the core point determination threshold ; Iterate through the point cloud dataset All sample points in , if the point It is judged as a core point and the dynamic neighborhood radius of the core point is calculated : Get of All points in the neighborhood , and Join cluster , loop processing Each point in ,like Then Join cluster , otherwise it is regarded as a potential noise point.

[0023] After the initial clustering is completed, the points that are not classified into any cluster are temporarily defined as noise points. , check whether there is a cluster The core point is of In the neighborhood, if there is and The weight of the field radius , then Divided into clusters .

[0024] Step 3: Multi-channel array construction based on radar and camera data fusion In order to accurately map the radar point cloud data, distance, speed, and intensity to the pixel locations of the visual image, it is first necessary to convert the radar point cloud from the radar coordinate system to the image coordinate system to achieve spatial alignment; Define the radar installation location as the origin of the radar coordinate system Radar coordinate system The axis is parallel to the ground and the forward direction is vertical, Parallel to the ground and pointing directly in front of the radar, Perpendicular to And Pointing directly upwards, defining the camera optical center as the origin of the camera world coordinate system , The coordinate axis direction is the same as the radar coordinate system, and the upper left corner of the image is defined as the origin of the image coordinate system , Image coordinate system Pointing to the image width direction, Pointing to the image height direction, defining the camera optical center as the origin of the camera coordinate system , Camera coordinate system Axis, The axis points in the same direction as the image coordinate system, The axis points directly in front of the camera; First, convert the radar coordinate system to the camera world coordinate system, as follows: Where , , is the distance between the origin of the radar coordinate system and the camera world coordinate system, then convert the camera world coordinate to the three-dimensional coordinate system inside the camera, the conversion matrix is as follows: Where, is the camera installation pitch angle, H is the installation height of the camera in the world coordinate system, and finally project the three-dimensional coordinates of the camera to the two-dimensional image plane to get the pixel coordinates , The formula is as follows: Where, And are the focal lengths of the camera x and y axes, And are the pixel coordinates of the image center.

[0025] In order to convert the continuous physical signals collected by the radar into standardized image features suitable for deep learning model processing, and realize cross-modal fusion with the RGB image of the camera, the information collected by the radar needs to be processed, set the distance D, the speed V, and the intensity I as the three channels of the original data of the millimeter wave radar after transformation, The specific conversion formula is as follows: Where d represents the relative distance between the millimeter-wave radar and the target, and the maximum measurement distance of the millimeter-wave radar is set to 30m; v represents the movement speed of the millimeter-wave radar, and the maximum speed of the radar is 33.3m / s; SNR represents the ratio of the radar echo signal power to the noise power, reflecting the signal quality; The three channels of the RGB image and the D, V, and I channels of the radar image are sequentially concatenated to form a multi-channel input array, which is then input into the target detection deep learning model.

[0026] For details, please refer to Figure 2 and Figure 3 In the figure, burr represents a large burr in the pipeline, lburr represents a long burr with a small area in the pipeline, sburr represents a short burr with a small area in the pipeline, and obstacle represents a foreign object in the pipeline.

[0027] For details, please refer to Figure 4 and Figure 5 The mean average precision (mAP@0.5) is the core indicator for measuring model performance in target detection tasks. The higher the indicator, the higher the accuracy of the target detection method. For the application scenario of foreign objects in pipes, the average precision curve shows that this solution improves the target detection accuracy by 8.3% compared with the pure vision solution.

[0028] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multimodal underground pipeline target detection system, characterized by: The system includes the following steps: S1: Denoising of radar point cloud based on point cloud region segmentation method; S2: Radar point cloud clustering based on improved DBSCAN method; S3: Multi-channel array construction based on radar and camera data fusion; Step S1 specifically includes: S11: Get radar point cloud dataset , for a point in the point cloud , search for the 30 points closest to the point in space, and use these points to form the point Neighborhood point set , calculate the neighborhood centroid And construct the covariance matrix C. The formula of the covariance matrix C is as follows: in the formula is the point in the neighborhood except the centroid; S12: Matrix Perform singular value decomposition and get 、 、 Three singular values, among which is the smallest singular value, defining the curvature of the point , calculate the average curvature of the entire point cloud as the threshold ,like , then the point is initially classified as a flat area; S13: Calculate points based on distance Neighborhood weight , and construct the weighted covariance matrix : in the formula Represents the distance attenuation parameter, which controls how fast the weight changes with distance; Pair Matrix Perform singular value decomposition and take the eigenvector corresponding to the minimum singular value as the normal vector , then calculate the point and its neighboring points The normal vector angle If the angle is less than the set threshold, the point is confirmed to belong to the flat area; otherwise, the point is classified into the non-flat area; S14: Flat areas have small curvature and noise is mostly outliers. Statistical filtering can be used to accurately remove noise through dynamic neighborhood and curvature weighting. Non-flat areas contain features such as edges, so bilateral filtering combines spatial distance and normal difference to remove noise while retaining geometric details. The two are adapted to different regional characteristics and noise distribution.

2. The multimodal underground pipeline target detection system according to claim 1, characterized in that: Step S2 specifically includes: S21: Set the initial neighborhood radius of the cluster , minimum number of neighborhood points MinPts, weight adjustment parameter , scale adjustment parameter , calculate the data set midpoint The distance of the MinPts-1th neighbor point , and As the new neighborhood radius, the statistical points The number of points in the new neighborhood And the average number of neighborhood points of all points globally , Calculate the range radius weight of each point in the point cloud dataset using the following formula: Arrange the weight values ​​of all sample points from small to large, and take the weight value at the 75% position as the core point determination threshold ; S22: Traversing the point cloud dataset All sample points in , if the point It is judged as a core point and the dynamic neighborhood radius of the core point is calculated : Get of All points in the neighborhood , and Join cluster , loop processing Each point in ,like Then Join cluster , otherwise it is regarded as a potential noise point.

3. The multimodal underground pipeline target detection system according to claim 2, characterized in that: Step S2 specifically also includes: S23: After the initial clustering is completed, the points that are not classified into any cluster are temporarily defined as noise points. , check whether there is a cluster The core point is of In the neighborhood, if there is and The weight of the field radius , then Divided into clusters .

4. The multimodal underground pipeline target detection system according to claim 1, characterized in that: Step S3 specifically includes: S31: In order to accurately map the radar point cloud data, distance, speed, and intensity to the pixel locations of the visual image, it is first necessary to convert the radar point cloud from the radar coordinate system to the image coordinate system to achieve spatial alignment; Define the radar installation location as the origin of the radar coordinate system , radar coordinate system The axis is parallel to the ground and the direction of travel is vertical, Parallel to the ground and pointing straight ahead of the radar, Vertical and and Pointing straight up, define the camera's optical center as the origin of the camera's world coordinate system , the coordinate axis direction is the same as the radar coordinate system, and the upper left corner of the image is defined as the origin of the image coordinate system , image coordinate system Pointing in the width direction of the image, Pointing to the image height direction, define the camera optical center as the origin of the camera coordinate system , camera coordinate system axis, The axes point in the same direction as the image coordinate system, The axis points directly in front of the camera; S32: First, convert the radar coordinate system to the camera world coordinate system. The formula is as follows: in , , is the distance between the radar coordinate system and the origin of the camera world coordinate system. Then the camera world coordinate is converted to the three-dimensional coordinate system inside the camera. The conversion matrix is ​​as follows: in, is the camera installation pitch angle, H is the camera installation height in the world coordinate system, and finally the camera's three-dimensional coordinates are projected onto the two-dimensional image plane to obtain the pixel coordinates , the formula is as follows: in, and is the focal length of the camera along the x-axis and y-axis, and is the pixel coordinate of the center of the image.

5. The multimodal underground pipeline target detection system according to claim 4, characterized in that: Step S3 specifically also includes: S33: In order to convert the continuous physical signals collected by the radar into standardized image features suitable for deep learning model processing and achieve cross-modal fusion with the camera's RGB image, it is necessary to process the information collected by the radar. Set distance D, speed V, and intensity I as the three channels output after the millimeter wave radar raw data is converted. The specific conversion formula is as follows: Where d represents the relative distance between the millimeter-wave radar and the target, and the maximum measurement distance of the millimeter-wave radar is set to 30m; v represents the movement speed of the millimeter-wave radar, and the maximum speed of the radar is 33.3m / s; SNR represents the ratio of the radar echo signal power to the noise power, reflecting the signal quality; S34: Concatenate the three channels of the RGB image and the D, V, and I channels of the radar image in sequence to form a multi-channel input array, and input the concatenated array into the target detection deep learning model.

Images