An intelligent inspection method and device based on image and point cloud fast matching

Abstract

The application discloses a kind of municipal facilities intelligentized inspection method and equipment based on image and point cloud matching.It relates to municipal facilities inspection technical field, method includes: municipal facilities database containing multi-view image template and point cloud grid model is constructed;Through the unmanned vehicle of optical camera, laser radar and GPS is synchronous collected scene image and point cloud;Respectively, image target detection and point cloud target detection are carried out;Cross-modal target matching stage, the 2D / 3D nearest neighbor vector of image mask centroid and point cloud cluster centroid is calculated, the direction included angle matching of y coordinate of point cloud vector after zero is matched with image vector, and the correctness of matching is verified by vector module length ratio mode distribution;When any modality detects defect, the repair work order containing defect position and GPS coordinate is automatically generated.The application realizes double mode independent detection and result fusion, eliminates sensor calibration requirement, improves inspection efficiency and accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of municipal facility inspection, and in particular to an intelligent inspection method and device based on image and point cloud fast matching. BACKGROUND

[0002] Periodic inspection and repair of municipal facilities is an important measure to maintain the healthy operation of the city and ensure the quality of life of citizens. The traditional municipal inspection method is mainly manual, that is, professional inspectors walk or drive on the road according to a certain route and frequency, observe and judge the damage of municipal facilities with the naked eye, take pictures and record them, and then submit an inspection report. The inspection results are affected by many factors such as personnel, equipment and environment, and are prone to subjectivity, bias, error, omission and repetition, making it difficult to ensure the efficiency, objectivity, accuracy and integrity of municipal inspection, and supporting the decision-making and implementation of municipal management.

[0003] The proposed method matches point cloud and image data with different representation methods and information characteristics, fully utilizes the advantages of both types of data, and realizes information enrichment and completeness. Traditional matching methods mainly use color information (RGB-D data) in point cloud and RGB channel in image for direct matching, or extract geometric features of point cloud and texture features of image respectively, establish association through feature similarity for matching, or project point cloud to a two-dimensional plane to generate a depth map or an intensity map, and then perform 2D-2D matching with the image. However, the above matching methods have some problems, such as: color information of LiDAR point cloud is greatly affected by light changes, and LiDAR point cloud does not have color information; the difference in feature dimension between different modalities is large, and the matching efficiency is low; the projection-based method requires that the sensor strictly matches the initial calibration parameters when collecting data, otherwise the projection error will increase significantly, resulting in matching failure or accuracy reduction.

[0004] Therefore, how to propose an intelligent inspection method based on image and point cloud fast matching to improve the inspection efficiency and accuracy while reducing the cost is a problem to be solved by those skilled in the art. SUMMARY

[0005] The technical solution of the present application to solve the above technical problems is to provide an intelligent inspection method for municipal facilities based on image and point cloud matching, comprising the following steps:

[0006] S1: Construct a municipal facility database:

[0007] Collect multi-view image templates of target objects, and label the accurate bounding box of the object in each template;

[0008] Collect multi-angle point cloud templates and convert them into grid models;

[0009] S2: Deploy unmanned inspection vehicle device: Collect scene images and point clouds by unmanned vehicles equipped with optical cameras, LiDAR, and GPS, wherein the distance between the camera and LiDAR is ≤20cm and hardware triggers synchronization;

[0010] S3: Image Object Detection: Preprocess the scene image, use a lightweight detection model to locate objects and generate masks, and perform crack detection, corrosion classification, color comparison, graffiti anomaly detection and blur determination based on defect type;

[0011] S4: Point cloud target detection: After voxel filtering, denoising, ground segmentation and clustering of the scene point cloud, the pose is verified by RANSACICP registration through FPFH feature matching database model, and defects such as depressions, tilts and structural missing parts are detected.

[0012] S5: Cross-modal target matching:

[0013] Calculate the centroid of the image mask and the centroid of the point cloud cluster respectively, and generate 2D / 3D nearest neighbor vectors;

[0014] After the y-coordinate of the point cloud vector is zeroed out, it is matched with the direction angle of the image vector, and candidate pairs with an angle of less than 5° are filtered out.

[0015] Verify the correctness of the matching based on the ratio of the vector magnitude to the mode distribution;

[0016] S6: Work Order Generation: When a defect is detected in any image or point cloud of the same object, a maintenance work order containing the defect location marker and GPS coordinates is automatically generated.

[0017] Furthermore, in step S1:

[0018] The acquisition of the image templates covers the front, side, and top views as well as different lighting conditions, with ≥20 templates for a single object;

[0019] The point cloud template is scanned by LiDAR from multiple angles, registered by ICP, and then reconstructed using Poisson to generate a mesh model.

[0020] Further, step S3 includes the following steps:

[0021] (1) Scene image preprocessing:

[0022] Unify the input image resolution, assuming the original image is... The target resolution is (H new W new Scaling is achieved through bilinear interpolation, and the output image I is obtained. resized The pixel value is calculated as follows:

[0023] I resized (x ′ ,y′ )=∑ i,j∈{0,1} I(x i ,y i )·(1-|x ′ ,x i |)·(1-|y ′ ,y i |);

[0024] Gaussian filtering and histogram equalization are performed to equalize the illumination and reduce environmental interference; for I resized Gaussian smoothing is performed, and the convolution kernel G is defined as:

[0025]

[0026] Output image I blur Calculated via convolution:

[0027] I blur (x,y)=(I resized *G)(x,y);

[0028] Calculate the cumulative distribution function (CDF):

[0029]

[0030] Where, n i Let N be the number of pixels at gray level i, N be the total number of pixels, and L be the number of gray levels.

[0031] The mapping function is:

[0032] I equalized (x,y)=round((L-1)·CDF(I blur (x,y)));

[0033] (2) Use a lightweight detection model to locate objects and generate masks:

[0034] A training set is constructed using labeled templates and scene images;

[0035] Choose the lightweight YOLOv8 model for transfer learning;

[0036] Add random rotation, occlusion, and brightness perturbation to improve generalization ability;

[0037] Random rotation: rotation transformation matrix parameter

[0038] Random occlusion: Fill the region (x,y,ω,h) with noise.

[0039] Brightness perturbation: Pixel value scaling I′=γI,

[0040] The model takes a scene image as input and outputs bounding boxes with category labels and confidence scores directly.

[0041] The confidence threshold is set to 0.6 for filtering, retaining only the predicted bounding boxes with a confidence level higher than the threshold;

[0042]

[0043] Non-maximum suppression (NMS) removes duplicate detections, based on confidence level p. i sorting in descending order Iterate through and remove boxes whose IoU with high-confidence bounding boxes exceeds a threshold τ. NMS The box:

[0044]

[0045] Output unique detection result Ensure that only the most accurate frame is retained for each object.

[0046] The GrabCut image segmentation algorithm is used to refine the bounding box and generate a binary mask.

[0047] Compare the original image I with the mask M i Multiply to generate result I result Set the background area to white.

[0048]

[0049] (3) Target object defect detection:

[0050] Cracks and fissures: Canny edge detection is used to detect cracks of 1-5mm, with a Gaussian kernel size of (5, 5) and a threshold range of (50, 150).

[0051] Morphological processing connects fractured edges. Crack edges output by Canny may be judged as fractured due to uneven lighting or surface stains. They can be repaired through morphological operations. When the fracture gap is ≤ the core radius, they can be connected.

[0052] Rust and corrosion: texture classification using LBP features + SVM classification;

[0053] Coating peeling and fading: Histogram comparison was performed, using a chi-square test of the color histograms of the test image I and the template T; the color histogram H of the test image I and the template T was calculated. I H T :

[0054]

[0055] When D chi A threshold of <85% for color similarity indicates an anomaly.

[0056] Graffiti Anomalies: Using anomaly detection, the Autoencoder reconstructs the error to locate the anomalous region, and the encoder E and decoder D minimize the reconstruction loss.

[0057]

[0058] Anomaly region location: Error map E map =|ID(E(I))|, thresholding yields the anomaly mask:

[0059]

[0060] Blurred text on traffic signs and road signs: edge sharpness and Laplacian variance <100 are considered blurry.

[0061] Calculate the Laplacian response L of image I:

[0062]

[0063] Variance σ 2 Measuring sharpness:

[0064]

[0065] If σ 2 <100, is judged as fuzzy.

[0066] Further, step S4 includes the following steps:

[0067] (1) Preprocessing:

[0068] Voxel grid filtering is used to reduce the number of point clouds and improve computational efficiency; the point cloud space is divided into cubic voxels with side length l, and a representative point (centroid) is retained within each voxel; for each voxel V... i Point set within Calculate the centroid:

[0069]

[0070] Radius filtering is used to quickly remove obviously isolated noise points. The radius is set to 0.2m, the minimum neighborhood number is set to 5, and N points are retained. i ≥5;

[0071] Statistical filtering is used to further refine the processing of complex noise for each point P. i Calculate the average distance μ of its nearest neighbor k = 40 neighbors. i and standard deviation σ iThe standard deviation multiplier α = 1.5 is used to refine noise reduction by adapting to local density changes and gradually adjusting to avoid excessive deletion.

[0072] Filtering conditions: retention point P i -μ i |||≤α·σ i

[0073] The PCA method is used to calculate the point cloud normals in preparation for subsequent feature extraction; for point P i The neighborhood point set N i Calculate the covariance matrix

[0074]

[0075] Pick The eigenvector n corresponding to the smallest eigenvalue i ;

[0076] The RANSAC algorithm is used to remove ground point clouds, and three points are randomly sampled to fit a plane π: ax + by + cz + d = 0; let point P be... j Distance d to the plane j Points less than the threshold τ are interior points:

[0077]

[0078] Select the planar model with the most interior points and remove all interior points (ground points) to reduce interference with subsequent detection;

[0079] The scene point cloud is segmented into multiple candidate object clusters using Euclidean clustering (DBSCAN); starting from the core point, points with reachable density are recursively merged to form cluster C. k :

[0080] C k ={P j P i -P j |||≤ε,P i ∈C k}

[0081] (2) Detect the target to be inspected:

[0082] Extract feature descriptors (FPFH) from object models and candidate clusters in the database;

[0083] Use nearest neighbor search KD-Tree to match feature points of object models with scene;

[0084] The transformation matrix (rotation + translation) for coarse matching is estimated using the RANSAC algorithm, and erroneous matches are filtered out to complete the coarse registration.

[0085] The ICP algorithm is applied to optimize the pose, minimize the distance error between the model and the scene point cloud, and perform fine registration; the point-to-point distance is minimized.

[0086]

[0087] Repeat the solution for R,t until convergence;

[0088] Calculate the mean square error (MSE) after registration, and set a threshold to determine whether the match is successful; Mean square error:

[0089]

[0090] If MSE≤τ MSE The match was successful.

[0091] Verify the overlap rate and size consistency between the model and the scene point cloud;

[0092] If the same object appears multiple times in the same scene, it is necessary to repeatedly match and verify different poses;

[0093] The principal orientation of the object is determined by principal component analysis, the minimum bounding box is calculated, the point cloud region corresponding to the object to be inspected in the scene is marked, and the 6D pose of the object is output.

[0094] Remove point clouds outside of objects;

[0095] (3) Target object defect detection:

[0096] Depressions and protrusions: Using height difference analysis, calculate the Hausdorff distance between the target surface and the ideal surface; the maximum bidirectional distance between the target surface point cloud P and the ideal surface Q:

[0097]

[0098] For each point p i ∈P, calculate the distance to the nearest point to Q:

[0099]

[0100] If |h i |>τ height Marked as a depression (h) i <0) or protrusion (h) i >0);

[0101] Tilt deformation: Using PCA principal direction analysis, calculate the covariance matrix and eigenvalues; covariance matrix calculation:

[0102]

[0103] Eigenvalue decomposition:

[0104]

[0105] Main direction: Eigenvector v1 corresponding to the largest eigenvalue λ1.

[0106] Inclination angle: relative to the ideal normal n ref The included angle:

[0107]

[0108] If θ>τ angle It was determined to be a tilting deformation;

[0109] Structural Missing: ICP registration standard model, calculate the nearest neighbor distance from each point in the target point cloud P to the template Q:

[0110]

[0111] An anomaly distance threshold is determined based on statistical distribution, and points with residuals exceeding the threshold are filtered out. Threshold setting:

[0112] τ missing =μ+kσ (k is adjustable);

[0113] If d i >τ missing , marked as missing regions.

[0114] Work orders are automatically generated based on missing tags.

[0115] Further, step S5 includes the following steps:

[0116] (1) Match the point cloud and image of the object to be inspected in the same scene.

[0117] Label the vectors between nearest neighbors: calculate 2D and 3D centroids in the image and point cloud respectively;

[0118] Image (2D) centroid: the mask region M for the i-th object i centroid (x) i ,y i The calculation is as follows:

[0119]

[0120] Where N i It is a mask M i The total number of pixels;

[0121] Point cloud (3D) centroid: For the point cloud cluster P of the i-th object i centroid (x) i ,y i ,z iThe calculation is as follows:

[0122]

[0123] Construct a list of coordinates for all centroids, denoted as the image centroid coordinate list and the point cloud centroid coordinate list;

[0124] Image centroid list:

[0125] C 2D ={(x1,y1),(x2,y2),…,(x N ,y N )};

[0126] Point cloud centroid list:

[0127] C 3D ={(x1,y1,z1),(x2,y2,z2),…,(x N ,y N ,z N )};

[0128] For each centroid, calculate the Euclidean distance to other centroids and find the nearest neighbor;

[0129] Mark the centroid location on the image;

[0130] Draw an arrow pointing from the current centroid to the nearest neighbor centroid;

[0131] From c i Point to its nearest neighbor c NN(i) vector:

[0132] v i =c NN(i) -c i =(x NN ( i )-x i ,y NN (i)-y i );

[0133] Store centroid coordinates, nearest neighbor index, and vector information;

[0134] (2) Match vectors with the same orientation in the point cloud and the image:

[0135] Change the y-coordinate value in the point cloud vector information to 0, and denote it as point cloud vector information 2. Unify all objects on the same plane so that their directions can be compared with the vectors in the image;

[0136] Compare two-dimensional vectors in the image Compared with the modified point cloud vector information 2, The included angle;

[0137] formula:

[0138]

[0139] If θ < 5°, the directions are consistent, avoiding matching failure caused by a strict 0° angle.

[0140] Iterate through all possible vector pairs. For each image vector, find all vectors in point cloud vector information 2, calculate the angle between them, and select the point cloud vector with the smallest angle as the matching candidate.

[0141] (3) Result verification:

[0142] For each pair of matched vectors, calculate the magnitude ratio of the image vector information and the point cloud vector information 2:

[0143]

[0144] Calculate the modulus ratio to the mode, and use a histogram to find the densely distributed intervals of the modulus ratio. Based on the range of the mode distribution, set the tolerance interval to ±10% of the mode, retain matching pairs whose modulus ratio values ​​fall within the threshold, and consider the rest as errors;

[0145] Threshold=[Mode×0.9,Mode×1.1];

[0146] After registration, the mean square error between the model and the scene point cloud is calculated. When the mean square error (MSE) is less than 0.1m, the matching is considered successful, and the overlap rate and the consistency of the bounding box size are verified.

[0147] The present invention also discloses an inspection unmanned vehicle device for implementing the above method, comprising:

[0148] Four-wheel drive chassis;

[0149] An industrial control computer, which is mounted on the four-wheel drive chassis, provides a variety of standard interfaces for connecting peripheral devices;

[0150] A lidar is mounted on the four-wheel drive chassis and connected to the industrial control computer via an Ethernet or USB interface.

[0151] An optical camera is mounted on the lidar and connected to the industrial control computer via a USB or Ethernet interface.

[0152] The radar is mounted on the top of the industrial control computer and connected to the industrial control computer via an Ethernet or USB interface.

[0153] An inertial sensor is mounted on the industrial computer and connected to the industrial computer via a serial port, USB, or I2C / SPI.

[0154] A GPS locator is installed on the industrial control computer and connected to the industrial control computer via a serial port.

[0155] A portable WIFI device, wherein the portable WIFI is installed on the industrial control computer and is directly connected to the industrial control computer via a USB interface;

[0156] The router is installed on the industrial control computer and connected to the robot's main power supply via a DC power cable.

[0157] Compared with the prior art, the technical solution of this application has the following technical effects:

[0158] 1. Independent Detection: The image and point cloud detect defects in the target object separately, without interdependence. The image provides high-resolution surface texture information (cracks, fading), while the point cloud provides accurate three-dimensional geometric information (depressions, tilting).

[0159] 2. Result matching: Associate the image with the detection results in the point cloud with the same object instance.

[0160] 3. Decision fusion: If any modality detects a defect, it is determined that a defect exists (logical "OR").

[0161] 4. No calibration required: Traditional point cloud projection onto a plane requires periodic sensor calibration, otherwise it will directly affect the accuracy. This method has no projection step and does not require periodic calibration. Attached Figure Description

[0162] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0163] Figure 1 This is a flowchart illustrating the steps of the intelligent inspection method based on rapid matching of images and point clouds as described in this invention.

[0164] Figure 2 This is a schematic diagram of the unmanned inspection vehicle device described in this invention;

[0165] Figure 3 This describes the system hardware and software coordination relationship of the unmanned vehicle device of the present invention.

[0166] The following are the symbols and their meanings: 1. Radar; 2. Portable WIFI device; 3. Router; 4. Industrial control computer; 5. Inertial sensor; 6. GPS locator; 7. Optical camera; 8. LiDAR; 9. Four-wheel drive chassis. Detailed Implementation

[0167] This invention proposes an intelligent inspection method and device based on rapid matching of images and point clouds. The aim is to design an intelligent inspection method based on rapid matching of images and point clouds to improve inspection efficiency and accuracy while reducing costs.

[0168] The intelligent inspection method based on rapid matching of images and point clouds proposed in this invention will be described below in specific embodiments:

[0169] Example 1:

[0170] An intelligent inspection method for municipal facilities based on image and point cloud matching, such as Figure 1 As shown, it includes the following steps:

[0171] S1: Build a municipal facilities database:

[0172] Collect multi-view image templates of the target object and annotate the precise bounding box of the object in each template;

[0173] Collect multi-angle point cloud templates and convert them into mesh models;

[0174] S2: Deploy unmanned inspection vehicle device: Collect scene images and point clouds by unmanned vehicles equipped with optical cameras, LiDAR, and GPS, wherein the distance between the camera and LiDAR is ≤20cm and hardware triggers synchronization;

[0175] S3: Image Object Detection: Preprocess the scene image, use a lightweight detection model to locate objects and generate masks, and perform crack detection, corrosion classification, color comparison, graffiti anomaly detection and blur determination based on defect type;

[0176] S4: Point cloud target detection: After voxel filtering, denoising, ground segmentation and clustering of the scene point cloud, the pose is verified by RANSACICP registration through FPFH feature matching database model, and defects such as depressions, tilts and structural missing parts are detected.

[0177] S5: Cross-modal target matching:

[0178] Calculate the centroid of the image mask and the centroid of the point cloud cluster respectively, and generate 2D / 3D nearest neighbor vectors;

[0179] After the y-coordinate of the point cloud vector is zeroed out, it is matched with the direction angle of the image vector, and candidate pairs with an angle of less than 5° are filtered out.

[0180] Verify the correctness of the matching based on the ratio of the vector magnitude to the mode distribution;

[0181] S6: Work Order Generation: When a defect is detected in any image or point cloud of the same object, a maintenance work order containing the defect location marker and GPS coordinates is automatically generated.

[0182] Specifically, defect detection is performed on the same object in both the image and point cloud. Based on the target matching results, if no defect is detected in either the image or the point cloud, no work order needs to be generated. However, if a defect is detected in either the image or the point cloud, a work order is generated and sent to the platform. The work order includes: image or point cloud data marking the defect location, and GPS positioning. This allows workers to accurately locate the object, improving maintenance efficiency. Defect detection in the image and point cloud is performed in parallel, with results independent of each other; detecting a defect in either modality triggers work order generation.

[0183] Furthermore, in step S1:

[0184] The acquisition of the image templates covers the front, side, and top views as well as different lighting conditions, with ≥20 templates for a single object;

[0185] The point cloud template is scanned by LiDAR from multiple angles, registered by ICP, and then reconstructed using Poisson to generate a mesh model.

[0186] Specifically, the lighting conditions were as follows: front light (coaxial with the camera), side light (45° / 90°), backlight, and top light; the light intensities were medium (800 lux) and high (2200 lux); and the color temperatures were neutral (4000-4500K) and cool (5000-6500K).

[0187] Further, step S3 includes the following steps:

[0188] (1) Scene image preprocessing:

[0189] Unify the input image resolution, assuming the original image is... The target resolution is (H new W new Scaling is achieved through bilinear interpolation, and the output image I is obtained. resized The pixel value is calculated as follows:

[0190] I resized (x ′ ,y ′ )=∑ i,j∈{0,1} I(x i ,y i )·(1-|x ′ ,x i |)·(1-|y ′ ,y i |);

[0191] Gaussian filtering and histogram equalization are performed to equalize the illumination and reduce environmental interference; for I resized Gaussian smoothing is performed, and the convolution kernel G is defined as:

[0192]

[0193] Output image I blur Calculated via convolution:

[0194] I blur (x,y)=(I resized *G)(x,y);

[0195] Calculate the cumulative distribution function (CDF):

[0196]

[0197] Where, n i Let N be the number of pixels at gray level i, N be the total number of pixels, and L be the number of gray levels.

[0198] The mapping function is:

[0199] I equalized (x,y)=round((L-1)·CDF(I blur (x,y)));

[0200] (2) Use a lightweight detection model to locate objects and generate masks:

[0201] A training set is constructed using labeled templates and scene images;

[0202] Choose the lightweight YOLOv8 model for transfer learning;

[0203] Add random rotation, occlusion, and brightness perturbation to improve generalization ability;

[0204] Random rotation: rotation transformation matrix parameter

[0205] Random occlusion: Fill the region (x,y,ω,h) with noise.

[0206] Brightness perturbation: pixel value scaling I ′ =γI,

[0207] The model takes a scene image as input and outputs bounding boxes with category labels and confidence scores directly.

[0208] The confidence threshold is set to 0.6 for filtering, retaining only the predicted bounding boxes with a confidence level higher than the threshold;

[0209]

[0210] Non-maximum suppression (NMS) removes duplicate detections, based on confidence level p. i sorting in descending order Iterate through and remove boxes whose IoU with high-confidence bounding boxes exceeds a threshold τ. NMS The box:

[0211]

[0212] Output unique detection result Ensure that only the most accurate frame is retained for each object.

[0213] The GrabCut image segmentation algorithm is used to refine the bounding box and generate a binary mask.

[0214] Compare the original image I with the mask M i Multiply to generate result I result Set the background area to white.

[0215]

[0216] (3) Target object defect detection:

[0217] Cracks and fissures: Canny edge detection is used to detect cracks of 1-5mm, with a Gaussian kernel size of (5, 5) and a threshold range of (50, 150).

[0218] Morphological processing connects fractured edges. Crack edges output by Canny may be judged as fractured due to uneven lighting or surface stains. They can be repaired through morphological operations. When the fracture gap is ≤ the core radius, they can be connected.

[0219] Rust and corrosion: texture classification using LBP features + SVM classification;

[0220] Coating peeling and fading: Histogram comparison was performed, using a chi-square test of the color histograms of the test image I and the template T; the color histogram H of the test image I and the template T was calculated. I H T :

[0221]

[0222] When D chi A threshold of <85% for color similarity indicates an anomaly.

[0223] Graffiti Anomalies: Using anomaly detection, the Autoencoder reconstructs the error to locate the anomalous region, and the encoder E and decoder D minimize the reconstruction loss.

[0224]

[0225] Anomaly region location: Error map E map =|ID(E(I))|, thresholding yields the anomaly mask:

[0226]

[0227] Blurred text on traffic signs and road signs: edge sharpness and Laplacian variance <100 are considered blurry.

[0228] Calculate the Laplacian response L of image I:

[0229]

[0230] Variance σ 2 Measuring sharpness:

[0231]

[0232] If σ 2 <100, is judged as fuzzy.

[0233] Further, step S4 includes the following steps:

[0234] (1) Preprocessing:

[0235] Voxel grid filtering is used to reduce the number of point clouds and improve computational efficiency; the point cloud space is divided into cubic voxels with side length l, and a representative point (centroid) is retained within each voxel; for each voxel V... i Point set within Calculate the centroid:

[0236]

[0237] Radius filtering is used to quickly remove obviously isolated noise points. The radius is set to 0.2m, the minimum neighborhood number is set to 5, and N points are retained. i ≥5;

[0238] Statistical filtering is used to further refine the processing of complex noise for each point P. i Calculate the average distance μ of its nearest neighbor k = 40 neighbors. i and standard deviation σ i The standard deviation multiplier α = 1.5 is used to refine noise reduction by adapting to local density changes and gradually adjusting to avoid excessive deletion.

[0239] Filtering conditions: retention point P i -μ i |||≤α·σ i

[0240] The PCA method is used to calculate the point cloud normals in preparation for subsequent feature extraction; for point P i The neighborhood point set N i Calculate the covariance matrix

[0241]

[0242] Pick The eigenvector n corresponding to the smallest eigenvalue i ;

[0243] The RANSAC algorithm is used to remove ground point clouds, and three points are randomly sampled to fit a plane π: ax + by + cz + d = 0; let point P be... j Distance d to the plane j Points less than the threshold τ are interior points:

[0244]

[0245] Select the planar model with the most interior points and remove all interior points (ground points) to reduce interference with subsequent detection;

[0246] The scene point cloud is segmented into multiple candidate object clusters using Euclidean clustering (DBSCAN); starting from the core point, points with reachable density are recursively merged to form cluster C. k :

[0247] C k ={P j P i -P j |||≤ε,P i ∈C k}

[0248] (2) Detect the target to be inspected:

[0249] Extract feature descriptors (FPFH) from object models and candidate clusters in the database;

[0250] Use nearest neighbor search KD-Tree to match feature points of object models with scene;

[0251] The transformation matrix (rotation + translation) for coarse matching is estimated using the RANSAC algorithm, and erroneous matches are filtered out to complete the coarse registration.

[0252] The ICP algorithm is applied to optimize the pose, minimize the distance error between the model and the scene point cloud, and perform fine registration; the point-to-point distance is minimized.

[0253]

[0254] Repeat the solution for R,t until convergence;

[0255] Calculate the mean square error (MSE) after registration, and set a threshold to determine whether the match is successful; Mean square error:

[0256]

[0257] If MSE≤τ MSE The match was successful.

[0258] Verify the overlap rate and size consistency between the model and the scene point cloud;

[0259] If the same object appears multiple times in the same scene, it is necessary to repeatedly match and verify different poses;

[0260] The principal orientation of the object is determined by principal component analysis, the minimum bounding box is calculated, the point cloud region corresponding to the object to be inspected in the scene is marked, and the 6D pose of the object is output.

[0261] Remove point clouds outside of objects;

[0262] (3) Target object defect detection:

[0263] Depressions and protrusions: Using height difference analysis, calculate the Hausdorff distance between the target surface and the ideal surface; the maximum bidirectional distance between the target surface point cloud P and the ideal surface Q:

[0264]

[0265] For each point p i ∈P, calculate the distance to the nearest point to Q:

[0266]

[0267] If |h i |>τ height Marked as a depression (h) i <0) or protrusion (h) i >0);

[0268] Tilt deformation: Using PCA principal direction analysis, calculate the covariance matrix and eigenvalues; covariance matrix calculation:

[0269]

[0270] Eigenvalue decomposition:

[0271] C vj =λ j v j (j=1,2,3),λ1≥λ2≥λ3;

[0272] Main direction: Eigenvector v1 corresponding to the largest eigenvalue λ1.

[0273] Inclination angle: relative to the ideal normal n ref The included angle:

[0274]

[0275] If θ>τ angle It was determined to be a tilting deformation;

[0276] Structural Missing: ICP registration standard model, calculate the nearest neighbor distance from each point in the target point cloud P to the template Q:

[0277]

[0278] An anomaly distance threshold is determined based on statistical distribution, and points with residuals exceeding the threshold are filtered out. Threshold setting:

[0279] τ missing =μ+kσ (k is adjustable);

[0280] If d i >τ missing , marked as missing regions.

[0281] Work orders are automatically generated based on missing tags.

[0282] Further, step S5 includes the following steps:

[0283] (1) Match the point cloud and image of the object to be inspected in the same scene.

[0284] Label the vectors between nearest neighbors: calculate 2D and 3D centroids in the image and point cloud respectively;

[0285] Image (2D) centroid: the mask region M for the i-th object i centroid (x) i ,y i The calculation is as follows:

[0286]

[0287] Where N i It is a mask M i The total number of pixels;

[0288] Point cloud (3D) centroid: For the point cloud cluster P of the i-th object i centroid (x) i ,y i ,z i The calculation is as follows:

[0289]

[0290] Construct a list of coordinates for all centroids, denoted as the image centroid coordinate list and the point cloud centroid coordinate list;

[0291] Image centroid list:

[0292] C 2D ={(x1,y1),(x2,y2),…,(x N ,y N )};

[0293] Point cloud centroid list:

[0294] C 3D ={(x1,y1,z1),(x2,y2,z2),…,(x N ,y N ,z N )};

[0295] For each centroid, calculate the Euclidean distance to other centroids and find the nearest neighbor;

[0296] Mark the centroid location on the image;

[0297] Draw an arrow pointing from the current centroid to the nearest neighbor centroid;

[0298] From c i Point to its nearest neighbor c NN(i) vector:

[0299] v i =c NN(i) -c i =(x NN ( i )-x i ,y NN (i)-y i );

[0300] Store centroid coordinates, nearest neighbor index, and vector information;

[0301] (2) Match vectors with the same orientation in the point cloud and the image:

[0302] Change the y-coordinate value in the point cloud vector information to 0, and denote it as point cloud vector information 2. Unify all objects on the same plane so that their directions can be compared with the vectors in the image;

[0303] Compare two-dimensional vectors in the image Compared with the modified point cloud vector information 2, The included angle;

[0304] formula:

[0305]

[0306] If θ < 5°, the directions are consistent, avoiding matching failure caused by a strict 0° angle.

[0307] Iterate through all possible vector pairs. For each image vector, find all vectors in point cloud vector information 2, calculate the angle between them, and select the point cloud vector with the smallest angle as the matching candidate.

[0308] (3) Result verification:

[0309] For each pair of matched vectors, calculate the magnitude ratio of the image vector information and the point cloud vector information 2:

[0310]

[0311] Calculate the modulus ratio to the mode, and use a histogram to find the densely distributed intervals of the modulus ratio. Based on the range of the mode distribution, set the tolerance interval to ±10% of the mode, retain matching pairs whose modulus ratio values ​​fall within the threshold, and consider the rest as errors;

[0312] Threshold=[Mode×0.9,Mode×1.1];

[0313] After registration, the mean square error between the model and the scene point cloud is calculated. When the mean square error (MSE) is less than 0.1m, the matching is considered successful, and the overlap rate and the consistency of the bounding box size are verified.

[0314] Example 2:

[0315] An unmanned inspection vehicle device is used to implement the method of Embodiment 1, such as... Figure 2 As shown, it includes:

[0316] Four-wheel drive chassis 9;

[0317] Industrial computer 4 is mounted on four-wheel drive chassis 9 and provides a variety of standard interfaces for connecting peripheral devices.

[0318] LiDAR 8 is mounted on a four-wheel drive chassis and connected to the industrial control computer 4 via Ethernet or USB interface.

[0319] Optical camera 7 is mounted on lidar 8 and connected to industrial computer 4 via USB or Ethernet interface;

[0320] Radar 1 is mounted on the top of industrial computer 4 and is connected to industrial computer 4 via Ethernet or USB interface;

[0321] Inertial sensor 5 is mounted on industrial computer 4 and connected to industrial computer 4 via serial port, USB or I2C / SPI.

[0322] GPS locator 6 is installed on industrial computer 4 and connected to industrial computer 4 via serial port;

[0323] Portable WIFI device 2 is installed on industrial computer 4 and is directly connected to industrial computer 4 via USB interface;

[0324] Router 3 is installed on industrial computer 4 and is connected to the robot's main power supply via a DC power cable.

[0325] The software architecture of the unmanned inspection vehicle adopts a modular design, relying on the Vehicle Operating System (ROS) framework for efficient collaboration. The entire system consists of five core modules: an inspection control mobile terminal, an autonomous driving algorithm cluster, a multi-source data acquisition and processing unit, and a cloud database. The autonomous driving algorithm, image data acquisition, and point cloud data acquisition and processing programs run as nodes within the ROS framework. The autonomous driving algorithm, including mapping, localization, path planning, and chassis control algorithms, serves as the system's decision-making center, providing the unmanned vehicle with centimeter-level positioning accuracy and adaptive obstacle avoidance capabilities. The image data acquisition and point cloud data acquisition and processing programs are responsible for processing municipal facility data collected by optical cameras and LiDAR, and sending the results to the cloud database for storage. The inspection control mobile terminal is an Android-based interactive interface. Users can send inspection start / stop commands to the unmanned vehicle, monitor equipment status in real time, and retrieve historical data for comparative analysis through the communication mode under the ROS framework. Command data is transmitted to the motion control node via the ROS service call mechanism, realizing full lifecycle management of the task. The various functional modules interact through the ROS node communication mechanism, and their hardware and software collaboration relationships are as follows: Figure 3 As shown.

[0326] Radar, inertial sensors, GPS, LiDAR, and image acquisition and processing programs will automatically start after the autonomous vehicle is powered on, preparing for the upcoming inspection. As the inspection proceeds, the cloud database will synchronously receive and save the images and point cloud data of the municipal facilities, supporting petabyte-level municipal facility data storage and backtracking, providing a complete technical solution for smart city operation and maintenance.

[0327] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for intelligent inspection of municipal facilities based on image and point cloud matching, characterized in that, Includes the following steps: S1: Build a municipal facilities database: Collect multi-view image templates of the target object and annotate the precise bounding box of the object in each template; Collect multi-angle point cloud templates and convert them into mesh models; S2: Deploy unmanned inspection vehicles: Collect scene images and point clouds by unmanned vehicles equipped with optical cameras, LiDAR, and GPS, wherein the distance between the camera and LiDAR is ≤20cm and hardware triggers synchronization; S3: Image Object Detection: Preprocess the scene image, use a lightweight detection model to locate objects and generate masks, and perform crack detection, corrosion classification, color comparison, graffiti anomaly detection and blur determination based on defect type; S4: Point cloud target detection: After voxel filtering, denoising, ground segmentation and clustering of the scene point cloud, the pose is verified by RANSACICP registration through FPFH feature matching database model, and defects such as depressions, tilts and structural missing parts are detected. S5: Cross-modal target matching: Calculate the centroid of the image mask and the centroid of the point cloud cluster respectively, and generate 2D / 3D nearest neighbor vectors; After the y-coordinate of the point cloud vector is zeroed out, it is matched with the direction angle of the image vector, and candidate pairs with an angle of less than 5° are filtered out. Verify the correctness of the matching based on the ratio of the vector magnitude to the mode distribution; S6: Work Order Generation: When a defect is detected in any image or point cloud of the same object, a maintenance work order containing the defect location marker and GPS coordinates is automatically generated.

2. The intelligent inspection method for municipal facilities based on image and point cloud matching according to claim 1, characterized in that, In step S1: The acquisition of the image templates covers the front, side, and top views as well as different lighting conditions, with ≥20 templates for a single object; The point cloud template is scanned by LiDAR from multiple angles, registered by ICP, and then reconstructed using Poisson to generate a mesh model.

3. The intelligent inspection method for municipal facilities based on image and point cloud matching according to claim 1, characterized in that, Step S3 includes the following steps: (1) Scene image preprocessing: Unify the input image resolution, assuming the original image is... The target resolution is (H new W new Scaling is achieved through bilinear interpolation, and the output image I is obtained. resized The pixel value is calculated as follows: Yo resized (x ′ ,and ′ )=∑ i,j∈{0,1} I(x i ,and i )·(1-|x ′ ,x i |)·(1-|y ′ ,and i |); Gaussian filtering and histogram equalization are performed to equalize the illumination and reduce environmental interference; for I resized Gaussian smoothing is performed, and the convolution kernel G is defined as: Output image I blur Calculated via convolution: I blur (x,y)=(I resized *G)(x,y); Calculate the cumulative distribution function: Where, n i Let N be the number of pixels at gray level i, N be the total number of pixels, and L be the number of gray levels. The mapping function is: I equalized (x,y)=round((L-1)·CDF(I blur (x,y))); (2) Use a lightweight detection model to locate objects and generate masks: A training set is constructed using labeled templates and scene images; Choose the lightweight YOLOv8 model for transfer learning; Add random rotation, occlusion, and brightness perturbation to improve generalization ability; Random rotation: rotation transformation matrix parameter Random occlusion: Fill the region (x,y,ω,h) with noise. Brightness perturbation: pixel value scaling The model takes a scene image as input and outputs bounding boxes with category labels and confidence scores directly. The confidence threshold is set to 0.6 for filtering, retaining only the predicted bounding boxes with a confidence level higher than the threshold; Non-maximum suppression removes duplicate detections, based on confidence level p. i sorting in descending order Iterate through and remove boxes whose IoU with high-confidence bounding boxes exceeds a threshold τ. NMS The box: Output unique detection result Ensure that only the most accurate frame is retained for each object. The GrabCut image segmentation algorithm is used to refine the bounding box and generate a binary mask. Compare the original image I with the mask M i Multiply to generate result I result Set the background area to white. (3) Target object defect detection: Cracks and fissures: Canny edge detection is used to detect cracks of 1-5mm, with a Gaussian kernel size of (5, 5) and a threshold range of (50, 150). Morphological treatment connects the fractured edges, and repair is achieved through morphological manipulation. Connection is possible when the fracture gap is ≤ the core radius. Rust and corrosion: texture classification using LBP features + SVM classification; Coating peeling and fading: Histogram comparison was performed, using a chi-square test of the color histograms of the test image I and the template T; the color histogram H of the test image I and the template T was calculated. I H T : When D chi A threshold of <85% for color similarity indicates an anomaly. Graffiti Anomalies: Using anomaly detection, the Autoencoder reconstructs the error to locate the anomalous region, and the encoder E and decoder D minimize the reconstruction loss. Anomaly region location: Error map E map =|ID(E(I))|, thresholding yields the anomaly mask: Blurred text on traffic signs and road signs: edge sharpness and Laplacian variance <100 are considered blurry. Calculate the Laplacian response L of image I: Variance σ 2 Measuring sharpness: If σ 2 <100, is judged as fuzzy.

4. The intelligent inspection method for municipal facilities based on image and point cloud matching according to claim 1, characterized in that, Step S4 includes the following steps: (1) Preprocessing: Voxel raster filtering is used to reduce the number of point clouds and improve computational efficiency; the point cloud space is divided into cubic voxels with side length l, and each voxel retains a representative point; for each voxel V i Point set within Calculate the centroid: Radius filtering is used to quickly remove obviously isolated noise points. The radius is set to 0.2m, the minimum neighborhood number is set to 5, and N points are retained. i ≥5; Statistical filtering is used to further refine the processing of complex noise for each point P. i Calculate the average distance μ of its nearest neighbor k = 40 neighbors. i and standard deviation σ i The standard deviation multiplier α = 1.5 is used to refine noise reduction by adapting to local density changes and gradually adjusting to avoid excessive deletion. Filtering conditions: retention point Calculate the point cloud normal using the PCA method; for point P i The neighborhood point set N i Calculate the covariance matrix Pick The eigenvector n corresponding to the smallest eigenvalue i ; The RANSAC algorithm is used to remove ground point clouds, and three points are randomly sampled to fit a plane π: ax + by + cz + d = 0; let point P be... j Distance d to the plane j Points less than the threshold τ are interior points: Select the planar model with the most interior points and remove all interior points; The scene point cloud is segmented into multiple candidate object clusters using Euclidean clustering (DBSCAN); starting from the core point, points with reachable density are recursively merged to form cluster C. k : C k ={P j ‖‖P i -P j ‖‖≤ε,P i ∈C k }; (2) Detect the target to be inspected: Extract feature descriptors (FPFH) from object models and candidate clusters in the database; Use nearest neighbor search KD-Tree to match feature points of object models with scene; The transformation matrix of the coarse matching is estimated by using the RANSAC algorithm, and erroneous matches are filtered out to complete the coarse registration. The ICP algorithm is applied to optimize the pose, minimize the distance error between the model and the scene point cloud, and perform fine registration; the point-to-point distance is minimized. Repeat the solution for R,t until convergence; Calculate the mean square error (MSE) after registration, and set a threshold to determine whether the match is successful; Mean square error: If MSE≤τ MSE The match was successful. Verify the overlap rate and size consistency between the model and the scene point cloud; If the same object appears multiple times in the same scene, it is necessary to repeatedly match and verify different poses; The principal orientation of the object is determined by principal component analysis, the minimum bounding box is calculated, the point cloud region corresponding to the object to be inspected in the scene is marked, and the 6D pose of the object is output. Remove point clouds outside of objects; (3) Target object defect detection: Depressions and protrusions: Using height difference analysis, calculate the Hausdorff distance between the target surface and the ideal surface; the maximum bidirectional distance between the target surface point cloud P and the ideal surface Q: For each point p i ∈P, calculate the distance to the nearest point to Q: If |h i |>τ height Marked as a depression (h) i <0) or protrusion (h) i >0); Tilt deformation: Using PCA principal direction analysis, calculate the covariance matrix and eigenvalues; covariance matrix calculation: Eigenvalue decomposition: C vj =λ j v j (j=1,2,3),λ1≥λ2≥λ3; Main direction: Eigenvector v1 corresponding to the largest eigenvalue λ1. Inclination angle: relative to the ideal normal n ref The included angle: If θ>τ angle It was determined to be a tilting deformation; Structural Missing: ICP registration standard model, calculate the nearest neighbor distance from each point in the target point cloud P to the template Q: An anomaly distance threshold is determined based on statistical distribution, and points with residuals exceeding the threshold are filtered out. Threshold setting: τ missing =μ+kσ (k is adjustable); If d i >τ missing , marked as missing regions. Work orders are automatically generated based on missing tags.

5. The intelligent inspection method for municipal facilities based on image and point cloud matching according to claim 1, characterized in that, Step S5 includes the following steps: (1) Match the point cloud and image of the object to be inspected in the same scene. Label the vectors between nearest neighbors: calculate 2D and 3D centroids in the image and point cloud respectively; Image centroid: The mask region M for the i-th object i centroid (x) i ,y i The calculation is as follows: Where N i It is a mask M i The total number of pixels; Point cloud centroid: for the point cloud cluster P of the i-th object i centroid (x) i ,y i ,z i The calculation is as follows: Construct a list of coordinates for all centroids, denoted as the image centroid coordinate list and the point cloud centroid coordinate list; Image centroid list: C 2D ={(x1,y1),(x2,y2),…,(x N ,y N )}; Point cloud centroid list: C 3D {(x1,y1,z1),(x2,y2,z2),…,(x N ,y N ,z N )}? For each centroid, calculate the Euclidean distance to other centroids and find the nearest neighbor; Mark the centroid location on the image; Draw an arrow pointing from the current centroid to the nearest neighbor centroid; From c i Point to its nearest neighbor c NN(i) vector: v i =c NN(i) -c i =(x NN ( i )-x i ,y NN (i)-y i ); Store centroid coordinates, nearest neighbor index, and vector information; (2) Match vectors with the same orientation in the point cloud and the image: Change the y-coordinate value in the point cloud vector information to 0, and denote it as point cloud vector information 2. Unify all objects on the same plane so that their directions can be compared with the vectors in the image; Compare two-dimensional vectors in the image Compared with the modified point cloud vector information 2, The included angle; formula: If θ < 5°, the directions are consistent to avoid matching failure caused by a strict 0° angle. Iterate through all possible vector pairs. For each image vector, find all vectors in point cloud vector information 2, calculate the angle between them, and select the point cloud vector with the smallest angle as the matching candidate. (3) Result verification: For each pair of matched vectors, calculate the magnitude ratio of the image vector information and the point cloud vector information 2: Calculate the modulus ratio to the mode, and use a histogram to find the densely distributed intervals of the modulus ratio. Based on the range of the mode distribution, set the tolerance interval to ±10% of the mode, retain matching pairs whose modulus ratio values ​​fall within the threshold, and consider the rest as errors; Threshold=[Mode×0.9,Mode×1.1]; After registration, the mean square error between the model and the scene point cloud is calculated. When the mean square error (MSE) is less than 0.1m, the matching is considered successful, and the overlap rate and the consistency of the bounding box size are verified.

6. An unmanned inspection vehicle device that implements the method of any one of claims 1-5, characterized in that, include: Four-wheel drive chassis; An industrial control computer, which is mounted on the four-wheel drive chassis, provides a variety of standard interfaces for connecting peripheral devices; A lidar is mounted on the four-wheel drive chassis and connected to the industrial control computer via an Ethernet or USB interface. An optical camera is mounted on the lidar and connected to the industrial control computer via a USB or Ethernet interface. The radar is mounted on the top of the industrial control computer and connected to the industrial control computer via an Ethernet or USB interface. An inertial sensor is mounted on the industrial computer and connected to the industrial computer via a serial port, USB, or I2C / SPI. A GPS locator is installed on the industrial control computer and connected to the industrial control computer via a serial port. A portable WIFI device, wherein the portable WIFI device is installed on the industrial control computer and is directly connected to the industrial control computer via a USB interface; The router is installed on the industrial control computer and connected to the robot's main power supply via a DC power cable.

Images