Method for collaborative inspection of roads, bridges, and third-party construction through drones

Abstract

The embodiment of the present invention discloses a method for collaborative inspection of road and bridge construction by unmanned aerial vehicles, which relates to the technical field of intelligent inspection of infrastructure. After preprocessing the image data obtained by shooting, the abnormality in the scene to be inspected is detected, and according to the result of the abnormality detection, a spatial distribution model corresponding to the scene to be inspected is established, and the key influencing factors of the abnormality in the scene to be inspected are determined by using the established spatial distribution model, so that the inspection of three specific scenes is completed by the collaborative cooperation of two unmanned aerial vehicles, a large amount of inspection human resources is saved, and the accuracy of the inspection is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent inspection of infrastructure, and in particular to a method for collaborative inspection of road and bridge construction by unmanned aerial vehicles. Background Art

[0002] With the acceleration of urbanization, the scale and complexity of urban roads, bridges and other infrastructure are constantly increasing, and third-party construction activities are becoming more frequent. Traditional manual inspection methods have problems such as low efficiency, limited coverage, and lack of timeliness. It is difficult to detect and deal with infrastructure diseases and safety hazards in a timely manner, and cannot meet the needs of modern cities for efficient management and safety monitoring of infrastructure.

[0003] The current manual inspection methods mostly focus on a single scene, which makes it difficult to timely and effectively implement simultaneous inspections of multiple targets such as roads, bridges, and construction sites. This limitation leads to low inspection efficiency and waste of resources, and cannot meet the needs of modern cities for efficient management of infrastructure. Due to the lack of the ability to conduct simultaneous inspections of multiple targets, some hidden dangers may not be discovered in time, increasing the risk of urban operations.

[0004] In addition, manual inspections mainly rely on visual inspections and require on-site personnel to operate portable detection equipment, which is easily affected by the environment, light, and personnel experience. If you want to improve the accuracy of inspections, you need to increase the frequency of inspections and organize multiple working groups to conduct multiple inspections. This will improve the accuracy of identifying and evaluating facility defects, which will take up a lot of human resources. Summary of the invention

[0005] The embodiments of the present invention provide a method for collaborative inspection of road and bridge construction by unmanned aerial vehicles, which can save a large amount of inspection manpower and improve the accuracy of inspection.

[0006] To achieve the above object, the embodiments of the present invention adopt the following technical solutions:

[0007] S1. Collecting data related to the scene to be inspected by using a drone, wherein the types of the scene to be inspected include roads, bridges, and third-party construction areas, and the drone includes a fixed-wing drone and a multi-rotor drone;

[0008] S2, after pre-processing the captured image data, detecting anomalies in the scene to be inspected;

[0009] S3. Establishing a spatial distribution model corresponding to the scene to be inspected according to the result of anomaly detection;

[0010] S4. Using the established spatial distribution model, determine the key influencing factors of the anomalies in the scene to be inspected and display them.

[0011] The method for collaborative inspection of road and bridge construction by unmanned aerial vehicles provided in an embodiment of the present invention pre-processes the captured image data, detects anomalies in the scene to be inspected, and establishes a spatial distribution model corresponding to the scene to be inspected based on the result of the anomaly detection. The established spatial distribution model is used to determine the key influencing factors of the anomalies in the scene to be inspected, thereby completing the inspection of three specific scenes through the collaborative cooperation of two unmanned aerial vehicles, saving a large amount of inspection human resources and improving the accuracy of the inspection. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0013] Figure 1 , 2 A schematic diagram of a method flow chart provided by an embodiment of the present invention;

[0014] Figure 3 , 4 A schematic diagram of a specific example provided for an embodiment of the present invention. DETAILED DESCRIPTION

[0015] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The embodiments of the present invention will be described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements with the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention. It can be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "one", "said" and "the" used herein may also include plural forms. It should be further understood that the term "including" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or their groups. It should be understood that when we say that an element is "connected" or "coupled" to another element, it can be directly connected or coupled to other elements, or there may also be intermediate elements. In addition, the "connection" or "coupling" used here may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items. It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meaning as generally understood by those of ordinary skill in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0016] The embodiment of the present invention provides a method for collaboratively inspecting roads, bridges, and third-party construction by using drones, such as Figure 1 , 2 As shown, including:

[0017] S1. Collecting data related to the scene to be inspected by using a drone, wherein the types of the scene to be inspected include roads, bridges, and third-party construction areas, and the drone includes a fixed-wing drone and a multi-rotor drone;

[0018] S2, after pre-processing the captured image data, detecting anomalies in the scene to be inspected;

[0019] S3. Establishing a spatial distribution model corresponding to the scene to be inspected according to the result of anomaly detection;

[0020] S4. Using the established spatial distribution model, determine the key influencing factors of the anomalies in the scene to be inspected and display them.

[0021] Among them, the data related to the scene to be inspected includes: data collected by the onboard equipment of the fixed-wing UAV, and the types of onboard equipment of the fixed-wing UAV include: LiDAR sensor, multispectral camera and high-definition camera; and, data collected by the onboard equipment of the multi-rotor UAV, and the types of onboard equipment of the multi-rotor UAV include: high-definition camera, multispectral camera, lidar, thermal imager and / or RFID positioning detection equipment.

[0022] In this embodiment, in S1, it includes: using the data collected by the onboard equipment of the fixed-wing UAV to perform terrain modeling, and confirming the reshoot area in the terrain modeling; starting the multi-rotor UAV to shoot the reshoot area, wherein the reshoot area includes: an area with abnormal terrain, an area with occlusion, and a specific area with suspected diseases. The UAV operator can determine whether it is a suspected abnormal area. For example: the data collected by the fixed-wing UAV includes: LiDAR point cloud data, spectral image data in the near-infrared and infrared bands, orthophoto image data and video data. Fixed-wing UAVs have the characteristics of high efficiency, wide range, fast speed, and strong wind resistance. They are used to uniformly collect data for three scenes: roads, bridges, and third-party construction. In the inspection scheme of this embodiment, the fixed-wing UAV is equipped with LiDAR (laser radar), high-definition cameras, multi-spectral cameras and other equipment. LiDAR sensor: collects three-dimensional LiDAR point cloud data for generating a large-scale three-dimensional ground model and preliminary detection of terrain changes. Multispectral camera: Acquires spectral image data in the near-infrared and infrared bands, mainly used to filter out surface vegetation in the lidar point cloud to provide support for the generation of accurate terrain models. HD camera: Collects orthophotos and video data to identify surface defects on roads and bridges (such as crack width and depth) and to record real-time conditions at third-party construction sites. For areas with abnormalities after modeling (such as subsidence, abnormal protrusions, etc.), or areas with obstructions or specific areas suspected of being damaged, the shortest path planning that can connect all the above areas in series is used, and multi-rotor drones are used for additional collection.

[0023] The image data captured by the multi-rotor drone includes: laser point cloud, image map, thermal imaging. The multi-rotor drone will make up for the data that cannot be collected at one time due to occlusion on the fixed-wing drone collection path, and call the data collection equipment carried according to the scene requirements. The road scene is equipped with high-definition cameras and LiDAR laser radars to collect image data for road condition analysis; the bridge scene multi-rotor drone undertakes refined inspection tasks, equipped with thermal imagers, LiDAR laser radars and high-definition cameras, combined with the existing urban lifeline bridge scene subsystem, to achieve comprehensive integration of inspection data and long-term monitoring data, specifically including: thermal imagers are used to detect structural diseases, such as concrete delamination, steel corrosion, etc.; LiDAR laser radar generates a three-dimensional point cloud model of the bridge to capture geometric deformation and structural anomalies; high-definition cameras record surface diseases to assist manual and AI in identifying the characteristics of diseases; link the monitoring equipment deployed in the bridge scene in the urban lifeline system to complete data upload, abnormal triggering, dynamic load analysis and report generation, and provide support for the health assessment of the bridge throughout its life cycle. Third-party construction uses laser radar and thermal imaging to scan the ground during night inspections to determine whether there are areas of abnormal terrain (such as temporary soil piles or deep foundation pit excavation, etc.). Thermal imaging is used to determine whether there are abnormal heat sources around the road (such as abnormal human activities or abnormally high temperatures). Finally, the construction machinery equipped with RFID positioning sensors is located to obtain real-time positioning of the equipment and personnel location information.

[0024] Intelligent flight plans are formulated for roads, bridges and third-party construction areas to be inspected to achieve efficient inspections while meeting specific needs. The formulation of flight plans includes the following steps: Automatic generation of basic flight routes: Through the built-in flight planning software of the drone, the basic flight route covering the target area is automatically generated according to the digital map information and task requirements of the inspection area. The flight route is planned according to the target area range, flight parameter settings and task priorities of the inspection task to ensure the comprehensiveness and coverage of data collection. Manual adjustment of flight plans: When the inspection unit raises special needs or there are personalized goals in the task, the flight plan is manually adjusted to refine the flight plan. For example, the sampling density can be increased for key areas, the flight path can be optimized, or the inspection time window and flight mode can be adjusted according to specific task requirements.

[0025] In this embodiment, in S2, the preprocessing includes: organizing the image data captured by the fixed-wing UAV and the multi-rotor UAV into a data set, and establishing a mapping relationship between the data set and the geographic coordinates; to facilitate further analysis. After the data of all projects are collected separately, in order to ensure the data quality, denoising, correction and cleaning will be performed to eliminate redundant and abnormal information, and the data will be integrated into a unified geographic coordinate system according to the needs of different projects.

[0026] Among them, for road scenes, the high-definition image data collected by fixed-wing drones and the multispectral data taken by multi-rotor drones are fused to establish a data set; for bridge scenes, the LiDAR point cloud data collected by fixed-wing drones and the thermal imaging data taken by multi-rotor drones are fused to establish a data set; for third-party construction area scenes, the LiDAR point cloud data and high-definition image data collected by fixed-wing drones and the RFID data collected by multi-rotor drones are fused to establish a data set.

[0027] Specifically, a data set is established for road scenes, including: removing blurry image frames in high-definition images and cleaning multispectral data; then, the high-definition images are aligned with the multispectral data through the registration function, and a textured road model is generated; for example, the specific operations for road projects are:

[0028] (1) The high-definition images and multispectral data will be exported, and the image resolution will be unified by removing blurry image frames. The set of high-definition images collected by the drone will be counted as:

[0029] D HD ={I1,I2,…,I n}

[0030] I i Represents the i-th frame image, i is the numerical label of the frame, 1≤i≤n, and n is the maximum number of frames.

[0031] Multispectral imagery collections are counted as:

[0032] D MS ={M1,M2,…,M i}

[0033] M i Represents the multispectral data of the i-th frame.

[0034] The blurred image frames are removed as follows:

[0035]

[0036] τ is the ambiguity threshold,

[0037] Sharpness i ) is the image clarity function.

[0038] (2) Perform multispectral data cleaning on the above sets:

[0039]

[0040] (3) Align the image with the multispectral data to generate a textured road model. The spatial alignment of the image and the multispectral data is completed through the alignment function Align(I,M):

[0041]

[0042] Among them, F fused Represents the fusion result of high-definition image and multispectral image.

[0043] (4) Finally, the fused data crack detection result set and spectral feature set are output:

[0044] 1. It is expressed as a crack distribution map (crack detection result set F crack ):

[0045] F crack ={(x,y,w,d)}, x,y are the coordinates of the crack center, w is the crack width, d is the crack depth

[0046] 2. Multispectral analysis results (spectral characteristic set F spectral ):

[0047] F spectral ={(x,y,λ1,λ2,…,λ k )},λ k Represents the reflectance values ​​of the multispectral band.

[0048] Establish a data set for the bridge scene, including: denoising the LiDAR point cloud data collected by fixed-wing drones, cleaning the thermal imaging data taken by multi-rotor drones, and spatially superimposing the thermal imaging data with the LiDAR point cloud data; for example, the specific operations for the bridge project are:

[0049] First, the LiDAR point cloud data and thermal imaging data will be exported, and then the data will be denoised, corrected, and cleaned to ensure the quality of the data and integrated into the 3D model of the bridge. The specific steps are as follows:

[0050] (1) Data Collection

[0051] LiDAR point cloud data collection:

[0052] P LiDAR ={(x i ,y i ,z i ,I i )|i=1,2,…,n}

[0053] in:

[0054] (x i ,y i ,zi ) is the three-dimensional coordinate (X, Y, Z) of the i-th point;

[0055] I i is the reflection intensity of the i-th point;

[0056] n is the total number of points in the point cloud data.

[0057] Thermal imaging data collection:

[0058] T thermal ={T1,T2,…,T n}

[0059] in:

[0060] T i represents the i-th frame of thermal image, including thermal imaging data and its corresponding temperature information;

[0061] (2) Data cleaning

[0062] LiDAR point cloud denoising (removing outliers):

[0063]

[0064] NoiseFilter is a denoising algorithm.

[0065] Thermal imaging data cleaning (eliminating invalid thermal images):

[0066]

[0067] Validity is a validity judgment function used to screen valid thermal images and eliminate invalid thermal images caused by equipment failure or poor acquisition conditions.

[0068] 3. Data Fusion

[0069] Spatial superposition of thermal imaging data and LiDAR point cloud data: The thermal imaging data and LiDAR point cloud data are spatially superimposed through the projection function Project(T,P) to form a fused data set:

[0070]

[0071] Project(T i ,(x i ,y i ,z i ) is a projection function, which means projecting the temperature data in the thermal image Ti to the corresponding three-dimensional coordinate point (x i ,y i ,z i ) to complete the spatial docking of the two.

[0072] 4. Output Data

[0073] (1) Bridge crack detection result collection

[0074] F crack =(x,y,z,w,d)

[0075] in:

[0076] (x, y, z) are the coordinates of the center of the crack;

[0077] w is the width of the crack;

[0078] d is the depth of the crack.

[0079] (2) Thermal anomaly area detection result collection

[0080] F thermal =(x,y,z,T)

[0081] (x, y, z) are the three-dimensional coordinates of the thermal anomaly point;

[0082] T is the temperature of that point.

[0083] Establish a data set for the third-party construction area scene, including: cleaning the noise and outliers in the LiDAR point cloud data through the RANSAC algorithm, and removing the blurred image frames in the high-definition image data through the image clarity function, and removing duplicate data points from the RFID data and calibrating the timestamp; then superimpose the LiDAR point cloud data, high-definition image data and RFID data into a unified coordinate system through spatial registration to form a data set corresponding to the unified coordinate system. For example, the specific operations for the third-party construction scene are:

[0084] The inspection goal is to identify abnormal behaviors in the construction area, such as construction beyond the scope, abnormal material stacking, and dynamic monitoring of equipment and personnel. To this end, comprehensive collection and analysis are carried out in combination with multiple data sources.

[0085] Data sources include LiDAR point cloud data, high-definition image data, and RFID data.

[0086] In LiDAR point cloud data, the low-density point cloud data collected by fixed-wing drones covers the overall terrain and structure of the construction area. LiDAR point cloud data format: P LiDAR ={(x i ,y i ,z i ,I i )}. In high-definition image data, high-resolution images obtained by fixed-wing drones are used for dynamic change monitoring and material stacking detection in construction areas. Image data format: D HD={I1,I2,…,I n}. RFID sensors obtain real-time location information of materials, equipment, and personnel. Data format: R RFID ={(x r ,y r ,t)}, where t represents the timestamp.

[0087] LIDAR point cloud cleaning, using the RANSAC algorithm to remove noise and outliers:

[0088] P clean ={(x i ,y i ,z i ,I i )|||NoiseFilter((x i ,y i ,z i ,I i ))=False}

[0089] HD image cleaning, remove blurry image frames through image clarity function:

[0090]

[0091] RFID data deduplication and spatiotemporal correction, removing duplicate data points and calibrating timestamps:

[0092] R clean ={(x r ,y r ,t)∣t=CorrectTime(t),DuplicateFilter(x r ,y r )}

[0093] Spatial data registration and fusion, LIDAR point cloud, high-definition image and RFID positioning data are superimposed into a unified coordinate system through spatial registration:

[0094]

[0095] The denoising and smoothing of point cloud data, as well as the geometric correction and format conversion of high-definition images, specifically include the following steps.

[0096] The following data processing process will be used for the “wide-area inspection” conducted by fixed-wing drones and the “fine collection” conducted by multi-rotor drones.

[0097] The first step is to collect high-resolution visible light images through the camera, collect point cloud data through the LiDAR laser radar, and collect infrared and near-infrared band image data through the multispectral camera. The collection band is (700-1300nm).

[0098] Step 2: Preprocess the above data. Since the processing measures to be taken have been briefly introduced in the data collection part, this part will focus on the principles of the processing measures adopted.

[0099] 1. Image Correction

[0100] Image correction includes two important steps: geometric correction and radiation correction, which ensure the accuracy and consistency of image data.

[0101] The main goal of geometric correction is to transform the image from the camera coordinate system to the geographic coordinate system (such as CGCS2000). During the flight, the image may be distorted due to the change of the camera's attitude (pitch, roll, yaw, etc.). Geometric correction adjusts the image's coordinate system to align with the coordinate system of the point cloud data by using a rotation matrix (R) and a translation vector (T).

[0102] The image is registered using known control points. Control points can be known points on the ground, usually obtained through GPS or RTK positioning. The image coordinates are aligned with the actual ground coordinates through rotation and translation matrices to ensure the accuracy of the image.

[0103] P image =R·O cloud +T

[0104] O image : The pixel position in the image coordinate system.

[0105] P cloud : Point coordinates (x, y, z) in the point cloud.

[0106] R: Rotation matrix, used to rotate the point cloud coordinate system to the image coordinate system.

[0107] T: Translation vector, used to adjust the offset of the point cloud.

[0108] Radiometric correction is used to eliminate image brightness variations caused by illumination variations, camera sensor errors, etc. Radiometric correction is usually performed through reflectance normalization, using known reference images (such as black field images and white field images) to correct the brightness of the image to meet standard lighting conditions.

[0109]

[0110] I corrected : Corrected pixel value;

[0111] I raw : pixel value of the original image;

[0112] I dark: Black field image (response under no light conditions);

[0113] I whit : Whiteboard reference image (totally reflective surface).

[0114] 2. Point cloud denoising

[0115] In the preprocessing of point cloud data, the removal of noise points is a key step to ensure data quality. Noise points in point cloud data may come from acquisition equipment errors, environmental interference or acquisition method limitations. These points usually appear as isolated points that deviate from the terrain or bridge structure and do not conform to the actual target model. Therefore, the point cloud data is denoised by the NoiseFilter function (based on the RANSAC algorithm) to construct accurate ground models and bridge models.

[0116] For the ground area, the fitting model is the ground model. The mathematical form of the model is the plane equation ax+by+cz+d=0, where a, b, c, and d are the parameters of the model, representing the direction and position of the normal vector of the ground respectively. During the fitting process, a small number of points are randomly selected from the point cloud as initial samples, and the parameters of the plane model are calculated based on these samples. The parameters of the plane model can be solved by the least squares method. The specific steps are:

[0117] First, for a given set of initial sample points {(x i ,y i ,z i )}, define the objective function E

[0118]

[0119] The goal is to determine a, b, c, and d by minimizing E. The function is solved using the least squares method to form a system of linear equations, and the best fitting model parameters are calculated using the matrix method.

[0120] After the model is fitted, the distance between each point in the point cloud and the fitting plane is calculated to determine whether the point is a noise point. The distance formula from the point to the model is:

[0121]

[0122] Among them, (x i ,y i ,z i ) is the 3D coordinate of a point in the point cloud, and a, b, c, d are the parameters of the fitting model. If the distance from a point to the model is i Less than the set threshold τ distance , then the point is considered to conform to the model (inside point); otherwise, the point is considered to be a noise point (outside point).

[0123] Threshold τ distance The basis for determination is as follows:

[0124] (1) According to the equipment accuracy, for example, the vertical accuracy of the laser radar is ±2cm, then τ distance =2cm;

[0125] (2) According to the scene experience, for example, in the complex curved area of ​​the bridge model, τ distance =10cm;

[0126] (3) Dynamic statistical analysis: by calculating the average deviation and standard deviation of the point cloud data, determine τ distance =μ+2σ. (Calculate the distance distribution from all points in the point cloud data to the preliminary fitting model, and obtain the average value μ and standard deviation σ)

[0127] In the fitting process, in order to ensure the reliability of the model, an iterative optimization method is used, that is, the point cloud data is randomly sampled multiple times and the above model fitting and point distance calculation process is repeated. After each iteration, the model with the most points that fit the model is selected as the final result. Finally, by removing all external points (i.e., noise points), a clean point cloud data set is obtained.

[0128] 3. NDVI filtering method

[0129] In the preprocessing of point cloud data, the presence of vegetation may interfere with the reconstruction of the ground model and the bridge model, resulting in the accumulation of errors. Therefore, by combining the NDVI (Normalized Difference Vegetation Index) filtering method, the points related to vegetation in the point cloud data are effectively removed, retaining the accurate data of the actual ground and structure. The specific process of the NDVI filtering method is as follows:

[0130]

[0131] NIR: reflectance in the near-infrared band;

[0132] RED: reflectivity of red light band;

[0133] NDVI: Normalized Difference Vegetation Index, ranging from -1 to +1.

[0134] The NDVI at each pixel is calculated using the above formula.

[0135] NDVI close to 1: indicates vegetation areas (such as trees, grasslands, etc.), because plant chlorophyll strongly absorbs red light and reflects near-infrared light;

[0136] NDVI close to 0 or negative values: indicates non-vegetated areas (such as bare land, roads, bridges, etc.).

[0137] For the area where points are removed, the surface fitting method is used to restore the terrain. The neighborhood points of the removed points are selected, a local surface (such as a plane or a curved surface) is fitted, and the elevation values ​​of the removed points are replaced by the fitted surface values.

[0138] Assume that the neighborhood point set is {(x i ,y i ,z i )}, the fitting surface equation is:

[0139] Z(x,y)=a+bx+cy+dx 2 +ey 2 +fxy

[0140] Solve the parameters a, b, c, d, e, f by the least square method and then remove the point (x i ,y i ) is replaced by the fitted surface value:

[0141]

[0142] After filtering out the vegetation, the new point cloud is used to reconstruct the surface.

[0143] 4. Thermal imaging

[0144] Through the validity judgment function Validity(T i ) Clean invalid images:

[0145]

[0146] Validity(T i ): Validity judgment function, returns True to indicate that the image T i is valid, and returning False means the image is invalid;

[0147] Criteria:

[0148] Image clarity (T i )>τ sharpness : Ensure image clarity through gradient or Laplace operator calculation;

[0149] Signal-to-noise ratio SNR(T i )>τ snr : Whether the signal quality of the image is higher than the set threshold;

[0150] Device Mark: Checks whether it is a faulty frame marked by the acquisition device.

[0151] τ sharpnes It is an empirical parameter, and its setting is usually based on the image acquisition conditions and quality requirements:

[0152] Set typical values ​​according to the acquisition device and target application. For example:

[0153] Scenes with high clarity requirements (such as detailed crack detection): τ sharpness =50~100 (the unit depends on the calculation method).

[0154] Scenarios with medium clarity requirements (such as third-party construction identification): τ sharpness =20~50

[0155] The setting of the signal-to-noise ratio threshold usually depends on the performance of the image acquisition device and the requirements of the target application:

[0156] High-quality analysis scenarios (such as crack or subtle defect detection): τ snr =30~40dB,

[0157] General temperature distribution analysis: τ snr =20~30dB.

[0158] 5. Spatial Registration

[0159] (1) Registration of point cloud data

[0160] Point cloud data is usually collected by LiDAR (laser radar), and preliminary location information is provided by GPS during collection. In order to further align with other data, the point cloud is geometrically corrected using ground control points. Using known control points on the ground, the point cloud is converted from the LiDAR coordinate system to the geographic coordinate system by calculating the rotation and translation between the point cloud coordinates and these control points.

[0161] (2) Image Registration

[0162] UAV images are taken by cameras and need to be converted from the camera coordinate system to the geographic coordinate system. The key to registration is camera calibration:

[0163] The camera internal parameters (such as focal length and principal point position) determine the relationship between image pixels and the camera coordinate system;

[0164] The camera external parameters (such as the camera's attitude and position) determine the relationship between the camera coordinate system and the geographic coordinate system. Feature points in the drone image (such as key nodes of a bridge) are aligned by matching the corresponding positions in the point cloud, and the image is finally projected into the geographic coordinate system.

[0165] (3) Registration of thermal imaging images

[0166] When thermal imaging images are collected, similar to visible light images, the pixel information in the camera coordinate system is recorded. In order to align with point cloud data and drone images, thermal imaging images need to: Use the camera's calibration information to convert the pixel data into a three-dimensional coordinate system.

[0167] Further, in S2, detecting anomalies in the scene to be inspected includes: obtaining a set of crack detection results and a set of spectral features corresponding to the road scene, and determining the surface damage of the road; and / or obtaining a set of bridge crack detection results and a set of thermal anomaly areas corresponding to the bridge scene, identifying cracks in the bridge and structural anomalies of the bridge, wherein the structural anomaly of the bridge includes exposed steel bars caused by concrete spalling; and / or detecting abnormal ground deformation through a data set corresponding to a unified coordinate system of a third-party construction area scene. For the third-party construction area scene, thermal imaging data is collected using a multi-rotor drone and imported into a data set corresponding to a unified coordinate system, and then abnormal heat sources in the third-party construction area scene are detected.

[0168] In this embodiment, in S3, the establishment of a spatial distribution model corresponding to the scene to be inspected includes: 3D ={(x,y,z,F geo ,F therm ,F texture ,F env )] where x, y, z represent three-dimensional coordinates, F geo ,F therm ,F texture and F env They represent geometric features, thermal imaging features, texture features, and environmental features, respectively. Specifically, the spatial identification of diseases and defects forms a high-precision analysis process through the combination of multiple data collection methods and advanced algorithm models. First, a large range of orthophotos and LIDAR point cloud data are obtained through fixed-wing drones to cover road and bridge areas and quickly identify suspected disease areas. Multi-rotor drones are responsible for collecting fine images, oblique images, and thermal imaging data, which are particularly suitable for obtaining local features of complex structures such as bridges. These data provide a multi-dimensional information basis for the spatial identification of diseases and defects. After data collection is completed, the data needs to be cleaned and preprocessed. In the process of disease identification, the extraction of image and point cloud features is particularly important.

[0169] Through the deep learning model (YOLO), cracks or potholes in the image can be quickly identified; combined with traditional texture analysis methods such as grayscale co-occurrence matrix, it can further distinguish between diseased areas and normal areas. In point cloud data, geometric feature extraction (such as elevation difference or curvature analysis) can be used to identify bridge deformation or road settlement. In addition, thermal imaging data is used to identify abnormal temperature areas and help mark steel corrosion or concrete delamination, such as Figure 3 shown.

[0170] Multi-source data fusion is the core link of disease identification. After spatially aligning the image and point cloud data through the ICP (Iterative Closest Point) algorithm, different types of data can be superimposed to generate a comprehensive three-dimensional distribution model. Subsequently, the geometric features, texture features and thermal imaging features are fused through a weighted calculation model to form a comprehensive disease score:

[0171] F fusion =w1·F geo +w2·F texture +w3·F thermal

[0172] Among them, F fusion is the comprehensive eigenvalue, w1, w2, w3 are the weights of each feature, F geo is a geometric feature; F texture is the texture feature; F therm It is a thermal imaging feature; in the scene of crack identification, the crack edge information in the image is first extracted by Canny edge detection, and then the depth and width of the crack are extracted by combining the point cloud data. After the image and point cloud data are registered, a three-dimensional distribution model containing the width, depth and position of the crack can be generated, providing a scientific basis for subsequent governance. Building a spatial model requires the integration of multi-source data, such as images, point clouds and thermal imaging data collected by fixed-wing and multi-rotor drones. These data provide texture features, geometric morphology and temperature anomaly information respectively, which is the basis for building the model. Data registration and fusion are the core steps in building a spatial model. Using the ICP (Iterative Closest Point) algorithm, the point cloud and image data are aligned to a unified geographic coordinate system. Subsequently, the high-definition image is mapped to the point cloud surface through texture mapping technology to generate a three-dimensional model with fine texture. At the same time, thermal imaging data can be superimposed on the geometric model to show the temperature distribution and add multi-dimensional information to the model.

[0173] Specifically, the solution of point cloud alignment method, ICP algorithm and texture mapping technology can be combined to achieve high-precision fusion of point cloud and image data. The initial registration of point cloud is optimized by polar coordinate representation and benchmark feature alignment, and the alignment process is further refined by ICP algorithm. On this basis, texture mapping technology is used to accurately project high-definition images onto the surface of point cloud, generating a 3D model with fine texture, such as Figure 4 The final 3D model is displayed through the GIS platform or 3D modeling tools, the disease or defect areas are marked, and a dynamic inspection map is generated. This model not only intuitively presents the distribution of diseases, but can also be used for subsequent analysis and governance decisions, providing a scientific basis for inspection tasks.

[0174] The early warning classification scheme realizes the quantification and classification of risks by analyzing the characteristics of diseases and defects and combining them with multivariate evaluation methods. Its core goal is to convert the impact of diseases or defects into quantifiable risk indicators, and based on quantitative evaluation, clarify the priority order of diseases according to risk levels, so as to provide a scientific basis for emergency response and resource allocation. Through a standardized classification system, it can effectively assist in the formulation of governance recommendations and make decisions more accurate and efficient. The early warning classification process includes data integration, risk score calculation and risk classification. First, through the multi-source data collected by drones, the geometric characteristics (such as crack width and depth), environmental characteristics (such as humidity, traffic load) and historical change trends of diseases or defects are summarized to construct a multivariate fusion risk indicator library. Subsequently, a comprehensive risk score R is generated for each disease point, and its calculation is based on the weighted scoring formula: Among them, R is the comprehensive risk score, F i represents the risk score of the ith variable (such as crack width or temperature anomaly), w i is the weight of the ith variable. The scoring results are divided into three levels: mild (R<4), moderate (4≤R<8), and severe (R≥8).

[0175] Taking bridge cracks as an example, the risk score combines four variables: crack width, depth, temperature difference in thermal anomaly area, and load historical fluctuation. Assuming that their weights are w1 = 0.4, w2 = 0.3, w3 = 0.2, w4 = 0.1, when the input data is crack width 15mm, depth 5cm, temperature difference 8℃, and load fluctuation 10%, the calculated comprehensive risk score is:

[0176] R=(0.4×15)+(0.3×5)+(0.2×8)+(0.1×10)=9.2

[0177] According to the scoring rules, the crack is considered a "serious risk" and immediate remediation measures are required.

[0178]

[0179]

[0180] The classification results are presented in a visual form. In the distribution map generated by GIS or 3D modeling tools, different risk levels are color-coded (such as red for severe risk and yellow for moderate risk) to intuitively display the problem areas. In addition, the risk analysis report lists in detail the location of each disease point, the scoring results, the main influencing factors and the corresponding management suggestions. This early warning classification method combines multivariate analysis with a dynamic update mechanism to provide comprehensive and scientific decision-making support for disease management.

[0181] Further, in S4, the process of determining the key influencing factors of the abnormality in the scene to be inspected by using the established spatial distribution model includes:

[0182] By analyzing the spatial hot spots in the spatial distribution model, high-risk areas are identified. The spatial hot spot analysis model is:

[0183]

[0184] Indicates the hotspot value of an area, w ij represents the weight, x j is the eigenvalue of the jth data point at a specific spatial position, n represents the total number of data points, and S represents the standard deviation of the eigenvalues ​​of all data points, which is defined as: Represents the mean of the eigenvalues ​​of all data points (i.e. the average value of all data points (i.e. all x j The average value of ) is defined as: If it is a road scene, then: x j is the elevation difference or crack width; for the bridge scenario, then: x j is the thermal imaging temperature difference or structural deformation; for the construction area scene, then: x j It is the ground deformation or abnormal heat source intensity. The higher the value, the greater the possibility of abnormality. Through the regression analysis model, the association between the key influencing factors of the abnormality and the specific variables is established. The regression analysis model is: Y is the anomaly score, X i is the variable value, β i is the regression coefficient, reflecting the contribution of each variable to the anomaly, and β0 is the intercept term, which means that when all explanatory variables ({X i}) are all zero, the predicted value of the target variable Y, ∈ represents the model predicted value The actual observed value Y actual For example:

[0185] In the process of multi-source data analysis based on the spatial distribution model, the model integrates multi-source information such as geometric data, image data, thermal imaging data and environmental data to form a high-precision three-dimensional model. In the data input stage, geometric data (such as elevation difference of road surface, deformation of bridge structure) and image data (such as surface features such as crack width and depth) provide the basis for spatial identification of defects. Thermal imaging data captures temperature anomalies and is used to detect steel corrosion and concrete delamination, while environmental data (such as traffic flow, humidity, wind speed and construction machinery activity trajectory) supplements external influencing factors for multivariate analysis. In order to determine the key influencing factors of abnormalities in road, bridge and third-party construction, this embodiment performs multivariate analysis based on the constructed spatial distribution model. The model integrates multi-source information such as geometric data, image data, thermal imaging data and environmental data to form a high-precision three-dimensional distribution model. In the data input stage, geometric data (such as elevation difference of road surface, deformation of bridge structure) and image data (such as surface features such as crack width and depth) provide the basis for spatial identification of defects. Thermal imaging data captures temperature anomalies for detecting steel corrosion and concrete delamination, while environmental data (such as traffic flow, humidity, wind speed, and construction machinery activity trajectories) supplements multivariate analysis with external influencing factors.

[0186] The model uses a unified expression to combine spatial coordinates with feature variables:

[0187] M 3D ={(x,y,z,F geo ,F thermal ,F texture ,F env )}

[0188] Where x, y, z represent three-dimensional coordinates, F geo ,F thermal ,F texture and F env They represent geometric features, thermal imaging features, texture features and environmental features respectively.

[0189] In the specific process of multivariate analysis, each variable is first standardized to eliminate the influence of different dimensions. The normalization formula is as follows:

[0190]

[0191] Through spatial hot spot analysis, high-risk areas are identified using hot spot analysis. The calculation formula is:

[0192]

[0193] in Indicates the hotspot value of a certain area, w ij represents the weight, xj is the characteristic value of the jth data point at a specific spatial position. For example: for roads: x j is the elevation difference or crack width; for bridges: x j It is the thermal imaging temperature difference or structural deformation; for the construction area: x j It is the ground deformation or abnormal heat source intensity. The higher the value, the greater the possibility of abnormality.

[0194] Furthermore, the influencing factors of the anomaly are associated with specific variables through the regression analysis model:

[0195]

[0196] Where Y is the anomaly score, X i is the variable value, β i is the regression coefficient, which reflects the contribution of each variable to the anomaly. By quantifying the risk contribution, a comprehensive risk scoring formula is constructed:

[0197]

[0198] w i is the weight of the i-th characteristic variable in the risk score, reflecting the influence of the variable on the abnormal risk;

[0199] F i It is the normalized value of the i-th risk characteristic variable, indicating the influence of the corresponding variable on the abnormal risk.

[0200] The scores are divided into three levels: low, medium and high according to the set thresholds, which are used to clarify the priority governance order of abnormal areas. The research results show that the constructed spatial distribution model can not only accurately quantify the key influencing factors of abnormalities, but also intuitively present the risk distribution, providing a scientific basis for governance and decision-making.

[0201] Taking bridge construction as an example, the thermal imaging temperature difference is 12°C, the elevation difference is 3.2cm, the crack width is 15mm, and the traffic flow is 1200 vehicles per hour. Through normalization, the variables are unified to the same analysis scale. For example, the normalized value of the thermal imaging temperature difference is calculated as:

[0202]

[0203] Similarly, the normalized values ​​of other variables are calculated using the corresponding formulas.

[0204] Next, these variables are associated with the anomaly score through a multivariate regression model, and the expression of the regression model is:

[0205]

[0206] in, Yes F geo The normalized value of is the normalized value of the other variables. The variable weight is set according to its influence on the anomaly. After substituting the normalized value into the model, the calculated comprehensive risk score is R = 8.5, indicating that the bridge construction area is at high risk.

[0207] According to the risk scoring grading criteria, the area is classified as a high-risk level and priority treatment measures are recommended. This approach not only quantifies the influencing factors of the anomaly, but also provides a scientific basis for the treatment sequence of the risk area. The final model results can be presented in 3D visualization, providing comprehensive decision support for bridge maintenance and construction management.

[0208] In S4 of this embodiment, the content that is finally displayed can be displayed in the form of an inspection report. The inspection report is based on a standardized template and includes a task overview, data source, inspection results, key anomaly analysis, governance recommendations, and appendixes. In the data aggregation and analysis stage, key indicators of diseases or anomalies (such as crack width, deformation, temperature difference, etc.) are extracted from the spatial distribution model, and combined with risk scores and classification results, high, medium, and low risk areas are identified. In terms of visualization, three-dimensional risk distribution maps and heat maps are used to intuitively display the distribution of diseases, and different levels of areas are marked by color coding (red indicates high risk, yellow indicates medium risk, and green indicates low risk). The formulation of governance recommendations is based on a grading strategy for risk levels. Regular inspections are recommended for low-risk areas, short-term maintenance such as crack repair or surface cleaning is required for medium-risk areas, and emergency measures are required to be taken immediately in high-risk areas, including support structure reinforcement, material replacement, and traffic diversion. In addition, according to the priority of risk scores, the diseases and areas to be treated are listed, and appropriate technical solutions and resources are matched. Through the analysis of multiple inspection reports, long-term governance plans can also be formulated, such as installing health monitoring sensors to obtain real-time data or increasing the frequency of inspections. Taking bridge inspections as an example, the results of a certain inspection showed that there were 5 high-risk areas with a total area of ​​20 square meters and 8 medium-risk areas with a total area of ​​30 square meters. The report analyzed in detail the cracks in the main beam and the temperature anomalies of the supporting structure, and put forward specific governance suggestions, such as using epoxy resin materials to repair cracks and anti-corrosion treatment of supporting structures. In addition, the appendix contains risk distribution maps and original data tables, which provide an intuitive basis for governance decisions. The entire process relies on GIS tools and analysis software, and realizes data extraction and template filling through an automated platform, converting complex data analysis into clear report content, providing efficient and scientific support for road, bridge and third-party construction governance.

[0209] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The above is only a specific implementation method of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A method for collaborative inspection of roads, bridges, and third-party construction using drones, characterized in that: include: S1. Collecting data related to the scene to be inspected by using a drone, wherein the types of the scene to be inspected include roads, bridges, and third-party construction areas, and the drone includes a fixed-wing drone and a multi-rotor drone; S2, after pre-processing the captured image data, detecting anomalies in the scene to be inspected; S3. Establishing a spatial distribution model corresponding to the scene to be inspected according to the result of anomaly detection; S4. Using the established spatial distribution model, determine the key influencing factors of the anomalies in the scene to be inspected and display them.

2. The method according to claim 1, characterized in that: The data related to the scene to be inspected includes: data collected by the onboard equipment of the fixed-wing UAV, and the types of onboard equipment of the fixed-wing UAV include: LiDAR sensor, multi-spectral camera and high-definition camera; and, data collected by the onboard equipment of the multi-rotor UAV, and the types of onboard equipment of the multi-rotor UAV include: high-definition camera, multi-spectral camera, lidar, thermal imager and / or RFID positioning detection equipment.

3. The method according to claim 2, characterized in that In S1, it includes: Using the data collected by the onboard equipment of the fixed-wing UAV to perform terrain modeling, and confirming the supplementary shooting area in the terrain modeling; The multi-rotor drone is started to shoot the reshoot area, wherein the reshoot area includes: an area with abnormal terrain, an area with obstruction, and a specific area with suspected disease.

4. The method according to claim 1, characterized in that: In S2, the preprocessing includes: The image data captured by the fixed-wing UAV and the multi-rotor UAV are sorted into a data set, and a mapping relationship between the data set and the geographic coordinates is established; For road scenes, the high-definition image data collected by fixed-wing drones and the multi-spectral data taken by multi-rotor drones are fused to create a data set. For bridge scenes, the LiDAR point cloud data collected by fixed-wing drones and the thermal imaging data taken by multi-rotor drones are fused to create a data set. For third-party construction area scenarios, the LiDAR point cloud data and high-definition image data collected by fixed-wing drones are fused with the RFID data collected by multi-rotor drones to establish a data collection.

5. The method according to claim 4, characterized in that Establish a data set for road scenes, including: removing blurry image frames in high-definition images and cleaning multispectral data; then, aligning the high-definition images with multispectral data through a registration function and generating a textured road model; Establish a data set for the bridge scene, including: denoising the LiDAR point cloud data collected by fixed-wing drones, cleaning the thermal imaging data taken by multi-rotor drones, and spatially superimposing the thermal imaging data with the LiDAR point cloud data; A data set is established for the third-party construction area scene, including: cleaning the noise and outliers in the LiDAR point cloud data through the RANSAC algorithm, and removing the blurred image frames in the high-definition image data through the image clarity function, and removing duplicate data points from the RFID data and calibrating the timestamp; then the LiDAR point cloud data, high-definition image data and RFID data are superimposed into a unified coordinate system through spatial registration to form a data set corresponding to the unified coordinate system.

6. The method according to claim 5, characterized in that In S2, detecting an abnormality in the scene to be inspected includes: Acquire a set of crack detection results and a set of spectral features corresponding to the road scene, and determine surface damage of the road; and / or, obtaining a bridge crack detection result set and a thermal anomaly region set corresponding to the bridge scene, identifying bridge cracks and bridge structural anomalies, wherein the bridge structural anomalies include exposed steel bars caused by concrete spalling; And / or, abnormal ground deformation is detected through a data set corresponding to a unified coordinate system of a third-party construction area scene.

7. The method according to claim 6, characterized in that Also includes: For third-party construction area scenes, multi-rotor drones are used to collect thermal imaging data and import them into a data set corresponding to a unified coordinate system, and then abnormal heat sources in third-party construction area scenes are detected.

8. The method according to claim 1, characterized in that In S3, the step of establishing a spatial distribution model corresponding to the scene to be inspected includes: M 3D ={(x,y,z,F geo ,F therm ,F texture ,F env )} Where x, y, z represent three-dimensional coordinates, F geo ,F thermal ,F texture and F env They represent geometric features, thermal imaging features, texture features and environmental features respectively.

9. The method according to claim 8, characterized in that In S4, the process of determining the key influencing factors of the abnormality in the scene to be inspected by using the established spatial distribution model includes: By analyzing the spatial hot spots in the spatial distribution model, high-risk areas are identified. The spatial hot spot analysis model is: Indicates the hotspot value of an area, w ij represents the weight, x j is the eigenvalue of the jth data point at a specific spatial position, n represents the total number of data points, and S represents the standard deviation of the data, which is used to measure the eigenvalue x j The overall distribution dispersion of represents the global mean of the eigenvalues; The association between the key influencing factors of the abnormality and the specific variables is established through the regression analysis model, and the regression analysis model is: W is the anomaly score, X i is the variable value, β i is the regression coefficient, which reflects the contribution of each variable to the anomaly; β0 is called the intercept term, which is used to indicate that when all explanatory variables ({X i }) are all zero; ∈ represents the model prediction value The actual observed value Y actual The deviation between .

Images