Unmanned aerial vehicle based road maintenance system and detection method

Abstract

The application discloses a kind of based on unmanned aerial vehicle's road maintenance system and detection method, belong to highway maintenance technical field, including the following steps: based on main surveying machine obtains the road surface image and three-dimensional point cloud data of target section;Based on the road surface image and three-dimensional point cloud data obtained by main surveying machine screening target section in road surface suspicious area set;Based on road surface suspicious area set planning re-survey path;Based on the optical image, thermal infrared image and three-dimensional point cloud data of road surface suspicious area set obtained from surveying machine;Based on the optical image, thermal infrared image and three-dimensional point cloud data obtained from surveying machine to road surface suspicious area set carries out defect verification.The application solves the technical problem that traditional road detection mode detects low efficiency, needs to close lane, hinders traffic, realizes the technical effect of low-cost high-precision rapid detection road surface.

Description

Technical Field

[0001] This invention relates to the field of highway maintenance technology, specifically to a road surface maintenance system and inspection method based on unmanned aerial vehicles (UAVs). Background Technology

[0002] When a vehicle is traveling at high speed, small potholes may cause safety accidents such as tire blowouts and loss of control. In addition, when there are cracks in the road surface, water seepage may cause the base layer to soften. Therefore, it is necessary to inspect and maintain the road surface in a timely manner to reduce the possibility of safety accidents.

[0003] In existing technologies, vehicle-mounted inspection and photography are commonly used to detect road conditions. However, this method suffers from problems such as low detection efficiency, the need to close lanes, and traffic obstruction. It is also prone to missed detections and over-detections, which affect the accuracy of the detection. Summary of the Invention

[0004] To address the above problems, the first aspect of this invention provides a road surface detection method based on unmanned aerial vehicles (UAVs), comprising the following steps:

[0005] The road surface images and 3D point cloud data of the target road section are obtained based on the main surveying machine;

[0006] Based on the road surface images and 3D point cloud data obtained by the main survey machine, a set of suspicious road surface areas in the target road section was selected;

[0007] Based on the planned re-survey route for suspicious road surface areas;

[0008] Based on optical images, thermal infrared images, and 3D point cloud data of suspicious road surface areas obtained from a surveying machine;

[0009] Defect verification was performed on a set of suspicious areas on the road surface based on optical images, thermal infrared images, and 3D point cloud data obtained from the surveying machine.

[0010] Preferably, the set of suspicious road surface areas in the target road segment selected based on road surface images and 3D point cloud data acquired by the main surveying machine includes:

[0011] Downsampling data is obtained based on road surface image data acquired by the main survey machine;

[0012] 3D raster data is obtained from the 3D point cloud data acquired by the main survey machine;

[0013] Linear texture features are extracted based on downsampled data using a first recognition model;

[0014] Voxel features are extracted from 3D raster data using a computational model.

[0015] Determining the confidence level of visible light cracks based on linear texture features;

[0016] Determining the crack confidence level of lidar based on voxel features;

[0017] The confidence level of doubt is calculated based on the normalized visible light crack confidence level and the lidar crack confidence level.

[0018] The set of suspicious areas on the road surface is determined based on the confidence level of suspicion and the dynamic threshold of suspicion.

[0019] Preferably, the suspicious dynamic threshold is set based on the road surface condition;

[0020] The calculation method for the suspicious dynamic threshold is as follows:

[0021] ;

[0022] in, For suspicious dynamic thresholds, This represents the average confidence level of all doubtful scenarios in the current context. This is the recall rate control coefficient. denoted as the fractional standard deviation.

[0023] Preferably, the resurvey route planning based on the set of suspicious road surface areas includes:

[0024] The number of surveying machines was determined based on the set of suspicious road surface areas.

[0025] Based on the number of surveying machines, spatial clustering is performed using a clustering model to generate task subsets corresponding to the number of surveying machines;

[0026] Based on task subsets, the task subsets of each slave surveying machine are adjusted through a cost calculation model;

[0027] The optimal survey path from the surveying machine is generated using a path planning model based on the adjusted task subset.

[0028] Preferably, adjusting the task subset of each slave surveyor through a cost calculation model includes:

[0029] The calculation of the virtual cost increment of transferring the task point source from the surveying machine to the target from the surveying machine;

[0030] Determine the task transfer status based on load balancing conditions and total cost constraints;

[0031] The iteration state is determined based on the convergence condition.

[0032] Preferably, the defect verification of the suspected road surface area set based on optical images, thermal infrared images, and three-dimensional point cloud data obtained from the surveying machine includes:

[0033] Linear texture features are extracted using a second recognition model based on optical images obtained from a surveying machine;

[0034] Temperature anomaly distribution features are extracted using a temperature recognition model based on infrared thermal images obtained from surveying machines.

[0035] Based on the 3D point cloud data obtained from the surveying machine, the 3D deformation distribution features are extracted through a deformation recognition model;

[0036] Weights are assigned based on the resurveyed environment;

[0037] Defect confidence is determined based on linear texture features, temperature anomaly distribution features, three-dimensional deformation distribution features, and assigned weights.

[0038] Defect confidence level normalization;

[0039] Based on the normalized defect confidence score and confidence threshold, the defect status of suspicious areas on the road surface is determined.

[0040] Preferably, the virtual cost increment is calculated as follows:

[0041] ;

[0042] in, For virtual cost increments, The current flight cost of the target reconnaissance aircraft, The cost of transferring the target from the surveying machine, To address the current flight costs of the survey aircraft, The cost of transferring the source from the surveying machine;

[0043] The load balancing conditions are as follows:

[0044] ;

[0045] The total cost constraint is:

[0046] ;

[0047] The convergence condition is:

[0048] ;

[0049] in, To survey machine The average virtual cost increment, To calculate the average virtual cost increment from the surveying machine j, This is the convergence threshold.

[0050] Preferably, the calculation model includes: an elevation difference calculation sub-model, a roughness calculation sub-model, and a density calculation sub-model;

[0051] The elevation difference calculation sub-model is as follows:

[0052] ;

[0053] in, For elevation difference, For the first The elevation coordinates of a laser point, where N is the set of all laser points falling into the voxel;

[0054] The roughness calculation sub-model is as follows:

[0055] ;

[0056] in, For roughness, For the first The elevation coordinates of the laser point This is the arithmetic mean of the elevations of all points within the voxel. The number of points within a voxel;

[0057] The density calculation sub-model is as follows:

[0058] ;

[0059] in, For density, This represents the total number of laser points within a voxel. The projected area of ​​the voxel;

[0060] The method for calculating the doubt confidence level based on visible light crack confidence and lidar crack confidence is as follows:

[0061] ;

[0062] in, Spatial location The degree of doubt at the location, For visible light learnable weights, These are the learnable weights for the LiDAR.

[0063] Preferably, it further includes: updating the screening parameters based on the doubt confidence level, the defect confidence level, and the confidence level deviation threshold;

[0064] The process of updating the screening parameters can be confirmed by calculating the range between the doubt confidence level and the defect confidence level;

[0065] The filtering parameters include: visible light learnable weights and lidar learnable weights;

[0066] The update method for the visible light learnable weights is as follows:

[0067] ;

[0068] in, For the updated learnable weights of visible light, For visible light learnable weights, For learning rate, This represents the absolute deviation between the doubt confidence level and the defect confidence level.

[0069] A second aspect of the present invention provides a road maintenance system based on unmanned aerial vehicles (UAVs) for performing the road detection method based on UAVs described in any of the above solutions, comprising: a maintenance vehicle, a first survey module and a second survey module disposed in the maintenance vehicle, and a cloud service module communicatively connected to the maintenance vehicle;

[0070] The maintenance vehicle includes a recovery compartment;

[0071] The first survey module includes: a main surveying machine, an aerial surveying camera and a lidar mounted on the main surveying machine;

[0072] The maintenance vehicle also includes a screening module, which is used to screen a set of suspicious road areas based on the road surface images and three-dimensional point cloud data of the target road section.

[0073] The inspection vehicle also includes a route planning module, which is used to plan a re-inspection route based on the selected suspicious road surface areas.

[0074] The second survey module includes: a slave surveying machine, a high-definition camera, a lidar, and an infrared thermal imager mounted on the slave surveying machine;

[0075] The inspection vehicle also includes a defect verification module, which is used to verify defects in a set of suspicious areas on the road surface based on optical images, thermal infrared images and three-dimensional point cloud data obtained from the surveying machine.

[0076] By adopting the above technical solution, the present invention mainly has the following technical effects:

[0077] By acquiring road surface images and 3D point cloud data of the target road section using a main surveying and mapping machine, and analyzing suspicious areas, optical images, thermal infrared images, and 3D point cloud data of the suspicious areas are used to verify defects in these areas. This method solves the technical problems of low detection efficiency, the need to close lanes, and traffic obstruction associated with traditional road surface inspection methods, achieving low-cost, high-precision, and rapid road surface inspection. Attached Figure Description

[0078] Figure 1 This is a schematic diagram of the structure of a road maintenance system based on an unmanned aerial vehicle (UAV) according to the present invention;

[0079] Figure 2 This is a schematic diagram illustrating the operation of a road maintenance system based on an unmanned aerial vehicle (UAV) according to the present invention.

[0080] Figure 3 This is a partial structural diagram of a maintenance vehicle in a road maintenance system based on an unmanned aerial vehicle (UAV) according to the present invention.

[0081] Figure 4 This is a schematic diagram of the main surveying machine in a road maintenance system based on an unmanned aerial vehicle (UAV) according to the present invention.

[0082] Figure 5 This is a schematic diagram of the operation of a road maintenance system based on an unmanned aerial vehicle (UAV) according to the present invention (from another perspective);

[0083] Figure 6 This is a schematic diagram of the operation of a road maintenance system based on an unmanned aerial vehicle (UAV) according to the present invention (another perspective);

[0084] Figure 7 This is a flowchart of a road surface detection method based on an unmanned aerial vehicle (UAV) according to the present invention.

[0085] The meanings of the reference numerals in the attached figures are as follows:

[0086] 1. Inspection vehicle; 11. Recovery bin; 12. Information interaction module; 13. Screening module; 14. Path planning module; 15. Defect verification module;

[0087] 2. First survey module; 21. Main surveying machine; 22. Aerial survey camera;

[0088] 3. Second survey module; 31. Surveying machine; 32. High-definition camera; 33. Infrared thermal imager;

[0089] 4. Cloud service module. Detailed Implementation

[0090] To enable those skilled in the art to better understand the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0091] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of the invention. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0092] Please see Figures 1-6 The first aspect of the present invention provides a road maintenance system based on unmanned aerial vehicles (UAVs), comprising: a maintenance vehicle 1, a first survey module 2 and a second survey module 3 disposed in the maintenance vehicle 1, and a cloud service module 4 communicatively connected to the maintenance vehicle 1.

[0093] In some embodiments, the maintenance vehicle 1 includes a recovery compartment 11 for accommodating a first survey module 2 and a second survey module 3. In some embodiments, the recovery compartment 11 can follow the maintenance vehicle 1 by being towed.

[0094] In some embodiments, the maintenance vehicle 1 further includes an information interaction module 12, which is used to realize information interaction between the maintenance vehicle 1 and the cloud service module 4.

[0095] In some embodiments, the first survey module 2 is used to acquire road surface images and 3D point cloud data of the target road section. In some embodiments, the first survey module 2 may include: a main surveying unit 21, an aerial surveying camera 22 and a lidar mounted on the main surveying unit 21. In some embodiments, the main surveying unit 21 may be any type of unmanned aerial vehicle controlled by remote control or autonomous program, the aerial surveying camera 22 may be any type of high-precision camera specifically designed for aerial photogrammetry, used to acquire ground images from the air, and the lidar may be any type of high-density LiDAR sensor specifically designed for aerial laser scanning, used to emit laser pulses from the air and receive echo signals to acquire high-precision 3D point cloud data of the ground and object surfaces. In some embodiments, the main surveying unit 21 may simultaneously carry the aerial surveying camera 22 and lidar and fly along the highway. The aerial surveying camera 22 is responsible for acquiring high-resolution road surface images, and the lidar is responsible for emitting laser pulses and receiving echoes to generate high-precision 3D point clouds. The two types of data are transmitted in real time to the maintenance vehicle 1 via a wireless link, and then subsequent road surface defect detection, geometric parameter extraction and multi-source fusion analysis are performed.

[0096] In some embodiments, the maintenance vehicle 1 further includes a screening module 13, which is used to screen a set of suspicious road areas based on the road surface image and three-dimensional point cloud data of the target road section; in some embodiments, the set of suspicious road areas refers to a set of candidate areas that are initially determined to have significant differences from normal road surface features and need further verification to determine whether there are actually defects.

[0097] In some embodiments, the maintenance vehicle 1 further includes a path planning module 14, which is used to plan a re-inspection path based on the selected suspicious areas of the road surface.

[0098] In some embodiments, the second survey module 3 is used to acquire optical images, thermal infrared images, and three-dimensional point cloud data of a set of suspected road surface areas. The second survey module 3 includes: a surveying machine 31, a high-definition camera 32, a lidar, and an infrared thermal imager 33 mounted on the surveying machine 31. In some embodiments, the surveying machine 31 can be any type of unmanned aerial vehicle controlled by remote control or autonomous program; the high-definition camera 32 can be any type of high-precision camera specifically designed for low-altitude photogrammetry, used to acquire optical images of the suspected road surface areas from above the set of suspected road surface areas; the lidar can be any type of high-density LiDAR sensor specifically designed for aerial laser scanning, used to emit laser pulses from the air and receive echo signals to acquire optical images of the set of suspected road surface areas; and the infrared thermal imager 33 can be any type of high-resolution thermal imager specifically designed for aerial thermal infrared imaging, used to collect thermal radiation signals from the air to acquire thermal infrared images of the suspected road surface areas.

[0099] In some embodiments, the maintenance vehicle 1 further includes a defect verification module 15, which is used to verify the defects of the suspicious area set of the road surface based on the optical images, thermal infrared images and three-dimensional point cloud data obtained from the surveying machine 31, and to classify the defective road surface into the road surface maintenance set.

[0100] In some embodiments, the maintenance vehicle 1 is communicatively connected to the cloud service module 4, which transmits the road surface maintenance set determined by the defect verification module 15 to the cloud service module 4, and determines the maintenance method through parameters such as damage type, damage degree, and risk factor.

[0101] Please see Figure 7 The second aspect of the present invention provides a road surface detection method based on unmanned aerial vehicles (UAVs), and a road surface maintenance system based on UAVs as described above, comprising the following steps:

[0102] S1. Obtain road surface images and 3D point cloud data of the target road section based on the main surveying machine;

[0103] In some embodiments, road surface images and 3D point cloud data can be acquired by the main surveying aircraft 21 simultaneously carrying an aerial survey camera 22 and a lidar while flying along the highway.

[0104] S2. Based on the road surface images and 3D point cloud data obtained by the main survey machine, filter the set of suspicious road surface areas in the target road segment;

[0105] In some embodiments, the suspicious road surface area set refers to the set of all candidate areas in the target road segment that are initially determined to have significant differences from normal road surface characteristics and require further verification to determine whether a defect actually exists.

[0106] In some embodiments, the screening process for the set of suspicious road areas can be determined by the maintenance vehicle 1 after performing a fusion analysis on the road image data and three-dimensional point cloud data acquired by the first survey module 2. The fusion analysis process will be further explained below.

[0107] In some embodiments, the set of suspicious road surface areas in the target road segment selected based on road surface images and 3D point cloud data acquired by the main surveying machine includes:

[0108] S201. Obtain downsampling data based on road surface image data acquired by the main survey machine;

[0109] In some embodiments, downsampled data can be obtained by performing average pooling on the road surface image data acquired in step S1.

[0110] In some embodiments, the average pooling process for road image data is performed as follows:

[0111] ;

[0112] in, To output a low-resolution mesh in the cell grayscale value at that location For the original high-resolution image in pixel coordinates grayscale value at that location To reduce the sampling factor, For the row and column indices of high-resolution pixels, This is the row and column index for the low-resolution grid.

[0113] S202. Obtain 3D raster data based on the 3D point cloud data acquired by the main surveying machine;

[0114] In some embodiments, three-dimensional raster data can be obtained by voxelizing the three-dimensional point cloud data acquired in step S1.

[0115] In some embodiments, the voxelization of 3D point cloud data is performed as follows:

[0116] ;

[0117] in, ;

[0118] voxels in two-dimensional grid coordinates The feature vector at that location, This represents the average elevation of all points within the voxel. The standard deviation of the elevation of all points within this voxel. The number of points that fall into that voxel. This represents the total number of points in the current voxel. For voxels Elevation values ​​of each point.

[0119] In some embodiments, by performing average pooling on the road surface image data acquired in step S1 and voxelization on the three-dimensional point cloud data, a real-time coarse screening process for large-scale road surface defects can be achieved by sacrificing a small amount of precision, reducing the amount of data, and increasing the calculation speed.

[0120] S203. Based on the downsampled data, extract linear texture features using the first recognition model;

[0121] In some embodiments, the first recognition model can be any neural network model trained through deep learning that is capable of efficiently classifying, detecting, or extracting features from input images or features. In some embodiments, the first recognition model can be a MobileNetV3 model.

[0122] In some embodiments, the linear texture features refer to geometric structures distributed in a linear pattern in a road surface image, which may include cracks, joints, or other stripes. Extracting such features aims to provide a basis for subsequent crack identification and analysis.

[0123] In some embodiments, the first identification model is:

[0124] ;

[0125] in, For the extracted feature map, For the model, This is downsampled data after average pooling.

[0126] S204. Extract voxel features based on 3D raster data using a computational model;

[0127] In some embodiments, the voxel features include:

[0128] Elevation difference Used to characterize the unevenness of the road surface, in some embodiments, elevation difference The larger the value, the more significant the unevenness defects such as pits, bulges, cracks, and uneven edges are present. A value close to 0 indicates that the small area has a uniform height and no obvious bumps or depressions.

[0129] roughness Roughness is used to characterize the roughness of the road surface microtexture. In some embodiments, roughness is... The larger the value, the rougher the road surface and the more potholes it has; roughness If the value is close to 0, the road surface will be smooth and even.

[0130] density Used to characterize point cloud density, in some embodiments, The higher the value, the more complete the laser irradiation, the richer the echo, the more complete the data, and the lower the noise. If the value is too low, it may be due to occlusion, distant edges, lack of reflection, or scanning blind spots caused by rapid movement, resulting in a decrease in the reliability of the geometric features in the corresponding area.

[0131] In some embodiments, the computational model can be any neural network model trained through deep learning that is capable of efficiently regressing, mapping, or extracting features from the input point cloud or features.

[0132] In some embodiments, the calculation model includes: an elevation difference calculation sub-model, a roughness calculation sub-model, and a density calculation sub-model.

[0133] The elevation difference calculation sub-model is as follows:

[0134] ;

[0135] in, For elevation difference, For the first The elevation coordinates of a laser point, where N is the set of all laser points falling into the voxel;

[0136] The roughness calculation sub-model is as follows:

[0137] ;

[0138] in, For roughness, For the first The elevation coordinates of the laser point This is the arithmetic mean of the elevations of all points within the voxel. The number of points within a voxel;

[0139] The density calculation sub-model is as follows:

[0140] ;

[0141] in, For density, This represents the total number of laser points within a voxel. Let be the projected area of ​​the voxel.

[0142] S205. Determine the confidence level of visible light cracks based on linear texture features;

[0143] In some embodiments, the method for determining crack confidence based on linear texture features is as follows:

[0144] ;

[0145] in, The confidence level for the visible light crack. This is the transpose of the weight vector. For activation function, This is the weight matrix. Spatial location Extract feature vectors from the location.

[0146] S206. Determine the confidence level of lidar cracks based on voxel features;

[0147] In some embodiments, the method for determining the lidar crack confidence level based on voxel features is as follows:

[0148] ;

[0149] in, For lidar crack confidence level, Hyperbolic tangent activation, , , These are the weights.

[0150] In some embodiments, weight , , It can be configured based on historical data, databases, etc.

[0151] S207. Calculate the doubt confidence level based on the normalized visible light crack confidence level and the lidar crack confidence level;

[0152] In some embodiments, the normalization method for the visible light crack confidence level may be:

[0153] ;

[0154] in, This represents the normalized confidence level for visible light cracks. It is a natural constant;

[0155] In some embodiments, the normalization method for the lidar confidence score may be:

[0156] ;

[0157] in, The normalized confidence level of the lidar;

[0158] In some embodiments, the doubt confidence level can be calculated based on the visible light crack confidence level and the lidar crack confidence level using a weighted fusion method.

[0159] In some embodiments, the doubt confidence level is used to characterize the likelihood that a candidate area of ​​the target road segment has significant differences from the characteristics of the normal road surface and needs further verification to determine whether a defect actually exists. In some embodiments, the higher the doubt confidence level, the greater the likelihood that a defective area exists.

[0160] In some embodiments, the method for calculating the doubt confidence level based on the visible light crack confidence level and the lidar crack confidence level is as follows:

[0161] ;

[0162] in, Spatial location The degree of doubt at the location, For visible light learnable weights, These are the learnable weights for the LiDAR.

[0163] In some embodiments, to facilitate subsequent calculations, weight normalization can be used when calculating the doubt confidence level to ensure that the doubt confidence level does not exceed the range of [0,1]. An exemplary weight normalization method could be... and and It is in the range [0,1].

[0164] S208. Determine the set of suspicious road areas based on suspicious confidence level and suspicious dynamic threshold;

[0165] In some embodiments, the suspected dynamic threshold refers to a threshold used to determine whether a candidate region is identified as having significantly different characteristics from normal road surfaces and requiring further verification to confirm the existence of a genuine defect. In some embodiments, the suspected dynamic threshold can be set based on road surface conditions. This is because road surface brightness, color, material, point cloud density, etc., can vary drastically with different road sections, weather conditions, and time periods. Setting a fixed threshold can easily lead to missed or over-detected defects. By setting a set of suspected dynamic thresholds based on road surface conditions, it is possible to effectively filter out the target regions most resembling cracks in the detected road section, thereby ensuring accuracy in different scenarios.

[0166] In some embodiments, the suspicious dynamic threshold is calculated as follows:

[0167] ;

[0168] in, For suspicious dynamic thresholds, This represents the average confidence level of all doubtful scenarios in the current context. This is the recall rate control coefficient. The standard deviation of the confidence level;

[0169] In some embodiments, to facilitate subsequent calculations, a method of normalizing the suspected dynamic threshold can be used when calculating the suspected dynamic threshold to ensure that the suspected dynamic threshold does not exceed the range of [0,1].

[0170] In some embodiments, by setting a recall control coefficient This allows for adjustment of the recall rate control coefficient. Relax or tighten the dynamic threshold for suspicious items; for example, during routine inspections, by reducing the recall rate control coefficient. To reduce the dynamic threshold for suspected issues and avoid missing crack areas; during the refinement stage, the recall rate control coefficient is increased. This increases the dynamic threshold for suspicious areas, enabling precise repairs of areas with significant cracks, reducing ineffective excavation, and improving efficiency.

[0171] In some embodiments, suspicious road surface areas can be identified by comparing a suspicion confidence level with a suspicious dynamic threshold set. For example, a suspicious dynamic threshold is generated based on the road surface condition of the road section under maintenance, and the suspicion confidence level is compared with the suspicious dynamic threshold. When the suspicion confidence level exceeds the suspicious dynamic threshold, the area is identified as a suspicious area and included in the set of suspicious road surface areas.

[0172] S3. Plan resurvey route based on suspicious road surface areas;

[0173] In some embodiments, after identifying a set of suspicious road surface areas, the road surface condition can be further confirmed by re-surveying the surveying machine 31.

[0174] In some embodiments, after obtaining the set of suspicious road surface areas of the target road segment, the number of surveying machines 31 and their flight paths can be determined by the path planning module 14.

[0175] In some embodiments, the planning of a resurvey route based on a set of suspicious road areas includes:

[0176] S301. Determine the number of surveying machines based on the set of suspicious road surface areas;

[0177] In some embodiments, the number of slave surveyors 31 can be determined based on the size and spatial distribution of the set of suspected road surface areas and the task capacity of a single slave surveyor.

[0178] S302. Based on the number of surveying machines, spatial clustering is performed using a clustering model to generate task subsets corresponding to the number of surveying machines.

[0179] In some embodiments, a clustering model can be used to assign a corresponding resurvey area to each group from the surveying machine 31, and then the road surface suspicious areas in the corresponding resurvey area can be resurveyed from the surveying machine 31.

[0180] In some embodiments, the clustering model can be any machine learning model that has been trained and is capable of effectively clustering or classifying input features. In some embodiments, the clustering model can be a spectral clustering model.

[0181] In some embodiments, the spectral clustering model is:

[0182] ;

[0183] in, For Laplace matrix, This is the clustering indicator matrix. It is the identity matrix; To add the diagonal elements of the square matrix, For matrix The transpose of .

[0184] S303. Based on task subsets, adjust the task subsets of each slave surveying machine through a cost calculation model;

[0185] In some embodiments, adjusting the task subsets of each slave surveyor using a cost calculation model includes:

[0186] S3031, Calculate the virtual cost increment of transferring the task point source from the survey machine to the target from the survey machine;

[0187] In some embodiments, a task point refers to a suspicious area of ​​the road surface in the resurvey area, and each suspicious area of ​​the road surface corresponds to a task point.

[0188] In some embodiments, the source slave surveyor refers to the slave surveyor at the transferred task point, and the target slave surveyor refers to the slave surveyor at the receiving task point.

[0189] In some embodiments, the virtual cost increment refers to the cost change caused by a shift in task points.

[0190] In some embodiments, the task subsets of each slave surveyor can be adjusted by transferring the source slave surveyor's task point to the target slave surveyor.

[0191] In some embodiments, the virtual cost increment is calculated as follows:

[0192] ;

[0193] in, For virtual cost increments, The current flight cost of the target reconnaissance aircraft, The cost of transferring the target from the surveying machine, To address the current flight costs of the survey aircraft, The cost after the source is transferred from the surveying machine.

[0194] S3032. Determine the task transfer status based on load balancing conditions and total cost constraints;

[0195] In some embodiments, when load balancing conditions and total cost constraints are met, the task point can be transferred from the source survey machine to the target survey machine.

[0196] In some embodiments, the load balancing conditions are:

[0197] ;

[0198] The reason for this is that the flight cost of a single reconnaissance aircraft must be equal to the shortest path length of the subset of tasks to which it is assigned, in order to ensure the instantaneous balance between the real-time total range and economy of the multi-aircraft system, and to prevent scheduling imbalances or mission timeouts caused by overloading or idling of some aircraft.

[0199] In some embodiments, the total cost constraint is:

[0200] ;

[0201] The reason for this is that the absolute value of the cost difference between the two aircraft must be between "zero" and "current difference", and the new total cost after the two aircraft are merged must not exceed 105% of the original total cost. This is to ensure that the total range of the system is always within a feasible economic range, neither exceeding the budget limit nor amplifying the total cost due to excessive pursuit of balance, thereby ensuring the operability and schedulability of the entire multi-aircraft mission allocation scheme.

[0202] S3033. Determine the iteration state based on convergence conditions;

[0203] In some embodiments, the task subsets of each slave surveying machine can be adjusted through iterative updates. When the convergence condition is met, the iterative optimization stops and the assigned task subsets are obtained. When the convergence condition is not met, the iterative optimization continues and the task points are transferred from the source surveying machine to the target surveying machine.

[0204] In some embodiments, the convergence condition is:

[0205] ;

[0206] in, To survey machine The average virtual cost increment, To calculate the average virtual cost increment from the surveying machine j, This is the convergence threshold;

[0207] In some embodiments, the convergence threshold can be obtained through historical data or a database.

[0208] In some embodiments, when the convergence condition is met, iterative optimization is stopped to obtain the optimal solution under the single transfer rule, thereby effectively reducing the operating cost of re-exploration from the surveying machine.

[0209] S304. Based on the adjusted task subset, the optimal survey path from the surveying machine is generated through a path planning model.

[0210] In some embodiments, after determining the task subsets of each slave surveyor, an optimal survey path can be generated based on the task points in the task subsets.

[0211] In some embodiments, the path planning model can be any combinatorial optimization model for solving the traveling salesman problem for a given set of points. In some embodiments, the path planning model can be the Christofides model, which in some embodiments is:

[0212] ;

[0213] in, For the shortest loop, To survey machine The task set, For the circular sequence, The Euclidean distance between two adjacent points. To find the shortest total length among all loops.

[0214] In some embodiments, a clustering model is used to divide the task subsets, a cost calculation model is used to adjust the task subsets, and a path planning model is used to generate the optimal survey path from the surveying machine. This enables dynamic and balanced allocation of the resurvey task area, ensuring the efficiency, spatial continuity, and non-overlapping nature of the tasks performed by each surveying machine, thereby improving the overall resurvey efficiency and path rationality.

[0215] S4. Based on optical images, thermal infrared images, and three-dimensional point cloud data of suspicious road surface areas obtained from the surveying machine;

[0216] In some embodiments, optical images, thermal infrared images, and three-dimensional point cloud data of the suspected road surface area set can be acquired by a high-definition camera 32, an infrared thermal imager 33, and a lidar simultaneously mounted on the surveying machine 31 above the suspected road surface area set.

[0217] S5. Defect verification is performed on the set of suspicious areas of the road surface based on optical images, thermal infrared images and three-dimensional point cloud data obtained from the surveying machine;

[0218] In some embodiments, defect verification refers to further verifying all candidate areas in the target road segment that are initially determined to be significantly different from normal road surface characteristics, in order to determine whether a defect actually exists.

[0219] In some embodiments, the defect verification process can be determined by the maintenance vehicle 1 through the fusion analysis of optical images, thermal infrared images and three-dimensional point cloud data of the suspicious road surface area set obtained by the second survey module 3. The fusion analysis process will be further explained below.

[0220] The defect verification of the suspected road surface area set based on optical images, thermal infrared images, and 3D point cloud data obtained from the surveying machine includes:

[0221] S501. Based on the optical image obtained from the surveying machine, linear texture features are extracted using a second recognition model;

[0222] In some embodiments, the second model can be any deep learning network model trained to extract linear texture features. In some embodiments, the second recognition model can be the MobileNetV3 model.

[0223] In some embodiments, the second recognition model extracts linear texture features in the following manner:

[0224] ;

[0225] in, For optical modal feature vectors, For the second recognition model, This is a visible light image data matrix;

[0226] S502. Based on the infrared thermal images obtained from the surveying machine, extract the temperature anomaly distribution features through a temperature recognition model;

[0227] In some embodiments, the temperature recognition model can be any deep learning network model that has been trained and has the ability to extract abnormal temperature distributions. In some embodiments, the temperature recognition model can be the MobileNetV3 model.

[0228] In some embodiments, the temperature identification model extracts temperature anomaly distribution features in the following way:

[0229] ;

[0230] in, This is the thermal infrared modal eigenvector. For temperature recognition model, This is the thermal infrared temperature distribution matrix;

[0231] S503. Based on the three-dimensional point cloud data obtained from the surveying machine, extract the three-dimensional deformation distribution features through the deformation recognition model;

[0232] In some embodiments, the point cloud can first be projected into a depth map or a voxel grid, and then the three-dimensional deformation distribution features can be extracted using a deformation recognition model. In some embodiments, the deformation recognition model can be any trained deep learning network model capable of extracting three-dimensional deformation distribution features. In some embodiments, the deformation recognition model can be the MobileNetV3 model.

[0233] In some embodiments, the deformation recognition model extracts the three-dimensional deformation distribution features in the following way:

[0234] ;

[0235] in, For laser mode feature vectors, For deformation recognition model, For point cloud projection, For laser point cloud data;

[0236] S504. Weighting based on the resurvey environment;

[0237] In some embodiments, the weight of the optical modal feature vector can be higher on sunny days; the weight of the three-dimensional deformation distribution feature can be higher on rainy or foggy days; and the weight of the temperature anomaly distribution feature can be higher at night.

[0238] The reasons mentioned above are as follows: when there is sufficient light and no rain scattering, the contrast of the crack edge is the best and the signal-to-noise ratio of the texture feature is the highest. Therefore, the optical modal feature vector can be given higher weights on sunny days. Water film, shadows, and reflections cause the contrast of the optical image to drop sharply, which can easily lead to the crack being blocked. Therefore, the three-dimensional deformation distribution feature can be given higher weights on rainy or foggy days. Without solar radiation, the road surface enters the radiation cooling stage. Due to the difference in heat capacity / thermal resistance, the voids, cracks, and water seepage areas inside the road surface form a significant temperature difference with the intact area. Therefore, the temperature anomaly distribution can be given higher weights at night, allowing the "most reliable" modality to dominate the fusion result, thereby reducing the possibility of false detection and false negative detection.

[0239] In some embodiments, to facilitate subsequent calculations, weight normalization may be used when assigning weights.

[0240] S505. Determine the defect confidence level based on linear texture features, temperature anomaly distribution features, three-dimensional deformation distribution features, and assigned weights.

[0241] In some embodiments, the confidence level of a defect can be determined by weighted summation;

[0242] In some embodiments, the confidence level of a defect is determined as follows:

[0243] ;

[0244] in, For the defect confidence level, , , These are the optical mode weights, thermal infrared mode weights, and laser mode weights, respectively.

[0245] S506, Defect confidence level normalization processing;

[0246] In some embodiments, the defect confidence normalization process is performed as follows:

[0247]

[0248] in, The defect confidence level after normalization. As the normalization factor, For the first The weights of each modality;

[0249] In some embodiments, normalization can eliminate dimensional differences between modes, ensuring that the output value range is [missing information]. .

[0250] S507. Based on the normalized defect confidence level and confidence threshold, determine the defect status of suspicious areas of the road surface.

[0251] In some embodiments, the confidence threshold refers to a threshold used to determine whether a candidate region is a defective pavement.

[0252] In some embodiments, defective pavement can be identified by comparing the normalized defect confidence level with a confidence threshold. For example, the normalized defect confidence level is compared with a confidence threshold, and if the normalized defect confidence level exceeds the confidence threshold, the area is identified as a defective pavement area and included in the pavement maintenance set.

[0253] S6. Update the filtering parameters based on the doubt confidence level, defect confidence level, and confidence deviation threshold;

[0254] In some embodiments, the confidence deviation threshold refers to a threshold used to determine whether the filtering parameters need to be updated. In some embodiments, the filtering parameters refer to parameters that can affect the determination result of suspicious road surface areas in the target road segment.

[0255] In some embodiments, the updating process of the screening parameters can be confirmed by calculating the range between the doubt confidence level and the defect confidence level. For example, the range between the normalized doubt confidence level and the defect confidence level can be calculated and compared with a confidence deviation threshold. If the range between the normalized doubt confidence level and the defect confidence level is greater than the confidence deviation threshold, the screening parameters need to be updated.

[0256] In some embodiments, the filtering parameters include: visible light learnable weights and lidar learnable weights;

[0257] In some embodiments, the visible light learnable weights are updated as follows:

[0258] ;

[0259] in, For the updated learnable weights of visible light, For visible light learnable weights, For learning rate, This represents the absolute deviation between the doubt confidence level and the defect confidence level.

[0260] In some embodiments, since visible light is highly susceptible to interference from sudden changes in illumination, shadows, water stains, lane line aging, lens contamination, etc., when there is a large absolute deviation between the confidence level of doubt and the confidence level of defect, the accuracy of screening suspicious areas of the road surface is improved by reducing the learnable weight of visible light.

[0261] In some embodiments, a weight normalization method is used when calculating the confidence level of suspicion. This ensures that the confidence level of suspicion does not exceed the range of [0,1], and while reducing the weight of visible light, it also increases the weight of lidar, thereby further improving the accuracy of screening suspicious areas on the road surface.

[0262] Finally, it should be noted that the embodiments disclosed in this invention are merely preferred embodiments of this invention and are only used to illustrate the technical solutions of this invention, not to limit it. Although this invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this invention.

Claims

1. A road surface detection method based on unmanned aerial vehicles (UAVs), characterized in that, Includes the following steps: The road surface images and 3D point cloud data of the target road section are obtained based on the main surveying machine; Based on the road surface images and 3D point cloud data obtained by the main survey machine, a set of suspicious road surface areas in the target road section was selected; Based on the planned re-survey route for suspicious road surface areas; Based on optical images, thermal infrared images, and 3D point cloud data of suspicious road surface areas obtained from a surveying machine; Defect verification is performed on a set of suspicious areas of the road surface based on optical images, thermal infrared images, and 3D point cloud data obtained from the surveying machine. The re-examination path planned based on the set of suspicious road surface areas includes: The number of surveying machines was determined based on the set of suspicious road surface areas. Based on the number of surveying machines, spatial clustering is performed using a clustering model to generate task subsets corresponding to the number of surveying machines; Based on task subsets, the task subsets of each slave surveying machine are adjusted through a cost calculation model; Based on the adjusted task subset, the optimal survey path from the surveying machine is generated through a path planning model. The adjustment of the task subsets of each slave surveyor through the cost calculation model includes: The calculation of the virtual cost increment of transferring the task point source from the surveying machine to the target from the surveying machine; Determine the task transfer status based on load balancing conditions and total cost constraints; Determine the iteration state based on convergence conditions; The virtual cost increment is calculated as follows: ； in, For virtual cost increments, The current flight cost of the target reconnaissance aircraft, The cost of transferring the target from the surveying machine, To address the current flight costs of the survey aircraft, The cost of transferring the source from the surveying machine; The load balancing conditions are as follows: ； The total cost constraint is: ； The convergence condition is: ； in, To survey machine The average virtual cost increment, To calculate the average virtual cost increment from the surveying machine j, This is the convergence threshold.

2. The road surface detection method based on unmanned aerial vehicles according to claim 1, characterized in that, The set of suspicious road surface areas in the target road segment, selected based on road surface images and 3D point cloud data acquired by the main surveying machine, includes: Downsampling data is obtained based on road surface image data acquired by the main survey machine; 3D raster data is obtained from the 3D point cloud data acquired by the main survey machine; Linear texture features are extracted based on downsampled data using a first recognition model; Voxel features are extracted from 3D raster data using a computational model. Determining the confidence level of visible light cracks based on linear texture features; Determining the crack confidence level of lidar based on voxel features; The confidence level of doubt is calculated based on the normalized visible light crack confidence level and the lidar crack confidence level. The set of suspicious areas on the road surface is determined based on the confidence level of suspicion and the dynamic threshold of suspicion.

3. The road surface detection method based on unmanned aerial vehicles according to claim 2, characterized in that, The suspicious dynamic threshold is set based on the road surface condition; The calculation method for the suspicious dynamic threshold is as follows: ； in, For suspicious dynamic thresholds, This represents the average confidence level of all doubtful scenarios in the current context. This is the recall rate control coefficient. denoted as the fractional standard deviation.

4. The road surface detection method based on unmanned aerial vehicles according to claim 2, characterized in that, The defect verification of the suspected road surface area set based on optical images, thermal infrared images, and 3D point cloud data obtained from the surveying machine includes: Linear texture features are extracted using a second recognition model based on optical images obtained from a surveying machine; Temperature anomaly distribution features are extracted using a temperature recognition model based on infrared thermal images obtained from surveying machines. Based on the 3D point cloud data obtained from the surveying machine, the 3D deformation distribution features are extracted through a deformation recognition model; Weights are assigned based on the resurveyed environment; Defect confidence is determined based on linear texture features, temperature anomaly distribution features, three-dimensional deformation distribution features, and assigned weights. Defect confidence level normalization; Based on the normalized defect confidence score and confidence threshold, the defect status of suspicious areas on the road surface is determined.

5. A road surface detection method based on unmanned aerial vehicles (UAVs) according to claim 2, characterized in that, The calculation model includes: an elevation difference calculation sub-model, a roughness calculation sub-model, and a density calculation sub-model; The elevation difference calculation sub-model is as follows: ； in, For elevation difference, For the first The elevation coordinates of a laser point, where N is the set of all laser points falling into the voxel; The roughness calculation sub-model is as follows: ； in, For roughness, For the first The elevation coordinates of the laser point This is the arithmetic mean of the elevations of all points within the voxel. The number of points within a voxel; The density calculation sub-model is as follows: ； in, For density, This represents the total number of laser points within a voxel. The projected area of ​​the voxel; The method for calculating the doubt confidence level based on the normalized visible light crack confidence level and the lidar crack confidence level is as follows: ； in, Spatial location The degree of doubt at the location, For visible light learnable weights, These are the learnable weights for the LiDAR.

6. The road surface detection method based on unmanned aerial vehicles according to claim 4, characterized in that, Also includes: The filtering parameters are updated based on the confidence level of doubt, the confidence level of defects, and the confidence deviation threshold; The updating process for screening parameters is confirmed by calculating the range between the doubt confidence level and the defect confidence level; The filtering parameters include: visible light learnable weights and lidar learnable weights; The update method for the visible light learnable weights is as follows: ； in, For the updated learnable weights of visible light, For visible light learnable weights, For learning rate, This represents the absolute deviation between the doubt confidence level and the defect confidence level.

7. A road maintenance system based on unmanned aerial vehicles (UAVs), characterized in that, The method for performing the road surface detection method based on UAV according to any one of claims 1-6 includes: a maintenance vehicle, a first survey module and a second survey module disposed in the maintenance vehicle, and a cloud service module communicatively connected to the maintenance vehicle. The maintenance vehicle includes a recovery compartment; The first survey module includes: a main surveying machine, an aerial surveying camera and a lidar mounted on the main surveying machine; The maintenance vehicle also includes a screening module, which is used to screen a set of suspicious road areas based on the road surface images and three-dimensional point cloud data of the target road section. The inspection vehicle also includes a route planning module, which is used to plan a re-inspection route based on the selected suspicious road surface areas. The second survey module includes: a slave surveying machine, a high-definition camera, a lidar, and an infrared thermal imager mounted on the slave surveying machine; The inspection vehicle also includes a defect verification module, which is used to verify defects in a set of suspicious areas on the road surface based on optical images, thermal infrared images and three-dimensional point cloud data obtained from the surveying machine.

Images