Road disease detection method and system based on deep learning and ground-air cooperation

Abstract

The invention provides a road disease detection method and system based on deep learning and ground-air cooperation, and relates to the technical field of road detection smart cities, and the system comprises an equipment scheduling and cooperation module which is used for obtaining a patrolled pavement crack image and underground disease radar data; the data processing module is used for preprocessing the acquired data, inputting a preprocessed pavement crack image into a pavement crack detection model, firstly, using depth separable convolution to deconvolution a convolution into depth convolution and point-by-point convolution, respectively extracting local features and global features of the crack through multi-scale feature extraction, and calculating the local features and the global features of the crack through multi-scale feature extraction; obtaining the disease type of the pavement crack; underground disease radar data are input to an underground disease detection model, the underground disease detection model uses a lightweight MobileNetV3 as a backbone network, a feature pyramid network and a path aggregation network are combined, multi-scale feature fusion is achieved, an anchor frame generation strategy is optimized through a K-Means clustering algorithm, and the type and position of the underground disease are obtained through output.

Description

Technical Field

[0001] The present disclosure relates to the technical field of road detection in smart cities, and in particular to a road disease detection method and system based on deep learning and ground-air collaboration. Background Art

[0002] The statements in this section merely provide background information related to the present disclosure and do not necessarily constitute prior art.

[0003] With the continuous advancement of urbanization, the number of municipal roads that need to be inspected and maintained is also increasing, and the workload of road inspection and maintenance is also increasing. In the process of urban road inspection, hidden diseases inside the road are difficult to detect because they are hidden inside the road, and they are sudden and destructive. In this case, three-dimensional ground penetrating radar is widely used in road inspection projects due to its non-contact, extremely fast scanning speed and three-dimensional imaging capabilities. However, urban roads have diverse functions and complex compositions, and there are many pipelines and other infrastructure under the roads. The radar echo signal is affected by many interference sources above and below the ground, and the signal-to-noise ratio is low, making the data interpretation of ground penetrating radar very challenging.

[0004] At present, the interpretation of ground penetrating radar data is highly dependent on the experience of experts. This expert interpretation method is time-consuming and labor-intensive, and the analysis results are highly subjective and unreliable. Therefore, there is an urgent need for an automatic interpretation method of 3D ground penetrating radar data to realize the automatic identification of urban road diseases. On the one hand, vehicle-mounted 3D ground penetrating radar is very suitable for checking underground diseases of urban roads such as voids, looseness, voids, poor layer adhesion and other structural problems without interfering with traffic flow. However, relying on vehicle-mounted detection makes it subject to certain limitations:

[0005] 1. Affected by the traffic environment, the vehicle-mounted 3D ground penetrating radar needs to maintain a stable speed and a low speed, which can easily affect traffic.

[0006] 2. Vehicle-mounted 3D ground-penetrating radar requires relatively smooth road conditions to achieve optimal operation, but not all inspection areas have this condition. In mountainous areas or rocky areas where vehicle travel is restricted, vehicle-mounted 3D ground-penetrating radar cannot perform monitoring operations.

[0007] 3. The vehicle-mounted 3D ground penetrating radar cannot fully detect the edge of the road and under the green belt. A method is needed to solve the driving limitations of the vehicle-mounted 3D ground penetrating radar.

[0008] On the other hand, the current method of using vehicle-mounted 3D ground-penetrating radar to detect road defects is that the driver first drives the vehicle equipped with the vehicle-mounted 3D ground-penetrating radar on the road section to be detected, and collects radar echo data from the space under the road. After the data collection is completed, the staff manually analyzes the radar echo image, finds abnormal waveforms, and uses experience to determine the type of defect corresponding to the abnormal waveform. Then, the inspection report is written, filling in the longitude and latitude location and type of each defect. This method cannot intuitively display the defect data, so there is a lack of an automated detection process, and there is also a lack of a visual and interactive method to intuitively display all the information such as the location, type, and size of road defects.

[0009] In summary, the current interpretation of ground penetrating radar echo data lacks an automatic identification method to detect and classify all disease data. The current vehicle-mounted three-dimensional ground penetrating radar method is limited by the terrain environment and traffic conditions. For the current process of manually driving a vehicle-mounted three-dimensional ground penetrating radar vehicle to collect radar echo data inside the road and manually summarizing and writing inspection reports, there is a lack of an automated and unmanned method to automate the disease data collection process, unmanned disease data analysis, and finally display all road internal disease information in front of people in an intuitive and interactive visualization method. Summary of the invention

[0010] In order to solve the above problems, the present invention proposes a road disease detection method and system based on deep learning and ground-air collaboration, which uses ground penetrating radar and deep learning models to realize efficient detection and precise positioning of road diseases. By introducing a lightweight network structure and an aggregated multi-channel coordinate attention mechanism, the computational complexity and resource requirements of the model are significantly reduced, while the accuracy and efficiency of disease detection are improved. The system supports deployment on embedded devices to realize real-time road disease detection and management, and can be widely used in the automated detection, maintenance and planning of road diseases in smart cities.

[0011] According to some embodiments, the present disclosure adopts the following technical solutions:

[0012] The road disease detection system based on deep learning and ground-air collaboration includes a smart platform, which includes:

[0013] Task management module, used to remotely configure and publish inspection tasks, dynamically schedule equipment resources, and monitor task progress in real time;

[0014] Equipment scheduling and coordination module, used to obtain images of road cracks and radar data of underground diseases during inspection;

[0015] The data processing module is used to preprocess the acquired data and input the preprocessed pavement crack image into the pavement crack detection model. First, the convolution is decomposed into deep convolution and point-by-point convolution using deep separable convolution, and then the local and global features of the cracks are extracted respectively through multi-scale feature extraction, and finally the disease type of the pavement crack is output; the preprocessed underground disease radar data is input into the underground disease detection model. The underground disease detection model uses a lightweight MobileNetV3 as the backbone network, combined with a feature pyramid network and a path aggregation network to achieve multi-scale feature fusion, and optimizes the anchor frame generation strategy through the K-Means clustering algorithm, and outputs the disease type and location of the underground disease;

[0016] Task log and historical data management module, used to generate task logs for each inspection task;

[0017] The remote decision-making and management module is used for the smart platform to adjust task parameters or view inspection results at any time.

[0018] According to some embodiments, the present disclosure adopts the following technical solutions:

[0019] The detection method of the road disease detection system based on deep learning and ground-air collaboration includes:

[0020] Obtain images of road cracks and radar data of underground damage during inspection;

[0021] The acquired data is preprocessed, and the preprocessed pavement crack image is input into the pavement crack detection model. First, the convolution is decomposed into deep convolution and point-by-point convolution using deep separable convolution, and then the local features and global features of the cracks are extracted respectively through multi-scale feature extraction, and finally the disease type of the pavement crack is output; the preprocessed underground disease radar data is input into the underground disease detection model, which uses a lightweight MobileNetV3 as the backbone network, combined with a feature pyramid network and a path aggregation network to achieve multi-scale feature fusion, and optimizes the anchor frame generation strategy through the K-Means clustering algorithm, and outputs the disease type and location of the underground disease;

[0022] Generate a task log for each inspection task, and adjust task parameters or view inspection results at any time based on the task log.

[0023] According to some embodiments, the present disclosure adopts the following technical solutions:

[0024] A computer program product includes a computer program, which, when executed by a processor, implements the detection method of the road damage detection system based on deep learning and ground-air collaboration.

[0025] According to some embodiments, the present disclosure adopts the following technical solutions:

[0026] A non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the detection method of the road damage detection system based on deep learning and ground-air collaboration is implemented.

[0027] According to some embodiments, the present disclosure adopts the following technical solutions:

[0028] An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the detection method of the road disease detection system based on deep learning and ground-air collaboration.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] The road disease detection system based on deep learning and ground-air collaboration disclosed in this disclosure designs an intelligent platform, which is based on the deployment of cloud servers. Through ground-air collaborative inspection tasks, it can realize real-time transmission and visual display of detection data. The intelligent platform is deployed based on cloud servers to provide reliable background support and data processing capabilities. The intelligent platform system architecture adopts a modular design, which enables the platform to quickly respond to user needs and perform inspection tasks. The main functions of the intelligent platform system include task release, data reception, data processing and visual interaction, which meet the real-time requirements of urban road disease detection.

[0031] The road disease detection system based on deep learning and ground-air collaboration disclosed in the present invention automatically processes the received inspection data, and after preprocessing, analysis and other steps, stores the processed data in the cloud database. The data processing module includes functions such as data calibration, anomaly detection, and feature extraction to ensure the accuracy and integrity of the inspection data and provide reliable data support for the front-end display.

[0032] The road disease detection system based on deep learning and ground-air collaboration disclosed in the present invention automatically identifies pavement cracks and underground diseases through a deep learning model, thereby improving the accuracy and comprehensiveness of disease detection. The pavement crack detection model uses an improved lightweight convolutional neural network (CNN) for image analysis, which can efficiently identify various types of cracks; the underground disease detection model analyzes ground penetrating radar image data based on an improved YOLOv4 model to accurately identify underground structural disease problems, such as voids and looseness. And feedback the detection results in real time. The combination of the two models not only improves the comprehensiveness and accuracy of disease detection, but also greatly improves the efficiency of inspection tasks, optimizes data processing and analysis processes, and provides accurate and real-time data support for road disease detection management. The deep learning model automatically identifies pavement cracks and underground diseases, thereby improving the accuracy and comprehensiveness of disease detection. The pavement crack detection model uses an improved lightweight convolutional neural network (CNN) for image analysis, which can efficiently identify various types of cracks; the underground disease detection model analyzes ground penetrating radar image data based on the improved YOLOv4 model to accurately identify underground structural disease problems, such as voids and looseness, and provide real-time feedback on the detection results. The combination of the two models not only improves the comprehensiveness and accuracy of disease detection, but also greatly improves the efficiency of inspection tasks, optimizes data processing and analysis processes, and provides accurate and real-time data support for road disease detection management.

[0033] The road disease detection system based on deep learning and ground-air collaboration disclosed in the present invention transmits data from the inspection robot to the server, and then transmits it to the front end after processing by the server, and finally realizes data visualization on the user interface. The data transmission process from the inspection device to the server adopts a two-way communication mechanism, allowing users to view and interact with data on the front-end page, realizing a friendly experience of human-computer interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0035] Figure 1 Schematic diagram of the structure and function of the smart platform module according to the embodiment of the present disclosure.

[0036] Figure 2 This is a diagram showing the intelligent platform for road disease detection according to an embodiment of the present disclosure;

[0037] Figure 3 It is a schematic diagram of the structure of a road disease detection device according to an embodiment of the present disclosure;

[0038] Figure 4 Schematic diagram of the working process of the road disease detection device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0039] The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0040] It should be noted that the following detailed descriptions are all illustrative and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0041] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0042] Example 1

[0043] In one embodiment of the present disclosure, a road disease detection system based on deep learning and ground-air collaboration is provided, including an intelligent platform. The intelligent platform is as follows: Figure 1 As shown, including:

[0044] Task management module, used to remotely configure and publish inspection tasks, dynamically schedule equipment resources, and monitor task progress in real time;

[0045] Equipment scheduling and coordination module, used to obtain images of road cracks and radar data of underground diseases during inspection;

[0046] The data processing module is used to preprocess the acquired data and input the preprocessed pavement crack image into the pavement crack detection model. First, the convolution is decomposed into deep convolution and point-by-point convolution using deep separable convolution, and then the local and global features of the cracks are extracted respectively through multi-scale feature extraction, and finally the disease type of the pavement crack is output; the preprocessed underground disease radar data is input into the underground disease detection model. The underground disease detection model uses a lightweight MobileNetV3 as the backbone network, combined with a feature pyramid network and a path aggregation network to achieve multi-scale feature fusion, and optimizes the anchor frame generation strategy through the K-Means clustering algorithm, and outputs the disease type and location of the underground disease;

[0047] Task log and historical data management module, used to generate task logs for each inspection task;

[0048] Remote decision-making and management module, used in smart platforms, to adjust task parameters or view inspection results at any time.

[0049] As an embodiment, the road disease detection system based on deep learning and ground-air collaboration disclosed in the present invention designs a smart platform. The smart platform is deployed based on a cloud server, adopts a modular design, and has multiple functions such as data processing, task release and management, and result visualization. The modular architecture of the platform enables it to have good flexibility and scalability, and can dynamically adjust the functional modules of the system to meet different business needs. Users interact with the system through the unified interface of the platform to complete the entire process from task configuration to result viewing, including:

[0050] (1) Task management module: The task management module allows users to remotely publish inspection tasks and configure detailed parameters of the tasks, including inspection area, inspection time, equipment type (drone or vehicle-mounted equipment), and task priority. The task management module also supports dynamic allocation of equipment resources, intelligently dispatching drones and unmanned vehicles according to inspection needs to ensure that inspection resources are optimally configured. Through the task status monitoring function, users can track the progress of task execution in real time, view the operating status of the equipment, and the inspection completion ratio.

[0051] (2) Equipment scheduling and coordination module, which coordinates the collaborative work of drones and unmanned vehicles through the task management module. The system dynamically plans the flight path of drones and the driving route of unmanned vehicles according to the inspection area and task requirements. During the inspection process, the platform receives data returned by the equipment in real time to ensure the efficiency of ground-air coordination and the comprehensiveness of the data. In addition, the platform's built-in intelligent scheduling algorithm can dynamically adjust the equipment's working parameters according to the task load, equipment power and weather conditions to improve the task completion rate.

[0052] Specifically, it can conduct a comprehensive inspection of road infrastructure based on ground-penetrating radars on drones and on unmanned vehicles. Drones are used for preliminary scans to identify potential problem areas from an aerial perspective, especially in hard-to-reach areas. The vehicle-mounted system conducts detailed follow-up scans on passable roads to further investigate detected anomalies.

[0053] Among them, drones and unmanned vehicles work together to conduct inspections, obtain images of road cracks and radar data of underground diseases to ensure comprehensive coverage of road diseases. Drones and unmanned vehicles share data in real time through wireless communication. The drone scanning results provide hot spot area prompts for on-board equipment, and the on-board equipment conducts in-depth high-resolution inspections. Through ground-air collaboration, not only can hard-to-reach areas be covered, but underground diseases can also be accurately located.

[0054] (3) Data processing module: After the raw data collected by the inspection equipment is uploaded to the cloud server, it is automatically processed by the data processing module. This module includes functions such as data cleaning, denoising, calibration, and feature extraction to ensure the accuracy and integrity of the data. For multimodal data (such as image data and radar data), the data processing module performs feature fusion to generate more comprehensive disease detection results. The processed data is stored in the cloud database and associated with the user's task records for subsequent query and analysis.

[0055] Specifically, the acquired data is preprocessed, and the preprocessed pavement crack image is input into the pavement crack detection model. First, the convolution is decomposed into deep convolution and point-by-point convolution using deep separable convolution, and then the local and global features of the cracks are extracted respectively through multi-scale feature extraction, and finally the disease type of the pavement crack is output; the preprocessed underground disease radar data is input into the underground disease detection model. The underground disease detection model uses a lightweight MobileNetV3 as the backbone network, combined with a feature pyramid network and a path aggregation network to achieve multi-scale feature fusion, and the anchor frame generation strategy is optimized through the K-Means clustering algorithm, and the disease type and location of the underground disease are output.

[0056] Among them, 1) preprocessing the inspected pavement crack images and underground disease radar data, including: removing noise from the collected data through Gaussian filtering and median filtering operations, unifying the resolution through scale adjustment to ensure the consistency of model input. At the same time, the image is rotated, cropped, flipped and other data enhancement operations to generate diversified training data and improve the robustness of the model. Specifically, the filtering formula is expressed as follows:

[0057] The weights of the Gaussian filter are defined by the following Gaussian function:

[0058]

[0059] The pixel value after Gaussian filtering is calculated by the following formula:

[0060]

[0061] Among them, G(x,y) defines the weight distribution of each pixel in the Gaussian filter, which is calculated based on the Gaussian function. x,y are the coordinates of the pixel relative to the center of the filter window. σ is the standard deviation, which determines the smoothness of the filter. G(i,j) is the Gaussian weight, I(x,y) is the original pixel value, and the filter window size is (2k+1)(2k+1).

[0062] For a filter window with a window size of (2k+1)(2k+1), the median filter formula is:

[0063] I filtered(x,y)=median{I(x+i,y+j)|i,j∈[-k,k]}

[0064] Where median is the median of the pixel values ​​in the filter window, and I(x+i,y+j) is the pixel value in the filter window. k is used to control the size of the filter window, affecting the filter range and effect.

[0065] The combination of Gaussian filtering and median filtering can effectively process various noise types in GPR data while protecting key feature information in the data. By properly selecting the filtering order and parameters, higher quality signal and image preprocessing can be achieved, providing high-quality input data for subsequent feature extraction and deep learning model training.

[0066] 2) Data annotation, including:

[0067] The types of pavement cracks are annotated into three categories: longitudinal cracks, transverse cracks, and irregular cracks. The location and characteristic parameters of each type of crack are annotated. The types of underground diseases are annotated into four categories based on radar echoes: voids, looseness, voids, and poor layer adhesion, and the location and range of the disease are indicated. After the annotation is completed, the data is divided into training sets, validation sets, and test sets to meet the needs of model training, optimization, and evaluation.

[0068] 3) Model training, including:

[0069] 1. An improved lightweight convolutional neural network (CNN) is used for pavement crack detection. The model improves detection performance through a variety of optimization measures, including:

[0070] First, use depthwise separable convolution to replace the traditional convolution operation, and decompose the convolution into depthwise convolution and point-by-point convolution. Specifically:

[0071] In traditional convolution, the convolution kernel not only slides on the spatial dimension (H×W), but also performs convolution operations on the input channel (C). The input feature map size is H×W×C, the convolution kernel size is K×K, and the number of output channels is N. The calculation formula of traditional convolution is:

[0072]

[0073] The computational complexity is:

[0074] O(H·W·C·N·K 2 )

[0075] Where X is the input feature map, which represents the input data or image, Y is the output feature map, which represents the result after the convolution operation, W is the weight of the convolution kernel, which represents the weight matrix of the filter, i, j are the spatial coordinates (rows, columns) of the pixels in the output feature map, n is the channel index of the output feature map, which represents the nth channel of the output, c is the channel index of the input feature map, which represents the cth channel of the input, and k is the channel index of the input feature map, which represents the cth channel of the input. h ,k w is the size of the convolution kernel in the spatial dimension (rows, columns), respectively representing the height and width of the convolution kernel, H, W are the spatial dimensions of the input feature map, height H and width W, N is the number of channels of the input feature map, the number of channels of the output feature map, and K represents the size of the convolution kernel.

[0076] In the depth convolution, convolution operation is performed independently on each channel of the input feature map, and cross-channel processing is no longer performed. The input feature map is The size of the deep convolution kernel is K×K, each convolution kernel only acts on one channel, and the output feature map is still H×W×C. The formula is:

[0077]

[0078] The computational complexity is:

[0079] O(H.W.C.K. 2 )

[0080] Among them, Y c The cth channel in the output feature map after depthwise convolution. The meanings of other symbols are the same as those of traditional convolution.

[0081] In point-by-point convolution, a convolution kernel of size 1×1 is used to linearly combine all channels at each spatial position to obtain a new feature map. Assume that the input of the deep convolution is The point-by-point convolution kernel size is 1×1, the number of output channels is N, and the point-by-point convolution formula is:

[0082]

[0083] The computational complexity is:

[0084] O(H·W·C·N)

[0085] Among them, Z is the output feature map after point-by-point convolution, W[1,1,c,n] is the weight of the point-by-point convolution kernel, and the meanings of other symbols are the same as those of traditional convolution.

[0086] Using depth-wise separable convolution, by first performing depth-wise convolution and then point-wise convolution, the total computational complexity is:

[0087] O(H.W.C.K. 2 +H·W·C·N)

[0088] Compared with the computational complexity of traditional convolution O(H·W·C·N·K 2 ), its value is 1 / N+1 / K 2 , when N>>K 2 When the amount of calculation is reduced to nearly K 2 times, greatly reducing the computational complexity and number of parameters while maintaining efficient feature extraction capabilities. To enhance the feature expression capabilities of the network, the model introduces residual connections. By means of jump connections, the input of the previous layer is added to the output of the current layer, alleviating the problem of gradient vanishing in deep networks while retaining low-level feature information. This design improves the training efficiency and feature transfer capabilities of the model. The model also uses a multi-scale feature extraction module, combined with convolution kernels of different sizes (such as 3×3, 5×5, 7×7) to capture local and global features of cracks. Local detail information and global structural information are integrated through fusion operations (such as channel superposition, splicing or attention mechanism). The fused feature map will undergo global average pooling, compress the spatial dimension to generate a fixed-size feature vector, and input it to the fully connected layer for classification mapping. Finally, the classification result is output as the probability distribution of each crack type through the Softmax function, thereby judging the existence and type of cracks. In this process, the cross entropy loss function is used to optimize the model classification performance, ensure the effective fusion of local and global features, and improve the classification accuracy and stability of the model. Specifically, the loss function formula is as follows:

[0089]

[0090] Where: N is the number of samples, C is the number of categories, y ij is the true label of the i-th sample in the j-th category, usually 1 (correct category) or 0 (other categories), is the predicted probability of the i-th sample in the j-th class (usually calculated by the Softmax function).

[0091] 2. The underground disease detection model is based on the improved YOLOv4 design and is optimized for computing resource limitations and diverse disease features in actual scenarios. The model uses a lightweight MobileNetV3 as the backbone network, combined with deep separable convolution and hard activation function (Hard Swish), which reduces the amount of calculation and parameters while maintaining efficient feature extraction capabilities. Specifically:

[0092] Among them, the operation principle of depth-separable convolution is consistent with the above content, and Hard Swish is an efficient activation function similar to Swish, and the formula is:

[0093]

[0094] Among them, ReLU6(x)=min(max(x,0),6) is a variant of the ReLU activation function, and (x+3) is an offset operation used to shift the input value range. Multiplying with the original input value x finally generates a nonlinear output. Compared with the traditional Swish function, Hard Swish uses piecewise linear operations instead of Sigmoid function 0, which greatly reduces the amount of calculation while retaining the smooth nonlinear effect of the Swish function, which helps the model extract features more efficiently.

[0095] In order to adapt to the diversity of underground diseases, the model optimizes the anchor frame generation strategy through the K-Means clustering algorithm to generate anchor frames that are more suitable for the size and shape of the disease. The specific clustering steps are as follows:

[0096] ① Initialize the cluster center, set the k value to the number of target categories for clustering (that is, the number of anchor boxes to be generated), and randomly select k annotation boxes as the initial cluster center;

[0097] ② Define the distance metric and use the IoU (Intersection over Union) distance as an indicator to measure the similarity between the anchor box and the annotation box:

[0098] Distance = 1-IoU(A,B)

[0099] Where A is the cluster center (anchor box), B is the annotation box, and IoU is the intersection over union of the anchor box and the annotation box;

[0100] ③ Assign samples to the nearest cluster center, calculate the IoU distance between each annotation box and all cluster centers, and assign the annotation box to the cluster corresponding to the nearest cluster center;

[0101] ④ Update the cluster center. For each cluster annotation box, calculate the mean of its width and height and update it to the new cluster center:

[0102]

[0103] Where N is the number of annotation boxes in the cluster, W is the width of the annotation box, and H is the height of the annotation box;

[0104] ⑤ Repeat steps ③ and ④ until the cluster centers converge or the maximum number of iterations is reached.

[0105] After K-Means clustering, k anchor frames are obtained, which represent the main size and shape distribution of the disease annotation frames in the data set. These optimized anchor frames are particularly suitable for the detection of small diseases (such as voids and loose areas) and large diseases (such as poor layer adhesion), which improves the positioning accuracy and recall rate of the model.

[0106] Furthermore, the model combines the Feature Pyramid Network (FPN) and the Path Aggregation Network (PANet) in the detection head to achieve multi-scale feature fusion. Specifically:

[0107] ① Feature Pyramid Network (FPN), the backbone network outputs a multi-scale feature map P i , FPN first performs 1×1 convolution on each layer of feature maps to unify the number of channels. The top-down path upsamples the high-level feature maps layer by layer to the size of the low-level feature maps through upsampling operations:

[0108] P i '=P i +Up(P i +1)

[0109] Among them, P i is the original feature map of the i-th layer output from the backbone network, P i 'It is the feature map obtained by the top-down feature fusion mechanism of FPN. Up() represents the bilinear interpolation upsampling operation.

[0110] ② Path Aggregation Network (PANet), in the bottom-up path, down-samples the low-level feature maps layer by layer and fuses them with the high-level feature maps:

[0111] F i = Down(F i-1 )+P i '

[0112] Among them, F i represents the fused feature map of the i-th layer in PANet, and Down() represents the convolution plus maximum pooling operation;

[0113] ③ The output of multi-scale feature fusion directly combines multiple layers of feature maps through skip connections, and finally generates a high-semantic, low-resolution comprehensive feature map.

[0114] FPN enhances the model's ability to detect defects of different sizes, and PANet further aggregates multi-layer feature maps to improve the accuracy of defect classification and bounding box prediction. In view of the particularity of radar data, simulated environmental noise and random deformation operations (such as signal scaling and time perturbations) are added to the training to enhance the model's adaptability to real complex scenes.

[0115] The improved lightweight CNN and YOLOv4 models greatly improve the detection accuracy and efficiency, while significantly reducing the computational complexity through technologies such as deep separable convolution, residual connection, multi-scale feature extraction and anchor box optimization. Combined with the optimization of the cross entropy loss function, the model shows efficient classification and positioning capabilities in crack detection and underground disease identification tasks. Data enhancement and multi-scale feature design enable the model to adapt to complex scenarios and achieve real-time and efficient road disease detection on embedded devices, providing strong technical support for road maintenance in smart cities.

[0116] 4) The trained model is deployed on the development board to ensure that the model can achieve real-time detection in actual inspections. The crack detection model deployed on drones and unmanned vehicles combines high-speed wide-angle cameras with GPS positioning devices to accurately record the geographical location information while detecting the disease. The improved YOLOv4 model is responsible for processing radar echo data and accurately identifying the waveform of underground diseases. Through the deployment of these lightweight and efficient models, rapid positioning and real-time detection of road diseases are achieved, providing accurate data support for road maintenance and repair.

[0117] (4) Task log and historical data management module generates a task log for each inspection task, recording information such as task execution time, equipment usage, inspection area coverage, and task completion. Users can view key nodes in the inspection process through the task log and evaluate the task effect. The platform also provides historical data management functions, supports filtering historical records by time, task type, or equipment, and provides users with long-term data accumulation and analysis support.

[0118] (5) Remote decision-making and management module provides users with remote management and decision-making support functions. Users can access the platform through mobile devices or PCs and adjust task parameters or view inspection results at any time. The platform also provides intelligent analysis tools to provide users with road repair priority recommendations through data statistics and prediction models, and assist in formulating maintenance plans.

[0119] Through the task management module, equipment scheduling function, data processing and other functions of the smart platform, users can achieve full process management and efficient execution of inspection tasks. The platform not only significantly improves the efficiency of inspection tasks, but also provides strong technical support for road maintenance and management in smart cities.

[0120] As an example, the disease detection data collected by the inspection equipment is transmitted to the platform server in real time through the WebSocket protocol, achieving low-latency data reception and two-way communication. After receiving the inspection data, the server automatically processes the data, including data calibration, anomaly detection, and feature extraction, to ensure the accuracy and integrity of the data. Figure 2As shown in the figure, the processed data is transmitted to the front end of the platform, and the inspection results are displayed in real time in various forms such as maps and charts through data rendering tools. Users can view information such as disease distribution and disease type on the front end interface, which supports real-time interaction with data and facilitates rapid analysis and decision-making.

[0121] Through the above implementation methods, the present disclosure realizes efficient disease detection through ground-air collaboration, fully utilizes the complementary advantages of drones and unmanned vehicles, and improves the coverage, accuracy and efficiency of road disease detection.

[0122] Example 2

[0123] In one embodiment of the present disclosure, a detection method of a road disease detection system based on deep learning and ground-air collaboration is provided, including:

[0124] Step 1: Obtain the inspected pavement crack images and underground disease radar data;

[0125] Step 2: Preprocess the acquired data, input the preprocessed pavement crack image into the pavement crack detection model, first use the depth separable convolution to decompose the convolution into depth convolution and point-by-point convolution, then extract the local features and global features of the cracks respectively through multi-scale feature extraction, and finally output the disease type of the pavement crack; input the preprocessed underground disease radar data into the underground disease detection model, the underground disease detection model uses the lightweight MobileNetV3 as the backbone network, combined with the feature pyramid network and the path aggregation network to achieve multi-scale feature fusion, and optimize the anchor frame generation strategy through the K-Means clustering algorithm, and output the underground disease disease type and location;

[0126] Step 3: Generate a task log for each inspection task. Based on the task log, you can adjust the task parameters or view the inspection results at any time.

[0127] As an embodiment, the detailed processing flow of the detection method of the road disease detection system based on deep learning and ground-air collaboration disclosed in the present invention is as follows:

[0128] Inspections are carried out using a three-dimensional ground-penetrating radar road defect detection device based on ground-air collaboration, such as Figure 3 As shown, the disease detection device includes a drone, an unmanned vehicle and a ground-penetrating radar. Both the drone and the unmanned vehicle are equipped with high-definition cameras, IMUs (inertial measurement units) and GPS positioning systems. The high-definition camera collects images of road cracks to assist in identifying surface diseases; the IMU is used to provide posture change data of the drone and the unmanned vehicle during flight and driving to ensure positioning accuracy; the GPS system records the geographic location and detection path of the drone and the unmanned vehicle in real time, so as to accurately locate the disease point later. Specifically, Figure 4 As shown:

[0129] S1: Preliminary inspection by drone: In this embodiment, drones and unmanned vehicles conduct road disease inspections through ground-air collaboration. The drone is first responsible for preliminary scanning of inaccessible areas or areas with complex terrain, especially urban fringe areas, bridges, tunnel entrances, narrow streets and other areas that are difficult to reach with conventional inspection equipment. The drone uses the ground-penetrating radar it carries to scan the structure below the road to obtain preliminary data on road diseases. Once the drone has scanned a specific area and identified potential disease points, these hot spots will be marked and transmitted to the unmanned vehicle, which will go to these areas according to the mission instructions for more detailed inspections and data collection;

[0130] S2: Detailed detection by unmanned vehicle: After receiving the inspection task from the cloud platform, the unmanned vehicle goes to the hot spot marked by the drone for detailed detection. The unmanned vehicle further obtains higher-resolution underground disease data through ground-penetrating radar to ensure the comprehensiveness and accuracy of disease diagnosis. For different types of diseases, the on-board radar system can more accurately identify and classify, especially for detailed scanning of underground diseases. After the detection is completed, the unmanned vehicle transmits the collected high-resolution data to the cloud platform, which performs real-time analysis and generates a disease detection report.

[0131] S3: Data upload and task feedback: The data collected by drones and unmanned vehicles in real time are uploaded to the smart platform through wireless communication. The platform monitors the entire inspection process and provides real-time feedback on the task status, inspection progress and data processing of drones and unmanned vehicles. Through real-time analysis of uploaded data, the platform can dynamically adjust tasks, optimize inspection routes, and ensure that key areas are fully inspected. All data during the inspection process will be stored and processed to form a complete disease data archive;

[0132] The uploaded data is preprocessed. The preprocessing process includes: removing noise through Gaussian filtering and median filtering operations, unifying the resolution through scale adjustment to ensure the consistency of model input, and performing data enhancement operations such as rotating, cropping, and flipping the image.

[0133] Furthermore, the preprocessed pavement crack image is input into the pavement crack detection model. First, the convolution is decomposed into deep convolution and point-by-point convolution using depthwise separable convolution, and then the local and global features of the cracks are extracted respectively through multi-scale features, and finally the disease type of the pavement crack is output; the preprocessed underground disease radar data is input into the underground disease detection model. The underground disease detection model uses a lightweight MobileNetV3 as the backbone network, combined with a feature pyramid network and a path aggregation network to achieve multi-scale feature fusion, and optimizes the anchor frame generation strategy through the K-Means clustering algorithm, and outputs the disease type and location of the underground disease.

[0134] S4: Subsequent task instructions: If an area that needs further inspection is found during the inspection, or if certain disease points have a high urgency to repair, the platform will issue new task instructions to drones and unmanned vehicles. These instructions include in-depth detection of specific areas, or confirmation of repair suggestions for identified disease points. The platform ensures the accurate execution of inspection tasks and timely response and handling of emergencies through real-time task management and scheduling systems.

[0135] This embodiment realizes efficient road defect detection through a ground-air collaborative three-dimensional ground-penetrating radar road defect detection device, combining the advantages of drones and unmanned vehicles. The device significantly improves the efficiency and accuracy of road defect detection and promotes the development of intelligent road management.

[0136] Example 3

[0137] In one embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the detection method of the road damage detection system based on deep learning and ground-air collaboration.

[0138] Example 4

[0139] In one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the detection method of the road damage detection system based on deep learning and ground-air collaboration is implemented.

[0140] Example 5

[0141] In one embodiment of the present disclosure, an electronic device is provided, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the detection method of the road defect detection system based on deep learning and ground-to-air collaboration.

[0142] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0143] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0144] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A road damage detection system based on deep learning and ground-air collaboration, characterized by: The invention comprises a smart platform, wherein the smart platform comprises: Task management module, used to remotely configure and publish inspection tasks, dynamically schedule equipment resources, and monitor task progress in real time; Equipment scheduling and coordination module, used to obtain images of road cracks and radar data of underground diseases during inspection; The data processing module is used to preprocess the acquired data and input the preprocessed pavement crack image into the pavement crack detection model. First, the convolution is decomposed into deep convolution and point-by-point convolution using deep separable convolution, and then the local and global features of the cracks are extracted respectively through multi-scale feature extraction, and finally the disease type of the pavement crack is output; the preprocessed underground disease radar data is input into the underground disease detection model. The underground disease detection model uses a lightweight MobileNetV3 as the backbone network, combined with a feature pyramid network and a path aggregation network to achieve multi-scale feature fusion, and optimizes the anchor frame generation strategy through the K-Means clustering algorithm, and outputs the disease type and location of the underground disease; Task log and historical data management module, used to generate task logs for each inspection task; Remote decision-making and management module, used in smart platforms, to adjust task parameters or view inspection results at any time.

2. The road damage detection system based on deep learning and ground-air collaboration as claimed in claim 1, characterized in that: Through the coordinated inspection of drones and unmanned vehicles, the ground penetrating radars on drones and unmanned vehicles conduct comprehensive inspections of road infrastructure. Drones and unmanned vehicles share data in real time through wireless communications. Through ground-to-air collaboration, images of road surface cracks and radar data of underground diseases are obtained. Road surface crack diseases include transverse cracks, longitudinal cracks and irregular cracks. Underground diseases are divided into voids, looseness, voids and poor layer adhesion.

3. The road disease detection system based on deep learning and ground-air collaboration as claimed in claim 1, characterized in that: The preprocessing process includes: removing noise through Gaussian filtering and median filtering operations, unifying the resolution through scale adjustment to ensure the consistency of model input, and performing data enhancement operations such as rotating, cropping, and flipping the image.

4. The road disease detection system based on deep learning and ground-air collaboration as claimed in claim 1, characterized in that: The pavement crack detection model is an improved lightweight convolutional neural network architecture. In the improved lightweight convolutional neural network architecture, depthwise separable convolution is used to replace the traditional convolution operation, and the convolution is decomposed into depthwise convolution and point-by-point convolution. Residual connection is introduced, and the input of the previous layer is added to the output of the current layer through jump connection to retain the low-level feature information. A multi-scale feature extraction module is used, combined with convolution kernels of different sizes, to capture the local and global features of the cracks.

5. The road damage detection system based on deep learning and ground-air collaboration as claimed in claim 1, characterized in that: The underground disease detection model is an improved YOLOv4 neural network architecture, using a lightweight MobileNetV3 as the backbone network, combined with deep separable convolution and hard activation functions to maintain efficient feature extraction capabilities; the anchor frame generation strategy is optimized through the K-Means clustering algorithm to generate anchor frames that are more suitable for the size and shape of the disease; the feature pyramid network and path aggregation network are combined in the detection head to achieve multi-scale feature fusion.

6. The road disease detection system based on deep learning and ground-air collaboration as claimed in claim 1, characterized in that: The task log and historical data management module generates a task log for each inspection task, recording the task execution time, equipment usage, inspection area coverage and task completion. Users can view the key nodes in the inspection process through the task log and evaluate the task effectiveness. It also provides historical data management functions, supports filtering historical records by time, task type or equipment, and provides long-term data accumulation and analysis support.

7. The detection method of the road disease detection system based on deep learning and ground-air collaboration according to any one of claims 1 to 6 is characterized in that: include: Obtain images of road cracks and radar data of underground damage during inspection; The acquired data is preprocessed, and the preprocessed pavement crack image is input into the pavement crack detection model. First, the convolution is decomposed into deep convolution and point-by-point convolution using deep separable convolution, and then the local features and global features of the cracks are extracted respectively through multi-scale feature extraction, and finally the disease type of the pavement crack is output; the preprocessed underground disease radar data is input into the underground disease detection model, which uses a lightweight MobileNetV3 as the backbone network, combined with a feature pyramid network and a path aggregation network to achieve multi-scale feature fusion, and optimizes the anchor frame generation strategy through the K-Means clustering algorithm, and outputs the disease type and location of the underground disease; Generate a task log for each inspection task, and adjust task parameters or view inspection results at any time based on the task log.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by the processor, the detection method of the road damage detection system based on deep learning and ground-air collaboration described in claim 7 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the detection method of the road damage detection system based on deep learning and ground-air collaboration as described in claim 7 is implemented.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the detection method of the road defect detection system based on deep learning and ground-air collaboration as described in claim 7.

Images