Lightweight unmanned aerial vehicle pavement disease detection method and device and storage medium

Abstract

The invention relates to the field of pavement detection, and discloses a lightweight unmanned aerial vehicle pavement disease detection method and device and a storage medium, and the method comprises the steps: obtaining a pavement image shot by an unmanned aerial vehicle; constructing an improved unmanned aerial vehicle pavement disease detection network LHP-YOLO based on YOLOv8, and performing pre-training by using the data set to obtain a trained network; and inputting a pavement image shot by the unmanned aerial vehicle into the trained network to obtain an unmanned aerial vehicle pavement disease detection result. The technical problems that a current pavement disease detection method is difficult to deploy at a mobile terminal and low in precision are solved.

Description

Technical Field

[0001] The present invention relates to the field of road surface detection, and in particular to a lightweight unmanned aerial vehicle road surface disease detection method, equipment and storage medium. Background Art

[0002] UAV-based pavement disease target detection is to locate and fine-grainedly classify diseases in road images. In modern urban management, the speed and accuracy of information acquisition directly affect the maintenance effect of urban infrastructure. Real-time acquisition of information such as the location, quantity, and distribution of road diseases is key to formulating maintenance strategies, evaluating road health status, and making rapid repair decisions.

[0003] Therefore, being able to quickly and accurately detect road surface disease targets in complex and diverse road environments is crucial to ensuring the smooth operation of urban traffic and public safety. Compared with other target detection, drone detection has the following characteristics: it is necessary to identify targets at multiple angles and heights at high altitudes, and accurately distinguish them in complex environments. The algorithm must have high real-time performance and be able to respond and detect targets quickly. Due to limited hardware resources, especially on embedded devices, the algorithm needs to be lightweight to run efficiently under limited computing resources and memory conditions. At the same time, low-energy design is the key to extending the flight time and mission execution capabilities of drones.

[0004] As one of the core technologies of drone detection, target detection algorithm models can usually be divided into a two-stage detection algorithm based on candidate regions and a single-stage detection algorithm based on regression.

[0005] Two-stage target detection algorithms include Faster R-CNN, Cascade R-CNN, R-FCN, etc., and single-stage target detection algorithms include YOLO, SSD, EfficientDet, DETR, etc. Two-stage target detection algorithms perform well in high-precision and complex scenes, while single-stage target detection algorithms have significant advantages in speed and real-time performance. Therefore, based on the characteristics of drone target detection, scholars mainly use single-stage detection algorithms for research. However, since the single-stage detection algorithm directly performs classification and regression on the feature map and does not generate candidate regions, it is easy to miss and misdetect when dealing with small targets and complex backgrounds.

[0006] In the research on model accuracy improvement technology, distillation technology (Knowledge Distillation) plays a significant role in improving the accuracy of target detection models. Current research on improving model accuracy and lightweighting mainly focuses on network design and distillation technology, but there is little work that effectively combines the two. Summary of the invention

[0007] The purpose of the present invention is to propose a lightweight unmanned aerial vehicle road surface disease detection method, equipment and storage medium to solve the technical problems that the current road surface disease detection method is difficult to deploy on the mobile terminal and has low accuracy.

[0008] This paper uses YOLOv8n as the baseline model, conducts research from two perspectives: lightweight network design and distillation technology, and proposes a lightweight and efficient UAV pavement disease detection algorithm LHP-YOLO.

[0009] Specifically, the present invention provides a lightweight unmanned aerial vehicle road disease detection method, comprising the following steps:

[0010] S1, obtaining road surface images taken by a drone;

[0011] S2. Build the improved UAV pavement disease detection network LHP-YOLO based on YOLOv8, and use the data set for pre-training to obtain a trained network;

[0012] S3. Input the road surface image taken by the drone into the trained network to obtain the drone road surface disease detection results.

[0013] A storage medium stores instructions and data for implementing a lightweight unmanned aerial vehicle road surface disease detection method.

[0014] A lightweight unmanned aerial vehicle road surface disease detection device comprises: a processor and the storage medium; the processor loads and executes instructions and data in the storage medium to implement a lightweight unmanned aerial vehicle road surface disease detection method.

[0015] The beneficial effects provided by the present invention are:

[0016] 1) Use RG-C2f to replace the C2f module in YOLOv8 to reduce the redundant calculation of the model and improve the calculation speed while maintaining the feature extraction capability.

[0017] 2) A lightweight detection head Detect-T3G was designed, which effectively reduced the number of model parameters by streamlining and merging the convolutional layers.

[0018] 3) CGAFusion is introduced in the feature fusion layer of the YOLOv8 neck network to dynamically adjust the weights according to the image content, effectively focusing on important areas in the image and ignoring irrelevant information, thereby improving the accuracy of small target detection.

[0019] 4) Use CWD to fine-tune the lightweight model to restore its detection accuracy.

[0020] 5) By introducing more online data enhancement methods during the training process, the risk of model overfitting is effectively reduced, the robustness of the model is increased, and the performance of the model on unseen data is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 It is a schematic flow chart of the method of the present invention;

[0022] Figure 2 This is a schematic diagram of the LHP-YOLO network structure;

[0023] Figure 3 This is a schematic diagram of the RG-C2f module structure;

[0024] Figure 4 It is a schematic diagram of the content-guided attention fusion CGA Fusion structure;

[0025] Figure 5 It is a schematic diagram of the Detect-T3G structure;

[0026] Figure 6 This is a schematic diagram of 6 typical pavement diseases captured by drones;

[0027] Figure 7 This is a schematic diagram comparing the visualization results of LHP-YOLO and YOLOv8n;

[0028] Figure 8 It is a working schematic diagram of the hardware device of the present invention. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described below with reference to the accompanying drawings.

[0030] Before formally explaining the present invention, relevant technical terms in the present invention are explained. In addition, the scheme of the present invention is also generally explained first for easy understanding.

[0031] Please refer to Figure 1 , Figure 1 It is a schematic flow diagram of the method of the present invention.

[0032] The present invention provides a lightweight unmanned aerial vehicle road surface disease detection method, comprising the following steps:

[0033] S1, obtaining road surface images taken by a drone;

[0034] S2. Build the improved UAV pavement disease detection network LHP-YOLO based on YOLOv8, and use the data set for pre-training to obtain a trained network;

[0035] Please refer to Figure 2 , Figure 2 This is a schematic diagram of the LHP-YOLO network structure;

[0036] It should be noted that the improved UAV pavement disease detection network LHP-YOLO based on YOLOv8 includes: CSPRGX structure, improved Neck part and Detect-T3G detection head.

[0037] Regarding the CSPRGX structure, as an example, since the flying altitude and angle of the drone often change, the captured images have different perspectives and scales. The C2f module in YOLOv8, which is based on the ELAN structure of YOLOv7, is not sufficient to capture the changes of targets of all scales in the rapidly changing drone shooting scene, and has problems such as a large number of parameters.

[0038] To address the above problems, this paper constructs the RG-C2f module using a residual structure consisting of reparameterizable convolution (RepConv), standard convolution, and auxiliary convolution, as shown in Figure 3 As shown, Figure 3 This is a schematic diagram of the RG-C2f module structure. Figure 3 RG-C2f replaces C2f in the original YOLOv8 network, and the new network structure is named CSPRGX structure.

[0039] It should be noted that the RG-C2f module uses cheap one-dimensional auxiliary convolution to generate some redundant feature maps to solve the widespread redundancy problem of intermediate feature maps calculated by mainstream CNN, thereby reducing the amount of calculation and parameters.

[0040] In addition, the RepConv structure uses a multi-branch structure in the training phase, that is, a standard convolution layer and multiple auxiliary convolution layers, and merges the multi-branch convolution layers in the training phase into a single convolution layer in the inference phase. Compared with standard convolution, RepConv achieves higher accuracy and lower computational cost and memory usage.

[0041] When the CSPRGX network structure performs feature extraction and feature fusion, it can effectively detect road defects photographed from different perspectives and scales, reduce redundant computing and access costs, and achieve network lightweighting.

[0042] Regarding the Neck part, as an example, when drones collect road surface defects, they usually work in complex environments, such as cities, forests, and suburbs, and the targets in the captured images may be relatively small and dense. The neck structure of YOLOv8 adopts a PAN network structure, which can effectively fuse feature maps of different scales, but lacks the design of enhancing image details. When processing complex scenes, small targets may be missed. In order to improve the model's detection of detailed information, this paper introduces content-guided attention fusion (CGAFusion) to the feature fusion network of the neck structure of YOLOv8, such as Figure 4 As shown, Figure 4 It is a schematic diagram of the content-guided attention fusion CGAFusion structure.

[0043] CGAFusion first extracts multi-level features of the image, then obtains global information through a content-guided mechanism, and then uses an attention mechanism to dynamically weight the features. Finally, these weighted features are multi-level fused to integrate low-level detail information and high-level semantic information. Compared with the original YOLOv8 neck structure, the introduction of CGAFusion's efficient feature fusion network can highlight important areas and suppress unimportant parts, thereby generating detailed and clear images and improving target detection accuracy.

[0044] Regarding the Detect-T3G detection head, as an example, in order to enable road maintenance departments to make quick decisions and arrange repair work in a timely manner, the real-time detection of road surface diseases by drones plays a vital role. At the same time, drones usually have limited computing power and storage resources, which greatly restricts the operation of complex algorithms.

[0045] The detection head of YOLOv8 adopts a decoupled structure and is divided into two independent branches: object classification and bounding box regression. The decoupled head can improve the overall detection accuracy by separating tasks. However, independent components to handle different tasks will make the network structure more complex, increase computing overhead, and affect real-time performance.

[0046] To solve the above problems, the present invention optimizes the detection head of YOLOv8, simplifies and merges the standard convolution of the two branches in the decoupling head structure, and replaces them with group convolution (GConv), such as Figure 5 As shown, the new detection head structure is named Detect-T3G.

[0047] GConv groups the input feature maps and convolution kernels, performs convolution operations independently in each group, and then merges the results. Each group can be regarded as a specialized module that focuses on a subset of the input data. Compared with standard convolution, GConv can significantly reduce the overall computational cost and improve the model's ability to express data. The Detect-T3G detection head not only effectively ensures real-time detection, but also improves target detection accuracy.

[0048] Finally, the present invention adopts channel-wise knowledge distillation (CWD) to improve the accuracy of pavement disease detection of lightweight model drone.

[0049] The formula is as follows:

[0050]

[0051] L total =L train +aL distill (3)

[0052] Where: t and s represent the teacher and student networks respectively, y t and s represents the corresponding feature activation, C is the channel index, i corresponds to the spatial position of the channel, T is the distillation temperature, and its default value is 1, L train is the training model loss, L distill is the knowledge distillation loss, and α is the distillation loss ratio. The distillation loss ratio determines the weight of knowledge transfer between the teacher model and the student model, which is used to balance the guidance of the teacher model to the student model, optimize the performance of the student model, and maintain a low model complexity.

[0053] S3. Input the road surface image taken by the drone into the trained network to obtain the drone road surface disease detection results.

[0054] As an embodiment, the present invention uses the RDD 2022 dataset to verify the effectiveness of the improved model. RDD2022 contains a total of 47,420 road images, including more than 55,000 road damage incidents from six countries (Japan, India, Czech Republic, Norway, the United States and China). Six main types of road damage in different complex environments are captured through different shooting forms, including longitudinal cracks (D00), transverse cracks (D10), mesh cracks (D20), patches (Repair), block cracks (Blockcrack) and potholes (D40). Typical road surface damage captured by drones are as follows: Figure 6 shown. Figure 6These are six typical pavement diseases captured by drones. This paper divides the dataset into training set, validation set and test set, with a division ratio of 7:1:2.

[0055] In order to reduce the risk of overfitting the model, increase the robustness of the model, and improve the performance of the model on unseen data, this paper implements online data augmentation techniques in the study: filtering the images with kernels of different sizes to blur; converting RGB images to grayscale and inverting them if their average pixel value exceeds 127; applying contrast-limited adaptive histogram equalization to the input images; randomly adjusting image brightness and contrast; using random gamma correction as a data augmentation technique; and reducing image quality through Jpeg and WebP compression techniques.

[0056] The hardware platform and environment parameters used in the experimental training phase are shown in Table 1.

[0057] Table 1 Training environment and hardware platform parameters

[0058]

[0059] The main parameters adjusted during the model training phase are shown in Table 2.

[0060] Table 2 Experimental parameter settings

[0061]

[0062] In order to evaluate the effectiveness of the improved model, this experiment uses precision (Precision, P), recall (Recall, R), parameter quantity (Params), memory usage (Model Size), mean average precision (mean Average Precision, mAP) and floating point operations (GFLOPs) evaluation indicators. Precision, Recall and mAP are used to judge the detection performance of the model; Params, Model Size and GFLOPs are used to judge the computational complexity of the model. For these evaluations, there are several key parameters, including TP (the number of correct targets in the detection results), FP (the number of wrong targets in the detection results), and FN (the number of targets that were not detected in the correct targets). The formula is as follows:

[0063]

[0064] The formula for calculating the average accuracy of n categories is as follows:

[0065]

[0066] In order to verify the improvement effect of various improvements in LHP-YOLO on the accuracy and lightweight of UAV road disease detection, a total of 5 groups of ablation experiments were carried out. The experimental results are shown in Table 3.

[0067] Experiment A uses CSPRGX as the new network structure, which greatly reduces the number of model parameters and computational complexity, reducing 0.81M and 2.0GFLOPs respectively, and increasing mAP50 by 1.3%.

[0068] Experiment B uses Detect-T3G as a new detection head based on Experiment A. The number of model parameters and the amount of computation are further reduced by 0.59M and 2.5G FLOPs respectively. However, the simplified model structure causes a 3.2% drop in mAP50.

[0069] Experiment C optimizes the neck structure based on Experiment B, and increases mAP50 by 2.3% while keeping it as lightweight as possible (increasing 0.16M and 0.3GFLOPs respectively compared to Experiment B), and significantly improves the detection accuracy, indicating that the new feature fusion structure effectively improves the accuracy of complex background and small-size target detection.

[0070] Experiment D uses the CWD fine-tuned lightweight improved model, and the mAP50 is improved by 3% compared with group C, and the number of parameters and calculations have not increased. Overall, compared with YOLOv8n, the parameters and calculations of the LHP-YOLO model proposed in this paper are reduced by 1.24M and 4.2GFLOPs respectively, and the mAP50 is increased by 3.4%, which not only improves the lightweight of the model, greatly reduces its deployment and operation difficulty on mobile devices, but also improves the model detection performance.

[0071] Table 3 Ablation experiment

[0072]

[0073] In order to visually test the effect of LHP-YOLO on drone road disease detection, 6 images in the test set were selected for target detection and visualization, and compared with the detection results of YOLOv8n. The detection results are shown in the figure. Figure 7 shown. Figure 7 The figure is a comparison diagram of the visualization results of LHP-YOLO and YOLOv8n. LHP-YOLO can correctly detect aircraft targets in different scenarios, proving that the algorithm has high accuracy while ensuring lightweight.

[0074] After the design of the RG-C2f module is completed, in order to determine the best replacement position to achieve the best effect after the model is improved, this paper determines the best replacement strategy through three groups of comparative experiments. The replacement strategies involved in the comparison are as follows. Replacement strategy A: global replacement, all 4 C2f modules in the backbone network and 4 C2f modules in the neck network are replaced with the RG-C2f modules in this section. Replacement strategy B: only replace in the backbone network, all 4 C2f modules in the backbone network are replaced with the RG-C2f modules in this section. Replacement strategy C: only replace in the neck network, all 4 C2f modules in the neck network are replaced with the RG-C2f modules in this section. Through experiments, it is found that global replacement, that is, the CSPRGX network structure proposed in this paper, can achieve the best effect. While improving the detection performance, it also achieves a lightweight effect. The model parameters and calculation amount are reduced by 0.81M and 2.0GFLOPs respectively, and mAP50 increases by 1.3%. The comparative experiments selected by the replacement strategy are shown in Table 4.

[0075] Table 4 Comparative experiment on replacement strategy selection

[0076]

[0077] Using YOLOv8s as the teacher and the lightweight improved model LHP-YOLO as the student, the detection heads in the teacher and student networks were selected for knowledge distillation. The experimental results are shown in Table 5. After knowledge distillation, the P and mAP50 of LHP-YOLO increased by 12.9% and 3% respectively. Table 6 reflects the impact of different distillation loss ratios on performance. When α = 1.5, the performance of the model reaches the best.

[0078] Table 5. Impact of knowledge distillation on model performance

[0079]

[0080] Table 6 Effect of different distillation loss ratios on performance

[0081]

[0082] In order to further verify the superiority of LHP-YOLO, this paper compares different target detection models, including not only classic models before YOLOv8, but also new models after YOLOv8, with a total of 9 groups of comparative experiments, as shown in Table 7. The comparison results show that LHP-YOLO has the least amount of calculation and parameters, and has the highest average target detection accuracy.

[0083] Table 7 Comparative test of different models

[0084]

[0085] See also Figure 8 , Figure 8 40 is a schematic diagram of the working of the hardware device of an embodiment of the present invention, and the hardware device specifically includes: a lightweight UAV road surface disease detection device 401, a processor 402 and a storage medium 403.

[0086] A lightweight UAV road surface disease detection device 401: The lightweight UAV road surface disease detection device 401 implements the lightweight UAV road surface disease detection method.

[0087] Processor 402: The processor 402 loads and executes the instructions and data in the storage medium 403 to implement the lightweight UAV road surface disease detection method.

[0088] Storage medium 403: The storage medium 403 stores instructions and data; the storage medium 403 is used to implement the lightweight UAV road surface disease detection method.

[0089] The beneficial effects of the present invention are:

[0090] 1) Use RG-C2f to replace the C2f module in YOLOv8 to reduce the redundant calculation of the model and improve the calculation speed while maintaining the feature extraction capability.

[0091] 2) A lightweight detection head Detect-T3G was designed, which effectively reduced the number of model parameters by streamlining and merging the convolutional layers.

[0092] 3) CGA Fusion is introduced in the feature fusion layer of the YOLOv8 neck network to dynamically adjust the weights according to the image content, effectively focusing on important areas in the image and ignoring irrelevant information, thereby improving the accuracy of small target detection.

[0093] 4) Use CWD to fine-tune the lightweight model to restore its detection accuracy.

[0094] 5) By introducing more online data enhancement methods during the training process, the risk of model overfitting is effectively reduced, the robustness of the model is increased, and the performance of the model on unseen data is improved.

[0095] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A lightweight unmanned aerial vehicle road surface disease detection method, characterized in that: The method comprises the following steps: S1, obtaining road surface images taken by a drone; S2. Build the improved UAV pavement disease detection network LHP-YOLO based on YOLOv8, and use the data set for pre-training to obtain a trained network; S3. Input the road surface image taken by the drone into the trained network to obtain the drone road surface disease detection results.

2. A lightweight unmanned aerial vehicle road surface disease detection method as claimed in claim 1, characterized in that: The improved UAV pavement disease detection network LHP-YOLO based on YOLOv8 includes: a CSPRGX structure, an improved Neck part and a Detect-T3G detection head.

3. A lightweight unmanned aerial vehicle road surface disease detection method as claimed in claim 2, characterized in that: The CSPRGX structure is composed of the RG-C2f module replacing the C2f module in the Backbone part of the YOLOv8 network.

4. A lightweight unmanned aerial vehicle road surface disease detection method as claimed in claim 3, characterized in that: The RG-C2f module consists of a residual structure consisting of a reparameterizable convolution RepConv, a standard convolution, and an auxiliary convolution.

5. A lightweight unmanned aerial vehicle road surface disease detection method as claimed in claim 2, characterized in that: The improved Neck part specifically refers to introducing a content-guided attention fusion CGAFusion module into the Neck network of YOLOv8.

6. A lightweight unmanned aerial vehicle road surface disease detection method as claimed in claim 2, characterized in that: The Detect-T3G detection head specifically merges the two branch standard convolutions of the decoupling head in the YOLOv8 detection head and replaces them with the group convolution GConv.

7. A storage medium, characterized in that: The storage medium stores instructions and data for implementing a lightweight unmanned aerial vehicle road defect detection method as described in any one of claims 1 to 6.

8. A lightweight drone road surface disease detection device, characterized by: include: Processor and storage medium; the processor loads and executes instructions and data in the storage medium to implement a lightweight unmanned aerial vehicle road surface disease detection method as described in any one of claims 1 to 6.

Images