Lightweight road disease real-time detection method and system facing unmanned aerial vehicle platform

Abstract

The invention relates to the technical field of road disease detection, in particular to a light-weight road disease real-time detection method and system for an unmanned aerial vehicle platform, and the method comprises the steps: collecting road surface image data through an unmanned aerial vehicle, and constructing an unmanned aerial vehicle road disease detection data set UAVRDD10k; inputting the image data into a lightweight road disease detection model LRDet-P for processing; outputting a road disease detection result; yOLOv11n is used as a basic model to construct an LRDet-P model, and the LRDet-P model comprises a backbone network which adopts a lightweight heavy parameter multi-scale sensing network RMANet and is used for extracting features of multi-scale road diseases; an MSSC module is introduced into the neck network and is used for acquiring characteristics of extreme length-width ratio road diseases; the detection head adopts a SlideHead module and is used for balancing learning of difficult samples and improving the detection capability of the difficult samples; and further compressing the volume of the model through a pruning algorithm. According to the lightweight model LRDet-P provided by the invention, the detection precision is improved and the real-time performance of 57.5 FPS is achieved while the model parameter quantity is reduced and the calculated amount is reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of road disease detection, and particularly relates to a lightweight road disease real-time detection method and system for a UAV platform. BACKGROUND

[0002] Roads are often affected by harsh environments and heavy vehicles, resulting in various road diseases. Road diseases not only destroy the stability of the internal structure of the road, reduce the service life of the road disease, but also increase the safety hazards of road traffic. In order to prolong the service life of the road and reduce the safety hazards, the municipal department needs to quickly obtain the information of the road disease in the early stage of the disease development, so as to take timely maintenance measures. Therefore, it is particularly important to obtain road disease information in a timely and accurate manner.

[0003] Compared with the vehicle-mounted platform and manual patrol, the UAV remote sensing platform has the characteristics of easy operation, low cost and wide coverage. The digital camera carried by the UAV can collect high-resolution road image data in a wide range and efficiently. In addition, due to the development of convolutional neural networks and the opening of large-scale high-quality natural image data sets, the detection accuracy and robustness of general object detection algorithms based on deep learning on natural images have made great progress. Therefore, some researches use general object detection algorithms to automatically search and locate road diseases in UAV images, greatly improving the detection efficiency of road diseases. However, there is a huge difference in shooting angle between UAV images and natural images, and the general object detection algorithm is not suitable for road disease detection tasks.

[0004] Therefore, some researches improve the general object detection algorithm based on the characteristics of the road disease detection task, further improving the detection accuracy of road diseases. However, the structure of the target detection algorithm designed by these methods is complex and the parameter quantity is huge, which cannot be deployed to edge devices such as UAVs, and requires a large amount of computing resources and time, making it difficult to meet the real-time target detection requirements. Some special tasks, such as post-disaster emergency response, require the municipal department to use the UAV platform to carry lightweight road disease detection algorithms to update road disease information in real time. In addition, countries with large land area usually have numerous road network infrastructure, and efficient road disease detection algorithms are also needed to reduce the cost of road disease detection.

[0005] Therefore, some researchers propose lightweight road disease detection algorithms to improve the detection efficiency of road diseases and enable them to be deployed to edge devices. Wu et al. proposed a lightweight road disease detection model YOLO-LWNet for mobile terminal devices based on YOLOv5. Xiang et al. proposed a lightweight GC-YOLOv5 by replacing the original convolution module of YOLOv5 with a Ghost Module. Xu et al. proposed a lightweight road disease detection model DGE-YOLO-P for complex natural scenes based on YOLOv8. Zhou et al. proposed a road disease detection algorithm YOLOv8-MS that combines precision and lightweight based on YOLOv8, and proposed Multipath Coordinate Attention (MPCA) and Slim-neck structure according to the characteristics of the road disease detection task. Although these methods can effectively reduce the parameter quantity and computational complexity of the model, there is a huge size and shape difference between road disease targets. The receptive field of the convolution layer in the lightweight target detection algorithm is limited and fixed in a square shape, which cannot obtain the features of multi-scale and multi-shaped road disease targets, reducing the detection accuracy.

[0006] In addition, the target detection algorithm based on deep learning is a typical data-driven algorithm, and the performance and generalization ability of the model depend not only on the design of the model, but also on the support of high-quality data sets. In particular, the expression ability of lightweight models decreases, and high-quality data sets are needed to improve the generalization of the model. However, due to commercial factors and high labeling costs, the open source data sets available to researchers have the disadvantages of small size, poor quality, and insufficient diversity. SUMMARY

[0007] The present application aims to solve the problems of lack of large-scale, high-quality, and diverse benchmark data sets, and the target detection model based on deep learning usually has large model parameter quantity and high computational complexity, making it difficult to meet the needs of model deployment and real-time detection, and the lightweight target detection model cannot accurately detect multi-scale and multi-shaped road diseases. A lightweight road disease real-time detection method and system for unmanned aerial vehicle platforms are proposed, which builds the largest and highest quality unmanned aerial vehicle road disease data set UAVRDD10k, and combines the innovative lightweight road disease detection model LRDet-P, achieving reduced model parameter quantity and computational complexity while improving detection accuracy, achieving a real-time performance of 57.5 FPS (57.5 images per second).

[0008] To achieve the above purpose, the technical scheme adopted is:

[0009] The application provides a lightweight road disease real-time detection method for a UAV platform, comprising the following steps:

[0010] Collecting road surface image data using a UAV to construct a UAV road disease detection data set UAVRDD10k;

[0011] Inputting the image data into a lightweight road disease detection model LRDet-P for processing;

[0012] Outputting a detection result of the road disease;

[0013] The LRDet-P model is constructed based on a YOLOv11n model, and the LRDet-P model comprises:

[0014] The backbone network adopts a lightweight reparameterization multi-scale perception network RMANet for extracting multi-scale road disease features;

[0015] An MSSC module is introduced in the neck network for obtaining features of road diseases with an extreme aspect ratio;

[0016] The detection head adopts a SlideHead module for balancing the learning of difficult and easy samples and improving the detection capability of difficult samples;

[0017] The model volume is further compressed through a pruning algorithm.

[0018] According to the lightweight road disease real-time detection method for the UAV platform, further, the process of constructing the UAVRDD10k data set is as follows:

[0019] Collecting road surface images using a UAV carrying a camera;

[0020] Deleting images without diseases or repeated redundancies, and supplementing samples through open source data for a small number of diseases;

[0021] Defining seven types of road diseases, labeling the positions and categories of the road diseases in all UAV images through LabelImg, and generating annotation files in VOC / YOLO / COCO three formats;

[0022] The data set provides 10,395 UAV images with a uniform size, containing 25,303 labeled instances.

[0023] According to the lightweight road disease real-time detection method for the unmanned aerial vehicle platform, further, the RMANet comprises a plurality of serial RepHGBlock modules, each RepHGBlock module comprises a plurality of stacked 3*3 re-parameter convolution, and the convolution output feature maps are spliced, and then the channel number is adjusted through 1*1 re-parameter convolution, and then the original input is added element by element to output.

[0024] According to the lightweight road disease real-time detection method for the unmanned aerial vehicle platform, further, the calculation formula of the re-parameter convolution of the RepHGBlock module is:

[0025]

[0026] Wherein, W is the convolution kernel weight, x is the input data, gamma and beta are training parameters, sigma 2 is the variance, mu is the mean, epsilon is a constant, and b is the bias.

[0027] According to the lightweight road disease real-time detection method for the unmanned aerial vehicle platform, further, the core part of the MSSC module is the MSS Bottleneck, and the working process of the MSS Bottleneck is:

[0028] The input feature map is divided into a plurality of parts in the channel dimension through a convolution layer;

[0029] The strip depth separable convolution of different size convolution kernels is sequentially performed on each part to obtain the extreme aspect ratio features in the horizontal direction and the vertical direction;

[0030] The processed features are spliced in the channel dimension, and are fused through 1*1 convolution;

[0031] The extreme aspect ratio features in any direction are further obtained through deformable convolution.

[0032] According to the lightweight road disease real-time detection method for the unmanned aerial vehicle platform, further, the SlideHead module adjusts the learning weight of the difficult and easy sample through the following weighting function:

[0033]

[0034] Wherein, x represents the IOU value of the prediction frame and the real frame, mu represents the average IOU of all samples; for difficult samples, the weight is fixed as 1, and the model is forced to pay attention to these samples which are easily misdetected or missed; for medium difficulty samples, the weight is set to a higher e 1-μ , and the model is promoted to improve the recognition ability of the medium difficulty samples; for easy samples, the weight increases exponentially with the increase of IOU e 1-xDown, suppress the model's overemphasis on easy samples.

[0035] According to the lightweight road disease real-time detection method for the unmanned aerial vehicle platform, further, the pruning algorithm is an amplitude pruning based on layer adaptive sparsity, and the score standard is:

[0036]

[0037] (W[u]) 2 indicates the weight square of the target channel, indicates the weight square sum of all remaining channels in the same layer; delete the channel whose score is lower than the preset threshold.

[0038] Further, the present application also provides a lightweight road disease real-time detection system for unmanned aerial vehicle platform, for realizing the lightweight road disease real-time detection method for unmanned aerial vehicle platform, comprising:

[0039] The data acquisition module is used for collecting road surface image data using the unmanned aerial vehicle, and constructing an unmanned aerial vehicle road disease detection data set UAVRDD10k.

[0040] The detection module is used for inputting the image data into the lightweight road disease detection model LRDet-P for processing.

[0041] The output module is used for outputting the detection result of the road disease.

[0042] The detection module comprises:

[0043] The RMANet module is used for extracting the features of multi-scale road diseases.

[0044] The MSSC module is used for obtaining the features of road diseases with extreme aspect ratio.

[0045] The SlideHead module is used for balancing the learning of difficult and easy samples.

[0046] The pruning module is used for further compressing the model volume through the pruning algorithm.

[0047] The above technical scheme has the beneficial effects that:

[0048] In view of the problems that the lightweight model in the field of unmanned aerial vehicle road disease detection cannot accurately detect multi-scale and multi-shape road diseases, and lacks large-scale, high-quality and diverse benchmark data sets, the present application provides a lightweight road disease real-time detection method for unmanned aerial vehicle platform.

[0049] (1) The present application discloses a currently largest, best quality and most diverse unmanned aerial vehicle road disease detection data set UAVRDD10k, which contains 10395 unmanned aerial vehicle images of 1280x1280 pixels, 7 categories and 25303 instances.

[0050] (2) The present application proposes a lightweight road disease detection model LRDet-P for real-time and accurate detection of multi-scale and multi-shape road diseases in unmanned aerial vehicle images. Specifically, four improvements are made to the YOLOv11n for the road disease detection task: first, the backbone network adopts a lightweight reparameterized multi-scale perception network (RMANet) to extract multi-scale road disease features; second, a multi-scale and multi-shape C3K (MSSC) is introduced in the neck network to obtain features of extreme aspect ratio road diseases; third, the detection head adopts a SlideHead module to balance the learning of difficult and easy samples, focusing on learning difficult road disease samples; finally, a pruning algorithm is used to further realize the lightweight of the model. Experimental results show that the proposed LRDet-P improves the mAP of the baseline by 4.1%, and the parameter quantity, model size and FLOPs are reduced to 50.3%, 60.7% and 65% of the baseline, respectively, and the FPS is 57.5. LRDet-P not only meets the lightweight requirements of deployment on unmanned aerial vehicle platforms, but also accurately and in real time detects multi-scale and multi-shape road diseases. BRIEF DESCRIPTION OF DRAWINGS

[0051] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings of the embodiments of the present application will be briefly introduced below. Among them, the drawings are only used to show some embodiments of the present application, and not to limit all embodiments of the present application to this.

[0052] Figure 1 is a flowchart of the lightweight real-time road disease detection method for unmanned aerial vehicle platform of the embodiments of the present application;

[0053] Figure 2 is a flowchart of the UAVRDD10k data set of the embodiments of the present application;

[0054] Figure 3 is a schematic diagram of the road disease categories in the UAVRDD10k data set of the embodiments of the present application;

[0055] Figure 4 is a road disease spatial distribution statistical chart of the embodiments of the present application;

[0056] Figure 5 Fig. 3 is a schematic diagram of the LRDet-P model structure of an embodiment of the present application;

[0057] Figure 6 Fig. 4 is a schematic diagram of the RMANet structure of an embodiment of the present application;

[0058] Figure 7 Fig. 5 is a schematic diagram of the MSSC module structure of an embodiment of the present application;

[0059] Figure 8 Fig. 6 is a schematic diagram of the SlideHead module structure of an embodiment of the present application;

[0060] Figure 9 Fig. 7 is a schematic diagram of the LRDet-P detection result of an embodiment of the present application, (a) small size target; (b) large size target; (c) multi-scale target; (d) extreme aspect ratio target;

[0061] Figure 10 Fig. 8 is a comparison diagram of the detection results of LRDet-P and other algorithms of an embodiment of the present application, (a) (b) extreme aspect ratio target; (c) (d) multi-scale target. DETAILED DESCRIPTION

[0062] In the following, the example schemes of the embodiments of the present application will be described clearly and completely with reference to the accompanying drawings of the specific embodiments of the present application. Unless otherwise defined, the technical terms or scientific terms used in the present application should be understood as the general meanings understood by those skilled in the art.

[0063] As shown in Figure 1 , the present embodiment discloses a lightweight road disease real-time detection method for unmanned aerial vehicle platform, which comprises the following steps:

[0064] Step S1, using an unmanned aerial vehicle to collect road surface image data, and constructing an unmanned aerial vehicle road disease detection data set UAVRDD10k.

[0065] The process of making the UAVRDD10k data set is shown in Figure 2 , which mainly includes three steps of data collection, data cleaning and supplementing, and disease labeling.

[0066] (1) Data collection

[0067] 10395 UAV images with road damages were collected, which mainly contain two parts. First, the road surface images were collected by DJI Mavic Air 2 UAV equipped with L1D-20c digital camera. The detailed aerial parameters are shown in Table 1. UAV is easily affected by the complex environment in the air, such as street trees, power poles, and power towers, so the flight height is dynamically changing. The size of a single road surface image collected by L1D-20c digital camera is 5472x3648 pixels.

[0068] Table 1. Aerial parameters of UAV for collecting road surface images

[0069]

[0070] To comprehensively cover different sizes and types of road damages, five main roads (Longxiang Avenue, Future Road, Chang'an Avenue, and Daxiangshan Road, Pingbao Avenue) in the urban area of Pingdingshan City, Henan Province were selected as experimental sites. Longxiang Avenue and Future Road were built the earliest, and there are serious surface damages. Chang'an Avenue and Pingbao Avenue were built the latest, and there are mainly small and weak linear damages. Daxiangshan Road was built between the three, and there are various types of road damages.

[0071] Second, high-quality images were selected from public datasets, mainly to supplement the rare road damage categories in the UAV collected data, to balance the number difference of different categories in the UAVRDD10k dataset, and to improve the diversity of the background in the dataset. In order to facilitate subsequent research, all image data were unified to 1280x1280 pixels.

[0072] (2) Data cleaning and supplement

[0073] The large-scale collection of UAV images ensures the wide distribution of road damages. However, not all UAV images contain road damages, and there are also a large number of redundant images. In addition, there is a significant imbalance in the number of different categories of road damages (for example, the number of inclined cracks and potholes is small, while the number of transverse cracks and longitudinal cracks is large). This phenomenon will lead to a decline in the overall detection accuracy of the model, for example, the mAP of YOLOv3 in the UAPD dataset is 56.6%, but the detection accuracy of inclined cracks is only 32.3%, and the detection accuracy of inclined cracks seriously hinders the overall detection accuracy.

[0074] Therefore, the method of data cleaning and supplementing is used to ensure the quality of the UAVRDD10k dataset. The data cleaning and supplementing method mainly includes three aspects: (1) deleting images without road damage; (2) considering that the overlap rate of UAV images is very high, and the road damage difference between adjacent images is very small, redundant images are also deleted; (3) selecting some high-quality images in the open source dataset to supplement the less road damage categories such as inclined cracks and potholes in the UAV acquisition process. Considering the labeling workload and area, finally 10395 UAV images for road damage detection are obtained, of which 1135 images come from the supplemented data and 9260 images come from the collected data.

[0075] (3) Disease annotation

[0076] The types of road damage are defined as seven categories: alligator crack, longitudinal crack, inclined crack, block crack, pothole, transverse crack and repair. The road damage categories in the UAVRDD10k dataset are shown in FIG. 1. Among them, pothole, alligator crack and block crack are all surface diseases, and alligator crack is more serious than block crack. While transverse, longitudinal and inclined cracks are linear cracks in different directions. Among them, the direction of the transverse crack is basically perpendicular to the road center line; the direction of the longitudinal crack is basically parallel to the road center line; the direction of the inclined crack is at an angle of 30-60° to the road center line. Figure 3

[0077] LabelImg is used to accurately label the position and category of road damage in all UAV images. The accuracy and continuity of data labeling are crucial to the quality of the dataset. Therefore, in order to ensure the continuity and high quality of data labeling, an economical and practical labeling process is designed.

[0078] The specific labeling process is as follows: first, assign the labeling task to three professional labelers, two of whom are data labelers and one is a supervisor. The supervisor has professional knowledge of road damage maintenance. Then, let the labelers learn the detailed definition of each road damage; second, in the process of labeling road damage by labelers, the supervisor constantly patrols to supervise the labeling quality of labelers, and once the category of a certain road damage is ambiguous, the best category is selected by voting of the three people; finally, the supervisor randomly checks the labeled data, thereby ensuring the high quality and continuity of the dataset.

[0079] Through this labeling process, 25303 instances of high-quality road damage of seven different categories are labeled in the collected UAV images, and the training set, validation set and test set are divided according to the ratio of 6:2:2. The detailed number of different road damage categories in the training set, validation set and test set is shown in Table 2.

[0080] ​Table 2 Information of training set, validation set and test set in UAVRDD10k dataset

[0081] Class name Training sample set Validation sample set Test sample set Total Crazing 1351 504 464 2319 Checking 3060 982 1064 5106 Longitudinal cracking 2612 775 882 4269 Transverse cracking 3999 1339 1316 6654 Repairing 2990 968 973 4931 Diagonal cracking 888 285 273 1446 Pitting 377 95 106 578 Total 15277 4948 5078 25303

[0082] (4) Data records

[0083] To ensure the fairness of subsequent research, not only the UAVRDD10k dataset is open-sourced, but also the division of training set, validation set and test set is disclosed. The folder of the open-sourced UAVRDD10k dataset contains four files, namely UAVRDD10k_JPEGImages, UAVRDD10k_VOC, UAVRDD10k_YOLO and UAVRDD10k_COCO. Among them, UAVRDD10k_JPEGImages contains three folders of train, val and test. They respectively contain the UAV image data of the training set, the validation set and the test set; UAVRDD10k_VOC contains three folders of train, val and test. They respectively contain the annotation box of VOC format of the training set, the validation set and the test set, in the format of.xml; among them, UAVRDD10k_YOLO contains three folders of train, val and test. They respectively contain the annotation box of YOLO format of the training set, the validation set and the test set, in the format of.txt; among them, UAVRDD10k_COCO contains three folders of train, val and test. They respectively contain the annotation box of COCO format of the training set, the validation set and the test set, in the format of.json.

[0084] (5) Data attributes

[0085] In order to determine the spatial distribution and characteristics of road diseases in UAVRDD10k dataset, a series of statistics and analysis are carried out, and the details are shown in Figure 4 Figure 4 (a) provides intuitive information of the distribution of the center point coordinates of the annotation box, which helps to understand the distribution characteristics of the location of road diseases in UAVRDD10k. From the figure, it can be seen that the distribution of the center of the annotation box is relatively uniform, and there is a tendency to concentrate on the center line. This shows that the distribution of road diseases in UAVRDD10k dataset is relatively uniform, which can train a more general model. Figure 4 (b) is the distribution histogram of the aspect ratio of the road disease annotation box (the longest side divided by the shortest side in the annotation box rectangle). From the figure, it can be seen that there are a large number of extreme aspect ratio targets in UAVRDD10k, which shows that the shape difference between road diseases in UAVRDD10k is huge. Figure 4 ​(c) is a road disease bounding box normalized spatial distribution diagram showing the spatial distribution characteristics of road disease targets in the UAVRDD10k dataset. As can be seen from the figure, the road diseases in the UAVRDD10k are evenly distributed, and the sizes of the road disease targets are greatly different, with obvious multi-scale characteristics.

[0086] In summary, the characteristics of road diseases in the UAVRDD10k dataset are as follows:

[0087] (1) Multi-scale: The size of road diseases on UAV images is easily affected by various factors. Differences in flight height and different sensor resolutions can cause the size of the same road disease in the image to be significantly different. In addition, different road diseases have different sizes in UAV images due to different damage levels. These factors interact with each other, making the size of road diseases in the UAVRDD10k vary greatly, which poses a great challenge to designing a model that accurately detects multi-scale road diseases.

[0088] (2) Extreme aspect ratio: Various road diseases exhibit significant differences in shape. For example, cracks and block cracks are approximately square in shape; while transverse cracks and longitudinal cracks are long and narrow, with lengths much greater than widths. This clear shape difference results in a wide variety of aspect ratios for road diseases in the UAVRDD10k, requiring the detection model to have strong shape adaptation capabilities to cope with complex and variable shapes.

[0089] (3) Rich diversity: Different roads have different environments and building materials, resulting in different characteristics of diseases in UAV images. If a deep learning model is trained only on data collected from one road, the model may only accurately detect diseases on that road, and the recognition accuracy for diseases in other areas will be greatly reduced. The types of roads in the UAVRDD10k include asphalt roads, concrete roads, and country roads; the backgrounds include downtown areas, residential areas, and farmland, etc., with high diversity. However, this also poses new challenges for the target detection model to resist background noise.

[0090] (4) Large scale and excellent quality: The UAVRDD10k dataset contains 10395 UAV images with a size of 1280x1280 pixels, covering 25303 instances and 7 road disease categories. Most importantly, each instance is annotated by professionals with high quality. Compared with other similar datasets, this dataset is unique in both scale and quality, fully meeting the stringent needs of training high-performance and strong generalization deep learning models.

[0091] Step S2, inputting the image data into the lightweight road disease detection model LRDet-P for processing.

[0092] The YOLO series of object detection algorithms are widely used in real-time drone road defect detection due to their high detection speed. YOLOv11 is a more mature and innovative version of the YOLO series of object detection algorithms. It strikes a balance between detection accuracy and speed and is widely used in engineering and scientific research. YOLOv11 can be categorized into the YOLOv11n, YOLOv11s, YOLOv11m, and YOLOv11l models based on the number of parameters. However, YOLOv11 cannot accurately detect multi-scale and multi-shape road defects in complex scenarios. Furthermore, its model size and computational cost are too large to be embedded in resource-constrained mobile devices, making it lacking practical application capabilities.

[0093] To address the above issues and ensure the lightweight and real-time performance of the road damage detection model, the YOLOv11n model with the smallest number of parameters and computational complexity was selected as the basic model. Based on this model, targeted improvements and redesigns were made based on the characteristics of the UAV road damage detection task to obtain a new road damage detection model LRDet-P. The detailed structure of LRDet-P is shown in the figure below. Figure 5 As shown in the figure, RMANet is designed as the backbone network of LRDet-P to extract features of multi-scale road damage. Secondly, the MSSC module is introduced in the neck network to obtain features of road damage with extreme aspect ratios. Then, the SlideHead module is used in the detection head to balance the difficulty and easy samples in the road damage detection task, with a particular focus on improving the model's ability to handle difficult samples. Finally, a pruning algorithm is used to further compress the model size, minimizing the computational cost while minimizing the loss of model detection accuracy, resulting in a lightweight road damage detection model.

[0094] (1)RMANet

[0095] In drone imagery, the sizes of different road damages vary significantly. However, the YOLOv11n backbone network lacks multi-scale feature extraction capabilities, making it ineffective and requiring a large number of parameters. To address this issue, inspired by the HGNetV2 and RepVGG backbone networks of RT-DETR, a re-parameterized multilayer awareness network (RMANet) was designed to extract multi-scale features of road damage while reducing the number of parameters and computational overhead.

[0096] The detailed structure of RMANet is as follows Figure 6(a) shows that it achieves a better balance between speed and accuracy through efficient convolutional structure design, the combination of lightweight modules, and optimization of GPU hardware, which makes it have a significant advantage in real-time applications and edge devices. RMANet includes multiple serial RepHGBlock modules, and the detailed structure of the RepHGBlock module is as shown in Figure 6 (b) shows. Each RepHGBlock module includes multiple stacked 3x3 RepConv, which splices the output feature maps of each convolution, adjusts the channel number through 1x1 RepConv, and then outputs after element-wise addition with the original input.

[0097] The RepConv in the RepHGBlock module obtains multi-scale receptive fields through a multi-branch structure during training, and merges these structures during inference to maintain efficient target detection. It can significantly improve the detection accuracy of the model without increasing the computational amount of model inference. Its detailed formula is as follows. First, the formula of convolution can be expressed as:

[0098] Conv(x)=W(x)+b

[0099] Where W is the convolution kernel weight, x is the input data, and b is the bias. Second, a parallel BN layer is integrated into the convolution layer as shown in the following formula:

[0100]

[0101] Where μ is the mean, σ 2 is the variance, and ε is a very small constant to prevent the denominator from being zero. Finally, during the inference stage, the outputs of multiple parallel branches are merged into a 3x3 convolution layer and simplified as shown in the following formula:

[0102]

[0103] Where γ and β are the parameter weight and trade-off parameter that need to be learned during the training process. After simplification, can be used as the weight of the fused convolution, can be used as the bias of the fused convolution.

[0104] (2) MSSC module

[0105] In UAV images, road disease targets not only vary greatly in size, but also vary greatly in shape, and there are many extreme aspect ratio targets. However, the receptive field of YOLOv11n is a regular rectangle, which cannot obtain the features of extreme aspect ratio targets. In order to solve this problem, inspired by PKINet and deformable convolution, a multi-scale and multi-shape C3K (MSSC) is proposed to obtain the features of extreme aspect ratio road disease targets.

[0106] The detailed structure of the MSSC module is shown in Figure 7 (a), Figure 7 (b) is the core part of the MSSC module, Multi-scale and multi-shape Bottleneck (MSS Bottleneck). First, the input feature map is enlarged by the multi-scale orthogonal strip depth separable convolution to obtain the features of extreme aspect ratio targets in the horizontal and vertical directions. Then, deformable convolution is used to obtain extreme aspect ratio targets in any direction.

[0107] The details of the MSS Bottleneck are as follows. First, the input features are divided into four parts (X i (i∈{0,1,2,3})) by a convolution layer in the channel direction, and then the four parts are sequentially processed by strip depth separable convolution with different size kernels to obtain the features of extreme aspect ratio targets in the horizontal and vertical directions F i (i∈{0,1,2,3}):

[0108] F i = DWConv nx1 (DWConv 1xn (X i ))

[0109] where DWConv 1xn represents a strip depth separable convolution with a width greater than a height, which mainly obtains the features of extreme aspect ratio targets in the horizontal direction. DWConv nx1 represents a strip depth separable convolution with a height greater than a width, which mainly obtains the features of extreme aspect ratio targets in the vertical direction.

[0110] Then, the processed features are concatenated in the channel dimension and fused by 1x1 convolution to obtain the features of extreme aspect ratio targets:

[0111] F = Conv 1x1 (Concat(F1,F2,F3,F4))

[0112] Although the multi-scale orthogonal strip depth separable convolution can increase the receptive field and obtain the features of the extreme aspect ratio target, it can only obtain the extreme aspect ratio features in the horizontal and vertical directions, and cannot adapt to the extreme aspect ratio target in any direction. Therefore, a deformable convolution is further used to obtain the features O of the extreme aspect ratio target in any direction:

[0113]

[0114] wherein N is the sampling point data of the convolution kernel, Δp n is the fixed offset of the nth sampling point, w(n) is the weight of the corresponding sampling point, is the offset learned through the network, and F(q) is the feature value of the input feature map at position q.

[0115] (3) SlideHead module

[0116] In the road disease detection task, due to the differences in size and shape of the disease targets and the occlusion, distortion and blur of the road diseases in the image taken by the unmanned aerial vehicle, the difficulty of identifying different road disease targets is different. Generally, the number of easy samples is large, and the number of difficult samples is relatively sparse. The loss function used in the head of YOLOv11n cannot effectively balance the attention of the model to easy and difficult samples, and the model will pay more attention to easy samples, which leads to the problem that difficult samples cannot be fully learned and the detection accuracy of difficult samples is low.

[0117] In order to solve this problem, Slide Loss is introduced into the head of YOLOv11n to design a SlideHead module to solve the problem of imbalance between easy and difficult samples in the model training process, and to pay special attention to improving the processing ability of the model to difficult samples. The detailed structure of the SlideHead module is shown in Figure 8 The core idea of the weighting function of the SlideHead module is to give greater weight to difficult samples. The difference between easy samples and difficult samples is based on the IoU size of the prediction box and the groundtruth box. The detailed formula of the Slide Loss weighting function is:

[0118]

[0119] wherein μ represents the average IOU of all samples, and samples less than μ are negative samples and samples greater than μ are positive samples. The weighting function sets the weight of the sample whose IOU is less than μ-0.1 to 1, sets the weight of the sample whose IOU is in the range of μ-0.1 and μ to a higher e 1 -μ and sets the weight of the sample whose IOU is greater than μ to a smaller e 1-xAnd it will gradually decrease with the increase of IOU. By adaptively adjusting the threshold parameter mu, the model's attention to more difficult to detect targets is improved to balance the model's detection ability for multi-size targets.

[0120] (4) LAMP pruning algorithm

[0121] The improved lightweight model is improved in terms of calculation amount and memory occupation, but there is still excessive parameter redundancy. In order to further compress the lightweight model and ensure that the model can be smoothly deployed on the unmanned aerial vehicle platform, the layer-adaptive sparsity for the magnitude-based pruning (LAMP) is used to further compress the model.

[0122] The pruning algorithm judges whether the channel structure needs to be removed through scoring. At present, the evaluation standard of most pruning algorithms is to compare the absolute value of the weight. The scoring standard of LAMP is to compare the square of the weight, which can significantly magnify the difference between the weights and improve the accuracy of pruning. The definition of LAMP score is as follows:

[0123]

[0124] Each weight tensor is fully unfolded into a one-dimensional vector and arranged in ascending order, and u and v represent the indexes of the channel weight. Then, (W[u]) 2 represents the weight item mapped by index u, and (W[v]) 2 represents the weight item mapped by index v. The larger the square of the weight mapped by index u, the higher the LAMP score. Conversely, the smaller the square of the weight mapped by index u, the lower the LAMP score. After calculating the LAMP score, the channel with the smallest LAMP score is globally pruned until the global sparsity constraint is met. Since there is an optimal channel with a score of 1 in each layer, the layer collapse phenomenon is effectively avoided. By applying the LAMP algorithm, the number of parameters of the model can be significantly reduced, making it more suitable for deployment on resource-constrained unmanned aerial vehicle platforms.

[0125] Step S3, outputting the detection result of the road disease.

[0126] Corresponding to the above method, the embodiment also discloses a lightweight real-time road disease detection system for an unmanned aerial vehicle platform, comprising:

[0127] A data acquisition module is configured to use an unmanned aerial vehicle to collect road surface image data and construct an unmanned aerial vehicle road disease detection data set UAVRDD10k.

[0128] The detection module is configured to input the image data into a lightweight road disease detection model LRDet-P for processing.

[0129] The output module is configured to output the detection result of the road disease.

[0130] The detection module comprises:

[0131] The RMANet module is configured to extract features of the multi-scale road disease.

[0132] The MSSC module is configured to obtain features of the road disease with an extreme aspect ratio.

[0133] The SlideHead module is configured to balance the learning of difficult and easy samples.

[0134] The pruning module is configured to further compress the model volume by using a pruning algorithm.

[0135] To verify the effectiveness of the present scheme, the following experimental data are further explained and described.

[0136] (1) Experimental environment and evaluation index

[0137] All experiments use the same hardware and software devices to ensure the fairness of the experiments. The hardware devices include AMD R76800H CPU, 16GB memory and NVIDIA GeForce RTX 3060Laptop GPU. The software devices include 64-bit Windows operating system, CUDA 12.4, PyTorch 2.6 and Python 3.10.

[0138] The mAP and F1 score are used as evaluation indexes for the improvement of the detection accuracy of the model, and the larger the value is, the higher the detection accuracy of the model is.

[0139] (2) Ablation experiment

[0140] To analyze the influence of the RMANet, MSSC and SlideHead proposed in the present application on the performance of the LRDet, the YOLOV11n is used as a baseline, and the RMANet, ShapeFPN and SlideHead are added in the baseline, respectively, so as to carry out detailed ablation experiments. The detailed experimental data are shown in Table 3.

[0141] Table 3: LRDet ablation experiment

[0142]

[0143] As shown in Table 3, the first row, the model size of the baseline is 5.6 MB, the parameter quantity is 2.60M, the mAP is 62.6%, the FLOPs is 6.3G, and the FPS is 64.2. As shown in Table 3, the second row, after replacing the backbone network of the baseline with RMANet, the model size is reduced by 0.8MB, the parameter quantity is reduced by 0.47M, the FLOPs is reduced by 0.6G, and the mAP is improved by 0.8%. This shows that the RMANet proposed in the present application can not only significantly reduce the parameter quantity and complexity of the model, but also improve the detection accuracy of road diseases. As shown in Table 3, the third row, after replacing the C3K2 module in the neck part of the baseline with MSSC. Although the model size, parameter quantity and FLOPs increase slightly, the model mAP is improved by 2.1%, which is the largest improvement compared with RMANet and SlideHead. This shows that the MSSC proposed in the present application can effectively improve the detection accuracy of the model, while slightly increasing the parameter quantity, calculation amount and FLOPs of the model. As shown in Table 3, the fourth row, after adding SlideHead, the parameter quantity, model size and FLOPs of the model do not increase, and the mAP is improved by 0.4%.

[0144] As shown in Table 3, the fifth row, after adding RMANet and MSSC at the same time, the mAP is improved by 2.7%, and the model size, parameter quantity and FLOPs are also reduced. As shown in Table 3, the sixth row, after continuing to add SlideHead, the model mAP is improved by 0.8% without increasing the model parameter quantity, size and FLOPs. However, due to the complex structure of RMANet and MSSC, the FPS is reduced after adding them to the baseline.

[0145] The experimental results show that RMANet, MSSC and SlideHead all make positive contributions to the performance improvement of LRDet, and they are all effective improvements for the road disease detection task.

[0146] (3) Model pruning

[0147] In order to reduce the model calculation cost to the greatest extent while reducing the loss of model detection accuracy as much as possible, channel pruning is performed on LRDet by setting different pruning rates, and the detailed experimental results are shown in Table 4. Among them, P represents pruning of the model (Pruning), and Speed-up represents different pruning rates.

[0148] Table 4: LRDet model pruning experiment

[0149]

[0150] The data in Table 4 shows that as the pruning ratio increases, model size, parameter count, FLOPs, and mAP gradually decrease, while FPS improves. Therefore, selecting an appropriate pruning ratio is crucial, striking a balance between model accuracy, lightweightness, and detection speed. Analysis shows that optimal model performance is achieved when the speed-up is 1.4. At this point, model accuracy is only 0.1% lower than when the speed-up is 1.1, and higher than when the speed-ups are 1.7 and 2.0. Furthermore, the FPS is only 1.2 lower than the highest FPS. The number of parameters, FLOPs, and model size are also relatively low.

[0151] Experimental results show that model pruning effectively reduces LRDet's model size, number of parameters, and number of FLOPs, while also improving its detection speed. LRDet-P achieves a 4.1% improvement in mAP compared to the baseline, with parameters and FLOPs reduced to 50.3% and 65% of the baseline, respectively, and an FPS of 57.5. This achieves a balance between detection accuracy and efficiency, effectively addressing the high computational cost of the model.

[0152] (4) Comparative experiment

[0153] To further validate the effectiveness of LRDet and LRDet-P and provide a performance benchmark for future research on UAV road damage detection, we compared their performance with 17 state-of-the-art object detection algorithms on the UAVRDD10k dataset. The object detection algorithms included are: YOLOv5n, YOLOv6, YOLOv8n, YOLOv10n, YOLOv11n, YOLOv12n, Hyper-YOLO, Gold-YOLO, YOLOv7-tiny, YOLOR, TOOD, Cascade R-CNN, RetinaNet, Deformable DETR, FreeAnchor, Mask R-CNN, and Faster R-CNN. All experiments were conducted on the same hardware, and the batch size for all object detection algorithms was set to 1 when testing FPS to ensure fairness and reliability. Detailed experimental results are shown in Table 5.

[0154] Table 5 Comparison experimental results of each model

[0155]

[0156] The LRDet proposed in the application achieves optimal effects in precision, recall rate and mAP, which are 65.9%, 64.3% and 66.1% respectively. The LRDet-P achieves optimal results in model size, parameter quantity and FLOPs, which are 2.6MB, 0.93M and 2.8G respectively. Compared with the LRDet model, the LRDet-P has a small decrease in precision indicators such as precision, recall rate and mAP, and a large decrease in lightweight indicators such as parameter quantity, model size and FLOPs, which are reduced by 49.7%, 39.3% and 35% respectively, and the FPS is increased by 21.9, which is more efficient in real-time detection. It is worth noting that the performance of the LRDet-P exceeds some lightweight target detection algorithms, such as YOLOv7-tiny, YOLOv5n, YOLOv8n, YOLOv10n, YOLOv11n and YOLOv12n.

[0157] In addition, as can be seen from the table, some two-stage target detection algorithms and Transformer-based target detection algorithms, such as Faster R-CNN, Mask R-CNN and Deformable DETR, although have high detection accuracy, have large model size, parameter quantity and FLOPs, and slow detection speed, which are difficult to meet the deployment requirements of unmanned aerial vehicles and other edge devices. On the contrary, one-stage target detection algorithms not only have fast detection speed, but also have high level of model lightweight, and have not low detection accuracy. For example, the FPS of YOLOv11n reaches 64.2, which is the fastest among all target detection algorithms. And the model size, parameter quantity and precision are 5.6MB, 2.6M and 62.6% respectively; Hyper-YOLO achieves 64.4% mAP with only 3.6M parameter quantity. One-stage target detection algorithms have more potential in future road disease detection tasks.

[0158] In summary, the lightweight road disease detection model LRDet-P proposed in the application can accurately and in real time detect multi-scale and multi-shape road diseases in complex scenes, and has high practicability by balancing the accuracy and model efficiency. In addition, through the comparison of 17 target detection algorithms on the UAVRDD10k dataset, a performance benchmark can be provided for future unmanned aerial vehicle road disease detection algorithms.

[0159] (5) Visual analysis

[0160] ① Detection result visual analysis: in order to visually display the effect of LRDet-P in detecting road diseases in unmanned aerial vehicle images, the detection results of LRDet-P on the test set of UAVRDD10k are analyzed, and the detailed results are shown in Figure 9 Figure 9 ​(a) shows that LRDet-P can accurately detect small-sized road damage targets, although these small-sized road damage targets have weak features and are in complex backgrounds such as branches, cars, water stains, etc. As shown in Figure 9 (b) shows that LRDet-P has good detection effect on large-sized road damage targets, and the area of these large-sized road damage targets basically occupies the whole UAV image, which shows that LRDet-P can obtain a large receptive field. As shown in Figure 9 (c) shows that LRDet-P can accurately detect multi-scale targets in a UAV image, which is mainly due to the multi-scale structure of MSSC and RMANet in LRDet-P, which can obtain the features of multi-scale road damage. As shown in Figure 9 (d) shows that LRDet-P can accurately detect extremely long and narrow ratio targets in a UAV image, which is mainly due to the ability of LRDet-P to obtain the features of extremely long and narrow ratio targets.

[0161] ②Comparison and analysis of detection results: In order to intuitively compare the differences in detection effect between LRDet-P and other algorithms, the detection results of LRDe-P and some of the latest target detection algorithms are compared in the same image area, including YOLOv8, YOLOv11 and GOLD-YOLO. As shown in Figure 10 (a) and (b) show that because YOLOv8 and GOLD-YOLO cannot obtain the features of extremely long and narrow ratio targets, they cause a large number of false detections on extremely long and narrow ratio cracks. Benefiting from the ability of LRDet-P to obtain the features of extremely long and narrow ratio targets, it can obtain the features of extremely long and narrow ratio targets, thereby reducing false detection. As shown in Figure 10 (c) and (d) show that because GOLD-YOLO, YOLOv8 and YOLOv11 cannot obtain the features of multi-scale targets, they cause false detection of large-sized targets and miss detection of small-sized targets. Benefiting from the multi-scale acquisition ability of LRDet-P, it can accurately obtain the features of multi-scale targets, so as to detect large-sized road damage targets and small-sized road damage targets.

[0162] The experimental results show that LRDet-P is significantly better than other algorithms in multi-scale and multi-shape target detection in complex background.

[0163] Finally, it should be noted that the above-described embodiments are merely specific embodiments of the present application, which are used to illustrate the technical solutions of the present application, but not to limit the same. The protection scope of the present application is not limited thereto. Although the present application has been described in detail with reference to the foregoing embodiments, it should be understood by those skilled in the art that any person skilled in the art can still modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacements to some of the technical features, within the technical scope disclosed by the present application. The modifications, changes or replacements do not cause the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A lightweight real-time road disease detection method for UAV platforms, characterized by: The following steps are involved: Use drones to collect road surface image data and build the UAV road damage detection dataset UAVRDD10k; The image data is input into the lightweight road damage detection model LRDet-P for processing; Output the detection results of road diseases; The LRDet-P model is constructed using YOLOv11n as the base model. The LRDet-P model includes: The backbone network uses a lightweight, heavy-parameter multi-scale perception network RMANet to extract the characteristics of multi-scale road damage; The MSSC module is introduced into the neck network to obtain the characteristics of road diseases with extreme aspect ratios; The detection head uses the SlideHead module to balance the learning of difficult and easy samples and improve the detection ability of difficult samples; The model size is further compressed through the pruning algorithm.

2. The lightweight real-time road damage detection method for UAV platforms according to claim 1 is characterized in that: The process of constructing the UAVRDD10k dataset is: Use drones with cameras to collect road surface images; Delete disease-free or redundant images, and supplement samples with open-source data for diseases with fewer cases. Define seven types of road damage, use LabelImg to annotate the locations and categories of road damage in all drone images, and generate annotation files in three formats: VOC, YOLO, and COCO. The dataset provides 10,395 drone images of uniform size, containing 25,303 annotated instances.

3. The lightweight real-time road damage detection method for UAV platforms according to claim 1 is characterized in that: The RMANet includes multiple serial RepHGBlock modules, each of which includes multiple stacked 3×3 multi-parameter convolutions. The feature maps output by each convolution are concatenated and then subjected to 1×1 multi-parameter convolution to adjust the number of channels. The convolution feature maps are then added element-by-element to the original input and output.

4. The lightweight real-time road damage detection method for UAV platforms according to claim 3 is characterized in that: The calculation formula of the re-parameter convolution of the RepHGBlock module is: Among them, W is the convolution kernel weight, x is the input data, γ and β are training parameters, σ 2 is the variance, μ is the mean, ε is a constant, and b is the bias.

5. The lightweight real-time road disease detection method for UAV platforms according to claim 1 is characterized in that: The core part of the MSSC module is the MSS Bottleneck. The working process of the MSS Bottleneck is as follows: The input feature map is divided into multiple parts according to the channel dimension through a convolution layer; Perform strip-like depth-separable convolution with kernels of different sizes on each part in turn to obtain extreme aspect ratio features in the horizontal and vertical directions; The processed features are concatenated in the channel dimension and fused through 1x1 convolution; Deformable convolution is used to further obtain features with extreme aspect ratios in any direction.

6. The lightweight real-time road damage detection method for UAV platforms according to claim 1 is characterized in that: The SlideHead module adjusts the learning weights of difficult and easy samples through the following weighting function: Among them, x represents the IOU value between the predicted box and the real box, and μ represents the average IOU of all samples. For difficult samples, the weight is fixed to 1, forcing the model to pay attention to these samples that are easy to be misdetected or missed. For medium-difficulty samples, the weight is set to a higher e 1-μ , which promotes the model to improve its recognition ability for medium-difficulty samples; for easy samples, the weight increases exponentially with the increase of IOU 1 -x Decrease, suppressing the model's excessive attention to easy samples.

7. The lightweight real-time road damage detection method for UAV platforms according to claim 1 is characterized in that: The pruning algorithm is an amplitude pruning based on layer-adaptive sparsity, and its scoring criteria are: Among them, (W[u]) 2 represents the square weight of the target channel, Represents the sum of squared weights of all remaining channels in the same layer; channels with scores below a preset threshold are removed.

8. A lightweight real-time road disease detection system for UAV platforms, characterized by: A method for implementing a lightweight real-time road damage detection method for an unmanned aerial vehicle platform as described in any one of claims 1 to 7, comprising: The data acquisition module is used to collect road surface image data using drones and construct the UAV road damage detection dataset UAVRDD10k; The detection module is used to input image data into the lightweight road damage detection model LRDet-P for processing; Output module, used to output the detection results of road diseases; The detection modules include: RMANet module, used to extract multi-scale road damage features; The MSSC module is used to obtain the characteristics of road damage with extreme aspect ratios; SlideHead module, used to balance learning of difficult and easy samples; The pruning module is used to further compress the model size through the pruning algorithm.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory, wherein: The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images