A lightweight target detection method for unmanned aerial vehicle (UAV) inspection of insulators
By improving the YOLOv5 model and introducing an attention mechanism and a lightweight feature extraction module, the real-time performance and efficiency issues of insulator detection algorithms in UAV inspections were resolved, enabling real-time inspection of insulator targets and improving detection efficiency and accuracy.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SHENYANG JIANZHU UNIVERSITY
- Filing Date
- 2023-04-11
- Publication Date
- 2026-05-19
AI Technical Summary
Existing insulator target detection algorithms have too many parameters and too much floating-point computation, making them impossible to deploy in real time in UAV inspections. Traditional manual inspections are dangerous and inefficient.
By improving the YOLOv5 model, introducing an attention mechanism and an angle loss function, designing a lightweight feature extraction module and a multi-scale feature fusion module, and combining the rotating rectangle annotation method, the model structure is optimized to meet the needs of UAV inspection.
It achieves real-time and high-efficiency insulator target detection, reduces the number of model parameters and floating-point calculations, is suitable for deployment on UAV mobile terminals, and improves detection efficiency and accuracy.
Smart Images
Figure CN116385911B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of fault detection and maintenance of insulators for power transmission lines, specifically a lightweight target detection method for insulator inspection by unmanned aerial vehicles (UAVs). Background Technology
[0002] Insulators, as a crucial component of power systems, play a vital role in the safe and stable operation of transmission lines. Due to their long-term outdoor operation, they inevitably experience faults or damage. Common fault types include insulator spontaneous explosion, insulator flashover, and insulator cracks. In addition, insulators operating in harsh climates or special environments are prone to flashover tripping during thunderstorms due to contamination. These faults mostly appear on the surface of the insulator and can be observed visually. Traditional inspection methods primarily involve manual inspection by personnel climbing the towers, which is not only laborious and time-consuming but also inherently dangerous. With the rapid development of artificial intelligence and deep learning in recent years, drone inspection equipped with target detection algorithms has gradually become the mainstream method.
[0003] Traditional object detection algorithms typically generate a series of candidate regions from an image, extract image features from these regions, classify these features using a classifier algorithm, and finally determine whether to draw an envelope for the candidate region based on the classification result. With the rapid development of deep learning, a series of deep learning-based object detection algorithms have emerged in recent years. These algorithms far outperform traditional object detection algorithms in terms of both inference speed and model size. Object detection algorithms for power transmission line insulators are mostly adaptive improvements based on these traditional algorithms.
[0004] Reddy proposed converting the original images into a laboratory color space, reproducing color features in the laboratory color space using K-means clustering, and finally processing the insulator images using an Adaptive Network Fuzzy Inference System (ANFIS). Yi proposed an improved Faster R-CNN algorithm for aerial images of transmission line insulators. By introducing multi-scale training and an adversary generation strategy for detecting difficult samples, it achieves better detection results for partially occluded transmission line insulators and insulators of different image sizes compared to the original Faster R-CNN algorithm. Yang improved the loss function of the basic YOLOv3 model, added an InceptionResnet module, and used the k-means algorithm for anchor box selection, enabling the model to maintain both detection speed and high recognition rate. These algorithms mostly focus on improving the model's detection performance. However, in the actual process of inspecting insulators using drones, mobile and embedded devices equipped with object detection algorithms do not have sufficient storage space and computing power to support the operation of these algorithms. Summary of the Invention
[0005] To address the problem of excessive parameter count and floating-point computation in existing insulator target detection algorithms, a lightweight target detection method for UAV insulator inspection is proposed. By adding an attention mechanism and angle loss function to the basic YOLOv5 model, and improving the feature extraction module and multi-scale feature fusion module, the method enables the deployment of UAVs and their supporting equipment for insulator target detection, and performs real-time detection of insulator images captured by the UAV during the inspection process.
[0006] A lightweight target detection method for unmanned aerial vehicle (UAV) inspection of insulators includes the following steps:
[0007] Step 1: Obtain insulator images containing faulty insulators, and design a multi-label rotating rectangle annotation method for insulator images to adapt to the data characteristics of long strip-shaped detection targets with arbitrary angles in aerial insulator images and the detection rate of fault samples; divide the labeled insulator images into training, testing, and validation sets according to proportions to train, test, and validate the insulator fault detection model. The insulator fault detection model is obtained in subsequent steps based on the improvement of the YOLOv5 model.
[0008] First, images of insulators containing faulty insulators are collected using drone aerial photography, and the collected insulator images should include most common insulator types. Since insulator fault image samples are difficult to collect, the fault samples are expanded using data augmentation methods, including vertical flipping, horizontal flipping, cropping, erasing pixel blocks, adjusting hue, adjusting saturation, adjusting exposure, and scaling.
[0009] Insulator image data was labeled using the RoLabelImg image annotation tool. In addition to the specific fault type, two extra labels were set: "normel" for normal insulators and "damaged" for faulty insulators. The generated label format was an XML file. This format cannot be directly used for model training. Therefore, an eight-label representation was used, and the data was converted to TXT format via the open-source project RoLabelImg_Transform. The generated data format is (x1, y1, x2, y2, x3, y3, x4, y4), where x and y are the coordinates of the four vertices of the labeled rectangle. To reduce the number of parameters during training, the labels were further converted to the long-side representation in a five-parameter representation. The specific conversion formula is as follows:
[0010] x1=xw / 2*cos(θ)-h / 2*sin(θ) (1)
[0011] y1=yh / 2*cos(θ)-w / 2*sin(θ) (2)
[0012] x2=xw / 2*cos(θ)+h / 2*sin(θ) (3)
[0013] y2=yh / 2*cos(θ)+w / 2*sin(θ) (4)
[0014] x3=x+w / 2*cos(θ)-h / 2*sin(θ) (5)
[0015] y3=y+h / 2*cos(θ)-w / 2*sin(θ) (6)
[0016] x4=x+w / 2*cos(θ)+h / 2*sin(θ) (7)
[0017] y4=y+h / 2*cos(θ)+w / 2*sin(θ) (8)
[0018] The converted coordinate format is (x,y,w,h,θ), where x and y are the coordinates of the center point of the rectangle, w is the length of the longest side, the adjacent side of w is defined as h, and θ represents the range of angles through which the x-axis rotates to w, θ∈[-π / 2,π / 2).
[0019] The long-side representation has fewer parameters than the eight-parameter representation, which can ensure the training speed of the model. In addition, it is more similar to the labeling method of the rectangular box, which makes it easier to make structural adaptive improvements to the original YOLOv5 model.
[0020] Step 2: Using the YOLOv5 model as the base model, an improved GhostConv module is introduced into the backbone network. The network parameters of the model are greatly reduced by using ghost convolution, and the numerical stability of the model is enhanced by using the Hardswish function.
[0021] The improved GhostConv module consists of a batch normalized Ghost module, Batch Normalization, and Hardswish activation function in sequence. The difference between the Ghost module and ordinary convolution is that the Ghost convolution divides ordinary convolution into two parts. First, a 1x1 convolution with half the number of channels is performed to integrate features and generate feature condensation for the input feature layer. Then, a depthwise separable convolution (layer-by-layer convolution) is performed to generate similar feature maps of the feature condensation. The similar feature maps and feature condensation maps are concatenated to obtain the output feature map with the same structure as ordinary convolution, but its computational cost is only half that of ordinary convolution. Batch normalization not only accelerates model convergence but also alleviates gradient vanishing to some extent. Compared to the SiLU function used in the original YOLOv5 model, Hardswish has the advantages of good numerical stability and fast computation. The specific process is as follows:
[0022] The calculations performed at the convolution point are as follows:
[0023]
[0024] in This represents the j'-th weight of the i-th convolutional kernel in the l-th layer. This represents the j-th convolutional local region in the l-th layer, where W is the width of the convolution kernel, and x... l(j+j,) Let y represent the j'-th weight of the j-th convolutional region in layer l. l(i,j) This represents the output value in the i-th row and j-th column of the l-th layer;
[0025] The calculations performed by the batch normalization department are as follows:
[0026]
[0027]
[0028]
[0029] z i =γY i +β (13)
[0030] Where m represents the number of channel weights in the batch dimension, y i μ represents the i-th weight on this channel. B This represents the corresponding mean. Y represents the corresponding variance, γ and β are the scaling and bias of the BN layer, and Y represents the variance. i z is an intermediate variable in the batch normalization process. i This represents the output of the BN layer, where ε is a constant term to ensure numerical stability.
[0031] The formula for calculating the Hardswish function is as follows:
[0032]
[0033] Step 3: Design an improved C3Ghost structure. Through lightweight network design, the number of parameters and floating-point calculations of the original YOLOv5 model are reduced. Compared with the original YOLOv5 model, the detection efficiency can be greatly improved at the cost of a small loss of accuracy, which is more conducive to the deployment of UAVs.
[0034] The improved C3Ghost module is derived from the CSP structure. First, the input data flows into two channel branches. One branch flows through a GhostConv module, where the convolutional kernel size is 1x1, the stride is 1, and the bias is 0. The other part flows sequentially through a GhostConv module and n BottleNeck modules. The size of n depends on the position of the C3Ghost module; the deeper the GhostConv module at the feature extraction point, the larger n becomes. At the feature fusion point, n is always 3 layers. Then, these two parts are concatenated using Concat, and finally connected to a 1x1 GhostConv structure with a stride of 1 and a bias of 0. Furthermore, the proposed GhostConv module differs in the BottleNeck modules of the algorithm's feature extraction and feature fusion parts. The BottleNeck in the feature extraction part is formed by concatenating the input data through two GhostConv modules, while the BottleNeck in the feature fusion part only goes through two GhostConv modules.
[0035] Step 4: Introduce an SE attention mechanism between the feature extraction and feature fusion modules. Use the SE attention mechanism to learn the importance of each channel and suppress the attention to feature channels that are not important to the current task by assigning weights.
[0036] The introduced SE attention mechanism structure first performs global average pooling to make the feature map size 1x1 while keeping the number of channels unchanged. Then, it sequentially connects a fully connected layer, a ReLU activation function, another fully connected layer, and a sigmoid activation function. Finally, it performs matrix multiplication with the number of channels learned by the sigmoid activation function and the original input feature map.
[0037] Step 5: By introducing CSL to replace regression with classification, the problem of sudden increase in loss caused by boundary issues when calculating rotation angle loss is solved;
[0038] By introducing circular smooth labeling (CSL), classification is used instead of regression to avoid the angular periodicity problem faced by angular regression prediction, thus obtaining more robust prediction results. The expression for CSL is as follows:
[0039]
[0040] g(x) should simultaneously possess the characteristics of periodicity, symmetry, monotonicity, and a maximum value of 1; therefore, we choose g(x) as a trigonometric function.
[0041] Step 6: Propose a high-precision rotated frame regression loss function R-EIoU Loss to accelerate the model training process and improve the model's detection accuracy; After steps 2-6, the final insulator fault detection model is obtained.
[0042] Since the annotation method used in step 1 is the rotating rectangle annotation method, it is only necessary to add a rotation angle prediction channel to the Head structure of the algorithm to achieve the prediction of the rotating box. When performing loss calculation, in addition to the position loss function, category loss function and confidence loss function, an additional angle loss function needs to be added. The improvement of the loss function accelerates the training process of the model and improves the detection accuracy of the model.
[0043] YOLOv5 uses the binary cross-entropy loss function for both class loss and confidence loss. The class loss only calculates the loss for positive samples, while the confidence loss calculates the loss for all samples. YOLOv5 uses the CIOU loss function for location loss; the formula for this loss function is as follows:
[0044]
[0045]
[0046]
[0047] CIOU_Loss = 1 - CIOU (19)
[0048] In the above formula, IOU is the intersection-union ratio of the predicted bounding box and the ground truth bounding box. IOU is obtained by dividing the intersection of the predicted bounding box and the ground truth bounding box by the union of the predicted bounding box and the ground truth bounding box. It is the ratio of the square of the distance between the center points of the ground truth box and the center point of the predicted box to the square of the length of the diagonal of the smallest box of the two. In αv, v is the normalization of the difference in the aspect ratio between the predicted box and the ground truth box, and α is the balance factor that weighs the loss caused by the aspect ratio and the IOU.
[0049] While CIOU, building upon DIOU, considers the aspect ratio by adding a penalty term for both predicted and ground truth boxes, it still has two problems. First, CIOU uses a relative aspect ratio; once the aspect ratios of the predicted and ground truth boxes are linear, the penalty term added by CIOU for this relative ratio no longer works. Second, the width and height of the predicted box cannot be scaled simultaneously; as the gradient formula shows, increasing one parameter will necessarily decrease the other. To address these issues, EIOU proposes a loss function that directly penalizes the predicted values of height and width. The EIOU calculation formula is as follows:
[0050]
[0051] Although EIOU is also composed of aspect ratio, overlap area and center point distance, it directly uses the side length as a penalty term when calculating the aspect ratio, which can address the two problems in CIOU and accelerate the convergence of the model.
[0052] During training, EIoU Loss is used to optimize the position and shape of the rotated rectangle, while Smooth L1 Loss is used to optimize the rotation angle of the rectangle. During inference, the two are combined to obtain a complete prediction result of the position information. The final overall loss function is as follows:
[0053] Loss=λ1L cls +λ2L obj +λ3L loc +λ4L theta (twenty one)
[0054] To balance the loss at different scales of confidence, different weights need to be assigned to the prediction feature layers for predicting large, medium, and small targets. Based on the characteristics of the insulator dataset, the prediction feature layer for predicting small targets uses a weight of 3.0, the prediction feature layer for predicting medium targets uses a weight of 1.0, and the prediction feature layer for predicting large targets uses a weight of 0.5.
[0055] Step 7: Use the training set and validation set obtained in Step 1 to train and validate the insulator fault detection model, quantize the trained model and deploy it to the UAV.
[0056] Beneficial technical effects of the present invention:
[0057] Compared to traditional manual inspection methods for transmission line insulators, this invention not only ensures the safety of inspection personnel but also significantly improves inspection efficiency. Compared to other deep learning-based target detection algorithms, the biggest advantage of this invention is its ability to meet the real-time requirements of mobile deployment. Other common insulator target detection algorithms, due to limitations in computing power and storage, mostly involve first acquiring insulator images and then uploading the collected image set to a computer containing the insulator detection algorithm for testing. Because the time interval between data acquisition and detection is long, detailed classification and grading of the acquired data are essential to determine which region, line, and insulator each image corresponds to. This process is cumbersome and time-consuming, failing to meet practical industrial needs. The improved YOLOv5 algorithm proposed in this invention has fewer parameters, less memory, and less floating-point computation, meeting the real-time requirements of mobile deployment. By deploying it on a mobile device and transmitting insulator images or videos captured during drone inspections, on-the-spot inspection of insulators can be achieved.
[0058] This invention utilizes a rotating rectangular bounding box three-label annotation method for insulator spontaneous explosion faults, which helps constrain the network's training direction, reduce convergence time, and improve detection performance. The proposed GhostConv and C3Ghost modules significantly reduce the model's network parameters, achieving algorithm lightweighting, while the Hardswish function enhances the model's numerical stability. The SE attention mechanism learns the importance of each channel, and weight allocation suppresses attention to feature channels unimportant to the current task, thereby improving the algorithm's accuracy. Improvements to the loss function enhance the algorithm's predicted bounding box regression performance. Attached Figure Description
[0059] Figure 1 This is a flowchart of the training process for the insulator fault detection model proposed in this invention.
[0060] Figure 2 This is a flowchart illustrating the workflow of the insulator fault detection model proposed in this invention.
[0061] Figure 3 This is a structural diagram of the insulator fault detection model proposed in this invention;
[0062] Figure 4 This is a schematic diagram of the three categories of sample labels proposed in this invention;
[0063] Figure 5 This is a structural diagram of the GhostConv module proposed in this invention;
[0064] Figure 6This is a structural diagram of the C3Ghost module proposed in this invention. Detailed Implementation
[0065] The present invention will be further described below with reference to the accompanying drawings and embodiments;
[0066] Most existing object detection algorithms focus on improving the detection performance of the model. Sufficient storage space and computing power are needed to support the operation of these algorithms. However, mobile GPUs, in order to meet users' needs for battery life and heat dissipation, have significantly reduced performance compared to PC GPUs; PC GPUs have computing power that is tens of times greater than mobile GPUs. Directly porting algorithms to mobile devices cannot meet the requirements of real-time detection. Therefore, most current drone-based insulator inspection solutions for power transmission lines first collect images of the insulators, and then upload the collected image set to a computer containing an insulator detection algorithm for inspection. This process is very cumbersome. In reality, most drones now support real-time transmission of images or videos captured by the drone to mobile phones. Therefore, this invention improves the YOLOv5 algorithm, making it have fewer parameters, less memory, and less floating-point computation. This improved lightweight YOLOv5 algorithm can then be deployed on mobile phones, enabling instant insulator inspection by image capture, which is more in line with the needs of practical engineering. YOLOv5 was chosen as the basis function because it has advantages over other existing object detection algorithms, such as a relatively simple model structure, fast model inference speed, low environmental requirements, and mature model deployment, which better meets the needs of practical engineering.
[0067] Figure 1 The flowchart for training the insulator fault detection model proposed in this invention mainly includes the following:
[0068] Step 1: Obtain insulator images containing faulty insulators, and design a multi-label rotating rectangle annotation method for insulator images to adapt to the data characteristics of long strip-shaped detection targets with arbitrary angles in aerial insulator images and the detection rate of fault samples; divide the labeled insulator images into training, testing, and validation sets according to proportions to train, test, and validate the insulator fault detection model. The insulator fault detection model is obtained in subsequent steps based on the improvement of the YOLOv5 model.
[0069] First, collect a sufficient number of insulator data images containing rich features using drones. At least 10,000 data images should be collected, and the number of tags should not be less than 40,000. If there are not enough insulator fault images, the number of fault samples can be expanded through data augmentation.
[0070] After collecting the data images, the dataset was labeled using the open-source project RoLabelImg on GitHub, resulting in labels for normal insulators ("normal"), damaged insulators ("damaged"), and faulty insulators ("faulty"). The data images and labels were then stored in separate folders according to a training set:validation set:test set ratio of 7:2:1. The purpose of using rotated bounding boxes for labeling is that insulator images captured by drones exhibit large scale variations, different angles, and the insulators themselves are mostly elongated. Compared to regular bounding boxes, rotated bounding boxes better fit the insulators, reducing background information within the labeled boxes. More precise labeling provides less redundant information for network training, which is more beneficial for constraining the network's training direction and reducing convergence time.
[0071] In addition, when targets are too closely packed, precise labeling can prevent some correct bounding boxes from being mistakenly identified by non-maximum suppression, thereby improving the model's detection accuracy. Compared to other labeling methods for insulator surface defects, the multi-label labeling method is advantageous because insulator self-explosion faults all occur on the insulator itself. The improved algorithm's superior feature extraction capabilities and added attention mechanism make its detection performance for faulty insulator labels containing fault locations not only better than labels for fault locations but also better than labels for normal insulators. Therefore, treating faulty insulators as a separate category of labels is valuable.
[0072] Step 2: Using the YOLOv5 model as the base model, an improved GhostConv module is introduced into the backbone network. The network parameters of the model are greatly reduced by using ghost convolution, and the numerical stability of the model is enhanced by using the Hardswish function.
[0073] Step 3: Design an improved C3Ghost structure. Through lightweight network design, the number of parameters and floating-point calculations of the original YOLOv5 model are reduced. Compared with the original YOLOv5 model, the detection efficiency can be greatly improved at the cost of a small loss of accuracy, which is more conducive to the deployment of UAVs.
[0074] Step 4: Introduce an SE attention mechanism between the feature extraction and feature fusion modules. Use the SE attention mechanism to learn the importance of each channel and suppress the attention to feature channels that are not important to the current task by assigning weights.
[0075] Step 5: By introducing CSL to replace regression with classification, the problem of sudden increase in loss caused by boundary issues when calculating rotation angle loss is solved;
[0076] Step 6: Propose a high-precision rotated frame regression loss function R-EIoU Loss to accelerate the model training process and improve the model's detection accuracy; After steps 2-6, the final insulator fault detection model is obtained.
[0077] Since the annotation method used in step 1 is the rotating rectangle annotation method, it is only necessary to add a rotation angle prediction channel to the Head structure of the algorithm to achieve the prediction of the rotating box. When performing loss calculation, in addition to the position loss function, category loss function and confidence loss function, an additional angle loss function needs to be added. The improvement of the loss function accelerates the training process of the model and improves the detection accuracy of the model.
[0078] YOLOv5 uses the binary cross-entropy loss function for both its class loss and confidence loss. The class loss only calculates the loss for positive samples, while the confidence loss calculates the loss for all samples. Both types of losses are relatively simple to calculate and perform well. A good bounding box regression function should comprehensively consider aspect ratio, overlap area, and the distance between the center points of the bounding box and the predicted bounding box. YOLOv5 uses the CIOU loss function for its location loss.
[0079] Step 7: Use the training set and validation set obtained in Step 1 to train and validate the insulator fault detection model, quantize the trained model and deploy it to the UAV.
[0080] To meet the deployment requirements of mobile devices, the trained model with high floating-point numbers is approximated as a low-bit-width neural network using tflite quantization. The official float16 quantization of tflite is adopted. This quantization method has both input and output float32, which can reduce the size by 50% and retain the model accuracy as much as possible while reducing the model size.
[0081] Then, the quantized model is deployed to the drone or its supporting application using Android Studio.
[0082] There are many common object detection algorithms available. While the YOLO series doesn't offer the highest accuracy, its model structure is relatively simple, its inference speed is fast, its environmental requirements are low, and its deployment is mature, supporting OpenVINO, TensorRT, and ncnn deployments, which is beneficial for future practical applications. We will use YOLOv5, which offers the best overall performance, as the base model. To configure the environment, open cmd, create a Python 3.8 virtual environment, install the GPU version of PyTorch from the PyTorch website, and finally install the required modules listed in the requirements.txt file of the algorithm package.
[0083] Open the algorithm package in PyCharm, select the environment you configured earlier, and choose the newyolov5s.yaml file in the models folder as the model configuration file for training, and the data location and class configuration file in the data folder as the data.yaml file.
[0084] In the PyCharm terminal, type `python train.py --data data.yaml --cfgnewyolov5s.yaml --weights pretrained / yolov5s.pt --epoch 200 --batch-size 8` to start training the model. The trained model will be automatically saved in the `train` folder under the `runs` folder.
[0085] The generated exp folder after training will contain 20 evaluation files, including a weights folder. The weights folder contains the best-performing model during training and the final trained model. Select the best-performing best.pt file as the final trained model.
[0086] Move the best.pt file to the root directory. Save the insulator images or video files to be detected to the images folder under the data folder. Use the detect.py file to test the model. Enter python detect.py --weights best.pt --source data / images in the terminal to perform the detection. Finally, you can view the detection results in the detect folder under the runs folder.
[0087] Figure 2 The flowchart illustrates the workflow of the insulator fault detection model proposed in this invention. First, the model is quantized and exported using float16 quantization. This quantization method reduces the model size by 50%, preserving model accuracy as much as possible while minimizing size. Next, Python reads the quantized model file into memory, retrieving input and output parameter details, primarily including the quantization parameters scale and zero-point. Finally, the quantized tflite file is deployed to Android Studio to generate an app, enabling real-time detection of images or videos transmitted by a drone.
[0088] Figure 3The diagram shows the structure of the insulator fault detection model proposed in this invention. The improved YOLOv5 network structure consists of four parts: Input, Backbone, Neck (multi-scale feature fusion module), and Prediction. The input side primarily performs data augmentation, adaptive anchor box calculation, and adaptive image scaling. The most important augmentation method is Mosaic data augmentation. Mosaic data augmentation essentially involves randomly stitching together four images. This method effectively increases the number of small samples, thereby improving the model's training performance. Other data augmentation methods in the algorithm include hue, saturation, exposure, rotation, translation, scaling, shearing, flipping, and perspective, which can be used as needed during actual training.
[0089] For different datasets, initial anchor boxes with defined widths and heights are used. Predicted bounding boxes are output based on these initial anchor boxes, then compared with the ground truth bounding boxes. The difference between the two is calculated, and the network parameters are updated iteratively. Adaptive image processing primarily utilizes the receptive field by performing proportional transformations of the rectangle's width and height. The backbone network is mainly used for aggregation, extracting feature maps at three scales. It employs a new CSP (Cross-Stage Partial Network) structure, but it's worth noting that the BottleNeck structure of the backbone network and the multi-scale fusion module is different. Furthermore, the initial Focus module of the model is replaced with a 6x6 convolutional layer. While theoretically equivalent, the latter is more efficient on some current GPU devices.
[0090] The multi-scale fusion module is primarily used for feature fusion, replacing the previous SPP with SPPF. Its purpose remains the same: to expand the receptive field. Compared to the SPP structure, SPPF is more efficient, improving the model's computational speed. It's worth noting that SPPF doesn't use 5x5, 9x9, or 13x13 MaxPool layers for fusion; instead, it uses multiple 5x5 MaxPool layers sequentially. Two sequential 5x5 MaxPool layers are equivalent to one 9x9 MaxPool layer, and three sequential 5x5 MaxPool layers are equivalent to one 13x13 MaxPool layer. This structure makes the algorithm's training more efficient. The multi-scale fusion module's role is to fuse features at different scales. The PAN (Pyramid Attention Network) structure of the multi-scale feature fusion module incorporates CSP to reduce the network's computational load and memory usage, while maintaining or slightly improving the network's capabilities.
[0091] The algorithm's output includes loss calculation and NMS (Non-Maximum Suppression). The loss calculation includes Classes loss, Objectness loss, Location loss, and theta loss. Classes loss and Confidence loss use BCE (Binary Cross-Entropy) Loss, while Location loss uses the EIOU loss function, which considers the overlap area between the predicted and ground truth bounding boxes, the distance between their centers, and the aspect ratio. The Angle loss uses Smooth L1 Loss. Non-Maximum Suppression is used to filter object boxes, predict bounding boxes based on image features, and predict the class.
[0092] Figure 4 This diagram illustrates the three-category sample labels proposed in this invention. Labels were generated using the RoLabelImg open-source project. Each label includes the storage path of the corresponding image, the fault category, the coordinates of the center point of the label rectangle, the length and width of the rectangle, and the rotation angle of the rectangle. Precise labeling helps constrain the network's training direction, reduce convergence time, and decrease the false negative rate of dense samples. The generated labels are in XML format and cannot be directly applied to the algorithm; they must be converted to TXT format before use.
[0093] Figure 5 The diagram shows the structure of the GhostConv module proposed in this invention. The improved GhostConv module consists of Ghostmodule, Batch Normalization, and Hardswish in that order. The Ghost module divides ordinary convolution into two parts. First, a 1x1 convolution is performed, with half the number of channels as in ordinary convolution, to integrate features and generate feature condensation for the input feature layer. Then, depthwise separable convolution is performed layer by layer to generate similar feature maps of the feature condensation. Finally, the similar feature maps and the feature condensation maps are concatenated.
[0094] Figure 6 The diagram shows the structure of the C3Ghost module proposed in this invention. The improved C3Ghost module is derived from the CSP structure. First, the input data is divided into two parts by a 1x1 convolution. Then, a GhostConv module and a BottleNeck module are connected after one of the parts. The two parts are then concatenated by Concat and finally connected by a GhostConv structure. Figure 6The two modules in the middle and on the right are called BottleNeck. The BottleNeck modules in the feature extraction part and the feature fusion part are different. The BottleNeck in the feature extraction part is formed by concatenating the input with two GhostConvs, while the BottleNeck in the feature fusion part only goes through two GhostConvs.
Claims
1. A lightweight target detection method for unmanned aerial vehicle (UAV) inspection of insulators, characterized in that, Includes the following steps: Step 1: Obtain images of insulators containing faulty insulators, and design a multi-label rotating rectangle annotation method for insulator images to adapt to the data characteristics of long strip-shaped detection targets with arbitrary angles in aerial insulator images and the detection rate of fault samples; The labeled insulator images are divided into training, testing, and validation sets according to the proportions to train, test, and validate the insulator fault detection model. The insulator fault detection model is obtained by subsequent steps based on the improvement of the YOLOv5 model. Step 2: Using the YOLOv5 model as the base model, an improved GhostConv module is introduced into the backbone network. The network parameters of the model are greatly reduced by using ghost convolution, and the numerical stability of the model is enhanced by using the Hardswish function. Step 3: Design an improved C3Ghost structure. Through lightweight network design, the number of parameters and floating-point calculations of the original YOLOv5 model are reduced. Compared with the original YOLOv5 model, the detection efficiency can be greatly improved at the cost of a small loss of accuracy, which is more conducive to the deployment of UAVs. Step 4: Introduce an SE attention mechanism between the feature extraction and feature fusion modules. Use the SE attention mechanism to learn the importance of each channel and suppress the attention to feature channels that are not important to the current task by assigning weights. Step 5: By introducing CSL to replace regression with classification, the problem of sudden increase in loss caused by boundary issues when calculating rotation angle loss is solved; Step 6: Propose a high-precision rotated frame regression loss function R-EIoU Loss to accelerate the model training process and improve the model's detection accuracy; After steps 2-6, the final insulator fault detection model is obtained. Step 7: Use the training set and validation set obtained in Step 1 to train and validate the insulator fault detection model, quantize the trained model and deploy it to the UAV; Step 1 is as follows: First, images of insulators containing faulty insulators are collected using drone aerial photography, and the collected insulator images should include most common insulator types. Since insulator fault image samples are difficult to collect, the fault samples are expanded using data augmentation methods, including vertical flipping, horizontal flipping, cropping, erasing pixel blocks, adjusting hue, adjusting saturation, adjusting exposure, and scaling. Insulator image data was labeled using the RoLabelImg image annotation tool. In addition to the specific fault type, two extra labels were set: "normel" for normal insulators and "damaged" for faulty insulators. The generated label format was an XML file. This format cannot be directly used for model training. Therefore, an eight-label representation was used, and the data was converted to TXT format via the open-source project RoLabelImg_Transform. The generated data format is (x1, y1, x2, y2, x3, y3, x4, y4), where x and y are the coordinates of the four vertices of the labeled rectangle. To reduce the number of parameters during training, the labels were further converted to the long-side representation in a five-parameter representation. The specific conversion formula is as follows: x1=xw / 2*cos(θ)-h / 2*sin(θ) (1) y1=yh / 2*cos(θ)-w / 2*sin(θ) (2) x2=xw / 2*cos(θ)+h / 2*sin(θ) (3) y2=yh / 2*cos(θ)+w / 2*sin(θ) (4) x3=x+w / 2*cos(θ)-h / 2*sin(θ) (5) y3=y+h / 2*cos(θ)-w / 2*sin(θ) (6) x4=x+w / 2*cos(θ)+h / 2*sin(θ) (7) y4=y+h / 2*cos(θ)+w / 2*sin(θ) (8) The converted coordinate format is (x,y,w,h,θ), where x and y are the coordinates of the center point of the rectangle, w is the length of the longest side, the adjacent side of w is defined as h, and θ represents the range of angles traversed by the x-axis to w, θϵ[-π / 2, π / 2). The long-side representation has fewer parameters than the eight-parameter representation, which ensures the training speed of the model. Furthermore, it is more similar to the labeling method of the rectangular box, making it easier to make structural adaptive improvements to the original YOLOv5 model.
2. The lightweight target detection method for insulator inspection by unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The improved GhostConv module described in step 2 consists of a batch normalized Ghost module, Batch Normalization, and the Hardswish activation function in sequence. The difference between the Ghost module and ordinary convolution is that the Ghost convolution divides ordinary convolution into two parts. First, a 1x1 convolution is performed with half the number of channels as in ordinary convolution to integrate features and generate feature condensation of the input feature layer. Then, a depthwise separable convolution, i.e., layer-by-layer convolution, is performed to generate similar feature maps of feature condensation. The similar feature maps and feature condensation maps are concatenated to obtain the output feature map with the same structure as ordinary convolution, but its computational cost is only half that of ordinary convolution. Batch normalization not only speeds up model convergence but also alleviates gradient vanishing to some extent. Compared to the SiLU function used in the original YOLOv5 model, Hardswish has the advantages of good numerical stability and fast computation. The specific process is as follows: The calculations performed at the convolution point are as follows: （9） in The j-th convolutional kernel of the l-th layer represents the i-th convolutional kernel. ‘ Each weight, Let W represent the j-th convolutional local region in the l-th layer, where W is the width of the convolution kernel. This represents the j'-th weight of the j-th convolutional region in the l-th layer. This represents the output value in the i-th row and j-th column of the l-th layer; The calculations performed by the batch normalization department are as follows: （10） （11） （12） （13） Where m represents the number of channel weights in the batch dimension, y i μ represents the i-th weight on this channel. B This represents the corresponding mean. Indicates the corresponding variance. and For scaling and biasing of the BN layer, Y i For intermediate variables in the batch normalization process. For the output of the BN layer, A constant term to ensure numerical stability; The formula for calculating the Hardswish function is as follows: 。 (14) 3. A lightweight target detection method for insulator inspection by unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The improved C3Ghost module described in step 3 is derived from the CSP structure. First, the input data flows into two channel branches. One branch flows through a GhostConv module, where the convolutional kernel size of the convolutional layer is 1x1, the stride is 1, and the bias is 0. The other part flows sequentially through a GhostConv module and n BottleNeck modules. The size of n depends on the position of the C3Ghost module; the deeper the GhostConv module at the feature extraction point, the larger n becomes. At the feature fusion point, n is always 3 layers. Then, these two parts are concatenated using Concat, and finally, a 1x1 GhostConv structure with a stride of 1 and a bias of 0 is connected. Furthermore, the proposed GhostConv module differs in the BottleNeck modules of the algorithm's feature extraction and feature fusion parts. The BottleNeck in the feature extraction part is formed by concatenating the input data through two GhostConv modules, while the BottleNeck in the feature fusion part only goes through two GhostConv modules.
4. A lightweight target detection method for insulator inspection by unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, The SE attention mechanism introduced in step 4 first performs global average pooling to make the feature map size 1x1 while keeping the number of channels unchanged. Then, it sequentially connects a fully connected layer, a ReLU activation function, another fully connected layer, and a sigmoid activation function. Finally, it performs matrix multiplication with the number of channels learned by the sigmoid activation function and the original input feature map.
5. A lightweight target detection method for insulator inspection by unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, Step 5 introduces circular smooth labeling (CSL) to replace regression with classification, avoiding the angular periodicity problem faced by angular regression prediction, thus obtaining more robust prediction results; the expression of CSL is as follows: （15） g(x) should simultaneously possess the characteristics of periodicity, symmetry, monotonicity, and a maximum value of 1. Therefore, we choose g(x) as a trigonometric function.
6. A lightweight target detection method for insulator inspection by unmanned aerial vehicles (UAVs) according to claim 1, characterized in that, Step 6 proposes a high-precision rotated frame regression loss function, R-EIoU Loss, which is as follows: Since the annotation method used in step 1 is the rotating rectangle annotation method, it is only necessary to add a rotation angle prediction channel to the Head structure of the algorithm to achieve the prediction of the rotating box. When calculating the loss, in addition to the position loss function, category loss function and confidence loss function, an additional angle loss function needs to be added. The improvement of the loss function accelerates the training process of the model and improves the detection accuracy of the model. YOLOv5 uses the binary cross-entropy loss function for both class loss and confidence loss. The class loss only calculates the loss for positive samples, while the confidence loss calculates the loss for all samples. YOLOv5 uses the CIOU loss function for location loss; the formula for this loss function is as follows: （16） （17） （18） （19） In the above formula, IOU is the intersection-union ratio of the predicted bounding box and the ground truth bounding box. IOU is obtained by dividing the intersection of the predicted bounding box and the ground truth bounding box by the union of the predicted bounding box and the ground truth bounding box. It is the ratio of the square of the distance between the center points of the ground truth bounding box and the center point of the predicted bounding box to the square of the length of the diagonal of the smallest bounding box of the two. In this context, v is the normalized value of the difference between the aspect ratio of the predicted box and the ground truth box, and α is the balance factor that weighs the loss caused by the aspect ratio and the IOU. While CIOU, building upon DIOU, considers the aspect ratio by adding a penalty term for both predicted and ground truth boxes, it still has two problems. First, CIOU uses a relative aspect ratio; once the aspect ratios of the predicted and ground truth boxes are linear, the penalty term added by CIOU for this relative ratio no longer works. Second, the width and height of the predicted box cannot be scaled simultaneously; as the gradient formula shows, increasing one parameter will necessarily decrease the other. To address these issues, EIOU proposes a loss function that directly penalizes the predicted values of height and width. The EIOU calculation formula is as follows: （20） Although EIOU is also composed of aspect ratio, overlap area and center point distance, it directly uses the side length as a penalty term when calculating the aspect ratio, which can address the two problems in CIOU and accelerate the convergence of the model. During training, EIoU Loss is used to optimize the position and shape of the rotated rectangle, while Smooth L1 Loss is used to optimize the rotation angle of the rectangle. During inference, the two are combined to obtain a complete prediction result of the position information. The final overall loss function is as follows: （21） To balance the loss at different scales of confidence loss, different weights need to be assigned to the prediction feature layers for predicting large, medium, and small targets. Based on the characteristics of the insulator dataset, the prediction feature layer for predicting small targets uses a weight of 3.0, the prediction feature layer for predicting medium targets uses a weight of 1.0, and the prediction feature layer for predicting large targets uses a weight of 0.5.