YOLO-based foreign body detection method, device, and drone for UAV power inspection

Abstract

The present invention belongs to the field of image analysis, and specifically relates to a method, device and drone for detecting foreign objects in UAV power inspection based on YOLO. The steps of the method include: S1: constructing a foreign object detection network based on YOLO, the network including an input layer, a backbone network, a feature fusion network and a prediction head. S2: obtaining a large number of real power inspection images collected by UAVs, and manually adding target selection boxes and category labeling information to the foreign objects in the images as sample images. S3: dividing the data set containing a large number of sample images into a training set and a test set, and using them to train and test the foreign object detection network. S4: retaining the model parameters of the network that meets the requirements after testing; and using them to perform real-time processing on the real-time image preprocessing collected during the UAV line inspection process to identify the foreign objects contained therein. The present invention solves the problem that the existing target detection algorithm is difficult to apply to the foreign object detection task in power inspection images.

Description

Technical Field

[0001] The present invention belongs to the field of image analysis, and specifically relates to a YOLO-based foreign object detection method, device, and drone for UAV power inspection. Background Art

[0002] Electricity is the foundation of modern society, providing a solid power guarantee for the normal operation of all industries. As a key link in power transmission, the power grid shoulders the important task of power delivery. With the construction of the power grid, the scale of transmission lines is constantly expanding, and the difficulty of maintenance and inspection is also increasing. Foreign objects such as bird nests, kites, and plastic bags commonly found on transmission lines have a high probability of causing short circuits, power outages, tripping, and even fires, posing serious safety hazards to people's lives and property. Therefore, foreign object detection on transmission lines is particularly important. However, transmission lines and transmission towers are distributed over a vast area. Traditional manual inspection methods are time-consuming and labor-intensive, and it is difficult to maintain high inspection efficiency for a long time. This is especially true for transmission lines located in mountainous areas, where inspection is often difficult.

[0003] Thanks to the rapid development of deep learning target detection technology and the significant advantages of drones, such as low cost, high flexibility, and small size, drones can be combined with target detection technology to conduct transmission line inspections, quickly reach the airspace above the transmission lines, and locate and identify foreign objects, providing key location and category information for subsequent foreign object cleaning and transmission line maintenance, saving a lot of manpower and material resources. When drones perform line inspection tasks, they usually need to shoot inspection images of transmission lines and towers from a high altitude. Therefore, drone power inspection images have the following characteristics: (1) A high proportion of small foreign objects. UAV power inspection images are captured by drones at high altitudes. Foreign objects occupy fewer pixels and show the characteristics of small objects. At the same time, the proportion of small foreign objects is high, which is prone to missed detection and false detection; (2) The foreign object target area is dense. Foreign objects in drone power inspection images are often concentrated in some areas of transmission lines and towers, and there are problems such as target occlusion and overlap that make it difficult to identify features; (3) The scale of foreign objects varies greatly. The shooting angles of drones during aerial inspections are different and the heights vary greatly, resulting in drastic changes in the scale of foreign objects in power inspection images.

[0004] However, existing target detection algorithms are typically designed for objects of conventional size on the ground and are unable to accurately identify small targets in drone-based power inspection images. Furthermore, target detection algorithms based on convolutional neural networks have numerous layers, each containing a large number of parameters, which places significant demands on the hardware resources required for actual drone deployment. Therefore, optimizing the design of target detection algorithms for power inspection images based on the characteristics of drone imaging, improving the detection accuracy of these algorithms while reducing hardware resource consumption, has become a pressing technical challenge in the practical application of drone-based foreign object detection in power inspections. Summary of the Invention

[0005] In order to solve the problem that existing target detection algorithms are difficult to apply to foreign object detection tasks in power inspection images, the present invention provides a YOLO-based UAV power inspection foreign object detection method, device and UAV.

[0006] The present invention is achieved by adopting the following technical solutions:

[0007] A YOLO-based foreign object detection method for UAV power inspection includes the following steps:

[0008] S1: Build an improved foreign object detection network based on YOLO. The foreign object detection network includes the input layer, backbone network, feature fusion network and prediction head. The core architecture of the foreign object detection network is as follows:

[0009] (1) The backbone network is used to perform feature extraction by downsampling the original image input to the input layer 4 times, 8 times, 16 times, and 32 times in sequence, and then obtain feature maps F2, F3, F4, and F5 with gradually reduced scales.

[0010] (2) The feature fusion network includes an upsampling channel, a downsampling channel, and a jump connection branch. Among them, F5, F4, and F3 are sequentially input into the upsampling channel and finally concatenated with F2. The feature maps of F3 and F4 processed by the CBS module are jump-connected to the corresponding layer of the downsampling channel. The downsampling channel concatenates and downsamples the feature maps from different input sources, and then outputs fused features of different scales, FP2, FP3, and FP4, through the three-level efficient layer aggregation network ELAN-Ⅱ.

[0011] (3) The prediction head includes three regression channels consisting of a Rep module and a Conv module. The three modules are used to process and predict the fusion features FP2, FP3 and FP4 of different scales output by the downsampling channel, and then obtain the bounding box regression results and categories of the detected targets.

[0012] S2: Obtain a large number of real power inspection images captured by drones, and manually add target selection boxes and category labeling information to the foreign objects in the collected power inspection images as sample images.

[0013] S3: The dataset containing a large number of sample images is divided into a training set and a test set, and used to train and test the foreign object detection network.

[0014] In the present invention, the loss function L used in the training phase is as follows:

[0015]

[0016] Among them, L SIoU represents the positioning loss, L obj represents the target confidence loss, L cla represents the classification loss; α, β and γ are the weights of the three types of losses, α+β+γ=1; IoU is the intersection over union ratio of the predicted box and the real box, L Dis is the distance loss, L Sp is the shape loss.

[0017] S4: Retain the model parameters of the trained foreign object detection network that meets the accuracy requirements; pre-process the real-time images collected by the drone during the line patrol to the required size and input them into the trained foreign object detection network to detect and locate foreign objects contained in the real-time images.

[0018] As a further improvement of the present invention, in the feature fusion network, the upsampling channel consists of 1 SPPCSPC module, 6 CBS modules, 3 upsampling modules UpSample, and 2 ELAN-Ⅱ modules.

[0019] Among them, F5 is processed by the SPPCSPC module, the CBS module, and the upsampling module UpSample, and then the feature map of F4 processed by the CBS module is first stitched together. The result of the first stitching is then processed by the ELAN-Ⅱ module, the CBS module, and the upsampling module UpSample, and then the feature map of F3 processed by the CBS module is second stitched together. The result of the second stitching is then processed by the ELAN-Ⅱ module, the CBS module, and the upsampling module UpSample, and then the feature map of F2 processed by the CBS module is third stitched together.

[0020] As a further improvement of the present invention, in the feature fusion network, the downsampling channel is composed of 3 ELAN-Ⅱ modules, 2 upsampling modules DownSample, and 2 MP modules.

[0021] The final output of the downsampling channel is processed sequentially by the ELAN-II module, the MP module, and the DownSample module. It is then concatenated with the output of the next ELAN-II module in the upsampling channel and the feature map of F3 processed by the CBS module via a skip connection. The fourth concatenation result is then processed sequentially by the ELAN-II module, the MP module, and the DownSample module. It is then concatenated with the output of the previous ELAN-II module in the upsampling channel and the feature map of F2 processed by the CBS module via a skip connection. The fifth concatenation result is finally processed by the ELAN-II module. In the downsampling channel, the three ELAN-II modules, from front to back, output the desired fused features, FP2, FP3, and FP4, respectively.

[0022] As a further improvement of the present invention, the skip connection branch in the feature fusion network realizes cross-layer fusion of features, and the expressions of the obtained weighted features O3 and O4 are as follows:

[0023]

[0024] In the above formula, I0, I1 and I2 represent the nodes of the skip connection part in level 0, level 1 and 2 respectively; and They represent the learnable weight parameters of the specified nodes in the corresponding levels, where the superscript represents the node number and the subscript is the level number; λ is a very small constant.

[0025] As a further improvement of the present invention, the backbone network sequentially includes: 4 CBS modules, 1 ELAN-I module, and three sets of alternating MP modules and ELAN-I modules. The four ELAN-I modules output feature maps F2, F3, F4, and F5 respectively.

[0026] As a further improvement of the present invention, the CBS module is composed of Conv convolution, batch normalization BN and SiLU activation function connected in series.

[0027] The MP module consists of one maximum pooling layer MaxPool and three CBS modules. The features input to the MP module are divided into two paths. One path is processed by the maximum pooling layer MaxPool and the CBS module in sequence, and the other path is processed by two CBS modules in sequence. Finally, the features of the processing results of the two paths are spliced ​​and output.

[0028] As a further improvement of the present invention, in step S3, the target confidence loss L in the loss function L is obj The binary cross entropy loss BCE is used, and its expression is as follows:

[0029]

[0030] In the above formula, y represents the true value, Indicates the prediction result;

[0031] Classification loss L cla The multi-label classification loss BWL is used, and its expression is as follows:

[0032]

[0033] In the above formula, N represents the total number of target categories, x i represents the true category of the i-th target; y i Represents the category prediction result of the i-th category target.

[0034] As a further improvement of the present invention, the positioning loss L in step S3 SIoU In the figure, the shape loss L Sp and distance loss L Dis The calculation formula is as follows:

[0035]

[0036] In the above formula, w and w GT Represent the width of the predicted box and the real box respectively; h and h GT Represent the height of the predicted box and the real box respectively; θ represents the preset weight value; C h and C w C is the height and width of a rectangle constructed by connecting the center points of the ground truth box and the predicted box and using the diagonals; y and C x are the height and width of the minimum bounding rectangle of the overall area corresponding to the two boxes; L ang is the angle loss, and σ represents the distance between the center points of the real box and the predicted box.

[0037] The present invention also includes a foreign object detection device based on power inspection images, comprising a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it creates a foreign object detection network trained as described in the aforementioned YOLO-based foreign object detection method for drone power inspections. The foreign object detection network is used to identify, locate, and classify foreign objects contained in input power inspection images.

[0038] The present invention also includes a power line inspection drone, which includes a body, a camera, and a data processing module. The body is a drone with positioning and height measurement capabilities. The camera is mounted on the body and is used to capture corresponding power line inspection images during the drone's power line inspection mission.

[0039] The data processing module synchronously receives the geographic information and height measurement information recorded by the machine body, as well as the power line inspection images captured by the camera. The data processing module includes a foreign object detection device based on the power line inspection images, as described above, and is used to identify, locate, and classify foreign objects contained in the input power line inspection images. The data processing module is used to package the foreign object information detected and identified based on the power line inspection images with the geographic information and height measurement information corresponding to the power line inspection images, and transmit the packaged information to the master station server.

[0040] The technical solution provided by the present invention has the following beneficial effects:

[0041] Based on the traditional YOLO algorithm model, the present invention adds a prediction head P2 applied to the shallow feature map FP2 to improve the solution's feature extraction capability for small foreign objects; and removes the prediction head P5 for the deep feature map FM5 to reduce the impact of deep features on the recognition of small foreign objects. In the feature fusion structure optimization part, the present invention uses the new EfficientDet to implement cross-level weight connections in the structure to improve the feature fusion effect of the network, thereby improving the model recognition and positioning accuracy. In the loss function optimization part, this embodiment adopts a loss function including SIoU, binary cross entropy loss and multi-label classification loss as a new regression loss function to increase the convergence efficiency of the network.

[0042] Compared with existing solutions, the proposed solution enhances the model's ability to recognize dense, small targets that often appear in power line inspection missions, significantly improving the solution's recognition accuracy and robustness. Furthermore, by reducing the deep feature map FM5, the proposed solution reduces the model's scale and parameters. This improved model is more suitable for use in devices with limited hardware computing power, such as drones. It improves the real-time detection of foreign objects in drone power line inspection images and reduces the network bandwidth requirements for data transmission, making it more practical and promising a broader market prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flowchart of the steps of the YOLO-based foreign object detection method for UAV power inspection provided in Example 1 of the present invention.

[0044] Figure 2 This is a model architecture diagram of the improved foreign object detection network constructed in Example 1 of the present invention.

[0045] Figure 3 This is a schematic diagram of three different types of CBS modules used in the foreign object detection network of Example 1 of the present invention.

[0046] Figure 4 This is a schematic diagram of the MP module used in the foreign object detection network of Example 1 of the present invention.

[0047] Figure 5 This is a schematic diagram of the ELAN-I module used in the foreign object detection network of Example 1 of the present invention.

[0048] Figure 6 This is a schematic diagram of the SPPCSPC module used in the foreign object detection network of Example 1 of the present invention.

[0049] Figure 7 This is a schematic diagram of the ELAN-Ⅱ module used in the foreign object detection network of Example 1 of the present invention.

[0050] Figure 8 This is a schematic diagram of the skip connection branches used in the feature fusion network of Example 1 of the present invention.

[0051] Figure 9 This is a schematic diagram of the Rep module used in the prediction head of Example 1 of the present invention.

[0052] Figure 10 This is a diagram showing the working principles of two line patrol drones with different working modes in Example 3 of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0054] Example 1

[0055] This embodiment provides a YOLO-based foreign object detection method for drone-based power line inspections. This method utilizes a trained network model to identify, locate, and classify objects in real-time power line inspection images. The network model in this embodiment is an improvement on the classic YOLO algorithm. This new approach enhances the model's detection performance for dense, small targets often encountered in power line inspections, while reducing the model's size and parameters, enabling its application in low-computing-power embedded devices such as drones.

[0056] Specifically, compared with the traditional YOLO algorithm model, this embodiment adds a prediction head P2 applied to shallow features to improve the solution's feature extraction capability for small foreign object targets; and removes the prediction head P5 for deep features to reduce the impact of deep features on the recognition of small foreign object targets. In the feature fusion structure optimization part, this embodiment uses the new EfficientDet to implement cross-level weight connections in the structure to improve the feature fusion effect of the network, thereby improving the model recognition and positioning accuracy. In the loss function optimization part, this embodiment uses a loss function including SIoU, binary cross entropy loss and multi-label classification loss as a new regression loss function to increase the convergence efficiency of the network. Compared with the original YOLO network, the improved model volume is effectively reduced, which is more conducive to the actual deployment of drone hardware, and the detection accuracy and real-time performance of foreign object targets in drone power inspection images are significantly improved.

[0057] Detailed, such as Figure 1 As shown, this embodiment provides a YOLO-based UAV power inspection foreign object detection method including the following steps:

[0058] S1: Build an improved foreign body detection network based on YOLO. Figure 2 As shown in the figure, the foreign object detection network includes an input layer, a backbone network, a feature fusion network, and a prediction head. Consistent with the traditional YOLO algorithm, the input layer in this embodiment supports input images with a size of 640×640×3. Specifically, the core architecture of the foreign object detection network is as follows:

[0059] 1. Backbone Network

[0060] Combine Figure 2 It can be found that the backbone network of this embodiment includes: 4 CBS modules, 1 ELAN-I module, and three sets of alternating MP modules and ELAN-I modules. Among them, the CBS module is an integrated module in the YOLO model. Each CBS module is composed of Conv convolution, batch normalization BN and SiLU activation function connected in series. According to the different convolution kernel k and step size s, the YOLO algorithm model provides three different CBS modules. Among them, Figure 3 As shown in the figure, k=1, s=1 means using a 1×1 convolution kernel and a CBS module with a stride of 1. This CBS module is used to change the number of channels of the feature; k=3, s=1 means using a 3×3 convolution kernel and a CBS module with a stride of 1. This CBS module is used to extract features; k=3, s=2 means using a 3×3 convolution kernel and a CBS module with a stride of 2. This CBS module is used for downsampling.

[0061] In the backbone network of this embodiment, the first and third of the four CBS modules use CBS modules with k=3 and s=1; the second and fourth use CBS modules with k=3 and s=2.

[0062] like Figure 4 As shown in the figure, the MP module consists of one MaxPool layer and three CBS modules. Two of the CBS modules are of the (k=1, s=1) type, and the other is of the (k=3, s=2) type. The features input to the MP module are divided into two paths. One path is processed sequentially by the MaxPool layer and the (k=1, s=1) type CBS module, and the other path is processed sequentially by two CBS modules, the former of the (k=1, s=1) type and the latter of the (k=3, s=2) type. Finally, the processed results of the two paths are concatenated and output.

[0063] The ELAN (Efficient QLayer Aggregation Networks) module is an important network structure in the YOLO algorithm. This module improves the accuracy and robustness of the target detection algorithm by effectively aggregating feature information from different layers. The structure of the ELAN-I module used in the backbone network of this embodiment is as follows: Figure 5 As shown, it includes seven CBS modules, three of which are (k=1, s=1) type and the other four are (k=3, s=1) type. The features entering the ELAN-I module are divided into four paths. The first path is processed by a (k=1, s=1) type CBS module; the second path is processed sequentially by a (k=1, s=1) type CBS module and two (k=3, s=1) type CBS modules; the third path is processed sequentially by a (k=1, s=1) type CBS module and four (k=3, s=1) type CBS modules; the fourth path is processed by another (k=1, s=1) type CBS module. The four paths are then concatenated and processed by a third (k=1, s=1) type CBS module before output.

[0064] In the backbone network of this embodiment, the four ELAN-I modules output feature maps F2, F3, F4, and F5, respectively. Therefore, the backbone network can downsample the original input image by 4, 8, 16, and 32 times, respectively, to extract features, thereby obtaining feature maps F2, F3, F4, and F5 at progressively smaller scales. The sizes of the four feature maps are 160×160, 80×80, 40×40, and 20×20, respectively.

[0065] 2. Feature Fusion Network

[0066] The feature fusion network provided in this embodiment includes an upsampling channel, a downsampling channel, and a jump connection branch. Among them, F5, F4, and F3 are sequentially input into the upsampling channel and finally feature-concatenated with F2. The feature maps of F3 and F4 processed by the CBS module jump to the corresponding layer of the downsampling channel. The downsampling channel performs feature concatenation and downsampling on the feature maps from different input sources, and then outputs fused features FP2, FP3, and FP4 of different scales through the three-level efficient layer aggregation network ELAN-Ⅱ.

[0067] (2.1) Upsampling channel

[0068] In the feature fusion network of this embodiment, the upsampling channel is composed of one SPPCSPC module, six CBS modules, three upsampling modules UpSample, and two ELAN-II modules.

[0069] Among them, F5 is processed by the SPPCSPC module, the CBS module, and the upsampling module UpSample, and then the feature map of F4 processed by the CBS module is first stitched together. The result of the first stitching is then processed by the ELAN-Ⅱ module, the CBS module, and the upsampling module UpSample, and then the feature map of F3 processed by the CBS module is second stitched together. The result of the second stitching is then processed by the ELAN-Ⅱ module, the CBS module, and the upsampling module UpSample, and then the feature map of F2 processed by the CBS module is third stitched together. Among them, the six CBS modules used in the upsampling channel are all (k=1, s=1) type CBS modules.

[0070] In the upsampling channel, the structure of the SPPCSPC module used in this embodiment is as follows: Figure 6 As shown. Each SPPCSPC module consists of 7 CBS modules and 3 maximum pooling layers MaxPool. Among them, the features input to the SPPCSPC module are divided into two paths. Among them, the first path is processed by the (k=1, s=1) type CBS module, and the second path is first processed by the (k=1, s=1) type CBS module, the (k=3, s=1) type CBS module and the (k=1, s=1) type CBS module in sequence, and then divided into three sub-paths. Each sub-path is processed by three pooling layers of different scales, and then spliced ​​with the original features. The splicing results are then processed by the (k=1, s=1) type CBS module and the (k=3, s=1) type CBS module respectively, and then spliced ​​with the features of one path. The splicing results are finally processed by the (k=1, s=1) type CBS module and output. Among them, the convolution kernels of the three pooling layers used in this embodiment are 5×5, 9×9 and 13×13 respectively.

[0071] (2.2) Downsampling channel

[0072] In the feature fusion network of this embodiment, the downsampling channel is composed of three ELAN-II modules, two downsampling modules DownSample, and two MP modules.

[0073] The final output of the downsampling channel is processed sequentially by the ELAN-II module, the MP module, and the upsampling module DownSample. It is then concatenated with the output of the next ELAN-II module in the upsampling channel and the feature map of F3 processed by the jump-connected CBS module for the fourth feature concatenation. The fourth concatenation result is then processed sequentially by the ELAN-II module, the MP module, and the downsampling module DownSample. It is then concatenated with the output of the previous ELAN-II module in the upsampling channel and the feature map of F2 processed by the jump-connected CBS module for the fifth feature concatenation. The fifth feature concatenation result is finally processed by the ELAN-II module. In the downsampling channel, the three ELAN-II modules, from front to back, output the desired fused features, FP2, FP3, and FP4, respectively.

[0074] In the downsampling channel, the ELAN-II module is also composed of 7 CBS modules, which are consistent with the ELAN-I module. Three of them are (k=1, s=1) type and the other four are (k=3, s=1) type. Figure 7 As shown, the features entering the ELAN-II module are divided into six paths. The first and second paths are each processed by only one (k=1, s=1) CBS module; the third path is processed sequentially by one (k=1, s=1) CBS module and one (k=3, s=1) CBS module; the fourth path is processed sequentially by one (k=1, s=1) CBS module and two (k=3, s=1) CBS modules; the fifth path is processed sequentially by one (k=1, s=1) CBS module and three (k=3, s=1) CBS modules; and the sixth path is processed sequentially by one (k=1, s=1) CBS module and four (k=3, s=1) CBS modules. The six feature maps are concatenated and then processed by one (k=3, s=1) CBS module before being output. In this embodiment, the MP module used in the downsampling stage is the same as that in the backbone network.

[0075] (2.3) Skip connection branch

[0076] Combine Figure 2It can be seen that the jump connection branch in this embodiment includes two paths, one of which is a jump connection between F3 and FP3, and the other is a jump connection between F4 and FP4. The jump connection can solve the gradient disappearance problem in the network of this embodiment and accelerate the training process. The upsampling channel and downsampling channel of this feature fusion network contain many layers. When the gradient signal is backpropagated, the gradient may become very small, making the network difficult to train. The jump connection used in this embodiment allows the gradient to be backpropagated more easily by passing the input information directly to the subsequent layers. This direct transfer helps to maintain the stability of the gradient and makes the network easier to train.

[0077] Specifically, the simplified diagram of the skip connection used in the feature fusion network of this embodiment is as follows: Figure 8 As shown. Combined Figure 8 It can be seen that the skip connection branch in the feature fusion network realizes cross-layer fusion between features, and the expressions of the obtained weighted features O3 and O4 are as follows:

[0078]

[0079] In the above formula, I0, I1 and I2 represent the nodes of the skip connection part in level 0, level 1 and 2 respectively; and They represent the learnable weight parameters of the specified nodes in the corresponding levels, where the superscript represents the node number and the subscript is the level number; λ is a very small constant.

[0080] 3. Prediction Head

[0081] The prediction head of this embodiment includes a data processing pipeline consisting of three routes, a Rep module and a Conv module. These three modules process and predict the fused features FP2, FP3, and FP4 at different scales output by the downsampling channel, thereby obtaining the bounding box regression results and categories of the detected objects. The REP module can improve the model's expressiveness and generalization capabilities by adding branches and reparameterization, while reducing the computational effort and the risk of overfitting.

[0082] Finally, after Conv convolution, the fused feature FP2 is input to the P2 prediction head of size 160×160×45, the fused feature FP3 is input to the P2 prediction head of size 80×80×45, and the fused feature FP2 is input to the P2 prediction head of size 40×40×45.

[0083] Among them Figure 9As shown in the figure, the Rep module includes two convolution modules Conv and two BN modules. The features input to the Rep module are divided into two paths, one of which is processed by a convolution module with a convolution kernel of 1 and a step size of 1 and a BN module, and the other is processed by a convolution module with a convolution kernel of 3 and a step size of 1 and a BN module; the features output by the two BN modules are spliced ​​as the output of the Rep module.

[0084] S2: Obtain a large number of real power inspection images captured by drones, and manually add target selection boxes and category labeling information to the foreign objects in the collected power inspection images as sample images.

[0085] In the practical solution of this embodiment, the collected power inspection images should be uniformly cropped to a pixel size of 640×640. In addition, to expand the number of sample images, various existing image enhancement methods can be used to amplify the original data. The amplified images and the original images are first labeled with their target selection boxes and target object categories. Image enhancement methods include geometric transformations, color space changes, and pixel operations. Geometric transformations include flipping, rotation, scaling, cropping, and translation. Geometric transformations simulate the effects of angle changes on objects in the field of view during drone image acquisition. Color space transformations include contrast changes, brightness changes, saturation changes, channel separation, grayscale processing, and histogram enhancement. Color space transformations simulate the effects of drone imaging quality under different ambient lighting conditions on images. Pixel operations include blurring and sharpening, which are designed to address the impact of different camera imaging effects on the solution. Using image enhancement methods to amplify sample data can improve the robustness of the trained network model in practical applications.

[0086] S3: Divide the dataset containing a large number of sample images into a training set and a test set, and use them to train and test the foreign object detection network. In practical applications, this embodiment divides the training set and the test set in a ratio of 6:4, and the images obtained by any image enhancement method and the original image of the same original image are both divided into the same training set or test set.

[0087] In the solution of this embodiment, the loss function L used in the training phase is as follows:

[0088]

[0089] In the above formula, L SIoU represents the positioning loss, L obj represents the target confidence loss, L cla represents the classification loss; α, β and γ are the weights of the three types of losses, α+β+γ=1; IoU is the intersection over union ratio of the predicted box and the real box, L Dis is the distance loss, LSp is the shape loss.

[0090] Among them, the target confidence loss L in the loss function L obj The binary cross entropy loss BCE is used, and its expression is as follows:

[0091]

[0092] In the above formula, y represents the true value, Indicates the prediction result;

[0093] Classification loss L cla The multi-label classification loss BWL is used, and its expression is as follows:

[0094]

[0095] In the above formula, N represents the total number of target categories, x i represents the true category of the i-th target; y i Represents the category prediction result of the i-th category target.

[0096] The positioning loss L in step S3 SIoU In the figure, the shape loss L Sp and distance loss L Dis The calculation formula is as follows:

[0097]

[0098] In the above formula, w and w GT Represent the width of the predicted box and the real box respectively; h and h GT Represent the height of the predicted box and the real box respectively; θ represents the preset weight value; C h and C w C is the height and width of a rectangle constructed by connecting the center points of the ground truth box and the predicted box and using the diagonals; y and C x are the height and width of the minimum bounding rectangle of the overall area corresponding to the two boxes; L ang is the angle loss, and σ represents the distance between the center points of the real box and the predicted box.

[0099] S4: Retain the model parameters of the trained foreign object detection network that meets the accuracy requirements; pre-process the real-time images collected by the drone during the line patrol to the required size and input them into the trained foreign object detection network to detect and locate foreign objects contained in the image.

[0100] Example 2.

[0101] Building on the solution in Example 1, this embodiment further provides a foreign object detection device based on power inspection images, comprising a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it creates a foreign object detection network trained in the YOLO-based foreign object detection method for drone power inspections in Example 1. The foreign object detection network is used to identify, locate, and classify foreign objects contained in input power inspection images.

[0102] The foreign object detection device based on power inspection images provided in this embodiment is essentially a computer device. The computer device can be an embedded computer device or a general-purpose computer device. For example, it can be a smartphone, tablet computer, laptop computer, desktop computer, rack server, blade server, tower server, or cabinet server (including a standalone server or a server cluster consisting of multiple servers) that can execute programs.

[0103] The computer device of this embodiment includes at least, but is not limited to, a memory and a processor that can be interconnected via a system bus. In this embodiment, the memory (i.e., a readable storage medium) includes flash memory, a hard disk, a multimedia card, a card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic storage, a magnetic disk, an optical disk, etc. In some embodiments, the memory can be an internal storage unit of the computer device, such as the hard disk or internal memory of the computer device. In other embodiments, the memory can also be an external storage device of the computer device, such as a plug-in hard disk equipped with the computer device, a smart media card (SMC), a secure digital (SD) card, a flash memory card, etc. Of course, the memory can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is generally used to store the operating system and various application software installed on the computer device. In addition, the memory can also be used to temporarily store various types of data that have been output or are about to be output.

[0104] In some embodiments, the processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip. The processor is generally used to control the overall operation of a computer device.

[0105] Example 3

[0106] Based on Examples 1 and 2, this embodiment further provides a line inspection drone. In actual application, this line inspection drone should have GPS positioning and altitude measurement functions. The line inspection drone can fly above the high-voltage power lines according to a specified inspection trajectory and capture power inspection images of the high-voltage power lines below. Foreign objects are then detected based on the power inspection images and reported to the backend server.

[0107] Considering that the foreign body detection solution of the power inspection image in this embodiment has relatively low requirements on hardware computing power, the real-time performance of detection and recognition is good. Figure 10 As shown, two processing methods can be used for the collected power inspection images: one is to transmit the power inspection images in real time to a server in the background control center via an onboard wireless communication module. The background server can deploy the foreign object detection network trained as in Example 1; in this case, the background server is the foreign object detection device based on power inspection images in Example 2. The background server can then identify, locate, and classify foreign objects contained in the input power inspection images, generating corresponding detection results.

[0108] In another embodiment, the drone includes a fuselage, a camera attached to its payload, and an embedded data processing module. The camera is used to capture corresponding power line inspection images during the drone's power line inspection mission and input them into the data processing module. The foreign object detection network trained in Example 1 is directly deployed in the data processing module. In this case, the embedded data processing module is the foreign object detection device based on power line inspection images in Example 2. In this state, the drone can directly identify, locate, and classify foreign objects contained in the input power line inspection images while acquiring them, and finally transmit the detection results back to the backend server.

[0109] Of the two different drone solutions provided in this embodiment, the former does not require local image data processing, but does require image data to be transmitted back to the backend. Therefore, this solution requires less computing power from the drone's processor, but relatively higher bandwidth from the communication module. The latter, on the other hand, can process the collected image data locally, so it requires relatively higher computing power from the drone's processor (still lowering hardware performance requirements compared to traditional solutions), but lower bandwidth requirements for the communication module. Both solutions can be reasonably selected based on the customer's specific needs.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A YOLO-based foreign object detection method for UAV power inspection, characterized by: It includes the following steps: S1: Build an improved foreign object detection network based on YOLO. The foreign object detection network includes an input layer, a backbone network, a feature fusion network, and a prediction head. The core architecture of the foreign object detection network is as follows: (1) The backbone network is used to perform downsampling of the original image input to the input layer by 4, 8, 16, and 32 times in sequence to achieve feature extraction, thereby obtaining feature maps F2, F3, F4, and F5 with gradually reduced scales; (2) The feature fusion network includes an upsampling channel, a downsampling channel, and a jump connection branch; wherein, F5, F4, and F3 are sequentially input into the upsampling channel and finally feature-concatenated with F2; the feature maps of F3 and F4 processed by the CBS module are jump-connected to the corresponding layer of the downsampling channel; the downsampling channel performs feature concatenation and downsampling on the feature maps of different input sources, and then outputs fusion features FP2, FP3, and FP4 of different scales through the three-level efficient layer aggregation network ELAN-Ⅱ; (3) The prediction head includes a regression channel consisting of three routes, a Rep module and a Conv module. The three modules are used to process and predict the fusion features FP2, FP3 and FP4 of different scales output by the downsampling channel, and then obtain the bounding box regression result and category of the detected target; S2: Obtain a large number of real power inspection images captured by drones. Manually add target selection boxes and category labeling information to the foreign objects in the collected power inspection images and use them as sample images. S3: The dataset containing a large number of sample images is divided into a training set and a test set, and used to train and test the foreign object detection network; the loss function L used in the training stage is as follows: Among them, L SIoU represents the positioning loss, L obj represents the target confidence loss, L cla represents the classification loss; α, β and γ are the weights of the three types of losses, α+β+γ=1; IoU is the intersection over union ratio of the predicted box and the real box, L Dis is the distance loss, L Sp is the shape loss; S4: retain the model parameters of the foreign object detection network that meets the accuracy requirements after training, and preprocess the real-time image collected by the drone during the line patrol to the required size and input it into the trained foreign object detection network to detect and locate foreign objects contained in the image.

2. The YOLO-based foreign object detection method for UAV power inspection according to claim 1, characterized in that: In the feature fusion network, the upsampling channel consists of 1 SPPCSPC module, 6 CBS modules, 3 upsampling modules UpSample, and 2 ELAN-Ⅱ modules; Among them, F5 is processed by the SPPCSPC module, the CBS module, and the upsampling module UpSample in sequence, and then the feature map of F4 processed by the CBS module is subjected to the first feature splicing; the first splicing result is processed by the ELAN-Ⅱ module, the CBS module, and the upsampling module UpSample in sequence, and then the feature map of F3 processed by the CBS module is subjected to the second feature splicing; the second splicing result is processed by the ELAN-Ⅱ module, the CBS module, and the upsampling module UpSample in sequence, and then the feature map of F2 processed by the CBS module is subjected to the third feature splicing.

3. The YOLO-based foreign object detection method for UAV power inspection according to claim 2, characterized in that: In the feature fusion network, the downsampling channel consists of three ELAN-Ⅱ modules, two downsampling modules DownSample, and two MP modules; Among them, the final output of the downsampling channel is processed by the ELAN-Ⅱ module, the MP module and the downsampling module DownSample in sequence, and then the fourth feature splicing is performed with the output of the next ELAN-Ⅱ module in the upsampling channel and the feature map of F3 processed by the jump connection and the CBS module; the fourth feature splicing result is processed by the ELAN-Ⅱ module, the MP module and the downsampling module DownSample in sequence, and then the fifth feature splicing is performed with the output of the previous ELAN-Ⅱ module in the upsampling channel and the feature map of F2 processed by the jump connection and the CBS module; the fifth feature splicing result is finally processed by the ELAN-Ⅱ module; in the downsampling channel, the three ELAN-Ⅱ modules from front to back output the required fusion features FP2, FP3 and FP4 respectively.

4. The YOLO-based foreign object detection method for UAV power inspection according to claim 3 is characterized by: The skip connection branch in the feature fusion network realizes cross-layer fusion of features, and the expressions of the obtained weighted features O3 and O4 are as follows: In the above formula, I0, I1 and I2 represent the nodes of the skip connection part in level 0, level 1 and 2 respectively; and They represent the learnable weight parameters of the specified nodes in the corresponding levels, where the superscript represents the node number and the subscript is the level number; λ is a very small constant.

5. The YOLO-based foreign object detection method for UAV power inspection as claimed in claim 3, characterized in that: The backbone network includes: 4 CBS modules, 1 ELAN-I module and three groups of alternating MP modules and ELAN-I modules; the 4 ELAN-I modules output feature maps F2, F3, F4 and F5 respectively.

6. The YOLO-based foreign object detection method for UAV power inspection according to claim 5, characterized in that: The CBS module is composed of Conv convolution, batch normalization BN and SiLU activation function connected in series; The MP module consists of one maximum pooling layer MaxPool and three CBS modules; the features input to the MP module are divided into two paths, one path is processed by the maximum pooling layer MaxPool and the CBS module in sequence, and the other path is processed by two CBS modules in sequence. Finally, the processing results of the two paths are spliced ​​and output.

7. The YOLO-based foreign object detection method for UAV power inspection according to claim 1, characterized in that: In step S3, the target confidence loss L in the loss function L is obj The binary cross entropy loss BCE is used, and its expression is as follows: In the above formula, y represents the true value, Indicates the prediction result; Classification loss L cla The multi-label classification loss BWL is used, and its expression is as follows: In the above formula, N represents the total number of target categories, x i represents the true category of the i-th target; y i Represents the category prediction result of the i-th category target.

8. The YOLO-based foreign object detection method for UAV power inspection according to claim 7, characterized in that: In the positioning loss L SIoU In the above equation, the shape loss L Sp and distance loss L Dis The calculation formula is as follows: In the above formula, w and w GT Represent the width of the predicted box and the real box respectively; h and h GT Represent the height of the predicted box and the real box respectively; θ represents the preset weight value; C h and C w C is the height and width of a rectangle constructed by connecting the center points of the ground truth box and the predicted box and using the diagonals; y and C x are the height and width of the minimum bounding rectangle of the overall area corresponding to the two boxes; L ang is the angle loss, and σ represents the distance between the center points of the real box and the predicted box.

9. A foreign body detection device based on power inspection images, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and running on the processor, and is characterized in that: when the processor executes the computer program, it creates a foreign object detection network that has been trained in the YOLO-based drone power inspection foreign object detection method as described in any one of claims 1 to 8, and the foreign object detection network is used to identify, locate and classify foreign objects contained in the input power inspection image.

10. A line patrol drone, characterized in that: It includes: The body adopts a drone with positioning and height measurement functions; A camera is mounted on the body and is used to collect corresponding power line inspection images when the UAV performs power line inspection tasks; A data processing module, which is used to synchronously receive the geographic information and height measurement information recorded by the body and the power line inspection image collected by the camera; the data processing module includes a foreign object detection device based on the power line inspection image as described in claim 9, and is used to identify, locate and classify foreign objects contained in the input power line inspection image; the data processing module is used to package the foreign object information detected and identified based on the power line inspection image with the geographic information and height measurement information corresponding to the power line inspection image and send them to the main station server.

Images