Unmanned aerial vehicle power inspection foreign matter detection method and device based on YOLO and unmanned aerial vehicle

Abstract

The invention belongs to the field of image analysis, and particularly relates to an unmanned aerial vehicle power inspection foreign matter detection method and device based on YOLO and an unmanned aerial vehicle. The method comprises the following steps: S1, constructing a foreign matter detection network based on YOLO, wherein the network comprises an input layer, a backbone network, a feature fusion network and a prediction head; s2, a large number of real electric power inspection images collected by the unmanned aerial vehicle are obtained, and a target selection box and category marking information are manually added to foreign matter existing in the images to serve as sample images; and S3, dividing the data set containing a large number of sample images into a training set and a test set, and training and testing the foreign matter detection network. S4, retaining the model parameters of the network meeting the requirements after the test; and the system is used for carrying out real-time processing on real-time images collected in the line patrol process of the unmanned aerial vehicle and identifying foreign matters contained in the real-time images. The problem that an existing target detection algorithm is difficult to be suitable for a foreign matter detection task in an electric power inspection image is solved.

Description

Technical Field

[0001] The present invention belongs to the field of image analysis, and in particular relates to a method, a device and a drone for detecting foreign objects in drone power inspection based on YOLO. Background Art

[0002] Electricity is the foundation of modern social development and provides a solid power guarantee for the normal operation of all walks of life. As a key link in power transmission, the power grid undertakes the important task of power transmission. With the construction of the power grid, the scale of transmission lines is also expanding, and the difficulty of maintenance and inspection is also escalating. Common foreign objects such as bird nests, kites, plastic bags, etc. on transmission lines are very likely to cause short circuits, power outages, tripping, and even fires in transmission lines, posing serious safety hazards to the lives and property of the people. Therefore, foreign object detection on transmission lines is particularly important. However, the distribution area of ​​transmission lines and transmission towers is vast, and the traditional manual inspection method is time-consuming and labor-intensive and it is difficult to maintain a high inspection efficiency for a long time, especially in mountainous areas. Transmission lines are often difficult to inspect.

[0003] Thanks to the rapid development of deep learning target detection technology and the significant advantages of drones, such as low cost, high flexibility and small size, drones can be combined with target detection technology to conduct transmission line inspections, quickly reach the airspace above the transmission line and locate and identify foreign objects, providing key location and category information for subsequent foreign object cleaning and transmission line maintenance, saving a lot of manpower and material resources. When drones perform line inspection tasks, they usually need to shoot inspection images of transmission lines and towers from high altitudes. Therefore, drone power inspection images have the following characteristics: (1) A high proportion of small foreign objects. Drone power inspection images are captured by drones at high altitudes. Foreign objects occupy fewer pixels and show the characteristics of small objects. At the same time, the proportion of small foreign objects is high, which is prone to missed detection and false detection; (2) The foreign object target area is dense. Foreign objects in drone power inspection images are often concentrated in some areas of transmission lines and towers, and there are problems such as target occlusion and overlap that are difficult to identify features; (3) The scale of foreign objects varies greatly. The shooting angles of drones during aerial inspections are different and the height varies greatly, resulting in dramatic changes in the scale of foreign objects in power inspection images.

[0004] However, existing target detection algorithms are usually designed for objects of regular size on the ground and cannot accurately identify UAV power inspection images that are mainly small targets. At the same time, the target detection algorithm based on convolutional neural networks has many network layers and each layer contains a large number of parameters, which places great demands on the hardware resources for the actual deployment of UAVs. Therefore, how to optimize the design of the target detection algorithm for power inspection images based on the characteristics of UAV imaging, improve the detection accuracy of the UAV power inspection foreign object target detection algorithm and reduce the consumption of hardware resources has become a technical problem that needs to be solved in the actual application field of UAV power inspection foreign object target detection. Summary of the invention

[0005] In order to solve the problem that the existing target detection algorithm is difficult to apply to the foreign object detection task in power inspection images, the present invention provides a UAV power inspection foreign object detection method, device and UAV based on YOLO.

[0006] The present invention is implemented by the following technical solutions:

[0007] A method for detecting foreign objects in UAV power inspection based on YOLO, comprising the following steps:

[0008] S1: Build an improved foreign body detection network based on YOLO. The foreign body detection network includes input layer, backbone network, feature fusion network and prediction head. The core architecture of the foreign body detection network is as follows:

[0009] (1) The backbone network is used to downsample the original image of the input layer by 4, 8, 16 and 32 times in sequence to achieve feature extraction, and then obtain feature maps F2, F3, F4 and F5 with gradually reduced scales.

[0010] (2) The feature fusion network includes upsampling channels, downsampling channels, and jump connection branches. Among them, F5, F4, and F3 are sequentially input into the upsampling channel and finally feature concatenated with F2. The feature maps of F3 and F4 processed by the CBS module are jump-connected to the corresponding layer of the downsampling channel. The downsampling channel performs feature concatenation and downsampling on the feature maps from different input sources, and then outputs fusion features FP2, FP3, and FP4 of different scales through the three efficient layer aggregation networks ELAN-Ⅱ.

[0011] (3) The prediction head includes three regression channels consisting of a Rep module and a Conv module. The three modules are used to process and predict the fused features FP2, FP3, and FP4 of different scales output by the downsampling channel, and then obtain the bounding box regression result and category of the detected target.

[0012] S2: Obtain a large number of real power inspection images collected from the perspective of drones, and manually add target selection boxes and category marking information to the foreign objects in the collected power inspection images as sample images.

[0013] S3: A dataset containing a large number of sample images is divided into a training set and a test set, and used to train and test the foreign object detection network.

[0014] In the present invention, the loss function L used in the training phase is as follows:

[0015]

[0016] Among them, L SIoU represents the positioning loss, L obj represents the target confidence loss, L cla represents the classification loss; α, β and γ are the weights of the three types of losses, α+β+γ=1; IoU is the intersection over union ratio of the predicted box and the true box, L Dis is the distance loss, L Sp is the shape loss.

[0017] S4: retain the model parameters of the foreign object detection network that meets the accuracy requirements after training; and pre-process the real-time images collected during the drone patrol process into the required size and input them into the trained foreign object detection network to detect and locate the foreign objects contained in the real-time images.

[0018] As a further improvement of the present invention, in the feature fusion network of the detection head, the upsampling channel is composed of 1 SPPCSPC module, 6 CBS modules, 3 upsampling modules UpSample, and 2 ELAN-Ⅱ modules.

[0019] Among them, F5 is processed by the SPPCSPC module, CBS module, and upsampling module UpSample in sequence, and then the feature map of F4 processed by the CBS module is spliced ​​for the first time. The result of the first splicing is then processed by the ELAN-Ⅱ module, CBS module, and upsampling module UpSample in sequence, and then the feature map of F3 processed by the CBS module is spliced ​​for the second time. The result of the second splicing is then processed by the ELAN-Ⅱ module, CBS module, and upsampling module UpSample in sequence, and then the feature map of F2 processed by the CBS module is spliced ​​for the third time.

[0020] As a further improvement of the present invention, in the feature fusion network of the detection head, the downsampling channel is composed of 3 ELAN-Ⅱ modules, 2 upsampling modules DownSample, and 2 MP modules.

[0021] Among them, the final output of the downsampling channel is processed by the ELAN-Ⅱ module, MP module and downsampling module DownSample in sequence, and then the fourth feature splicing is performed with the output of the next ELAN-Ⅱ module in the upsampling channel and the feature map of F3 processed by the CBS module of the jump connection. The fourth feature splicing result is processed by the ELAN-Ⅱ module, MP module and downsampling module DownSample in sequence, and then the fifth feature splicing is performed with the output of the previous ELAN-Ⅱ module in the upsampling channel and the feature map of F2 processed by the CBS module of the jump connection. The fifth feature splicing result is finally processed by the ELAN-Ⅱ module. In the downsampling channel, the three ELAN-Ⅱ modules from front to back output the required fusion features FP2, FP3 and FP4 respectively.

[0022] As a further improvement of the present invention, the skip connection branch in the feature fusion network realizes cross-layer fusion between features, and the obtained weighted feature O 3 and O 4 The expression is as follows:

[0023]

[0024] In the above formula, I 0 , I 1 and I 2 Represent the nodes of the skip connection part in level 0, level 1 and 2 respectively; and They respectively represent the learnable weight parameters of the specified nodes in the corresponding levels, where the superscript represents the node number and the subscript is the level number; λ is a very small constant.

[0025] As a further improvement of the present invention, the backbone network includes: 4 CBS modules, 1 ELAN-I module and three groups of alternating MP modules and ELAN-I modules. The 4 ELAN-I modules output feature graphs F2, F3, F4 and F5 respectively.

[0026] As a further improvement of the present invention, the CBS module is composed of Conv convolution, batch normalization BN and SiLU activation function connected in series in sequence.

[0027] The MP module consists of one maximum pooling layer MaxPool and three CBS modules. The features input to the MP module are divided into two paths. One path is processed by the maximum pooling layer MaxPool and the CBS module in sequence, and the other path is processed by two CBS modules in sequence. Finally, the processing results of the two paths are concatenated and output.

[0028] As a further improvement of the present invention, in step S3, the target confidence loss L in the loss function L objThe binary cross entropy loss BCE is used, and its expression is as follows:

[0029]

[0030] In the above formula, y represents the true value, Indicates the prediction result;

[0031] Classification loss L cla The multi-label classification loss BWL is used, and its expression is as follows:

[0032]

[0033] In the above formula, N represents the total number of target categories, x i represents the true category of the i-th target; y i Represents the category prediction result of the i-th target object.

[0034] As a further improvement of the present invention, the positioning loss L in step S3 SIoU In the figure, the shape loss L Sp and distance loss L Dis The calculation formula is as follows:

[0035]

[0036] In the above formula, w and w GT Represent the width of the predicted box and the real box respectively; h and h GT Represent the height of the predicted box and the real box respectively; θ represents the preset weight value; C h and C w C is the height and width of a rectangle constructed by connecting the center points of the ground truth box and the predicted box and using the diagonals; y and C x are the height and width of the minimum circumscribed rectangle of the overall area corresponding to the two boxes; L ang is the angle loss, and σ represents the distance between the center points of the real box and the predicted box.

[0037] The present invention also includes a foreign object detection device based on power inspection images, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, a foreign object detection network trained in the aforementioned UAV power inspection foreign object detection method based on YOLO is created. The foreign object detection network is used to identify, locate and classify foreign objects contained in the input power inspection image.

[0038] The present invention also includes a line inspection drone, which includes a body, a camera and a data processing module. The body is a drone with positioning and height measurement functions. The camera is mounted on the body and is used to collect corresponding power inspection images during the drone's power line inspection mission.

[0039] The data processing module synchronously receives the geographic information and height measurement information recorded by the body and the power line inspection image collected by the camera. The data processing module includes a foreign object detection device based on the power line inspection image as described above, and is used to identify, locate and classify foreign objects contained in the input power line inspection image. The data processing module is used to package the foreign object information detected and identified according to the power line inspection image with the geographic information and height measurement information corresponding to the power line inspection image and send them to the main station server.

[0040] The technical solution provided by the present invention has the following beneficial effects:

[0041] Based on the traditional YOLO algorithm model, the present invention adds a prediction head P2 applied to the shallow feature map FP2 to improve the solution's feature extraction capability for small foreign target objects; and removes the prediction head P5 for the deep feature map FM5 to reduce the impact of deep features on the recognition of small foreign target objects. In the feature fusion structure optimization part, the present invention uses the new EfficientDet to implement cross-level weight connections in the structure to improve the feature fusion effect of the network, thereby improving the model recognition and positioning accuracy. In the loss function optimization part, this embodiment adopts a loss function including SIoU, binary cross entropy loss and multi-label classification loss as a new regression loss function to increase the convergence efficiency of the network.

[0042] Compared with the existing solutions, the solution provided by the present invention enhances the model's ability to identify dense and small targets that often appear in power line inspection tasks, and can significantly improve the recognition accuracy and robustness of the solution. In addition, the solution of the present invention reduces the scale and parameters of the model by reducing the deep feature map FM5. The improved model is more suitable for use in devices with limited hardware computing power such as drones, improves the real-time detection of foreign objects in drone power inspection images, and reduces the device's requirements for network bandwidth for transmitting data. It is more practical and has a broader market prospect. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] Figure 1 This is a flowchart of the steps of the YOLO-based UAV power inspection foreign object detection method provided in Example 1 of the present invention.

[0044] Figure 2 This is a model architecture diagram of the improved foreign body detection network constructed in Example 1 of the present invention.

[0045] Figure 3 This is a schematic diagram of three different types of CBS modules used in the foreign body detection network of Example 1 of the present invention.

[0046] Figure 4 This is a schematic diagram of the MP module used in the foreign object detection network of Example 1 of the present invention.

[0047] Figure 5 This is a schematic diagram of the ELAN-I module used in the foreign object detection network of Example 1 of the present invention.

[0048] Figure 6 This is a schematic diagram of the SPPCSPC module used in the foreign body detection network of Example 1 of the present invention.

[0049] Figure 7 This is a schematic diagram of the ELAN-Ⅱ module used in the foreign object detection network of Example 1 of the present invention.

[0050] Figure 8 This is a schematic diagram of the skip connection branches used in the feature fusion network of Example 1 of the present invention.

[0051] Fig. 9 This is a schematic diagram of the Rep module used in the detection head of Example 1 of the present invention.

[0052] Fig.10 This is a working principle diagram of two line patrol drones with different working modes in Example 3 of the present invention. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0054] Example 1

[0055] This embodiment provides a method for detecting foreign objects in UAV power line inspection based on YOLO, which can use a trained network model to identify, locate and classify targets in real-time collected power line inspection images. The network model in this embodiment is improved based on the classic YOLO algorithm model. The new scheme enhances the detection performance of the model for dense and small targets that often appear in power line inspection tasks, and reduces the scale and parameters of the model, so that the scheme can be applied in low-computing embedded devices (such as drones).

[0056] Specifically, compared with the traditional YOLO algorithm model, in the detection head part, the scheme of this embodiment adds a prediction head P2 applied to shallow features to improve the scheme's feature extraction capability for small foreign target objects; and removes the prediction head P5 for deep features to reduce the impact of deep features on the recognition of small foreign target objects. In the feature fusion structure optimization part, this embodiment adopts the new EfficientDet to realize the cross-level weight connection of the structure, which is used to improve the feature fusion effect of the network, thereby improving the model recognition and positioning accuracy. In the loss function optimization part, the scheme of this embodiment adopts a loss function including SIoU, binary cross entropy loss and multi-label classification loss as a new regression loss function to increase the convergence efficiency of the network. Compared with the original YOLO network, the improved model volume is effectively reduced, which is more conducive to the actual deployment of drone hardware, and the detection accuracy and real-time performance of foreign targets in drone power inspection images are significantly improved.

[0057] Detailed, such as Figure 1 As shown, this embodiment provides a UAV power inspection foreign object detection method based on YOLO, which includes the following steps:

[0058] S1: Build an improved foreign body detection network based on YOLO. Figure 2 As shown, the foreign body detection network includes an input layer, a backbone network, a feature fusion network, and a prediction head. Consistent with the traditional YOLO algorithm, the input layer in this embodiment supports an input image size of 640×640×3. In particular, the core architecture of the foreign body detection network is as follows:

[0059] 1. Backbone network

[0060] Combination Figure 2 It can be found that the backbone network of this embodiment includes: 4 CBS modules, 1 ELAN-I module, and three groups of alternating MP modules and ELAN-I modules. Among them, the CBS module is an integrated module in the YOLO model. Each CBS module is composed of Conv convolution, batch normalization BN and SiLU activation function connected in series. According to the different convolution kernels k and step sizes s, three different CBS modules are provided in the YOLO algorithm model. Among them, Figure 3 As shown, k=1, s=1 means using a 1×1 convolution kernel and a CBS module with a stride of 1. This CBS module is used to change the number of feature channels; k=3, s=1 means using a 3×3 convolution kernel and a CBS module with a stride of 1. This CBS module is used to extract features; k=3, s=2 means using a 3×3 convolution kernel and a CBS module with a stride of 2. This CBS module is used for downsampling.

[0061] In the backbone network of this embodiment, the first and third of the four CBS modules use CBS modules with k=3 and s=1; the second and fourth use CBS modules with k=3 and s=2.

[0062] like Figure 4 As shown in the figure, the MP module consists of a maximum pooling layer MaxPool and three CBS modules. Two of the CBS modules are of (k=1, s=1) type, and the other CBS module is of (k=3, s=2) type. The features input to the MP module are divided into two paths. One path is processed by the maximum pooling layer MaxPool and the CBS module of (k=1, s=1) type in sequence, and the other path is processed by two CBS modules in sequence. The former is of (k=1, s=1) type, and the latter is of (k=3, s=2) type. Finally, the processing results of the two paths are spliced ​​and output.

[0063] The ELAN (Efficient QLayer Aggregation Networks) module is an important network structure in the YOLO algorithm. This module improves the accuracy and robustness of the target detection algorithm by effectively aggregating feature information from different layers. The structure of the ELAN-I module used in the backbone network of this embodiment is as follows: Figure 5 As shown, it includes 7 CBS modules, three of which are (k=1, s=1) type and the other four are (k=3, s=1) type. The features entering the ELAN-Ⅰ module are divided into four paths. The first path is processed by a (k=1, s=1) type CBS module; the second path is processed by a (k=1, s=1) type CBS module and two (k=3, s=1) type CBS modules in turn; the third path is processed by a (k=1, s=1) type CBS module and four (k=3, s=1) type CBS modules in turn; the fourth path is processed by another (k=1, s=1) type CBS module; the four paths of features are spliced ​​and then processed by the third (k=1, s=1) type CBS module before output.

[0064] In the backbone network of this embodiment, the four ELAN-I modules output feature maps F2, F3, F4, and F5 respectively. Therefore, the backbone network can perform 4, 8, 16, and 32 downsampling of the original image input to the input layer to achieve feature extraction, and then obtain feature maps F2, F3, F4, and F5 with gradually reduced scales. The sizes of the four feature maps are 160×160, 80×80, 40×40, and 20×20 respectively.

[0065] 2. Feature Fusion Network

[0066] The feature fusion network provided in this embodiment includes an upsampling channel, a downsampling channel and a jump connection branch. Among them, F5, F4, and F3 are sequentially input into the upsampling channel and finally feature spliced ​​with F2. The feature maps of F3 and F4 processed by the CBS module jump to the corresponding layer of the downsampling channel. The downsampling channel performs feature splicing and downsampling on the feature maps of different input sources, and then outputs fusion features FP2, FP3 and FP4 of different scales through the three efficient layer aggregation networks ELAN-Ⅱ.

[0067] (2.1) Upsampling channel

[0068] In the feature fusion network of this embodiment, the upsampling channel is composed of 1 SPPCSPC module, 6 CBS modules, 3 upsampling modules UpSample, and 2 ELAN-Ⅱ modules.

[0069] Among them, F5 is processed by the SPPCSPC module, CBS module, and upsampling module UpSample in sequence, and then the feature map of F4 processed by the CBS module is spliced ​​for the first time. The result of the first splicing is then processed by the ELAN-Ⅱ module, CBS module, and upsampling module UpSample in sequence, and then the feature map of F3 processed by the CBS module is spliced ​​for the second time. The result of the second splicing is then processed by the ELAN-Ⅱ module, CBS module, and upsampling module UpSample in sequence, and then the feature map of F2 processed by the CBS module is spliced ​​for the third time. Among them, the 6 CBS modules used in the upsampling channel are all (k=1, s=1) type CBS modules.

[0070] In the up-sampling channel, the structure of the SPPCSPC module used in this embodiment is as follows: Figure 6 As shown. Each SPPCSPC module is composed of 7 CBS modules and 3 maximum pooling layers MaxPool. Among them, the features input to the SPPCSPC module are divided into two paths. Among them, the first path is processed by the (k=1, s=1) type CBS module, and the second path is first processed by the (k=1, s=1) type CBS module, the (k=3, s=1) type CBS module and the (k=1, s=1) type CBS module in sequence, and then divided into three sub-paths. Each sub-path is processed by three different scales of pooling layers, and then spliced ​​with the original features. The splicing results are then processed by the (k=1, s=1) type CBS module and the (k=3, s=1) type CBS module, and then spliced ​​with the features of one path. The splicing results are finally processed by the (k=1, s=1) type CBS module and output. Among them, the convolution kernels of the three pooling layers used in this embodiment are divided into 5×5, 9×9 and 13×13.

[0071] (2.2) Downsampling channel

[0072] In the feature fusion network of this embodiment, the downsampling channel is composed of 3 ELAN-Ⅱ modules, 2 downsampling modules DownSample, and 2 MP modules.

[0073] Among them, the final output of the downsampling channel is processed by the ELAN-Ⅱ module, MP module and upsampling module DownSample in sequence, and then the fourth feature splicing is performed with the output of the next ELAN-Ⅱ module in the upsampling channel and the feature map of F3 processed by the CBS module of the jump connection. The fourth feature splicing result is processed by the ELAN-Ⅱ module, MP module and downsampling module DownSample in sequence, and then the fifth feature splicing is performed with the output of the previous ELAN-Ⅱ module in the upsampling channel and the feature map of F2 processed by the CBS module of the jump connection. The fifth feature splicing result is finally processed by the ELAN-Ⅱ module. In the downsampling channel, the three ELAN-Ⅱ modules from front to back output the required fusion features FP2, FP3 and FP4 respectively.

[0074] In the downsampling channel, the ELAN-II module is also composed of 7 CBS modules, which are consistent with the ELAN-I module, three of which are (k=1, s=1) type and the other four are (k=3, s=1) type. Figure 7 As shown, the features entering the ELAN-Ⅱ module are divided into six paths. The first and second paths are processed by only one (k=1, s=1) type CBS module; the third path is processed by one (k=1, s=1) type CBS module and one (k=3, s=1) type CBS module in turn; the fourth path is processed by one (k=1, s=1) type CBS module and two (k=3, s=1) type CBS modules in turn; the fifth path is processed by one (k=1, s=1) type CBS module and three (k=3, s=1) type CBS modules in turn; the sixth path is processed by one (k=1, s=1) type CBS module and four (k=3, s=1) type CBS modules in turn. The six-path feature map is processed by feature splicing and then processed by one (k=3, s=1) type CBS module before output. In this embodiment, the MP module used in the downsampling stage is the same as the MP module in the backbone network.

[0075] (2.3) Skip connection branch

[0076] Combination Figure 2It can be seen that the jump connection branch in this embodiment includes two paths, one of which is a jump connection between F3 and FP3, and the other is a jump connection between F4 and FP4. The jump connection can solve the gradient vanishing problem in the network of this embodiment and accelerate the training process. The upsampling channel and downsampling channel of the feature fusion network contain many layers. When the gradient signal is back-propagated, the gradient may become very small, making the network difficult to train. The jump connection used in this embodiment allows the gradient to be back-propagated more easily by passing the input information directly to the subsequent layers. This direct transfer helps to maintain the stability of the gradient and makes the network easier to train.

[0077] Specifically, a simplified diagram of the skip connection used in the feature fusion network of this embodiment is as follows: Figure 8 As shown. Combined Figure 8 It can be seen that the skip connection branch in the feature fusion network realizes cross-layer fusion between features, and the obtained weighted feature O 3 and O 4 The expression is as follows:

[0078]

[0079] In the above formula, I 0 , I 1 and I 2 Represent the nodes of the skip connection part in level 0, level 1 and 2 respectively; and They respectively represent the learnable weight parameters of the specified nodes in the corresponding levels, where the superscript represents the node number and the subscript is the level number; λ is a very small constant.

[0080] 3. Prediction Head

[0081] The prediction head of this embodiment includes a data processing channel consisting of three routes, namely, the Rep module and the Conv module, which respectively process and predict the fusion features FP2, FP3 and FP4 of different scales output by the downsampling channel, and then obtain the bounding box regression result and category of the detected target. Among them, the REP module can improve the expression and generalization ability of the model by adding branches and re-parameterization, while reducing the amount of calculation and the risk of overfitting.

[0082] Finally, after Conv convolution, the fused feature FP2 is input to the P2 prediction head of size 160×160×45, the fused feature FP3 is input to the P2 prediction head of size 80×80×45, and the fused feature FP2 is input to the P2 prediction head of size 40×40×45.

[0083] Among them Fig. 9As shown in the figure, the Rep module includes two convolution modules Conv and two BN modules. The features input to the Rep module are divided into two paths, one of which is processed by a convolution module with a convolution kernel of 1 and a step size of 1 and a BN module, and the other is processed by a convolution module with a convolution kernel of 3 and a step size of 1 and a BN module; the features output by the two BN modules are concatenated as the output of the Rep module.

[0084] S2: Obtain a large number of real power inspection images collected from the perspective of drones, and manually add target selection boxes and category marking information to the foreign objects in the collected power inspection images as sample images.

[0085] In the actual scheme of this embodiment, the collected power inspection images should be uniformly cropped to a pixel size of 640×640. In addition, in order to expand the number of sample images, various existing image enhancement methods can also be used to amplify the original data. The amplified image and the original image are first marked with their target selection boxes and target categories. Among them, the image enhancement methods include geometric transformation, color space change and pixel point operation. Geometric transformation includes flipping, rotation, scaling, cropping and translation. The operation of geometric change is to simulate the effect of angle changes on the target in the field of view during the drone image acquisition process. Color space transformation includes contrast change, brightness change, saturation change, channel separation, grayscale processing, histogram enhancement, etc. Color space transformation is to simulate the effect of the imaging quality of the drone under different ambient light conditions on the image. Pixel point operations include blurring, sharpening, etc. This operation is to cope with the impact of the imaging effects of different cameras on the scheme. After the sample data is amplified by the image enhancement method, the robustness of the trained network model in the actual application process can be improved.

[0086] S3: Divide the data set containing a large number of sample images into a training set and a test set, and use them to train and test the foreign body detection network. In practical applications, this embodiment divides the training set and the test set in a ratio of 6:4, and the image obtained by any image enhancement method and the original image of the same original image are divided into the same training set or test set.

[0087] In the solution of this embodiment, the loss function L used in the training phase is as follows:

[0088]

[0089] In the above formula, L SIoU represents the positioning loss, L obj represents the target confidence loss, L cla represents the classification loss; α, β and γ are the weights of the three types of losses, α+β+γ=1; IoU is the intersection over union ratio of the predicted box and the true box, L Dis is the distance loss, LSp is the shape loss.

[0090] Among them, the target confidence loss L in the loss function L obj The binary cross entropy loss BCE is used, and its expression is as follows:

[0091]

[0092] In the above formula, y represents the true value, Indicates the prediction result;

[0093] Classification loss L cla The multi-label classification loss BWL is used, and its expression is as follows:

[0094]

[0095] In the above formula, N represents the total number of target categories, x i represents the true category of the i-th target; y i Represents the category prediction result of the i-th target object.

[0096] The localization loss L in step S3 SIoU In the figure, the shape loss L Sp and distance loss L Dis The calculation formula is as follows:

[0097]

[0098] In the above formula, w and w GT Represent the width of the predicted box and the real box respectively; h and h GT Represent the height of the predicted box and the real box respectively; θ represents the preset weight value; C h and C w C is the height and width of a rectangle constructed by connecting the center points of the ground truth box and the predicted box and using the diagonals; y and C x are the height and width of the minimum circumscribed rectangle of the overall area corresponding to the two boxes; L ang is the angle loss, and σ represents the distance between the center points of the real box and the predicted box.

[0099] S4: retain the model parameters of the foreign object detection network that meets the accuracy requirements after training; and pre-process the real-time images collected during the drone patrol process into the required size and input them into the trained foreign object detection network to detect and locate the foreign objects contained in the image.

[0100] Example 2.

[0101] Based on the solution of Example 1, this embodiment further provides a foreign object detection device based on power inspection images, which includes a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, a foreign object detection network trained in the UAV power inspection foreign object detection method based on YOLO in Example 1 is created. The foreign object detection network is used to identify, locate and classify foreign objects contained in the input power inspection image.

[0102] The foreign body detection device based on power inspection images provided in this embodiment is essentially a computer device. The computer device can be an embedded computer device or a general computer device. For example, it can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a rack server, a blade server, a tower server or a cabinet server (including an independent server or a server cluster composed of multiple servers) that can execute programs.

[0103] The computer device of this embodiment at least includes but is not limited to: a memory and a processor that can be connected to each other through a system bus. In this embodiment, the memory (i.e., a readable storage medium) includes a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory can be an internal storage unit of a computer device, such as a hard disk or a memory of the computer device. In other embodiments, the memory can also be an external storage device of a computer device, such as a plug-in hard disk equipped on the computer device, a smart memory card (Smart Media Card, SMC), a secure digital (SecureDigital, SD) card, a flash card (Flash Card), etc. Of course, the memory can also include both the internal storage unit of the computer device and its external storage device. In this embodiment, the memory is generally used to store an operating system and various application software installed on the computer device. In addition, the memory can also be used to temporarily store various types of data that have been output or are to be output.

[0104] The processor may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chip in some embodiments. The processor is generally used to control the overall operation of a computer device.

[0105] Example 3

[0106] On the basis of embodiments 1 and 2, this embodiment further provides a line inspection drone. In actual application, the line inspection drone should have GPS positioning and height measurement functions. The line inspection drone can fly above the high-voltage line according to the specified inspection track and take power inspection images of the high-voltage line below. Then, foreign objects are detected based on the power inspection images and reported to the background server.

[0107] Considering that the foreign body detection scheme of the power inspection image in this embodiment has relatively low requirements on hardware computing power, the real-time performance of detection and recognition is good. Fig.10 As shown, two processing methods can be used for the collected power inspection images: one is to transmit the power inspection images in real time to the server of the background control center through the onboard wireless communication module. The background server can deploy the foreign object detection network trained as in Example 1; at this time, the background server is the foreign object detection device based on power inspection images in Example 2. Next, the background server can identify, locate and classify the foreign objects contained in the input power inspection images to obtain the corresponding detection results.

[0108] In another solution, the drone includes a fuselage and a camera on its load and an embedded data processing module. The camera is used to collect corresponding power inspection images during the drone's power line inspection mission and input them into the data processing module; the foreign object detection network trained as in Example 1 is directly deployed in the data processing module. At this time, the embedded data processing module is the foreign object detection device based on power inspection images in Example 2. In this state, the drone can directly identify, locate and classify foreign objects contained in the input power inspection images while acquiring the power inspection images, and finally transmit the detection results back to the background server.

[0109] In the two different drone solutions provided in this embodiment, the former does not need to process the image data locally, but needs to transmit the image data back to the background. Therefore, this solution has lower requirements on the computing power of the processor in the drone, but relatively higher requirements on the bandwidth of the communication module. The latter can process the collected image data locally, so the computing power requirements of the processor in the drone are relatively high (compared with the traditional solution, the hardware performance requirements are still reduced), but the bandwidth requirements of the communication module are lower. The two solutions can be reasonably selected according to the specific needs of the customer.

[0110] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A YOLO-based UAV power inspection foreign object detection method, characterized by: It includes the following steps: S1: Based on YOLO, an improved foreign body detection network is constructed. The foreign body detection network includes an input layer, a backbone network, a feature fusion network and a prediction head. The core architecture of the foreign body detection network is as follows: (1) The backbone network is used to perform 4, 8, 16 and 32 downsampling of the original image input to the input layer in turn to achieve feature extraction, thereby obtaining feature maps F2, F3, F4 and F5 with gradually reduced scales; (2) The feature fusion network includes an upsampling channel, a downsampling channel and a jump connection branch; among them, F5, F4, and F3 are sequentially input into the upsampling channel and finally feature concatenated with F2; the feature maps of F3 and F4 processed by the CBS module are jump-connected to the corresponding layer of the downsampling channel; the downsampling channel performs feature concatenation and downsampling on the feature maps of different input sources, and then outputs fusion features FP2, FP3, and FP4 of different scales through the three efficient layer aggregation networks ELAN-Ⅱ; (3) The prediction head includes three regression channels consisting of a Rep module and a Conv module, which are used to process and predict the fusion features FP2, FP3, and FP4 of different scales output by the downsampling channel, and then obtain the bounding box regression result and category of the detected target; S2: A large number of real power inspection images collected from the perspective of drones are obtained, and target selection boxes and category marking information are manually added to the foreign objects in the collected power inspection images as sample images; S3: Divide the data set containing a large number of sample images into a training set and a test set, and use them to train and test the foreign body detection network; the loss function L used in the training stage is as follows: Among them, L SIoU represents the positioning loss, L obj represents the target confidence loss, L cla represents the classification loss; α, β and γ are the weights of the three types of losses, α+β+γ=1; IoU is the intersection over union ratio of the predicted box and the true box, L Dis is the distance loss, L Sp is the shape loss. S4: retain the model parameters of the foreign object detection network that meets the accuracy requirements after training, and pre-process the real-time image collected during the drone line patrol process into the required size and input it into the trained foreign object detection network to detect and locate the foreign objects contained in the image.

2. The UAV power inspection foreign body detection method based on YOLO as claimed in claim 1 is characterized by: In the feature fusion network of the detection head, the upsampling channel is composed of 1 SPPCSPC module, 6 CBS modules, 3 upsampling modules UpSample, and 2 ELAN-Ⅱ modules; Among them, F5 is processed by the SPPCSPC module, the CBS module, and the upsampling module UpSample in sequence, and then the feature map of F4 processed by the CBS module is spliced ​​for the first time; the first splicing result is processed by the ELAN-Ⅱ module, the CBS module, and the upsampling module UpSample in sequence, and then the feature map of F3 processed by the CBS module is spliced ​​for the second time; the second splicing result is processed by the ELAN-Ⅱ module, the CBS module, and the upsampling module UpSample in sequence, and then the feature map of F2 processed by the CBS module is spliced ​​for the third time.

3. The YOLO-based UAV power inspection foreign body detection method according to claim 2, characterized in that: In the feature fusion network of the detection head, the downsampling channel is composed of 3 ELAN-Ⅱ modules, 2 downsampling modules DownSample, and 2 MP modules; Among them, the final output of the downsampling channel is processed by the ELAN-Ⅱ module, the MP module and the downsampling module DownSample in sequence, and then the fourth feature splicing is performed with the output of the next ELAN-Ⅱ module in the upsampling channel and the feature map of F3 processed by the CBS module of the jump connection; the fourth feature splicing result is processed by the ELAN-Ⅱ module, the MP module and the downsampling module DownSample in sequence, and then the fifth feature splicing is performed with the output of the previous ELAN-Ⅱ module in the upsampling channel and the feature map of F2 processed by the CBS module of the jump connection; the fifth feature splicing result is finally processed by the ELAN-Ⅱ module; in the downsampling channel, the three ELAN-Ⅱ modules from front to back output the required fusion features FP2, FP3 and FP4 respectively.

4. The YOLO-based foreign body detection method for UAV power inspection as claimed in claim 3 is characterized by: The skip connection branch in the feature fusion network realizes cross-layer fusion between features, and the expressions of the obtained weighted features O3 and O4 are as follows: In the above formula, I0, I1 and I2 represent the nodes of the skip connection part in level 0, level 1 and 2 respectively; and They respectively represent the learnable weight parameters of the specified nodes in the corresponding levels, where the superscript represents the node number and the subscript is the level number; λ is a very small constant.

5. The YOLO-based UAV power inspection foreign body detection method according to claim 3, characterized in that: The backbone network includes: 4 CBS modules, 1 ELAN-I module and three groups of alternating MP modules and ELAN-I modules; the 4 ELAN-I modules output feature maps F2, F3, F4 and F5 respectively.

6. The UAV power inspection foreign body detection method based on YOLO as claimed in claim 5 is characterized by: The CBS module is composed of Conv convolution, batch normalization BN and SiLU activation function connected in series in sequence; The MP module consists of one maximum pooling layer MaxPool and three CBS modules; the features input to the MP module are divided into two paths, one path is processed by the maximum pooling layer MaxPool and the CBS module in sequence, and the other path is processed by two CBS modules in sequence, and finally the processing results of the two paths are spliced ​​and output.

7. The YOLO-based foreign body detection method for UAV power inspection as claimed in claim 1, characterized in that: In step S3, the target confidence loss L in the loss function L is obj The binary cross entropy loss BCE is used, and its expression is as follows: In the above formula, y represents the true value, Indicates the prediction result; Classification loss L cla The multi-label classification loss BWL is used, and its expression is as follows: In the above formula, N represents the total number of target categories, x i represents the true category of the i-th target; y i Represents the category prediction result of the i-th target object.

8. The YOLO-based foreign body detection method for UAV power inspection as claimed in claim 7, characterized in that: In the positioning loss L SIoU In the above equation, the shape loss L Sp and distance loss L Dis The calculation formula is as follows: In the above formula, w and w GT Represent the width of the predicted box and the real box respectively; h and h GT Represent the height of the predicted box and the real box respectively; θ represents the preset weight value; C h and C w C is the height and width of a rectangle constructed by connecting the center points of the ground truth box and the predicted box and using the diagonals; y and C x are the height and width of the minimum circumscribed rectangle of the overall area corresponding to the two boxes; L ang is the angle loss, and σ represents the distance between the center points of the real box and the predicted box.

9. A foreign body detection device based on power inspection images, characterized in that: It includes a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: when the processor executes the computer program, it creates a foreign object detection network that has been trained in the YOLO-based UAV power inspection foreign object detection method as described in any one of claims 1-8, and the foreign object detection network is used to identify, locate and classify foreign objects contained in the input power inspection image.

10. A line patrol drone, characterized in that: It includes: The body adopts a drone with positioning and height measurement functions; A camera is mounted on the body, and is used to collect corresponding power line inspection images when the UAV performs a power line inspection task; A data processing module, which is used to synchronously receive the geographic information and height measurement information recorded by the body and the power line inspection image collected by the camera; the data processing module contains a foreign object detection device based on the power line inspection image as described in claim 9, and is used to identify, locate and classify foreign objects contained in the input power line inspection image; the data processing module is used to package the foreign object information detected and identified according to the power line inspection image with the geographic information and height measurement information corresponding to the power line inspection image and send them to the main station server.

Images