A foreign object detection method for transmission lines based on inter-frame correlation learning

Abstract

The present invention proposes a foreign object detection method for transmission lines based on inter-frame correlation learning, including the following methods. Method 1: Through image features, register the inspection template video frames of the overhead transmission line without foreign objects with the video frames of the transmission line to be detected. Method 2: Identify the differences between the registered video frames through inter-frame differences, detect the static foreign objects on the transmission line, and then use the image attention mechanism to establish the channel information features of the foreign objects, enabling the detection model to focus on the feature extraction of the foreign objects. Method 3: Utilize the characteristics of inter-frame correlation of a single video in the video of a single inspection, and adopt a moving object detection method to detect the moving foreign objects in the video. Through the foreign object detection method for transmission lines based on inter-frame correlation learning, the present invention comprehensively considers the detection results of static and dynamic foreign objects, making the foreign object detection of overhead transmission lines conform to the actual application scenario.

Description

Technical Field

[0001] The present invention relates to the technical fields of deep learning, instance segmentation, image processing, and foreign object detection in UAV power transmission lines, and in particular to a foreign object detection method for power transmission lines based on inter-frame correlation learning. Background Art

[0002] In recent years, due to the rise of UAV technology, UAVs have begun to be used for power transmission line inspections. Compared with manual inspections, UAV inspections have the advantages of low cost and high efficiency, can effectively save inspection time, and also provide certain protection for the safety of inspection personnel. However, detecting whether there are foreign objects in the inspection return images has encountered a bottleneck. Taking pictures at key positions, saving the data, and transmitting it back to the background for manual detection is time-consuming and laborious. However, due to the characteristics of foreign objects in power transmission lines being widely distributed and having complex and variable types in the power transmission lines, it makes their detection more difficult. But for the situation of a large number of foreign object types and the inability to determine the occurrence location, inspection personnel can only check the UAV inspection images one by one by visual judgment, which may lead to extremely high workloads and missed detection rates. Moreover, some foreign objects in aerial images have extremely small visual differences from the inspection background, making the foreign object detection in power transmission lines more challenging.

[0003] Thanks to the development of deep learning, currently in power inspections, most UAV inspection images use object detection methods. After training common foreign object types in a convolutional neural network, the purpose of detecting foreign objects can be achieved. Currently, the foreign object detection methods for power transmission lines based on deep learning can be roughly divided into four categories: object detection methods such as Faster R-CNN and YOLO, methods based on GANs (Generative Adversarial Networks), and methods based on instance segmentation such as Mask R-CNN and YOLACT. However, the current methods have the problem that the performance of the detection model requires a large amount of foreign object image data support, and the number and types of foreign objects are numerous, so it is impossible to train a model that can detect all foreign objects. When the detected foreign object object changes, the algorithm or model needs to be redesigned.

[0004] Currently, the detection of foreign objects is to perform separate detection on individual frames, and only a specific few types of foreign objects can be analyzed and detected, ignoring the connection between frames. When the detected foreign object object changes, the algorithm or model needs to be redesigned, and there is a problem of poor generalization. Summary of the Invention

[0005] The present invention proposes a foreign object detection method for power transmission lines based on inter-frame correlation learning. Through the foreign object detection method for power transmission lines based on inter-frame correlation learning, the detection results of static and dynamic foreign objects are comprehensively considered to make the foreign object detection in overhead power transmission lines conform to the real application scenario.

[0006] The present invention adopts the following technical solutions.

[0007] A foreign object detection method for transmission lines based on inter-frame correlation learning, including the following methods:

[0008] Method 1: Through image features, register the inspection template video frames of the overhead transmission line without foreign objects with the video frames of the transmission line to be detected.

[0009] Method 2: Identify the registered video frames through inter-frame differences, detect the static foreign objects on the transmission line by determining the differences from the inspection video without foreign objects, and then use the image attention mechanism to establish the channel information features of the foreign objects, enabling the detection model to focus on the feature extraction of foreign objects.

[0010] Method 3: Utilize the characteristics of inter-frame correlation in a single inspection video and adopt a moving object detection method to detect moving foreign objects in the video.

[0011] For the image registration in Method 1, use the drone inspection images of the transmission line.

[0012] For the image registration in Method 1, based on the YOLACT instance segmentation algorithm, adopt a feature point matching algorithm.

[0013] The detection method includes the following steps:

[0014] Step S1: Use the drone-mounted camera to collect multiple images of the specified inspection area.

[0015] Step S2: Divide the images taken at the same position in Step S1 into a template video without foreign object suspension and a video of unknown presence of foreign objects. Use the video of unknown presence of foreign objects as the video to be detected, and use the template video and the video to be detected taken at the same position as the video pair for registration.

[0016] Step S3: Make a dataset. Specifically, extract one frame per second from the inspection video to make the image training data for inspection instance segmentation. Similarly, pair the images extracted from the template video and the images extracted from the video to be detected one by one as the image pairs for registration.

[0017] Step S4: Classify the areas where foreign objects may be suspended in all the images collected from the transmission line into tower areas and line areas, and label their pixel categories according to the classification. Divide this part of the data into a training set and a validation set.

[0018] Step S5: Initialize the instance segmentation model parameters. The model includes but is not limited to the YOLACT instance segmentation model, and use the training set data labeled in Step S4 to train the model.

[0019] Step S6: Validate the model using the validation set marked in Step S4, and solidify the YOLACT instance segmentation model for the inspection images of the transmission line when the expected value is reached;

[0020] Step S7: Feed the image pairs at the same position in Step S2 into the YOLACT instance segmentation model to obtain the positions of the tower region and the line region;

[0021] Step S8: Perform image morphological processing on the results of Step S7 to improve the accuracy of image registration, and further improve the foreign object detection effect of the transmission line;

[0022] Step S9: Establish an image registration framework for the overhead transmission line. Adopt the feature point extraction algorithm, the feature point matching algorithm, and the outlier filtering algorithm to match the same objects in the image pair and map them to the same spatial position to complete the registration of the image pair;

[0023] Step S10: After the image pair is completed, slice it into sub-image pairs of the same size, and perform the image registration in Step S9 on the sliced sub-images again to map each sub-image pair to the same spatial position;

[0024] Step S11: Use the convolutional neural network to extract the images of the image registration framework, remove the downsampling operation of each layer, and adopt the dilated convolution operation on the last two convolutional layers to sample the image features onto the 256×256 feature map;

[0025] Step S12: Input the results of the sub-image pairs obtained in Step S10 into the convolutional neural network in Step S11 to extract the 256×256 size feature maps of the sub-image pairs respectively;

[0026] Step S13: Build an attention mechanism module to establish the weights of the image features, so that the network model pays more attention to the features that have a greater impact on the results. Specifically: perform global average pooling operation on the 256×256×512 size feature map extracted in Step S12 to convert the feature map into 1×1×512, select appropriate convolutional operation parameters, perform one-dimensional convolution calculation and keep the output dimension as 1×1×512, and then obtain the weights with channel attention information after Sigmoid calculation;

[0027] The dimension calculation formula of the one-dimensional convolution output is as follows:

[0028]

[0029] In the formula, n is the size of the image, p is the space between the filled element border and the element content, f is the convolution kernel size, and s is the convolution stride;

[0030] The Sigmoid calculation formula is as follows:

[0031]

[0032] Step S14: For the sub - graph pair feature maps extracted in Step S12 through the convolutional neural network, send the feature maps into the attention mechanism module to obtain the sub - graph pair feature maps carrying weight information;

[0033] Step S15: Calculate the distances at the corresponding positions of the sub - graph pair feature maps carrying weight information in Step S14, and determine whether there are foreign objects according to the distances, and confirm the positions of static foreign objects;

[0034] Step S16: After the above - mentioned detection of static foreign objects, use the moving object detection method to detect the dynamic foreign objects floating in the inspection video.

[0035] Step S17: Synthesize the static foreign object detection in Steps S11 to S15 and the dynamic foreign object detection in Step S16, fuse the results of the two judgments, and further improve the accuracy of the judgment.

[0036] In Step S8, the processing methods of image morphology erosion and dilation are used to retain more edge details of the transmission tower and line areas.

[0037] In Step S9, the algorithms used include but are not limited to the AKAZE feature point extraction algorithm, the Brute - force algorithm is used to match feature points, the AdaLAM algorithm is used to filter out outliers, and the image pair registration is completed by solving the mapping matrix.

[0038] Step S11 uses the resnet50 residual convolutional neural network.

[0039] The method used in Step S13 includes but is not limited to the ECA (Efficient Channel Attention) channel attention mechanism module to establish the weights of image features.

[0040] The moving object detection method in Step S16 includes but is not limited to the optical flow method, the inter - frame difference method, or the background subtraction method.

[0041] The present invention is divided into two parts: the registration of UAV inspection images and the foreign object detection of transmission lines based on inter - frame correlation learning. Both parts include two stages: training and testing. Through the foreign object detection method of transmission lines based on inter - frame correlation learning, considering the detection results of static and dynamic foreign objects comprehensively, the foreign object detection of overhead transmission lines conforms to the actual application scenario.

[0042] In view of the situation that there are a wide variety of foreign object images collected by current UAV inspections, but the total number is not large, the present invention proposes a method for registering video frames of an overhead transmission line inspection template without foreign objects and video frames to be detected by using feature registration, which can identify the differences through inter-frame differences, detect static foreign objects on the transmission line by determining the differences from the foreign object-free inspection video, and can detect moving foreign objects in the video by using a moving target detection method.

[0043] The beneficial effects of the present invention also lie in: by learning the inter-frame correlation, making full use of the inter-frame correlation information between videos, establishing a foreign object detection model with high detection accuracy and good generalization performance. It adapts to the actual situation that there are a wide variety of foreign object images collected by UAV inspections, but the total number is not large. It solves the problem that the current foreign object detection of transmission lines uses individual frames for detection, ignores the connection between frames, and when the detected foreign object object changes, it is necessary to re-design the algorithm or train the model, resulting in poor generalization. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The following further details the present invention in conjunction with the drawings and specific embodiments:

[0045] Att Figure 1 is a schematic diagram of the principle of a foreign object detection method for transmission lines based on inter-frame correlation learning;

[0046] Att Figure 2 Schematic diagram of the principle of image registration for UAV inspection of transmission lines;

[0047] Att Figure 3 Schematic diagram of the principle of the ECA channel attention mechanism. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0048] As shown in the figure, a foreign object detection method for transmission lines based on inter-frame correlation learning includes the following methods:

[0049] Method 1: Through image features, register the video frames of the inspection template of the overhead transmission line without foreign objects and the video frames of the transmission line to be detected;

[0050] Method 2: Identify the differences between the registered video frames through inter-frame differences, detect static foreign objects on the transmission line by determining the differences from the foreign object-free inspection video, and then use the image attention mechanism to establish the channel information features of the foreign objects, so that the detection model focuses on the feature extraction of the foreign objects;

[0051] Method 3: Utilize the characteristics of the inter-frame correlation of a single video in the video of a single inspection, and adopt a moving target detection method to detect moving foreign objects in the video.

[0052] The image registration in Method 1 uses the UAV inspection images of the transmission line.

[0053] The image registration in Method 1 is based on the YOLACT instance segmentation algorithm and adopts the feature point matching algorithm.

[0054] The detection method includes the following steps:

[0055] Step S1: Use the on-board camera of the drone to collect multiple images of the specified inspection area.

[0056] Step S2: Divide the images taken at the same position in Step S1 into a template video without foreign objects hanging and a video of unknown whether there are foreign objects. Regard the video of unknown whether there are foreign objects as the video to be detected, and regard the template video and the video to be detected taken at the same position as the video pair for registration.

[0057] Step S3: Make a data set. Specifically, extract one frame per second from the inspection video to make the image training data for inspection instance segmentation. Similarly, pair the images extracted from the template video and the images extracted from the video to be detected one by one as the image pairs for registration.

[0058] Step S4: Classify the areas where foreign objects may hang in all the images collected from the transmission line into tower areas and line areas, and label their pixel categories according to the classification. Divide this part of the data into a training set and a validation set.

[0059] Step S5: Initialize the instance segmentation model parameters, that is, the YOLACT instance segmentation model, and use the training set data labeled in Step S4 to train the model.

[0060] Step S6: Use the validation set labeled in Step S4 to verify the model. When the expected value is reached, solidify the YOLACT instance segmentation model of the inspection image transmission line.

[0061] Step S7: Send the image pair at the same position in Step S2 into the YOLACT instance segmentation model to obtain the positions of the tower area and the line area.

[0062] Step S8: Perform image morphological processing on the result of Step S7 to improve the accuracy of image registration and further improve the foreign object detection effect of the transmission line.

[0063] Step S9: Establish an image registration framework for the overhead transmission line. Adopt the feature point extraction algorithm, the feature point matching algorithm, and the outlier filtering algorithm to match the same objects in the image pair and map them to the same spatial position to complete the registration of the image pair.

[0064] Step S10: After the image pair is completed for matching, slice it into sub-image pairs of the same size, and perform the image registration in Step S9 on the sliced sub-images again to map each sub-image pair to the same spatial position.

[0065] Step S11: Use a convolutional neural network to extract the image registration framework image, remove the downsampling operation in each layer, and use dilated convolution operations on the last two convolutional layers to sample the image features onto a 256×256 feature map;

[0066] Step S12: Input the results of the sub-image pairs obtained in Step S10 into the convolutional neural network in Step S11, and respectively extract the 256×256-sized feature maps of the sub-image pairs;

[0067] Step S13: Build an attention mechanism module to establish the weights of the image features, enabling the network model to pay more attention to the features that have a greater impact on the results. Specifically: perform global average pooling on the 256×256×512-sized feature map extracted in Step S12 to convert the feature map into 1×1×512, select appropriate convolutional operation parameters, perform one-dimensional convolutional calculation and keep the output dimension as 1×1×512, and then obtain the weights with channel attention information after Sigmoid calculation;

[0068] The formula for calculating the output dimension of the one-dimensional convolution is as follows:

[0069]

[0070] In the formula, n is the size of the image, p is the space between the filled element border and the element content, f is the convolution kernel size, and s is the convolution stride;

[0071] The Sigmoid calculation formula is as follows:

[0072]

[0073] Step S14: Pass the feature maps of the sub-image pairs extracted through the convolutional neural network in Step S12 into the attention mechanism module to obtain the sub-image pair feature maps carrying weight information;

[0074] Step S15: Calculate the distances at the corresponding positions of the sub-image pair feature maps carrying weight information in Step S14, and determine whether there are foreign objects based on the distances, and confirm the positions of the static foreign objects;

[0075] Step S16: After the above detection of static foreign objects, use a moving object detection method to detect the floating dynamic foreign objects in the inspection video.

[0076] Step S17: Integrate the static foreign object detection in Steps S11 to S15 and the dynamic foreign object detection in Step S16, fuse the results of the two judgments, and further improve the accuracy of the judgment.

[0077] In step S8, an image morphological erosion and dilation processing method is adopted to retain more edge details of the transmission tower and line areas.

[0078] In step S9, the AKAZE feature point extraction algorithm is adopted, the Brute-force algorithm is used to match the feature points, the AdaLAM algorithm is used to filter out the outliers, and the registration of the image pair is completed by solving the mapping matrix.

[0079] Step S11 uses the resnet50 residual convolutional neural network.

[0080] Step S13 uses the ECA (Efficient Channel Attention) channel attention mechanism module to establish the weights of the image features.

[0081] The moving target detection method in step S16 includes the optical flow method, the inter-frame difference method or the background elimination method.

Claims

1. A foreign object detection method for transmission lines based on inter-frame correlation learning, characterized in that: including the following methods, Step 1: Through image features, register the inspection template video frames of the overhead transmission line without foreign objects with the video frames of the transmission line to be detected; Step 2: Identify the differences between the registered video frames through inter-frame differences, detect the static foreign objects on the transmission line, and then use the image attention mechanism to establish the channel information features of the foreign objects, so that the detection model focuses on the feature extraction of the foreign objects; Step 3: In the video of a single inspection, use the characteristics of the inter-frame correlation of a single video and adopt the moving object detection method to detect the moving foreign objects in the video; The specific content of the detection method includes the following steps; Step S1: Use the on-board camera of the unmanned aerial vehicle to collect multiple images of the designated inspection area; Step S2: Divide the images taken at the same position in Step S1 into a template video without foreign object suspension and a video of unknown whether there are foreign objects. Regard the video of unknown whether there are foreign objects as the video to be detected, and regard the template video and the video to be detected taken at the same position as the video pair for registration; Step S3: Make a data set. Specifically: Extract one frame per second from the inspection video to make the image training data for inspection instance segmentation. Similarly, extract the images from the template video and the images from the video to be detected, and pair them one by one as the image pairs for registration; Step S4: Classify the areas where foreign objects may be suspended in all the images collected from the transmission line into tower areas and line areas, and label their pixel categories according to the classification. Divide the image data of the areas where foreign objects may be suspended into a training set and a validation set; Step S5: Initialize the instance segmentation model parameters, that is, the YOLACT instance segmentation model, and use the training set data labeled in Step S4 to train the model; Step S6: Use the validation set labeled in Step S4 to verify the model, and solidify the YOLACT instance segmentation model of the inspection image transmission line when the expected value is reached; Step S7: Send the image pairs at the same position in Step S2 into the YOLACT instance segmentation model to obtain the positions of the tower area and the line area; Step S8: Perform image morphological processing on the results of Step S7 to improve the accuracy of image registration, and further improve the foreign object detection effect of the transmission line; Step S9: Establish an image registration framework for the overhead transmission line. Adopt the feature point extraction algorithm, the feature point matching algorithm, and the outlier filtering algorithm to match the same objects in the image pair, and map them to the same spatial position to complete the registration of the image pair; Step S10: After the image pair is matched, slice it into sub-image pairs of the same size, and perform the image registration in Step S9 on the sliced sub-images again to map each sub-image pair to the same spatial position; Step S11: Use the convolutional neural network to extract the images of the image registration framework, remove the downsampling operation of each layer, and adopt the dilated convolution operation on the last two convolutional layers to sample the image features onto the 256×256 feature map; Step S12: Input the results of the sub - graph pairs obtained in Step S10 into the convolutional neural network in Step S11, and extract feature maps of size 256×256 for the sub - graph pairs respectively; Step S13: Build an attention mechanism module to establish the weights of image features, enabling the network model to pay more attention to the features that have a greater impact on the results. Specifically: perform global average pooling operation on the feature map of size 256×256×512 extracted in Step S12 to convert the feature map into 1×1×512, select appropriate convolutional operation parameters, perform one - dimensional convolution calculation and keep the output dimension as 1×1×512, and then obtain the weights with channel attention information after Sigmoid calculation; The dimension calculation formula for the one - dimensional convolution output is as follows: In the formula, n is the size of the image, p is the space between the filled element border and the element content, f is the convolution kernel size, and s is the convolution stride; The Sigmoid calculation formula is as follows: Step S14: Feed the feature maps of the sub - graph pairs extracted through the convolutional neural network in Step S12 into the attention mechanism module to obtain the sub - graph pair feature maps carrying weight information; Step S15: Calculate the distances at corresponding positions of the sub - graph pair feature maps carrying weight information in Step S14, and determine whether there are foreign objects based on the distances and confirm the positions of static foreign objects; Step S16: After the detection of the above - mentioned static foreign objects, use the moving object detection method to detect the dynamic foreign objects floating in the inspection video; Step S17: Integrate the static foreign object detection from Step S11 to Step S15 and the dynamic foreign object detection in Step S16, fuse the results of the two judgments, and further improve the accuracy of the judgment.

2. The foreign object detection method for transmission lines based on inter-frame correlation learning according to claim 1, wherein: In Step 1, for image registration, use the UAV inspection images of transmission lines.

3. A foreign object detection method for transmission lines based on inter-frame correlation learning according to claim 1, characterized in that: In Step 1, for image registration, based on the YOLACT instance segmentation algorithm, use the feature point matching algorithm.

4. A foreign object detection method for transmission lines based on inter-frame correlation learning according to claim 1, characterized in that: In Step S8, use the image morphological erosion and dilation processing methods to retain more edge details of the transmission towers and line areas.

5. The foreign object detection method for transmission lines based on inter-frame correlation learning according to claim 1, wherein: In Step S9, use the AKAZE feature point extraction algorithm, use the Brute - force algorithm to match feature points, use the AdaLAM algorithm to filter out outliers, and complete the registration of the image pair by solving the mapping matrix.

6. The foreign object detection method for transmission lines based on inter-frame correlation learning according to claim 1, characterized in that: Step S11 uses the resnet50 residual convolutional neural network.

7. A foreign object detection method for transmission lines based on inter-frame correlation learning according to claim 1, characterized in that: Step S13 uses the ECA channel attention mechanism module to establish the weights of image features.

8. A foreign object detection method for transmission lines based on inter-frame correlation learning according to claim 1, characterized in that: The moving object detection method in Step S16 includes the optical flow method, the frame - difference method or the background subtraction method.

Images