A power transmission line suspension foreign matter detection method based on multi-network feature fusion

Abstract

The application discloses a power transmission line suspension foreign matter detection method based on multi-network feature fusion, first constructs a power transmission line suspension foreign matter image dataset; secondly extracts fusion features of power transmission line suspension foreign matter images; then constructs and trains a random forest power transmission line suspension foreign matter classification and identification model based on Bayesian optimization; power transmission line suspension foreign matter classification and identification: the fusion features X of the test set are used to test M CART decision trees in the random forest classification model, and predicted categories {C1(X), C2(X),..., Cm(X)} are obtained; voting is performed on the classified results obtained through testing, and the one with the highest frequency in the M CART decision tree classification results is selected as the final classification result; the application has good detection precision and detection speed, is more favorable to practical application, and can provide reference for power grid operation and maintenance personnel to carry out power transmission line suspension foreign matter obstacle removal work.

Description

Technical Field

[0001] This invention belongs to the field of image data processing technology for power transmission line monitoring, and specifically relates to a method for detecting suspended foreign objects on power transmission lines based on multi-network feature fusion. Background Technology

[0002] The operation and maintenance of power transmission lines are of great significance to the safe and stable operation of the power grid. my country's power transmission lines are complex and vary significantly across regions. Some lines are located near cities and construction sites, where foreign objects such as bird nests, balloons, and kites may hang on the lines, shortening the discharge distance and causing transmission faults. However, due to the unpredictable and sudden nature of these faults, relying on manual inspections suffers from problems such as untimely detection and low efficiency. Summary of the Invention

[0003] Therefore, the purpose of this invention is to provide a method for detecting foreign objects suspended on transmission lines based on multi-network feature fusion, which can accurately identify foreign objects suspended on transmission lines and provide a reference for power grid operation and maintenance personnel to carry out the work of clearing foreign objects suspended on transmission lines.

[0004] To achieve the above objectives, the present invention provides the following technical solution: a method for classifying and identifying bird species that pose a threat to power grids based on deep convolutional features, comprising the following steps:

[0005] A method for detecting suspended foreign objects on transmission lines based on multi-network feature fusion includes the following steps:

[0006] S1. Construct a dataset of images of foreign objects hanging on power transmission lines: For images of foreign objects hanging on power transmission lines taken by manual inspection and drone inspection of power grids, the Otus maximum inter-class variance threshold segmentation method and morphological methods are used to extract the target regions of foreign objects. The Poisson fusion algorithm is used to fuse the real background of the power transmission line with the target regions of foreign objects to generate virtual foreign object image augmentation image samples. A dataset of images of foreign objects hanging on power transmission lines containing N types is constructed and divided into training set and test set according to a certain ratio.

[0007] S2. Feature extraction from images of foreign objects suspended on transmission lines: Convolutional neural network models GoogLeNet, DarkNet-53, VGG-19, ResNet-50, and Xception were built and pre-trained using the large-scale dataset MS COCO. The structure of the five pre-trained network models was fine-tuned, and the five fine-tuned network models were retrained using the expanded training set of foreign object images.

[0008] Five retrained network models were used to extract features from a dataset of images of foreign objects suspended on power transmission lines. The Grad-CAM++ algorithm was used to generate class activation heatmaps for the feature extraction layers of each network model, obtaining the decision regions of each feature layer. Based on this, the best feature extraction layer of each network model was selected. The decision regions of the best feature extraction layers were analyzed, and complementary features were selected for cascade fusion to obtain the best foreign object image fusion features.

[0009] S3. Constructing and Training a Bayesian-Optimized Random Forest Classification and Recognition Model for Foreign Objects Hanging on Transmission Lines: The Bootstrap resampling method is used to randomly sample the fusion feature Z of the training set M times with replacement, generating M sample sets for training M CART decision trees, while ensuring that each sample set has the same number of samples. The fusion feature variable of the training set is defined as S. From these S feature variables, s feature variables are randomly selected to form a subset of the splitting feature variables for that node. The optimal splitting feature variable and optimal splitting value are selected based on the Gini coefficient, generating a random forest classification model containing M CART decision trees. Simultaneously, the hyperparameters of the random forest are optimized using Bayesian methods. The model is trained using 5x cross-validation, with the root mean square error (RMSE) as the control function to find the optimal hyperparameters.

[0010]

[0011] In the formula: n is the number of samples; y i This represents the true value of the i-th sample. For the i-th th Predicted values ​​for each sample;

[0012] S4. Classification and identification of foreign objects suspended on transmission lines: The fused feature X of the test set is used to test M CART decision trees in the random forest classification model to obtain the predicted categories {C1(X), C2(X), ..., C...}. m (X)}; A vote is taken on the classification results obtained from the test, and the result with the highest frequency among the classification results of the M CART decision trees is selected as the final classification result:

[0013]

[0014] In the formula: f RF (x) represents the classification result, f l (x) represents the classification result of the l-th CART decision tree for sample x, M is the number of CART decision trees, c is the number of foreign object classification categories, i is one of the c categories, and I(·) represents the case when the classification result is i.

[0015] Furthermore, in step S1, the foreign object image is augmented by combining Poisson fusion and Otsu's maximum inter-class variance thresholding method with morphological methods: the input foreign object image of size W×H is converted to grayscale, and the range of grayscale values ​​is G={0,1,2,…,L-1}, with the number of grayscale values ​​i (i∈G) set to n. i The probability of grayscale value i appearing is p i =n i / W×H; Set the initial threshold T to 0, classify the pixels in the gray value range [0, T] as the target region, and the pixels in the gray value range [T+1, L-1] as the background region. By changing the value of T, the inter-class variance between the target region and the background region is maximized, and the optimal segmentation threshold is obtained to achieve image segmentation. The segmented binary image is opened, eroded and then dilated to obtain connected regions and color different regions. The centroid and bounding rectangle of the largest connected region are calculated. Finally, the target region Z is extracted by cropping.

[0016] The target region Z is defined as the target scene, the real image of the transmission line is defined as the background scene K, and v is defined as the guiding field. The gradient fields of the target scene Z and the background scene K are calculated separately. The gradient fields of Z and K are then fused to obtain the gradient field of the fused scene. The minimum value between the gradient field of the fused scene and the guiding field is solved by constructing a Poisson equation with Dirichlet boundary conditions, thus completing the Poisson fusion process between the foreign object target region and the real background of the transmission line image. The dataset of images of foreign objects suspended on the transmission line is divided into training and testing sets in an 8:2 ratio.

[0017] Furthermore, in step S2, the structures of the five pre-trained network models are fine-tuned. The dimension of the fully connected layers of GoogLeNet, DarkNet-53, VGG-19, ResNet-50, and Xception is adjusted from 1000 to N (the number of heterogeneous species hanging on the transmission line), and the final classification output layer is adjusted to classify N (the number of heterogeneous species hanging on the transmission line). The five fine-tuned network models are retrained using the training set divided in S1, and the loss function is optimized using batch gradient descent to find the optimal parameters.

[0018] Furthermore, features were extracted from the image dataset of foreign objects suspended on transmission lines using five retrained convolutional neural network models. The Grad-CAM++ feature visualization algorithm was used to determine that the global average pooling layers of GoogLeNet, DarkNet-53, ResNet-50, and Xception, and the last dropout layer of VGG-19, were used as feature extraction layers. The features extracted by GoogLeNet, DarkNet-53, VGG-19, ResNet-50, and Xception were named F... G F D F V F R F X After analyzing the classification decision regions of each convolutional neural network, the final decision was made to use formula (2) to classify F. G F D and F E Cascade fusion is performed to obtain the optimal foreign object image fusion feature F.

[0019] F = concatenate(F) G ,F D ,F X (3)

[0020] Compared with the prior art, the present invention has the following advantages:

[0021] By combining Poisson fusion and Otsu's maximum inter-class variance thresholding method with morphological methods to augment foreign object images, multiple convolutional neural network models and the Grad-CAM++ feature visualization algorithm are constructed to jointly obtain the optimal fusion features. A Bayesian-optimized random forest classification model is then used to classify and identify foreign objects hanging on transmission lines. The proposed technical solution offers good detection accuracy and speed, and is more suitable for practical applications, providing a reference for power grid maintenance personnel in clearing foreign objects from transmission lines. Attached Figure Description

[0022] Figure 1 This is a flowchart of the method of the present invention.

[0023] Figure 2 This is a structural diagram of the random forest classification model in this invention. Detailed Implementation

[0024] The present invention will be further described below with reference to the embodiments. It should be noted that the following embodiments are only used to further illustrate the present invention and should not be construed as limiting the scope of protection of the present invention. Some non-essential improvements and adjustments made by those skilled in the art based on the above-described invention are still within the scope of protection of the present invention.

[0025] A method for detecting suspended foreign objects on transmission lines based on multi-network feature fusion is illustrated in the flowchart below. Figure 1 As shown, it includes the following steps:

[0026] S1. Construct a dataset of images of foreign objects hanging on power transmission lines: For images of foreign objects hanging on power transmission lines taken by manual inspection and drone inspection of power grids, the Otus maximum inter-class variance threshold segmentation method and morphological methods are used to extract the target regions of foreign objects. The Poisson fusion algorithm is used to fuse the real background of the power transmission line with the target regions of foreign objects to generate virtual foreign object image augmentation image samples. A dataset of images of foreign objects hanging on power transmission lines containing N types is constructed and divided into training set and test set according to a certain ratio.

[0027] In this embodiment, images of foreign objects hanging on transmission lines captured during power grid inspections were collected. Four types of foreign objects—plastic, bird nests, kites, and balloons—were selected as the research objects, with 60 images for each type, totaling 240 image samples. The images were augmented using Poisson fusion, Otsu's maximum inter-class variance thresholding method, and morphological methods. The input images of size W×H were converted to grayscale, with the grayscale value range G = {0, 1, 2, ..., 255}, and the number of grayscale values ​​i (i∈G) set to n. i The probability of grayscale value i appearing is p i =n i / W×H; Set the initial threshold T to 0, classify the pixels in the gray value range [0, T] as the target region, and the pixels in the gray value range [T+1, L-1] as the background region. By changing the value of T, the inter-class variance between the target region and the background region is maximized, and the optimal segmentation threshold is obtained to achieve image segmentation. The segmented binary image is opened, eroded and then dilated to obtain connected regions and color different regions. The centroid and bounding rectangle of the largest connected region are calculated. Finally, the target region Z is extracted by cropping.

[0028] Let Z be the target scene, K be the real image of the transmission line as the background scene, and v be the guiding field. The gradient fields of the target scene Z and the background scene K are calculated separately. The gradient fields of Z and K are then fused to obtain the gradient field of the fused scene. A Poisson equation with Dirichlet boundary conditions is constructed to solve for the minimum value between the gradient field of the fused scene and the guiding field, completing the Poisson fusion process between the foreign object target area and the real background of the transmission line image. The generated virtual foreign object images are used to expand the existing image samples, constructing a dataset containing images of four types of foreign objects suspended on transmission lines, with 100 images for each type, totaling 400 image samples. These are randomly divided into training and test sets in an 8:2 ratio; therefore, the training set contains 320 samples, and the test set contains 80 samples.

[0029] S2. Feature extraction from images of foreign objects suspended on transmission lines: Convolutional neural network models GoogLeNet, DarkNet-53, VGG-19, ResNet-50, and Xception were built and pre-trained using the large-scale dataset ImageNet. The structure of the five pre-trained network models was fine-tuned, and the five fine-tuned network models were retrained using the expanded training set of foreign object images.

[0030] Five retrained network models were used to extract features from the image dataset of foreign objects suspended on power transmission lines. The Grad-CAM++ algorithm was used to generate class activation heatmaps for the feature extraction layers of each network model, obtaining the decision regions of each feature layer. Based on this, the best feature extraction layer of each network model was selected. The decision regions of the best feature extraction layers were analyzed, and complementary features were selected for cascade fusion to obtain the best foreign object image fusion features.

[0031] In this embodiment, the large-scale open image dataset MS COCO is used to pre-train GoogLeNet, DarkNet-53, VGG-19, ResNet-50, and Xception. The structures of the five pre-trained network models are fine-tuned. The dimension of the fully connected layers of GoogLeNet, DarkNet-53, VGG-19, ResNet-50, and Xception is adjusted from 1000 to N (the number of foreign species hanging on the transmission line), and the final classification output layer is adjusted to classify N (the number of foreign species hanging on the transmission line). The five fine-tuned network models are retrained using the partitioned training set, and the loss function is optimized using batch gradient descent with a batch size of 5.

[0032] Feature extraction was performed on a dataset of images of foreign objects suspended on power transmission lines using five retrained convolutional neural network models. The Grad-CAM++ feature visualization algorithm was used to determine that the global average pooling layers of GoogLeNet, DarkNet-53, ResNet-50, and Xception, and the last dropout layer of VGG-19, were used as feature extraction layers. The features extracted by GoogLeNet, DarkNet-53, VGG-19, ResNet-50, and Xception were named F... G F D F V F R F X After analyzing the classification decision regions of each convolutional neural network, the final decision was made to use formula (1) to classify F. G FD and F X Cascade fusion is performed to obtain the optimal foreign object image fusion feature F.

[0033] F = concatenate(F) G ,F D ,F X (1)

[0034] S3. Constructing and Training a Random Forest-Based Classification and Recognition Model for Foreign Objects Hanging on Transmission Lines: The Bootstrap resampling method is used to randomly sample the fusion feature Z of the training set M times with replacement, generating M sample sets for training M CART decision trees, while ensuring that each sample set has the same number of samples. The fusion feature variable of the training set is defined as S. From these S feature variables, s feature variables are randomly selected to form the splitting feature variable subset for that node. The optimal splitting feature variable and optimal splitting value are selected based on the Gini coefficient, generating a random forest classification model containing M CART decision trees. Bayesian optimization of the random forest's hyperparameters is used, and the model is trained using 5x cross-validation. The root mean square error (RMSE) is used as the control function to find the optimal hyperparameters.

[0035]

[0036] In the formula: n is the number of samples; y i This represents the true value of the i-th sample. For the i-th th The predicted values ​​for each sample; the model parameters are as follows: the number of decision trees is 100;

[0037] The random forest classification model constructed in this embodiment is as follows: Figure 2 As shown, firstly, the fusion feature Z of the training set is sampled 100 times with replacement using the Bootstrap resampling method to generate 100 sample sets for training 100 CART decision trees. The fusion feature variables of the training set are defined as 320. 17 feature variables are randomly selected from the 320 feature variables to form the split feature variable subset of the node. The optimal split feature variable and the optimal split value are selected according to the Gini coefficient to generate a random forest classification model containing 100 CART decision trees.

[0038] S4. Classification and identification of foreign objects suspended on transmission lines: The fused feature X of the test set is used to test 100 CART decision trees in the random forest classification model to obtain the predicted categories {C1(X), C2(X), ..., C...}. 100 (X)}; A vote is taken on the classification results obtained from the test, and the classification result with the highest frequency among the 100 CART decision tree classification results is selected as the final classification result:

[0039]

[0040] In the formula: f RF (x) represents the classification result, f l (x) represents the classification result of the l-th CART decision tree for sample x, M is the number of CART decision trees, c is the number of foreign object classification categories, i is one of the c categories, and I(·) represents the case when the classification result is i.

[0041] In this embodiment, the optimal fusion features of the training and test sets are used to train and test the random forest classification model, respectively, to achieve the classification and identification of foreign objects hanging on transmission lines. Accuracy is used to evaluate its classification performance. The results are shown in Table 1.

[0042] Table 1 Performance indicators of different feature fusion schemes

[0043]

[0044] As shown in Table 1, the results of various fusion schemes indicate that fusing the foreign object features extracted by GoogLeNet, DarkNet-53, and Xception is the best fusion scheme, achieving an optimal accuracy of 95.98%. This verifies that the classification model proposed in this invention can accurately identify foreign objects hanging on transmission lines, providing a reference for power grid maintenance personnel to carry out the work of clearing foreign objects from transmission lines.

[0045] The above description merely illustrates preferred embodiments of the present invention and is not intended to limit the invention in any other way. Any person skilled in the art may make modifications or alterations to the above-disclosed content to create equivalent embodiments. However, any simple modifications, equivalent changes, and alterations made to the above embodiments based on the technical essence of the present invention, without departing from the scope of the present invention, shall still fall within the protection scope of the present invention.

Claims

1. A method for detecting suspended foreign objects on transmission lines based on multi-network feature fusion, characterized in that, Includes the following steps: S1. Constructing a dataset of images of foreign objects suspended on power transmission lines: Based on images of foreign objects suspended on power transmission lines captured by manual and drone inspections, the Otus maximum inter-class variance thresholding method and morphological methods are used to extract the target regions of the foreign objects. The Poisson fusion algorithm is then applied to fuse the real background of the transmission lines with the target regions of the foreign objects, generating virtual image augmentation samples. This dataset is used to construct a dataset containing... N A dataset of images of foreign objects hanging on power transmission lines was created and divided into training and testing sets according to a certain ratio. S2. Feature extraction from images of foreign objects suspended on transmission lines: Convolutional neural network models GoogLeNet, DarkNet-53, VGG-19, ResNet-50, and Xception were built and pre-trained using the large-scale dataset MS COCO. The structure of the five pre-trained network models was fine-tuned, and the five fine-tuned network models were retrained using the expanded training set of foreign object images. Five retrained network models were used to extract features from a dataset of images of foreign objects suspended on power transmission lines. The Grad-CAM++ algorithm was used to generate class activation heatmaps for the feature extraction layers of each network model, obtaining the decision regions of each feature layer. Based on this, the best feature extraction layer of each network model was selected. The decision regions of the best feature extraction layers were analyzed, and complementary features were selected for cascade fusion to obtain the best foreign object image fusion features. S3. Constructing and training a Bayesian optimization-based random forest model for classifying and identifying foreign objects suspended on transmission lines: The Bootstrap resampling method is used to fuse features from the training set. Z conduct M Generating a random sample with replacement M A sample set is used for training. M Use CART decision trees, while ensuring that the number of samples in each sample set is the same; define the fusion feature variable of the training set as... S Randomly in S Select from the feature variables s A subset of splitting feature variables is formed by several feature variables. The optimal splitting feature variable and the optimal splitting value are selected based on the Gini coefficient to generate a set containing... M A random forest classification model based on CART decision trees was developed, and Bayesian optimization of the hyperparameters of the random forest was used. The model was trained using 5x cross-validation, and the root mean square error (RMSE) was used as the control function to find the optimal hyperparameters. (1) In the formula: n The number of samples; For the first i The true value of each sample; For the first Predicted values ​​for each sample; S4. Classification and identification of foreign objects suspended on transmission lines: utilizing the fusion features of the test set. X In the random forest classification model M Test with CART decision trees to obtain predicted categories ; The classification results obtained from the test are voted on to select the correct one. M The classification result with the highest frequency among the CART decision tree classification results is taken as the final classification result: In the formula: For the classification results, For the first l CART decision tree for samples x The classification results M The number of CART decision trees, c For the number of foreign object classification categories, i for c One of the classes, The representative classification result is i The situation at that time; Specifically, step S2 includes: Using the Grad-CAM++ feature visualization algorithm, the global average pooling layers of GoogLeNet, DarkNet-53, ResNet-50, and Xception, and the last dropout layer of VGG-19 were selected as feature extraction layers. The features extracted by GoogLeNet, DarkNet-53, VGG-19, ResNet-50, and Xception were named as follows: F G , F D , F V , F R , F X After analyzing the classification decision regions of each convolutional neural network, the final decision was made using formula (3). F G , F D and F X Perform cascaded fusion to obtain the optimal foreign object image fusion features. F ; (3)。 2. The method for detecting suspended foreign objects on transmission lines based on multi-network feature fusion according to claim 1, characterized in that, In step S1, the foreign object image is augmented by combining Poisson fusion and Otsu's maximum inter-class variance thresholding method with morphological methods: the input size is... W × H The image of the foreign object is converted to grayscale, and the range of grayscale values ​​is G={0, 1, 2, ..., L -1}, grayscale value The number of items is set to grayscale value i The probability of occurrence is Set initial threshold T If it is 0, then [0, T Pixels within the grayscale range are classified as the target region. T +1, L Pixels in the grayscale range of -1] are classified as background areas, and this can be achieved by changing... T The value of maximizes the inter-class variance between the target and background regions, thus determining the optimal segmentation threshold for image segmentation. The segmented binary image is then subjected to an opening operation, followed by erosion and dilation to obtain connected regions. Different regions are then colored, and the centroid and bounding rectangle of the largest connected region are calculated. Finally, the target region is extracted through cropping. Z ; Set the target area to extract Z The target scene is a real image of a power transmission line, which serves as the background scene. K , v As a guiding field, calculate the target scene separately. Z and background scene K The gradient field will Z gradient field and K The gradient fields of the two scenes are fused to obtain the gradient field of the fused scene. By constructing a Poisson equation with Dirichlet boundary conditions and solving for the minimum value between the gradient field and the guiding field of the fused scene, the Poisson fusion process of the foreign object target area and the real background of the transmission line real image is completed.

3. The method for detecting suspended foreign objects on transmission lines based on multi-network feature fusion according to claim 1, characterized in that, In step S2, the structures of the five pre-trained network models are fine-tuned, and the dimension of the fully connected layers in GoogLeNet, DarkNet-53, VGG-19, ResNet-50, and Xception is adjusted from 1000 to [missing value]. N And adjust the final classification output layer to N Classification; using the training set divided in S1, the five finely tuned network models are retrained, and the loss function is optimized using batch gradient descent to find the optimal parameters of the network models.

Images