Visible light image inspection identification method for power transmission line hardware fitting based on deep network

Abstract

The invention discloses a deep network-based visible light image inspection identification method for power transmission line hardware fittings, which introduces an innovative unmanned aerial vehicle inspection mechanism, realizes automatic feature extraction of targets in images by capturing visible light images of key equipment of a power transmission line and depending on an improved Faster RCNN algorithm, and improves the detection accuracy of the target features. And finally, intelligent identification of the visible light inspection image of the power transmission unmanned aerial vehicle is realized. The visible light image inspection recognition method for the power transmission line hardware fitting based on the deep network is suitable for hardware fitting picture detection with a poor shooting effect, is not easily influenced by a shooting background, light and noise, is high in robustness, and remarkably improves the accuracy of hardware fitting detection in a complex scene.

Description

Technical Field

[0001] The present invention relates to the field of intelligent detection technology, and in particular to a visible light image inspection and recognition method for power transmission line fittings based on a deep network. Background Art

[0002] Electricity is the cornerstone of modern society, covering industry, agriculture, transportation and national defense, and is a necessity for people's daily lives. It is as important as food, air and water, supporting modern civilization. The power industry is the foundation of the national economy, and its development level is a symbol of national economic prosperity. The per capita electricity consumption reflects the level of modern life. Power grid line inspection is the core work to ensure the safe operation of transmission lines and their equipment. Through careful inspection, we can grasp the operating status of the line and the changes in its environment, and timely detect the defects and potential safety risks of the equipment. The purpose of this work is to ensure the reliability of the transmission and distribution lines and maintain the stability of the power system.

[0003] Transmission equipment faces harsh environments such as exposure, acid rain, freezing and typhoons, which can easily lead to failures and affect the economy and safety. To prevent failures, damaged equipment needs to be inspected and replaced regularly. Traditional inspection methods such as manual and manned helicopters have problems such as high cost, low efficiency or difficulty in operation. Drone inspections are popular because of their low cost, no blind spots and low risks, and can effectively improve the efficiency of power grid inspections. However, in existing technologies, the detection of transmission line hardware has the following problems: it is easily affected by the shooting background, light and noise, and the robustness is not strong, which brings challenges to the detection and positioning of hardware. Therefore, it is of great practical significance to study a reliable transmission line hardware target detection technology. Summary of the invention

[0004] The purpose of the present invention is to provide a visible light image inspection and recognition method for transmission line fittings based on a deep network to solve the above defects.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] The visible light image inspection and recognition method of transmission line fittings based on deep network includes the following steps:

[0007] S1. UAV inspection to obtain visible light images of hardware on the transmission line;

[0008] S2. Preprocess and annotate the hardware images obtained by the UAV inspection, and use the annotated hardware images as a data set;

[0009] S3. Improve the Faster R-CNN network model and use Inception-ResNetV2 instead of VGG for feature extraction;

[0010] S4, the feature map is input into the RPN network to generate a candidate region, and after the RPN generates the candidate region, BAM is inserted to enhance the attention of the candidate region;

[0011] S5. Use the improved network model for training to obtain a trained hardware detection model for the transmission line;

[0012] S6. Use the trained detection model to detect the visible light images of transmission line fittings obtained by drone inspection and output the detection results.

[0013] Preferably, in step S2, the preprocessing includes unifying the size of the hardware images obtained by the drone inspection into a size of 299*299*3 to facilitate subsequent training.

[0014] Preferably, in step S3, the Inception-ResNetV2 network model performs feature extraction through a combination of an Inception module and a residual connection; the Inception module captures features of different scales through multi-size convolution kernels to improve feature diversity; the residual connection effectively transfers gradients by adding direct jump connections in each Inception module, alleviates gradient vanishing, simplifies the training process, accelerates convergence, and avoids overfitting.

[0015] Preferably, the step S4 is specifically as follows:

[0016] The BAM attention mechanism enables the model to focus on more important feature areas by modeling the channel and spatial dimension information in the feature map, thereby achieving precise enhancement of the candidate areas; in the channel dimension, BAM uses global average pooling and fully connected layers to capture the dependencies between different channels and generate a channel attention weight vector; in the spatial dimension, BAM uses convolution operations to capture local spatial information and generate a spatial attention weight matrix.

[0017] Preferably, in step S4, for a given input feature map BAM infers a 3D attention map The calculation formula of the refined feature map F′ is as follows:

[0018]

[0019] Where M(F) represents the attention map; represents element-wise multiplication; F represents the input feature map; F′ represents the feature map.

[0020] A residual learning scheme and an attention mechanism are used to promote gradient flow. First, the channel attention of two independent branches is calculated. and spatial attention Then the attention map M(F) is calculated, and its calculation formula is:

[0021] M(F)=σ(M c (F))+M s (F),

[0022] Where σ represents the sigmoid function.

[0023] Preferably, in step S5, the hardware detection model includes an Inception-ResNetV2 feature extraction network, a region extraction network RPN, a BAM attention mechanism, a ROIPooling layer and a fully connected layer; the fully connected layer is used for classification and regression calculations.

[0024] Preferably, in step S5, the improved network model is used for training. During the model training process, in order to optimize its performance for the target detection task, a loss function is needed to measure the error between the model prediction and the actual result. The loss function includes the classification loss L cls and the regression loss L reg Two parts; classification loss L cls , used to determine whether the category of the anchor box is a positive sample or a negative sample, and the regression loss L reg , is used for bounding box regression training.

[0025] Preferably, the loss function, the total loss L is calculated as follows:

[0026]

[0027]

[0028]

[0029]

[0030] Where i is the index of the anchor box, which can include the region proposal box and the real target box for target detection; p i represents the probability that the i-th anchor box predicted by the model is a positive class, and It represents the true classification label corresponding to the anchor box; t i Represents the position parameter of the i-th anchor box predicted by the model; Represents the position parameters of the true target box corresponding to the i-th anchor box; x, y represent the horizontal and vertical coordinates of the center of the anchor box or the target center box; w, h represent the width and height of the anchor box or the target box.

[0031] Preferably, in step S6, the visible light images of transmission line fittings obtained by drone inspection are detected, and the detection process includes: inputting a picture to be detected, and using the Inception-ResNetV2 network to extract features of the input image; using the RPN network on the extracted feature map to generate candidate detection areas; introducing the BAM attention mechanism to enhance the attention of the generated candidate areas; the BAM-enhanced candidate areas are sent to the classifier and bounding box regressor to classify the target and accurately locate the position; and finally outputting the detection results.

[0032] The beneficial effects of the present invention are:

[0033] The invention discloses a visible light image inspection and recognition method for transmission line fittings based on a deep network. In the feature extraction stage, a deeper feature extraction network Inception-ResNetV2 is used to extract features from the data set. The Inception-ResNetV2 network combines the advantages of the Inception structure and the residual network, and can significantly improve the feature representation capability while maintaining the network depth. The BAM attention mechanism is used to enhance the candidate areas generated by the RPN network, improve the model's attention to the target object, and suppress background information. The method of the invention is suitable for detecting fittings with poor shooting effects, is not easily affected by the shooting background, light, and noise, has strong robustness, and significantly improves the accuracy of fitting detection in complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 : Flow chart of the method of the present invention;

[0035] Figure 2 : The improved Faster R-CNN network model structure diagram of the method of the present invention;

[0036] Figure 3 : Schematic diagram of the Inception-ResNetV2 feature extraction network of the method of the present invention;

[0037] Figure 4 : Diagram of the BAM attention mechanism of the method of the present invention. DETAILED DESCRIPTION

[0038] The present invention is further described below in conjunction with the embodiments. It should be noted that these are merely examples and illustrations of the concept of the present invention. Those skilled in the art may make various modifications or additions to the described specific embodiments or replace them in a similar manner. As long as they do not deviate from the concept of the invention or exceed the scope defined by the claims, they should be deemed to fall within the protection scope of the present invention.

[0039] Embodiment 1:

[0040] Visible light image inspection and recognition method for transmission line fittings based on deep network, such as Figure 1 As shown, the following steps are included:

[0041] S1. Use drone inspection to obtain visible light images of hardware on transmission lines.

[0042] Operate drones to inspect transmission lines, capture visible light images of major power equipment, and obtain a large number of transmission line inspection image samples.

[0043] S2. Preprocess and annotate the hardware images obtained by drone inspection, and use the annotated hardware images as data sets.

[0044] Among them, preprocessing includes resizing the hardware images obtained by drone inspection to make the image size uniform to 299*299*3. This can reduce the impact on data images with poor shooting effects and meet the input of the feature extraction network for subsequent training. After collecting the images of the transmission line inspection, the LabelImg data annotation software is used to annotate the data set.

[0045] S3. Improve the Faster R-CNN network model and use Inception-ResNetV2 instead of VGG for feature extraction. Through a deeper structure, it can capture multi-level and multi-scale feature information in the image.

[0046] The improved Faster R-CNN network model structure diagram is as follows Figure 2 As shown in the figure, the convolutional layer of the Faster R-CNN network model uses the vgg16 network, and the Inception-ResNetV2 is used as the feature extraction network in the convolutional layer.

[0047] Inception-ResNetV2 feature extraction network diagram, such as Figure 3As shown. The Inception-ResNetV2 network model, through the combination of Inception modules and residual connections, performs feature extraction, which can significantly improve the feature representation capability while maintaining the network depth. The network model includes stem, Inception-resnet1-A, Inception-resnet1-B and Inception-resnet1-C modules. The network input is 299*299*, and then it will perform three convolutions, maximum pooling, two convolutions, maximum pooling, four branches, and stacking; the structure in Inception-resnet1-A, Inception-resnet1-B and Inception-resnet1-C is divided into four branches, and different modules have different convolution processing.

[0048] Among them, the Inception module processes input data in parallel through multiple convolution kernels of different sizes, effectively capturing feature information of different scales, thereby improving the diversity and richness of feature extraction. The residual connection, by adding direct jump connections in each Inception module, enables the network to transfer gradients more effectively, alleviating the common gradient vanishing problem in deep networks. In addition, the residual connection simplifies the network training process, accelerates the convergence speed, and avoids the risk of overfitting due to the increase in network depth.

[0049] S4. The feature map is input into the RPN network to generate a candidate region. After RPN generates the candidate region, BAM is inserted to enhance the attention of the candidate region.

[0050] The RPN network predicts a large number of candidate detection regions, each of which is equipped with a classification score and a bounding box regression value.

[0051] The introduced BAM attention mechanism diagram, such as Figure 4 As shown in Figure 2, the BAM attention mechanism performs weighted processing on the feature map by modeling the channel and spatial dimension information in the feature map, so that the model can focus on more important feature areas while suppressing irrelevant background information, thereby achieving accurate enhancement of the candidate area.

[0052] In the channel dimension, BAM uses global average pooling and fully connected layers to capture the dependencies between different channels and generate a channel attention weight vector that can highlight channel features that contribute more to target detection. The BAM module introduces a channel attention mechanism that can adaptively strengthen or weaken the feature responses of different channels, thereby improving the performance of the model.

[0053] In the spatial dimension, BAM uses convolution operations to capture local spatial information, focusing on key areas related to the target object in the image and generating a spatial attention weight matrix.

[0054] Through these two aspects of optimization, the model can more accurately capture the features of the target object when processing the candidate area, while reducing the influence of irrelevant background information, thereby significantly improving the accuracy and robustness of target detection.

[0055] For a given input feature map BAM infers a 3D attention map The calculation formula of the refined feature map F′ is as follows:

[0056]

[0057] Where M(F) represents the attention map, represents element-wise multiplication, F represents the input feature map, and F′ represents the feature map.

[0058] A residual learning scheme and an attention mechanism are used to promote gradient flow. First, the channel attention of two independent branches is calculated. and spatial attention Then the attention map M(F) is calculated, and its calculation formula is:

[0059] M(F)=σ(M c (F))+M s (F),

[0060] Where σ represents the sigmoid function.

[0061] The candidate regions enhanced by BAM are sent to the classifier for target classification. The classifier determines whether each candidate region contains a specific target object and gives a corresponding classification score. At the same time, the candidate region is sent to the bounding box regressor for accurate positioning. The bounding box regressor predicts the precise bounding box of each target object. When the classifier predicts that multiple candidate regions belong to the same category, non-maximum suppression (NMS) is used to eliminate candidate regions with high overlap and retain one or more candidate regions with the highest confidence.

[0062] S5. Use the improved network model for training to obtain a trained hardware detection model for the transmission line.

[0063] Among them, the hardware detection model includes Inception-ResNetV2 feature extraction network, region extraction network RPN, BAM attention mechanism, ROIPooling layer and fully connected layer; the fully connected layer is used for classification and regression calculation.

[0064] The improved network model is used for training. During the model training process, in order to optimize its performance for target detection tasks, a loss function is needed to measure the error between the model prediction and the actual result.

[0065] Loss function, including classification loss L cls and the regression loss L reg Two parts; classification loss L cls , used to determine whether the category of the anchor box is a positive sample or a negative sample, and the regression loss L reg , is used for bounding box regression training.

[0066] The loss function, the total loss L calculation formula is as follows:

[0067]

[0068]

[0069]

[0070]

[0071] Where i is the index of the anchor box, which can include the region proposal box and the real target box for target detection; p i represents the probability that the i-th anchor box predicted by the model is a positive class, and It represents the true classification label corresponding to the anchor box; t i Represents the position parameter of the i-th anchor box predicted by the model; Represents the position parameters of the true target box corresponding to the i-th anchor box; x, y represent the horizontal and vertical coordinates of the center of the anchor box or the target center box; w, h represent the width and height of the anchor box or the target box.

[0072] S6. Use the trained detection model to detect the visible light images of transmission line fittings obtained by drone inspection and output the detection results.

[0073] The visible light images of transmission line fittings obtained by drone inspection are detected, and the detection process includes: inputting the picture to be detected, and using the Inception-ResNetV2 network to extract features of the input image; using the RPN network on the extracted feature map to generate candidate detection areas; introducing the BAM attention mechanism to enhance the attention of the generated candidate areas; the BAM-enhanced candidate areas are sent to the classifier and bounding box regressor to classify the target and accurately locate the position; and finally outputting the detection results, which include the classification label, bounding box position and corresponding confidence score of each target object.

[0074] The invention discloses a visible light image inspection and recognition method for power transmission line fittings based on a deep network. In the feature extraction stage, a deeper feature extraction network Inception-ResNetV2 is used to extract features of the data set. The Inception-ResNetV2 network combines the advantages of the Inception structure and the residual network, and can significantly improve the feature representation capability while maintaining the network depth. The BAM attention mechanism is used to enhance the candidate areas generated by the RPN network, improve the model's attention to the target object, and suppress background information.

[0075] The method of the present invention is suitable for detecting hardware images with poor shooting effects, is not easily affected by shooting background, light and noise, has strong robustness, and significantly improves the accuracy of hardware detection in complex scenes.

[0076] The above is an exemplary description of the invention. Obviously, the specific implementation of the present invention is not limited to the above-mentioned method. As long as such non-substantial improvements are made by adopting the method concept and technical solution of the present invention, or the concept and technical solution of the invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A visible light image inspection and recognition method for transmission line fittings based on deep network, characterized in that: The following steps are involved: S1. UAV inspection obtains visible light images of hardware on the transmission line; S2. Preprocess and label the hardware images obtained by the UAV inspection, and use the labeled hardware images as a data set; S3. Improve the Faster R-CNN network model and use Inception-ResNetV2 instead of VGG for feature extraction; S4, the feature map is input into the RPN network to generate a candidate region, and after the RPN generates the candidate region, BAM is inserted to enhance the attention of the candidate region; S5. Use the improved network model for training to obtain a trained hardware detection model for the transmission line; S6. Use the trained detection model to detect the visible light images of transmission line fittings obtained by drone inspection and output the detection results.

2. According to claim 1, the visible light image inspection and recognition method of power transmission line fittings based on deep network is characterized in that: In step S2, the preprocessing includes unifying the size of the hardware images obtained by the drone inspection to 299*299*3 for subsequent training.

3. According to claim 1, the visible light image inspection and recognition method of power transmission line fittings based on deep network is characterized in that: In step S3, the Inception-ResNetV2 network model performs feature extraction by combining the Inception module and the residual connection; the Inception module captures features of different scales through multi-size convolution kernels to improve feature diversity; The residual connection effectively transfers gradients by adding direct skip connections in each Inception module, alleviates gradient vanishing, simplifies the training process, accelerates convergence, and avoids overfitting.

4. The visible light image inspection and recognition method for power line fittings based on deep network according to claim 1 is characterized in that: The S4 step is specifically as follows: The BAM attention mechanism enables the model to focus on more important feature areas by modeling the channel and spatial dimension information in the feature map, thereby achieving precise enhancement of the candidate areas; in the channel dimension, BAM uses global average pooling and fully connected layers to capture the dependencies between different channels and generate a channel attention weight vector; in the spatial dimension, BAM uses convolution operations to capture local spatial information and generate a spatial attention weight matrix.

5. The visible light image inspection and recognition method for power line fittings based on deep network according to claim 1 or 4, characterized in that: In step S4, for a given input feature map BAM infers a 3D attention map The calculation formula of the refined feature map F′ is as follows: Where M(F) represents the attention map, represents element-wise multiplication, F represents the input feature map, and F′ represents the feature map. A residual learning scheme and an attention mechanism are used to promote gradient flow. First, the channel attention of two independent branches is calculated. and spatial attention Then the attention map M(F) is calculated, and its calculation formula is: M(F)=σ(M c (F))+M s (F), Where σ represents the sigmoid function.

6. The visible light image inspection and recognition method for power transmission line fittings based on deep network according to claim 1 is characterized in that: In step S5, the hardware detection model includes an Inception-ResNetV2 feature extraction network, a region extraction network RPN, a BAM attention mechanism, a ROIPooling layer and a fully connected layer; the fully connected layer is used for classification and regression calculations.

7. The method for inspection and identification of transmission line fittings based on visible light images based on deep network according to claim 1 is characterized in that: In step S5, the improved network model is used for training. During the model training process, in order to optimize its performance for the target detection task, a loss function is needed to measure the error between the model prediction and the actual result. The loss function includes the classification loss L cls and the regression loss L reg Two parts; classification loss L cls , used to determine whether the category of the anchor box is a positive sample or a negative sample, and the regression loss L reg , is used for bounding box regression training.

8. The method for inspection and identification of transmission line fittings based on visible light images based on deep network according to claim 7 is characterized in that: The loss function, the total loss L calculation formula is as follows: Where i is the index of the anchor box, which can include the region proposal box and the real target box for target detection; p i represents the probability that the i-th anchor box predicted by the model is a positive class, and It represents the true classification label corresponding to the anchor box; t i Represents the position parameter of the i-th anchor box predicted by the model; Represents the position parameters of the true target box corresponding to the i-th anchor box; x, y represent the horizontal and vertical coordinates of the center of the anchor box or the target center box; w, h represent the width and height of the anchor box or the target box.

9. The method for inspection and identification of transmission line fittings based on visible light images based on deep network according to claim 1 is characterized in that: In step S6, the visible light image of the transmission line fittings obtained by the drone inspection is detected, and the detection process includes: Input the image to be detected, use the Inception-ResNetV2 network to extract features of the input image; use the RPN network on the extracted feature map to generate candidate detection areas; introduce the BAM attention mechanism to enhance the attention of the generated candidate areas; send the BAM-enhanced candidate areas to the classifier and bounding box regressor to classify the target and accurately locate the position; finally output the detection results.

Images