An intrusion target detection model and its detection method in power transmission line inspection

Abstract

An intrusion target detection model and its application in power transmission line inspection belong to the field of target detection under deep learning. It comprises three modules: a backbone network, a feature pyramid, and a cascaded region proposal network (RPN). The backbone network is implemented using a multi-layer MobileNetV2 architecture, extracting features from the input image to obtain multi-layer feature maps. These features are fused through the feature pyramid and then input into the cascaded region proposal network. Feature fusion involves upsampling the feature maps from higher layers of the MobileNetV2 architecture and then adding them element-wise to the feature maps of lower layers. The cascaded region proposal network consists of a region proposal network and N cascaded detection heads, each representing a level. The region proposal network generates several candidate boxes that may be intrusion targets based on the fused features. The detection is then refined starting from the first level by progressively increasing the IoU threshold. Finally, the Nth detection head outputs the intrusion target detection classification and bounding box regression results.

Description

Technical Field

[0001] This invention belongs to the field of target detection technology under deep learning, specifically involving a detection model and method for detecting and outlining intrusion targets such as tower cranes, cranes, pile drivers, construction trucks and excavators in power transmission line scenarios. Background Technology

[0002] With the continuous development of the State Grid, the number of power transmission facilities such as transmission poles and transmission lines is increasing daily, and their interconnections are becoming increasingly close. A problem in any facility can lead to a large-scale power outage in the vicinity, resulting in severe economic losses. Therefore, real-time monitoring of transmission line status is necessary to ensure their safe and stable operation. However, transmission lines are frequently intruded upon by construction equipment such as excavators and pile drivers, as well as transportation equipment such as engineering trucks, causing transmission lines to break or transmission poles to collapse. Such intrusion incidents not only cause significant economic losses but also trigger numerous electric shock accidents, endangering the lives of pedestrians and power grid workers. Therefore, research on intrusion target detection for transmission lines is essential. It is not only a fundamental component of routine power network inspections but can also assist in subsequent early warning work regarding whether intrusion targets are approaching transmission lines. Currently, intrusion target detection mainly relies on regular manual inspections, a method that is unsuitable for the increasing number of transmission facilities and suffers from low efficiency and poor real-time performance.

[0003] To address the aforementioned issues, some studies have proposed using deep learning-related methods to detect intrusion targets near power transmission lines. Deep learning-based object detection methods typically extract image features through a backbone network, then fuse and enhance these features using a feature pyramid module, finally outputting the coordinates of the target bounding box (i.e., the coordinates of the top-left and bottom-right vertices) and its category. Generally, object detection algorithms combine a series of pre-set prior boxes, predicting offsets and categories for each prior box to arrive at the final result. These object detection methods can be categorized into one-stage and two-stage methods. One-stage methods, such as SSD, YOLOv3, and RetinaNet, directly predict prior boxes without filtering. Two-stage methods additionally use a region proposal network to filter out prior boxes that are more likely to be targets. For example, Faster R-CNN first extracts features through a backbone network and feature pyramid, then uses a region proposal network to determine whether the prior box is an object or background. If it is background, the bounding box is refined through regression to obtain candidate boxes. For intrusion targets near power transmission lines, existing methods mainly achieve intrusion target detection by fine-tuning mature target detection methods such as SSD, YOLOv3, and Faster R-CNN.

[0004] Although the above methods have made significant progress compared to manual periodic inspections, the following problems still exist: the number of intrusion target types that can be detected is relatively small; there are relatively dense or small-scale intrusion targets in the power transmission line scenario, and the above methods are not accurate in locating these targets, and the false negative rate is high; the detection effect drops significantly when the number of intrusion targets in the dataset is small; although the two-stage method based on Faster R-CNN can achieve good detection results, the detection speed is slow and far from meeting the real-time requirements. Summary of the Invention

[0005] This invention proposes an intrusion target detection model, comprising three modules: a backbone network, a feature pyramid, and a cascaded region proposal network. The backbone network is implemented using a multi-layer MobileNetV2 architecture, extracting features from the input image to obtain multi-layer feature maps. These features are fused through the feature pyramid and then input into the cascaded region proposal network. Feature fusion involves upsampling the feature maps from higher layers of the MobileNetV2 architecture and then adding them element-wise to the feature maps of lower layers. The cascaded region proposal network includes a region proposal network and N cascaded detection heads, each representing a level. The region proposal network generates several candidate bounding boxes that may be intrusion targets based on the fused features. The bounding boxes are then refined starting from the first level by progressively increasing the IoU threshold. Finally, the Nth detection head outputs the intrusion target detection classification and bounding box regression results. Preferably, N = 3.

[0006] Furthermore, the MobileNetV2 has 9 layers, from the lowest to the highest, namely conv1, layer-i and conv2, i = 1, 2, ..., 7. Among them, conv1 and conv2 are 3*3 convolutional layers with batch normalization and ReLU6 activation function, layer-i consists of inverted residual blocks, and the features are feature maps obtained from layer-2, layer-3, layer-5 and conv2.

[0007] Furthermore, the inverted residual block first performs a 1*1 convolution on the input to increase the number of channels, then performs a 3*3 grouped convolution, where the number of groups in the grouped convolution is the current number of channels, and finally performs a 1*1 convolution to decrease the number of channels.

[0008] Furthermore, there are two MobileNetV2 systems with the same structure but no shared parameters. One is an auxiliary backbone and the other is an output backbone. For each layer of the output backbone at or above layer 2, the features of each layer of the auxiliary backbone at or above layer 2 are added to the features of the output backbone.

[0009] Furthermore, the auxiliary backbone and the output backbone's layer-7 layer consists of only one inverted residual block A. After the 3*3 grouped convolution of the inverted residual block A, an SE module is inserted. The SE module first performs average pooling on the input feature map, then obtains the weight of each channel of the feature map through two fully connected operations and the Sigmoid function, and finally multiplies it with the feature map to obtain the weighted feature map.

[0010] Furthermore, during the fusion process, the feature map of the auxiliary backbone is upsampled so that the size of the feature map of the auxiliary backbone is the same as the size of the feature map of the output backbone; the feature map of the corresponding layer of the auxiliary backbone is also input into the feature pyramid that shares the parameters with the feature pyramid, and participates in the calculation of the loss function together with the output backbone.

[0011] Furthermore, the feature fusion process involves upsampling the feature maps of the higher layers of MobileNetV2 and then adding them element-wise to fuse them into the feature maps of the lower layers. Then, a second round of fusion is performed from the bottom up: the lower-layer feature maps are downsampled and added element-wise to the feature maps of the higher layers. The resulting feature maps are then subjected to a 3x3 convolution, and the output is then subjected to the same operation as the feature maps of the higher layers. In this final bottom-up fusion, both the downsampling convolution and the 3x3 convolution with a stride of 1 employ deformable convolutions.

[0012] Furthermore, the deformable convolution learns additional offsets and weights of each sampling point in the x and y directions on top of the traditional 3*3 convolution kernel, and performs convolution calculations using the new sampling points and weights.

[0013] Furthermore, max pooling is applied to the output feature map of the deformable convolution as another output of the feature pyramid.

[0014] In another aspect of the present invention, an intrusion target detection method for use in power transmission line inspection includes the following steps: acquiring power grid scene images and constructing a dataset; designing and training the intrusion target detection model and saving model parameters; inputting the real-time acquired power grid scene images into the intrusion target detection model with saved model parameters to obtain detection results.

[0015] Beneficial Effects: Power transmission lines are a crucial component of the national smart grid, playing a vital role in the national economy and people's lives. However, power transmission lines are vulnerable to intrusions by engineering vehicles, tower cranes, excavators, etc., and traditional inspection methods suffer from poor real-time performance and low accuracy, failing to adequately address these intrusion scenarios. This invention proposes a lightweight intrusion target detection model based on MobileNetV2 and its application in power transmission line inspection, utilizing a composite MobileNetV2 backbone network, a bottom-up combined with a top-down feature pyramid network, and a cascaded region proposal network. On a dataset of 8177 power grid scene images collected from surveillance cameras, a mAP-50 value of 74.60 was achieved, the highest among compared algorithms. Furthermore, this algorithm minimizes the number of parameters and computational cost, ensuring real-time performance. Attached Figure Description

[0016] Figure 1 This is an overall framework diagram of the present invention.

[0017] Figure 2 This is a structural diagram of the inverted residual block.

[0018] Figure 3 This is a structural diagram of the backbone network in this invention.

[0019] Figure 4 This is a structural diagram of the SE module.

[0020] Figure 5 It is a structural diagram of the traditional feature pyramid.

[0021] Figure 6 This is a structural diagram of the deformable convolution used in this invention.

[0022] Figure 7 This is a flowchart of the detection method of the intrusion target detection model of the present invention in the inspection of power transmission lines. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0024] The overall framework of the present invention is as follows Figure 1 As shown, an intrusion target detection model comprises three modules: a backbone network, a feature pyramid, and a cascaded region proposal network. First, features are extracted from the input image through the backbone network, and then these features are fused using the feature pyramid. Simultaneously, the cascaded region proposal network generates candidate boxes by progressively increasing the IoU threshold, and the final classification and bounding box regression results are output after passing through the detection head.

[0025] This invention uses MobileNetV2 as a crucial component of its backbone network. MobileNetV2 is a mainstream lightweight backbone network, consisting of nine layers, from lowest to highest: conv1, layer-i (i = 1, 2... 7), and conv2. conv1 and conv2 are 3x3 convolutional layers with batch normalization and ReLU6 activation. Layer-i (i = 1, 2... 7) comprises inverse residual blocks. In the inverse residual block, the input is first subjected to a 1x1 convolution to increase the number of channels, then to a 3x3 grouped convolution where the number of groups is the current number of channels (i.e., depthwise convolution), and finally, a 1x1 convolution is performed to decrease the number of channels.

[0026] Inverted residual block such as Figure 2 As shown, where BN represents batch normalization; for auxiliary backbone and output backbone, each layer-i (i = 1, 2... 7) contains m inverted residual blocks A and n inverted residual blocks B, 1 ≤ m + n ≤ 4, 0 ≤ n ≤ 1;

[0027] The number of input and output channels and the feature map dimensions of each layer of MobileNetV2 are shown in Table 1; the number of output channels of each inverted residual block in each layer-i (i = 1, 2... 7) is shown in the table.

[0028] Table 1. Number of input / output channels and feature map dimensions for each layer of MobileNetV2.

[0029]

[0030] The backbone network structure of the present invention is as follows: Figure 3 As shown. This invention uses two identical MobileNetV2 networks as backbone networks, one called the auxiliary backbone and the other the output backbone. The parameters of the two backbones are not shared. The connection lines between the auxiliary backbone and the output backbone represent upsampling operations and feature addition. For simplicity, only a portion of the connection lines are shown; hollow blocks represent deleted conv1 and layer-1. For each stage of the output backbone from layer-2 onwards, the features of each stage of the auxiliary backbone at or above this stage are added to the features of the output backbone. During the fusion process, the feature map of the auxiliary backbone is upsampled so that the feature map size of the auxiliary backbone is the same as that of the output backbone. In the layer-7 part of the auxiliary and output backbones, a squeeze-excitation (SE) module is additionally used. The SE module belongs to the attention mechanism module, and its structure is as follows. Figure 4As shown. Specifically, layer-7 consists of only one inverted residual block A. An SE module is inserted after a 3x3 depthwise convolution of inverted residual block A. The SE module first performs average pooling on the input feature map, then obtains the weights (i.e., attention values) of each channel of the feature map through fully connected operations and the Sigmoid function, and finally multiplies them with the feature map to obtain a weighted feature map. To further reduce the number of model parameters and computational cost, conv1 and layer1 of the output backbone are removed. Finally, the feature maps obtained from the layer2, layer3, layer5, and conv2 stages of the output backbone are input into the feature pyramid. Simultaneously, the feature maps of the corresponding stages of the auxiliary backbone are also input into the feature pyramid, which shares parameters with the aforementioned feature pyramid, and participate in the subsequent calculation of classification and localization losses along with the output backbone. The classification loss uses cross-entropy loss, and the localization loss uses smoothed L1 loss. The loss weight of the auxiliary backbone is 0.5 times that of the output backbone.

[0031] Figure 5 The traditional feature pyramid structure only contains a top-down architecture, where high-level feature maps are upsampled and fused to lower-level feature maps. To improve the fusion effect and achieve bidirectional interaction between high-level and low-level features, the feature pyramid of this invention is composed of... Figure 5 The feature map obtained after fusion in this way is then followed... Figure 1 The model employs a bottom-up fusion approach. The lower-level feature maps are downsampled and element-wise added to the higher-level feature maps. The resulting feature maps are then subjected to a 3x3 convolution, and the output is then processed in the same way as the higher-level feature maps. To further reduce the total number of parameters and computational cost, the model limits the number of output channels in the feature pyramid to 48. Therefore, before fusion, a 1x1 convolution kernel is used to reduce the number of input channels in each layer to 48. To improve downsampling accuracy, a 3x3 convolution with a stride of 2 is then used for downsampling.

[0032] Meanwhile, in the final bottom-up fusion, both the downsampling convolution and the 3x3 convolution with a stride of 1 employ deformable convolution, such as... Figure 1 As shown, deformable convolution can learn the offset of sampling points, and then select appropriate sampling points to obtain features, thereby increasing the receptive field of the model and obtaining higher-quality features. The offsets in the x and y directions learned by deformable convolution may be decimals, so bilinear interpolation is needed to finally determine the values ​​of the sampling points.

[0033] A schematic diagram of deformable convolution is shown below. Figure 6As shown. To avoid deformable convolution focusing excessively on useless feature points in the background, additional weights are learned for each sampling point. Based on the traditional 3x3 convolution kernel, deformable convolution additionally learns the offsets and weights of each sampling point in the x and y directions, and performs convolution calculations using the new sampling points and weights. To further utilize the features extracted by deformable convolution, max pooling is applied to the output feature map of deformable convolution, serving as the fifth output of the feature pyramid.

[0034] The cascaded region proposal network of this invention comprises a region proposal network (RPN) and three detection heads. First, the RPN generates several candidate boxes (RoIs). These candidate boxes are filtered by an IoU threshold and then processed by pooling and detection heads to output classified and refined candidate boxes. Subsequently, these candidate boxes are passed through cascaded detection heads with progressively increasing IoU thresholds to output the final classification and bounding box results. The thresholds for each filtering step are 0.5, 0.6, and 0.7.

[0035] Figure 7 The following is a flowchart of the detection method of the intrusion target detection model of the present invention in the inspection of power transmission lines.

[0036] 1. Acquire images of the power grid scene using surveillance cameras. After acquisition, use Python's OpenCV library to segment the video into frame images. Then, randomly divide the images into training and validation sets, placing them in two folders, train2017 and val2017, respectively.

[0037] 2. Use VGG Image Annotator software to annotate the images in the train and val folders respectively. First, click Add Files to load the images, then click Atributes and enter the superclass name, which can be set to intrusion. After entering the name, set the Type to radio. An id will then pop up; enter the English names of the intrusion targets in sequence as categories. In this invention, there are 5 categories of intrusion targets: engineering trucks, excavators, cranes, tower cranes, and pile drivers, corresponding to the English words trans, engine, lift, crane, and machine, respectively. Then, drag the mouse to outline the intrusion targets and classify them. Finally, after all images are annotated, click Export as COCO to save the annotations to a JSON file. Place both JSON files obtained above in the Annotations folder.

[0038] 3. Download the PyTorch deep learning framework and the mmdetection object detection framework. Write the backbone network class CMobile according to the backbone network structure of this invention. The backbone network class needs to specify BN as the normalization function and ReLU6 as the activation function, and specify that the feature maps of layer 2, layer 3, layer 5, and conv2 are output to the feature pyramid in the next step. Simultaneously, download the MobileNetV2 pre-trained model from mmdetection and specify that the backbone class loads this model. Finally, place the CMobile class file cmobile.py in the mmdet / models / backbones folder under the mmdetection framework, and modify the mmdet / models / backbones / __init__.py file, adding `from.cmobile import CMobile` to the first line and adding CMobile to the __all__ list.

[0039] 4. Write the feature pyramid class CFPN according to the feature pyramid structure of this invention. In the feature pyramid, specify the input feature map channel count as [24, 32, 96, 1280], the output count as 5, and the output channel count as 48. Place the CFPN class file cfpn.py in the mmdet / models / necks folder under the mmdetection framework, and modify the mmdet / models / necks / __init__.py file, adding `from.cfpn import CFPN` to the first line and adding CFPN to the __all__ list.

[0040] 5. Use `configs / _base_ / models / cascade_rcnn_r50_fpn.py` as a template to write the configuration file. Change the `type` of the backbone dictionary to `CMobile`, and keep the other settings consistent with the `__init__()` function in the `CMobile` class; change the `type` of the neck to `CFPN`, and keep the other settings consistent with the `__init__()` function in the `CFPN` class; refer to `configs / _base_ / schedules / schedule_2x.py` to set the learning rate and optimizer, and change the `lr` (learning rate) of the optimizer to 0.005, and set `grad_clip` (gradient clipping) of `optimizer_config` to `dict(max_norm=50, norm_type=2)`; refer to `configs / _base_ / schedules / coco_detection.py` to set the data set reading method, and specify `data_root` as the location where the data set is stored. Simultaneously, specify `int_class = ('trans', 'engine', 'lift', 'crane', 'machine')`, and specify `classes = int_class` under the `train`, `val`, and `test` sub-dictionaries of the `data` dictionary. Set the `samples_per_gpu` parameter according to the available server memory; generally, when `samples_per_gpu = 1`, it requires 7000–8000 MB of GPU memory. Finally, in the summarized `config.py` file, specify `work_dir` as the path to where the model files are to be saved, assuming it is `work_dirs / cm`.

[0041] 6. Run the command (assuming a server with a single graphics card): python tools / train.py config.py --gpus 1, and then start training.

[0042] 7. After training is complete, you can select the epoch with the highest mAP-50 value based on the logs, such as epoch_24, and run python-m tools / test.py config.py work_dirs / cm / epoch_24.pth --show --evalbbox --show_dir work_dirs / cm / visualize to obtain the final detection results with target boxes.

[0043] 8. To obtain detection results in real time, you can purchase a smart network camera and install it near the power grid. Then, ensure the camera and server are on the same network segment. Next, use the camera's provided SDK and its programming language to write a script. This script should include the following: access the camera via its IP address, causing the camera to periodically capture images and save them to a specific path on the server (e.g., / snapshots); call the command `python demo / image_demo.py snapshots / xxx.jpg epoch_24.pth --device cpu --out-file xxx_result.jpg`, where `xxx.jpg` is the name of the captured image, `xxx_result.jpg` is the name of the displayed image with the detection boxes, and `epoch_24.pth` is the model file with the highest mAP-50 obtained from training in step 7. The path to the `config.py` file from step 7 needs to be specified in the `image_demo.py` file.

Claims

1. A detection method for an intrusion target detection model in power transmission line inspection, characterized in that, Including the following steps: Acquire images of power grid scenes and construct a dataset; Design and train the intrusion target detection model, and save the model parameters; The intrusion target detection model, which stores model parameters, is input into the real-time acquired power grid scene image to obtain the detection result; The intrusion target detection model comprises three modules: a backbone network, a feature pyramid, and a cascaded region proposal network. The backbone network is implemented using a multi-layer MobileNetV2 architecture, extracting features from the input image to obtain multi-layer feature maps. These features are then fused through the feature pyramid and input into the cascaded region proposal network. Feature fusion involves upsampling the feature maps from higher layers of the multi-layer MobileNetV2 architecture and then adding them element-wise to the feature maps of lower layers. The cascaded region proposal network includes a region proposal network and... N The system consists of cascaded detection heads, each representing a level. The region proposal network generates several candidate boxes that may be intrusion targets based on the fused features. Then, starting from the first level, the network refines the detection by progressively increasing the IoU threshold. Finally, the system optimizes the detection by... N Each detection head outputs the results of intrusion target detection classification and bounding box regression; The MobileNetV2 has 9 layers, from the lowest to the highest: conv1, layer-... i and conv2, i =1,2,..,7, where conv1 and conv2 are 3x3 convolutional layers with batch normalization and ReLU6 activation function, layer- i It consists of inverted residual blocks, and the features are feature maps obtained from layer-2, layer-3, layer-5, and conv2 layers; The inverted residual block first performs a 1*1 convolution on the input to increase the number of channels, then performs a 3*3 grouped convolution, where the number of groups in the grouped convolution is the current number of channels, and finally performs a 1*1 convolution to decrease the number of channels. The MobileNetV2 has two versions, which have the same structure but do not share parameters. One version is an auxiliary backbone and the other is an output backbone. For each layer of the output backbone at or above layer 2, the features of each layer of the auxiliary backbone at or above layer 2 are added to the features of the output backbone. The auxiliary backbone and the output backbone's layer-7 consists of only one inverted residual block A. An SE module is inserted after the 3*3 grouped convolution of the inverted residual block A. The SE module first performs average pooling on the input feature map, then obtains the weight of each channel of the feature map through two fully connected operations and the Sigmoid function, and finally multiplies it with the feature map to obtain the weighted feature map.

2. The detection method as described in claim 1, characterized in that, During the fusion process, the feature map of the auxiliary backbone is upsampled so that the size of the feature map of the auxiliary backbone is the same as the size of the feature map of the output backbone; the feature map of the corresponding layer of the auxiliary backbone is also input into the feature pyramid that shares the parameters with the feature pyramid, and participates in the calculation of the loss function together with the output backbone.

3. The detection method as described in claim 1, characterized in that, The feature fusion process involves upsampling the feature maps from higher layers of the multi-layered MobileNetV2 and then adding them element-wise to fuse them into the feature maps of lower layers. Then, a second round of fusion is performed from the bottom up: the lower-layer feature maps are downsampled and added element-wise to the feature maps of higher layers. The resulting feature maps are then subjected to a 3x3 convolution, and the output is then subjected to the same operation as the feature maps of higher layers. In this final bottom-up fusion, both the downsampling convolution and the 3x3 convolution with a stride of 1 employ deformable convolutions.

4. The detection method as described in claim 3, characterized in that, The deformable convolution learns additional offsets and weights of each sampling point in the x and y directions on top of the traditional 3*3 convolution kernel, and performs convolution calculations using the new sampling points and weights.

5. The detection method as described in claim 3, characterized in that, The output feature map of the deformable convolution is subjected to max pooling as another output of the feature pyramid.

Images