Power line detection method based on multi-scale characteristics

Abstract

The invention provides a power line detection method based on multi-scale characteristics. The method comprises the following steps: preprocessing a data set; performing multi-scale feature extraction by adopting an encoder; processing a multi-scale feature map Mi extracted by an encoder through a CoCBAM module to realize attention enhancement optimization of features, and obtaining an enhanced and optimized feature map; adding up-sampled results of the enhanced and optimized feature maps E4 and E5, processing through a BiLKB decoder, and selecting and refining effective features by using a large kernel convolution and spatial selection mechanism and assisted by small kernel convolution correction to obtain decoding information; and fusing deep semantic information and shallow detail information by using the deep decoding information O4, the shallow enhanced and optimized feature maps E1, E2 and E3 and an input dense scale feature fusion structure to obtain a final power transmission line edge extraction map.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning, and specifically to a method for edge detection of power lines based on drone-assisted image capture, which is essentially an edge detection problem for specific objects. Background Art

[0002] With the development of micro-UAV and image sensor technology, automatic inspection of power lines based on UAVs has received widespread attention in the power industry. UAVs equipped with optical imaging or infrared thermal imaging sensors can capture images of power lines, thereby detecting power line faults and potential dangers, and subsequently adjusting the flight position and posture in real time based on the power lines in the identified images. Therefore, the key to the technical realization of fully automatic inspection is power line detection.

[0003] Although traditional image-based power line detection methods can capture some features of power lines, due to the complex background, changing lighting and the relatively small power lines in the natural environment, these methods usually find it difficult to distinguish between power lines and the background, resulting in low detection accuracy.

[0004] With the development of deep learning, neural networks that can efficiently extract features have been introduced into power line detection tasks. Power line detection is usually regarded as an application task of edge detection. The multi-scale features of power lines are learned through a large number of parameters of deep neural networks to achieve the perception and expression of power line features. Although power line detection based on deep learning has shown excellent feature extraction capabilities, can handle complex backgrounds and improve detection accuracy, there are still some problems that need to be solved in practical applications:

[0005] (1) Since power lines are usually thin and have complex backgrounds, the network is easily affected by noise when processing the edges of power lines, resulting in blurred or discontinuous edges;

[0006] (2) Deep convolutional networks may not be able to fully capture multi-scale information when extracting transmission line features, thus affecting the comprehensiveness and accuracy of detection. Summary of the invention

[0007] In view of the above problems in the prior art, the present invention provides a method for detecting power lines, which can realize fine extraction of power lines. The technical solution is as follows:

[0008] A method for detecting power lines based on multi-scale features, characterized in that it comprises the following steps:

[0009] Step 1, data set preprocessing;

[0010] Step 2: Use the encoder to extract multi-scale features, and the output feature map is represented as Mi (i=1,2,3,4,5); from M 1 To M 5 The extracted feature information is from shallow to deep;

[0011] Step 3: extract the multi-scale feature map M from the encoder i , through the CoCBAM module processing, the feature attention enhancement optimization is realized, and the enhanced and optimized feature map E is obtained i (i=1, 2, 3, 4, 5);

[0012] Step 4: Enhance the optimized feature map E 4 and E 5 The up-sampled results are added and then processed by the BiLKB decoder. The large kernel convolution and space selection mechanism are used, supplemented by the small kernel convolution correction, to select and refine the effective features, and the decoded information is obtained as O 4 ;

[0013] Step 5: Decode the deep layer information O 4 And the feature map E after shallow enhancement optimization 1 ,E 2 ,E 3 , the input dense scale feature fusion structure realizes the fusion of deep semantic information and shallow detail information, and obtains the final transmission line edge extraction map M edae ;

[0014] Step 6: Train the network model using the loss function.

[0015] Furthermore, step 1 includes: adopting a data enhancement strategy, randomly rotating 24 angles and flipping the image-label pair, and mirror-filling the cropped image; the enhanced training set is used as input data for training the network model.

[0016] Furthermore, in the encoder part, EfficientNet b7 is selected as the backbone network to process the input image and extract feature maps of different scales.

[0017] Furthermore, in step 3, the processing steps of the CoCBAM module structure are as follows: feature map M i The residual module composed of Convblock is first processed, and then input into the residual module composed of CBAM for feature enhancement. CBAM combines channel attention and spatial attention to focus on the spatial position of feature channels and input data, enhances the feature expression ability of the network, and outputs the enhanced and optimized feature map E. i (i=1, 2, 3, 4, 5).

[0018] Furthermore, ConvBlock contains a 3×3 convolution kernel, a batch normalization layer and a Relu activation function.

[0019] Furthermore, the method of step 4 is as follows:

[0020] After the CoCBAM module enhances and optimizes the feature map E 5 The deconvolution method is used for upsampling to obtain the feature map E after enhancement and optimization with the previous layer. 4 The result of size alignment; E 5 The result image after upsampling and the enhanced feature map E of the previous layer 4 Add element by element, and the result is I 4 ;

[0021] I 4 Through the decoder BiLKB module processing, the receptive field is expanded and spatial selection is performed, and then supplemented with detail information correction, the optimization and selection of multi-scale features are realized, and the corresponding decoding information graph O is obtained. 4 ;

[0022] The decoder BiLKB module consists of two LKB blocks, batchnorm and relu sequentially connected residual structures;

[0023] The LKB module consists of a 1×1 convolution, an activation function GELU, two parallel modules, namely the large kernel convolution fusion module LKF and the small kernel convolution correction module SKR, and a 1×1 convolution;

[0024] Among them, the LKF module uses two large-core convolutions to expand the receptive field, extracts feature maps containing more information, and then uses the spatial selection mechanism to select effective features. The specific processing steps of the LKF module are: using a 5×5 deep hole large-core convolution (dilation=1), and connecting it with a 7×7 deep hole large-core convolution (dilation=3) in sequence to extract features of different scales and obtain features F1 and F2; using the spatial selection mechanism, the features F1 and F2 extracted by convolution kernels of different sizes are connected, and average pooling and maximum pooling operations are applied based on the channel dimension. Feature interaction is achieved through convolution operations, and the sigmoid activation function is applied to obtain a tensor after spatial selection processing; the tensor is decomposed into separate spatial selection masks of features of different scales along the channel as spatial selection of feature maps of different scales;

[0025] The processing steps of the SKR module in parallel with LKF are as follows: 3×3 convolution, batch normalization layer and Relu activation function are used to correct the detail information of the input feature map;

[0026] Multiply the features extracted by each large kernel convolution by its spatial selection mask element by element, and then restore the number of channels through convolution operation to obtain the feature map after the spatial selection mechanism;

[0027] I 4 After two LKB module processing, the final feature selection optimized deep decoding information O is obtained. 4 .

[0028] Furthermore, the method of step 5 is as follows:

[0029] The deep decoding information O 4 Perform deconvolution upsampling to achieve scale matching with the third-layer feature map and the enhanced and optimized feature map E of the third layer. 3 The result of adding each element is used as the input of the decoder BiLKB for decoding to obtain the decoding information O of the third layer. 3 ;

[0030] The decoding result of the deep network O 4 ,O 3 Perform deconvolution upsampling to achieve scale matching with the second layer feature map, and then with the enhanced feature map E of the second layer 2 Add element by element and input into the decoder to get the second layer decoding information O 2 ;

[0031] The decoding result of the deep network O 4 ,O 3 ,O 2 Perform deconvolution upsampling to achieve scale matching with the shallowest feature map, and then enhance and optimize the feature map E with the shallowest layer. 1 Add element by element and input into the decoder to get the final decoded information O 1 , achieving dense scale feature fusion;

[0032] The final decoded information O 1 After a layer of 1×1 convolution processing, the final transmission line edge detection map M is obtained. edge .

[0033] Furthermore, the method of step 6 is as follows: in the training stage, the Adam optimization algorithm is used to guide the gradient descent process; the weighted cross entropy loss function is used, and back propagation is performed to guide each round of training and parameter adjustment of the model to achieve accurate edge detection of the network.

[0034] The present invention aims at the problems faced in the edge detection of power lines in the field of power inspection, and proposes a power line detection method based on multi-scale features. The attention mechanism is introduced, and the CoCBAM module is proposed to make up for the problem of blurred edge line extraction caused by complex background noise. The encoder BiLKB composed of large kernel convolution is introduced, and a dense scale feature fusion structure is designed to solve the problem of insufficient extraction and utilization of multi-scale features, and realize more precise extraction of the edge of the transmission line. Therefore, the present invention has a high application value in the field of automatic inspection of transmission lines. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 This is the network architecture diagram of this algorithm;

[0036] Figure 2 It is the structural diagram of the CoCBAM module;

[0037] Figure 3 This is the structure diagram of the BiLKB module;

[0038] Figure 4 It is a transmission line edge detection result diagram of the method of the present invention on the PLDU data set;

[0039] Figure 5 It is a transmission line edge detection result diagram of the method of the present invention on the PLDM data set;

[0040] Figure 6 It is the comparison of edge detection results of the present invention and other networks on PLDU and PLDM data sets; DETAILED DESCRIPTION

[0041] The present invention is described below in conjunction with the accompanying drawings and embodiments.

[0042] Step 1: Dataset preprocessing and image enhancement:

[0043] The dataset used in this method is PLD-UAV data. This public dataset contains two power line detection sub-datasets with pixel-level annotations, namely PLDU dataset and PLDM dataset. Since the data and sample images provided are not sufficient to support our model training, we adopted a data augmentation strategy, which randomly rotated 24 angles and flipped the image-label pairs, and mirrored and filled the cropped images. The enhanced training sets of PLDU and PLDM contain 43488 and 22752 images respectively as input data for training network models.

[0044] Step 2: Use encoder to extract multi-scale features

[0045] This embodiment provides a power line detection method based on multi-scale features. The overall algorithm framework is as follows: Figure 1As shown in the figure, in the encoder part of this algorithm, EfficientNet b7 is selected as the backbone network to process the input image and extract feature maps of different scales. The output feature map is represented by M i (i=1,2,3,4,5). 1 To M 5 The extracted feature information is from shallow to deep.

[0046] Step 3: The multi-scale feature map M extracted by the encoder i The feature map is processed by the CoCBAM module to enhance and optimize the extracted features and obtain the output E i (i=1, 2, 3, 4, 5)

[0047] Feature maps of different scales extracted by the encoder M i , feature attention enhancement is achieved through jump connection CoCBAM (Correlated-Convolutional-Block-Attention-module) module processing.

[0048] Here is the specific CoCBAM module structure, such as Figure 2 As shown in (a), the feature map M i First, the residual module composed of Convblock is processed. The specific structure of ConvBlock is as follows Figure 2 As shown in (b), it contains a 3×3 convolution kernel, batch normalization layer and ReLU activation function to improve the accuracy of the feature map extracted by the encoder. Then it is input into the residual module composed of CBAM for feature enhancement processing, and the output result is E i Among them, CBAM is an existing attention enhancement module, and its specific structure is as follows Figure 2 (c) By combining channel attention (Channel Attention Module) and spatial attention (Spatial Attention Module), we focus on the spatial position of feature channels and input data to enhance the feature expression ability of the network.

[0049] Step 4: Feature map E after feature enhancement 4 and E 5 The up-sampled results are added and then processed by the BiLKB decoder. The large kernel convolution and space selection mechanism are used, supplemented by the small kernel convolution correction, to select and refine the effective features, and the decoded information is obtained as O 4 .

[0050] Feature map E after CoCBAM module enhancement 5 Contains the deepest semantic information, 5The deconvolution method is used for upsampling to obtain the enhanced feature map E of the previous layer. 4 The result of size alignment can restore the details as much as possible. 5 The result image after upsampling and the enhanced feature map E of the previous layer 4 Add element by element, and the result is I 4 .

[0051] I 4 The decoder BiLKB (Bi-Large-Kernel-Block) module is then used to expand the receptive field and perform spatial selection. Detailed information correction is then used to optimize and select multi-scale features, and the corresponding decoding information graph is obtained. 4 .O 4 The specific expression is as follows:

[0052] O 4 =D(E 4 +U 2 (E 5 ))

[0053] Where D(·) represents the decoding operation, U 2 (·) Deconvolution with 2x upsampling.

[0054] The decoder BiLKB module here is as follows Figure 3 As shown in (a), it is composed of two LKB (Large-Kernel-Block) blocks, batchnorm and relu sequentially connected residual structures.

[0055] LKB modules, such as Figure 3 (b) It consists of a 1×1 convolution, an activation function GELU, two parallel modules: Large Kernel Convolution Fusion Module (LKF) and Small Kernel Convolution Rectification Module (SKR), and a 1×1 convolution.

[0056] The LKF module is the core module of the LKB module. It uses two large kernel convolutions to expand the receptive field, extract feature maps containing more information, and then use the spatial selection mechanism to select effective features.

[0057] The specific processing steps of the LKF module are as follows: first, through the sequentially connected 5×5 deep hole convolution (dilation=1) and 7×7 deep hole convolution (dilation=3) large kernel convolution, the features of different scales are extracted to obtain F1 and F2; then, the spatial selection mechanism is used: the features F1 and F2 extracted by the convolution kernels of different sizes are connected, and then the average pooling and maximum pooling operations are applied based on the channel dimension, and then the feature interaction is realized through the convolution operation, and finally the sigmoid activation function is applied to obtain the tensor after the spatial selection processing. This tensor can be decomposed into a separate spatial selection mask for each large convolution kernel along the channel, and the mask can be used as a spatial selection in the feature map of the large convolution kernels of different scales.

[0058] The processing steps of the SKR module in parallel with LKF are as follows: after 3×3 convolution, batch normalization layer and Relu activation function, the feature map of the input module is corrected for detail information.

[0059] Finally, the features extracted by each large kernel convolution are multiplied element by element with its spatial selection mask, and the number of channels is restored through convolution operation to obtain the feature map after the spatial selection mechanism. 4 After two LKB module processing, the final feature selection optimization result can be obtained. 4 .

[0060] Step 5: Decode the deep layer information O 4 And the feature map E after feature enhancement 1 ,E 2 ,E 3 , input the multi-scale feature fusion structure, which can realize the fusion of deep semantic information and shallow detail information, and obtain the final transmission line edge extraction map M edge .

[0061] The shallow features are enhanced into Figure E 1 ,E 2 ,E 3 With deep decoding information O 4 Input the structure of dense scale feature fusion designed by this method. The specific steps of fusing information of different scales are as follows:

[0062] First, the decoding result of the deep network O 4 Perform deconvolution upsampling to achieve scale matching with the third-layer feature map and the enhanced feature map E of the third layer. 3 The result of adding each element is used as the input of the decoder BiLKB for decoding to obtain the decoding information O of the third layer. 3 The decoding graph realizes the fusion of deep decoding information and third-layer feature information. 3 The specific expression is as follows:

[0063] O3 =D(E 3 +U 2 (O 4 ))

[0064] The decoding result of the deep network O 4 ,O 3 Perform deconvolution upsampling to achieve scale matching with the second layer feature map, and then with the enhanced feature map E of the second layer 2 Add element by element and input into the decoder to get the second layer decoding information O 2 , O 2 The specific expression is as follows:

[0065] O 2 =D(E 2 +U 2 (O 3 )+U 4 (O 4 ))

[0066] Among them, U 4 (·) Deconvolution 4 times upsampling. The decoding result is O 2 , making full use of deep decoding information O 4 ,O 3 And shallow feature information E 2 .

[0067] The decoding result of the deep network O 4 ,O 3 ,O 2 Perform deconvolution upsampling to achieve scale matching with the shallowest feature map, and then with the enhanced feature map E of this layer 1 Add element by element and input into the decoder to get the final decoded information O 1 .O 1 The specific expression is as follows:

[0068] O 1 =D(E 1 +U 2 (O 2 )+U 4 (O 3 )+U 8 (O 4 ))

[0069] Among them, U 8 (·) Deconvolution 8 times upsampling. The final decoded information O 1 This achieves feature fusion at dense scales.

[0070] By using this dense scale feature fusion structure, the decoding results of each layer can be obtained one by one, and the decoding results contain the semantic information extracted by the deep network to suppress the noise pixels, which can achieve more accurate edge distinction. The final decoded information graph is processed by a layer of 1×1 convolution to obtain the final transmission line edge detection graph M. edge . M edge The specific representation is as follows:

[0071] M edge =Conv(O 1 )

[0072] Among them, Conv(·) represents a 1×1 convolution operation.

[0073] Step 6: Train the network model using the loss function

[0074] During the training process, the Pytorch deep learning framework was used. During the training phase, the hyperparameters were set as follows: the batch size was 4, the initial learning rate was 1×10 -4 , the learning rate decay rate is 0.1, and the weight decay is 5×10 -4 The number of training cycles is 30, and the Adam optimization algorithm is used to guide the gradient descent process. The experiments are conducted on NVIDIA A40 GPU.

[0075] The weighted cross entropy loss function is used to perform back propagation to guide each round of model training and parameter adjustment to achieve accurate edge detection of the network.

[0076] The weighted cross entropy loss function formula is as follows:

[0077]

[0078] Among them, α, β are the introduced weights, y i is the true value of the i-th pixel (0 or 1); p i is the probability that the i-th pixel is predicted to be an edge pixel; N is the total number of pixels.

[0079] Step 7: Evaluate the trained transmission line edge detection algorithm

[0080] Use the images in the test set to evaluate the performance of the trained network.

[0081] The commonly used comprehensive evaluation index F for edge detection is used. Score , which is divided into two types: global optimal threshold (ODS) F and image optimal threshold (OIS) F. The calculation formula is:

[0082]

[0083] Among them, Percision is the precision rate and Recall is the recall rate. Figure 6 By comparing this method with other classic methods, it can be seen that this method has achieved performance indicators of ODS=0.969 and ODS=0.943 in the power line datasets PLDU and PLDM, respectively, which fully proves the effectiveness of the method and also shows that this algorithm has high application value in the field of automatic inspection of transmission lines.

[0084] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the relevant field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A power line detection method based on multi-scale features, characterized in that: The following steps are involved: Step 1, data set preprocessing; Step 2: Use the encoder to extract multi-scale features, and the output feature map is represented as M i (i=1, 2, 3, 4, 5); the feature information extracted from M1 to M5 is from shallow to deep; Step 3: extract the multi-scale feature map M from the encoder i , through the CoCBAM module processing, the feature attention enhancement optimization is realized, and the enhanced and optimized feature map E is obtained i (i=1, 2, 3, 4, 5); Step 4, add the up-sampled results of the enhanced and optimized feature maps E4 and E5, and then process them through the BiLKB decoder, using large kernel convolution and spatial selection mechanism, supplemented by small kernel convolution correction, to select and refine the effective features, and obtain the decoded information O4; Step 5: The deep decoding information 04 and the shallow enhanced optimized feature maps E1, E2, E3 are input into the dense scale feature fusion structure to achieve the fusion of deep semantic information and shallow detail information, and the final transmission line edge extraction map M is obtained. edge ; Step 6: Train the network model using the loss function.

2. The power line detection method based on multi-scale features according to claim 1, characterized in that: Step 1 includes: adopting a data enhancement strategy, randomly rotating 24 angles and flipping image-label pairs, and mirror-filling the cropped images; the enhanced training set is used as input data for training the network model.

3. The power line detection method based on multi-scale features according to claim 1, characterized in that: In the encoder part, EfficientNet b7 is selected as the backbone network to process the input image and extract feature maps of different scales.

4. The method for detecting power lines based on multi-scale features according to claim 1, characterized in that: In step 3, the processing steps of the CoCBAM module structure are as follows: Feature map M i The residual module composed of Convblock is first processed, and then input into the residual module composed of CBAM for feature enhancement. CBAM combines channel attention and spatial attention to focus on the spatial position of feature channels and input data, enhances the feature expression ability of the network, and outputs the enhanced and optimized feature map E. i (i=1, 2, 3, 4, 5).

5. The method for detecting power lines based on multi-scale features according to claim 4, characterized in that: ConvBlock contains a 3×3 convolution kernel, a batch normalization layer, and a ReLU activation function.

6. The method for detecting power lines based on multi-scale features according to claim 1, characterized in that: The method for step 4 is as follows: The feature map E5 enhanced and optimized by the CoCBAM module is upsampled using the deconvolution method to obtain a result that is aligned with the size of the feature map E4 enhanced and optimized in the previous layer; The result image after upsampling E5 is added element by element to the enhanced feature image E4 of the previous layer, and the addition result is I4; I4 is processed by the decoder BiLKB module to expand the receptive field and perform spatial selection, and then supplemented with detail information correction to achieve optimization and selection of multi-scale features, and obtain the corresponding decoding information graph O4; The decoder BiLKB module consists of two LKB blocks, batchnorm and relu sequentially connected residual structures; The LKB module consists of a 1×1 convolution, an activation function GELU, two parallel modules, namely the large kernel convolution fusion module LKF and the small kernel convolution correction module SKR, and a 1×1 convolution; Among them, the LKF module uses two large kernel convolutions to expand the receptive field, extracts feature maps containing more information, and then uses the spatial selection mechanism to select effective features. The specific processing steps of the LKF module are: using a 5×5 deep hole large kernel convolution (dilation=1), and sequentially connecting it with a 7×7 deep hole large kernel convolution (dilation=3) to extract features of different scales and obtain features F1 and F2; Using the spatial selection mechanism, the features F1 and F2 extracted by the convolution kernels of different sizes are connected, and the average pooling and maximum pooling operations are applied based on the channel dimension. The feature interaction is realized through the convolution operation, and the sigmoid activation function is applied to obtain the tensor after spatial selection processing; the tensor is decomposed into separate spatial selection masks of features of different scales along the channel as the spatial selection of feature maps of different scales; The processing steps of the SKR module in parallel with LKF are as follows: 3×3 convolution, batch normalization layer and Relu activation function are used to correct the detail information of the input feature map; Multiply the features extracted by each large kernel convolution by its spatial selection mask element by element, and then restore the number of channels through convolution operation to obtain the feature map after the spatial selection mechanism; After I4 is processed twice by the LKB module, the final feature selection optimized deep decoding information O4 is obtained.

7. The method for detecting power lines based on multi-scale features according to claim 1, characterized in that: The method for step 5 is as follows: The deep decoding information O4 is deconvoluted and upsampled to achieve scale matching with the feature map of the third layer, and is added element by element with the enhanced and optimized feature map E3 of the third layer. The result is decoded as the input of the decoder BiLKB to obtain the decoding information O3 of the third layer; The decoding results O4 and O3 of the deep network are deconvoluted and upsampled to achieve scale matching with the second-layer feature map, and then added element by element with the enhanced feature map E2 of the second layer, and input into the decoder to obtain the second-layer decoding information O2; The decoding results O4, O3, and O2 of the deep network are deconvoluted and upsampled to match the scale of the shallowest feature map, and then added element by element with the shallowest enhanced and optimized feature map E1, and input into the decoder to obtain the final decoding information O1, realizing dense scale feature fusion; The final decoded information O1 is processed through a layer of 1×1 convolution to obtain the final transmission line edge detection map M edge .

8. The method for detecting power lines based on multi-scale features according to claim 1, characterized in that: The method of step 6 is as follows: During the training phase, the Adam optimization algorithm is used to guide the gradient descent process; the weighted cross entropy loss function is used to perform back propagation to guide each round of training and parameter adjustment of the model to achieve accurate edge detection of the network.

Images