A power equipment corrosion defect image detection method based on SwinT and size adaptive convolution

Abstract

The present application relates to a kind of power equipment corrosion defect image detection methods based on SwinT and size self-adapting convolution, compared with prior art, solve the difficulty of substation equipment defect model feature extraction, difficult to identify, low detection precision, the defect of poor robustness.The present application includes the following steps: the acquisition of power equipment defect image sample;Power equipment defect target detection model is constructed;Power equipment defect target detection model training;Obtain the power equipment defect image to be detected and carry out pretreatment;Power equipment image defect detection.The present application is stronger in feature extraction capability, model training is simple, while the target detection method of more optimal identification precision is realized, and the accurate detection of power equipment defect image target is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power equipment image recognition, in particular to a power equipment rust defect image detection method based on SwinT and size adaptive convolution. BACKGROUND

[0002] As a place for power transportation and distribution, it is very important to maintain the stable operation of the equipment in the substation for human production and life. Due to the long-term operation of the substation in a harsh and complex natural environment, the fittings on the line are affected by climate, terrain, external force and other factors to appear different types of defects. At present, the anti-rust layer of the power equipment may lose its protective effect due to aging, causing safety hazards. Therefore, it is crucial to detect and maintain the rusted equipment. Using the target detection technology based on deep learning to identify the rust defect detection of substation equipment has the advantages of high accuracy, safety, wide application range and the like compared with the artificial inspection method.

[0003] At present, although the target detection technology based on deep learning is widely used in power equipment defect detection, when the image background is complex and the background is large and the foreground is small, there are problems such as mis-detection, missed detection and low detection accuracy when facing irregularly shaped and various rust targets. The general target detection algorithm has weak discrimination ability for similar targets and irregularly shaped targets, and it is difficult to distinguish different types of targets.

[0004] Therefore, how to make the network have stronger discrimination ability for rust defect targets and further improve the detection accuracy of the model has become a technical problem to be solved. SUMMARY

[0005] The purpose of the present application is to solve the defects of difficult feature extraction, difficult recognition, low detection accuracy and poor robustness of the substation equipment defect model in the prior art, and to provide a power equipment rust defect image detection method based on SwinT and size adaptive convolution to solve the above problems.

[0006] In order to achieve the above purpose, the technical scheme of the present application is as follows:

[0007] A power equipment rust defect image detection method based on SwinT and size adaptive convolution, comprising the following steps:

[0008] Acquisition of power equipment defect image samples: acquire power equipment defect images of different scenes and different categories and pre-process them to form a power equipment defect image dataset;

[0009] Constructing a power equipment defect target detection model: constructing a power equipment defect target detection model based on SwinT and size adaptive convolution;

[0010] Training of the power equipment defect target detection model: the power equipment defect target detection model is trained by using the power equipment defect image dataset;

[0011] Obtaining and preprocessing the power equipment defect image to be detected;

[0012] Power equipment image defect detection: inputting the power equipment defect image to be detected into the trained power equipment defect target detection model to obtain the target classification and positioning result in the power equipment defect image, and detecting the rust area.

[0013] The construction of the power equipment defect target detection model includes the following steps:

[0014] The power equipment defect target detection model includes a feature extraction network, a feature fusion network and a detection head network;

[0015] The feature extraction network is constructed based on Swin-Transformer and size adaptive convolution module,

[0016] Wherein, Swin-Transformer is composed of Patch Partition layer, Linear Embedding layer, SwinTransformer Block layer and Patch Merging layer, and size adaptive convolution is composed of convolutional neural network, and size adaptive convolution module is inserted in the 3rd and 4th stages of SwinT to further complete the feature extraction of objects with different scales and shapes; the input is the enhanced power equipment defect image, and the output is the overall feature map of the power equipment defect image;

[0017] The input of the Patch Partition layer is an image with a given size of HxW, the Patch Partition layer performs block dimension reduction operation, the Linear Embedding layer performs linear transformation, and the Patch Merging layer is used for down sampling;

[0018] The size adaptive convolution module first uses the ordinary convolution layer to learn the offset shift and adjust the weight parameters, enhances the adaptability of convolution to the defect shape characteristics, then outputs the feature map and offset adjustment weight, and finally inputs the different scale feature maps generated in the four stages into the feature fusion network;

[0019] The feature fusion network is set as:

[0020] The set feature fusion network is constructed based on a FPN network, an input of which is an overall feature map output by the feature extraction network, and the fusion of the overall features of the input defect image is performed to obtain a prediction feature map by up-sampling the high-level features and horizontally connecting and fusing the high-resolution features of the low-level features, so as to detect small objects that are lost in high-level semantics but still exist in low-level feature maps, and an output is a fusion feature map after feature fusion.

[0021] The set detection head network is constructed based on a TOOD network, an input of which is the fusion feature map output by the feature fusion network, and whether it is a detection target is determined through classification and positioning of the task, and the boundary coordinates of the measured target are determined through a boundary box regression algorithm.

[0022] The set detection head network is constructed based on a TOOD network, an input of which is the fusion feature map output by the feature fusion network, and whether it is a detection target is determined through classification and positioning of the task, and the boundary coordinates of the measured target are determined through a boundary box regression algorithm.

[0023] The training of the power equipment defect target detection model includes the following steps:

[0024] The training of the feature extraction network: input the power equipment defect image dataset into the feature extraction network, and use the feature extraction network to train the overall feature map of the power equipment defect image;

[0025] The training of the feature fusion network: input the overall feature map extracted by the feature extraction network into the feature fusion network from bottom to top, to obtain F1, F2, F3 and F4 feature maps, the width and height of which are 1 / 4, 1 / 8, 1 / 16 and 1 / 32 of the original image respectively, and the dimension is d;

[0026] In the top-down process, the new feature map and the lower layer feature map are kept consistent in scale through scale change;

[0027] In the long and wide directions, the width and height of the lower layer feature map are pulled to the same size by up-sampling; in the depth direction, the depth of the upper layer feature map is compressed to the same depth as the lower layer feature map by a 1x1 convolution, so that the new feature map and the original lower layer feature map have the same scale; then the corresponding elements of the new feature map and the original lower layer feature map are added to realize the fusion of the upper layer feature and the lower layer feature;

[0028] The training of the detection head network:

[0029] The detection head network first takes the fused feature map as input, uses N consecutive conv layers with activation functions to calculate the task interaction feature F k inter :

[0030]

[0031] F fpnFor the feature map output by the fusion network, conv k and δ represent the kth convolutional layer and the relu activation function, respectively;

[0032] After obtaining the task interaction features, they are input into the TAP module. Then, the features are flattened through a fully connected layer and a sigmoid activation function to obtain the weight of each task interaction feature. The classification or positioning task features are obtained by multiplying the weights.

[0033]

[0034] where ω k is the kth element of the layer attention, which is calculated from the task interaction features and can capture the dependency between layers:

[0035]

[0036] where, and are fully connected layers, and σ is a sigmoid activation function. F inter is obtained by using the average pooling operation to obtain f inter .

[0037] Finally, the classification or positioning result Z task is obtained:

[0038] Z task = conv2(δ(conv1(F task ))) (4)

[0039] where F task is the connection feature of , conv1 is a 1×1 convolution for dimension reduction, and then the sigmoid activation function is used to convert Z task into dense classification scores or object bounding boxes

[0040] The training of the feature extraction network includes the following steps:

[0041] The input substation equipment image is divided into patches by the Patch Partition layer for dimensionality reduction operation, the input image size is HxWxC, wherein H and W represent the height and width of the image respectively, and C is the channel number of the image, the size of each patch (P, P) = HW / N, the divided patch is linearly projected by the Linear Embedding layer and input into the Swin Transformer Block module for multi-head self-attention calculation to generate a feature map F1 with a size of the feature map F1;

[0042] The above size of the feature map F1 is The feature map F1 is input into the Patch Merging layer for downsampling operation and then into the Swin Transformer Block module for multi-head self-attention calculation to generate a feature map F2 with a size of the feature map F2;

[0043] The above size of the feature map F2 is The feature map F2 is input into the Patch Merging layer for downsampling operation and multi-head self-attention calculation to generate a feature map F3 with a size of the feature map F3;

[0044] Training of the size adaptive convolution module:

[0045] The feature map F3 is input into the size adaptive convolution module, first passes through ordinary convolution, the convolution padding is same, that is, the input and output sizes are unchanged, and the corresponding feature output size is The offset is denoted as offset, which represents the offset of each pixel index in the original feature map;

[0046] wherein the sampling position set G is obtained by AL n , AL n = 1, 2,..., N, the sampling position offset is realized to increase the receptive field range, N is the number of pixels in the grid, 2C3 represents the offset in x and y directions, and 1C3 channel represents the adjustment weight value Am, which is in the range of 0 to 1 after the Sigmoid function;

[0047] The index value of the pixel in the input feature map F3 is added to the offset to obtain the offset position, that is, the coordinate value of each pixel in F3, and the value of position is limited in the picture size range;

[0048] lo is any position on the output feature map:

[0049]

[0050] Δl n = 1, 2,..., N

[0051] wherein, l n respectively represent the pre-designated offset of the nth position; ω(l n ) is the weight information of the sampling position; G = {(-1, -1), (-1, 0),..., (1, 0), (1, 1)} is the set of sampling positions; Δl n is the learnable offset of the nth position; Δm n is the adjustment weight parameter; x(l0+l n + Δl n ) and y(l0) respectively represent the features at any position l0 obtained from the input feature map x and the output feature map y;

[0052] Since the sampling points are performed at l n + Δl n after irregular offset, and the offset Δl n is a floating-point number, accurate pixel values cannot be obtained. In order to obtain accurate pixel values and perform back propagation, a bilinear interpolation method is adopted to obtain the pixel corresponding to the coordinates;

[0053] Take a coordinate (a, b), and convert it into four integers floor(a), ceil(a), floor(b), and ceil(b), wherein the floor() and ceil() operations are to take a and b to the floor or ceiling to obtain four coordinates (floor(a), floor(b)), (floor(a), ceil(b)), (ceil(a), floor(b)), and (ceil(a), ceil(b));

[0054] Each of the four coordinates corresponds to a pixel value in F3, and the pixel value x(l) at any position is calculated by a bilinear interpolation method:

[0055]

[0056] wherein, l = l0+l n + Δl n represents an arbitrary position after offset; m is the enumeration of the overall spatial position coordinates in the feature map; x(m) is the pixel value of the four adjacent integer coordinates around the feature map F3; H() is the weight corresponding to the four integer coordinates, and it is two-dimensional, composed of two one-dimensional kernels:

[0057] H(m, l) = g(m x , l x )·g(m y , l y) (7)

[0058] wherein, (m x ,l x ), (m y ,l y ) are coordinate values in x, y offset direction respectively;

[0059] g(a,b)=max(0,1-|a-b|) limits the distance between the interpolation point and the adjacent four integer coordinates to not more than 1 pixel, a, b are coordinate values;

[0060] After obtaining all the pixels of position, the weight value parameter is adjusted by Δm, and finally the size of The new feature map F'3 is input to the next stage;

[0061] The feature map F'3 with the size of is input to the Patch Merging layer for downsampling operation and then calculated by the multi-head self-attention calculation in the Swin Transformer Block module to generate a feature map F4 with the size of The feature map F4 learns the high-level semantic information of the image by the size adaptive convolution module, so that the sampling position is more consistent with the shape and size of the object itself rather than the fixed geometric structure sampling, which is more conducive to the extraction of defect features;

[0062] Wherein, the core module Swin Transformer Block contains multi-head attention for shifting windows, and the attention calculation of each head in the multi-head attention is as follows:

[0063]

[0064] Wherein, Q, K, V are query, key, and value matrices respectively, K T is the transpose matrix of K; d is the vector dimension, B is the bias matrix, and SoftMax is a multi-classification activation function;

[0065] 45) The stage1, stage2, stage3, and stage4 layers of the Swin-Transformer backbone network represent the features of each scale in the overall feature map, and the power equipment defect detection image is subjected to data enhancement processing. By adding Gaussian and salt and pepper noise and adjusting the image brightness, the enhanced data is used as the input of the feature extraction network, and the overall feature map is input into the next stage of the feature fusion network.

[0066] The training of the power equipment defect target detection model also includes an explicit alignment operation, which is used to obtain better classification performance and positioning accuracy; the explicit alignment operation includes the following steps:

[0067] aligning classification task Q is obtained using spatial probability map N align :

[0068]

[0069] where dense classification score is calculated from interaction feature to learn the consistency of spatial position; spatial bias is used to obtain positioning task prediction B align , which is realized by bilinear interpolation:

[0070] B align (i,j,c) = B(i+P(i,j,2×c),j+P(i,j,2×c+1),c) (10)

[0071] where object bounding box is spatial bias map calculated from interaction feature to adjust the predicted bounding box of each position; (i,j,c) represents the (i,j)th spatial position on the cth channel in the tensor, since the offset of each channel is learned independently, which means that each object's boundary has its own offset;

[0072] Aligning mapping N and P are automatically learned from interaction feature stack:

[0073] N = σ(conv2(δ(conv1(F inter )))) (11)

[0074] P = conv4(δ(conv3(F inter ))) (12)

[0075] conv1 and conv3 are 1×1 convolution for dimension reduction operation; F inter is task interaction feature; δ represents relu activation function; σ is sigmoid activation function;

[0076] Classification loss function QualityFocalLoss is used to adjust the parameters in the network, and finally the accurate defect target in the substation equipment image is output, and the loss QFL(σ) formula is as follows:

[0077] QFL(σ) = -a i *|x-σ| β *[(1-x)log(1-σ)+xlog(σ)] (13)

[0078] where a i is the balance of positive and negative samples, σ is the predicted value, x is the continuous label, and β is the proportion factor to control the weight decay speed;

[0079] The CIOU loss is used as the regression loss function, and the loss Loss CIOU As follows:

[0080]

[0081] Wherein, d represents the minimum circumscribed rectangle surrounding the prediction frame and the real frame, c, c gt Respectively represent the center point of the prediction frame and the real frame, p represents the Euclidean distance between the two center points, and beta and w are the aspect ratio.

[0082] Advantages

[0083] The power equipment corrosion defect image detection method based on SwinT and size adaptive convolution has stronger feature extraction capability compared with the prior art, the model training is simple, and the target detection method has better recognition accuracy, and realizes accurate detection of power equipment defect image targets.

[0084] The application constructs a feature extraction network based on Swin-Transformer and size adaptive convolution through image enhancement technology, so that the network extracts features more in line with the shape and size of the object itself, and the fixed geometric structure sampling is more conducive to the extraction of defect features; a feature fusion network is constructed by fusing different levels of features, which increases the detection accuracy of small targets; and the TOOD detection head obtains targets with better classification performance and positioning accuracy. BRIEF DESCRIPTION OF DRAWINGS

[0085] Figure 1 The method sequence diagram of the application;

[0086] Figure 2 The power equipment defect target detection model structure diagram of the application;

[0087] Figure 3 The corrosion detection effect diagram using the method of the application. DETAILED DESCRIPTION

[0088] In order to have a further understanding and understanding of the structural features and effects achieved by the application, the following detailed description is given in combination with preferred embodiments and drawings:

[0089] As Figure 1 shown, the power equipment corrosion defect image detection method based on SwinT and size adaptive convolution comprises the following steps:

[0090] First, the acquisition of power equipment defect image samples: acquire power equipment defect images of different scenes and different categories and pre-process them to form a power equipment defect image dataset.

[0091] Second step, build power equipment defect target detection model: based on SwinT and size adaptive convolution to build power equipment defect target detection model.

[0092] For the problem that the surface rust defect of power equipment is small, the shape is variable and the background is complex and not easy to identify, the original SwinT is not good at modeling the shape of the variable rust target, and cannot achieve good adaptive defect shape characteristics. Due to the fixed rectangular structure, only the fixed position on the feature map is sampled, but the environment of power equipment is complex, and the shape of rust defect is different. For different positions, different scales and shapes of defect targets, increasing size adaptive convolution is needed for adaptive learning of different scales or receptive fields to accurately detect rust defect targets. Because the feature map size of the third and fourth stages of SwinT is small, the receptive field is large, and the semantic information is rich, adding size adaptive convolution in this stage is more conducive to learning the overall shape category of the object, and the small size feature input is relatively fast when calculating the offset.

[0093] The specific steps are as follows:

[0094] (1) As shown in Figure 2 , the power equipment defect target detection model is set to include a feature extraction network, a feature fusion network and a detection head network.

[0095] (2) The feature extraction network is set to be constructed based on Swin-Transformer and size adaptive convolution module,

[0096] Wherein, Swin-Transformer is composed of Patch Partition layer, Linear Embedding layer, SwinTransformer Block layer and Patch Merging layer, and size adaptive convolution is composed of convolutional neural network. The size adaptive convolution module is inserted in the 3rd and 4th stages of SwinT to further complete the feature extraction of objects with different scales and shapes. The input is the enhanced power equipment defect image, and the output is the overall feature map of the power equipment defect image.

[0097] The input of the Patch Partition layer is an image with a given size of HxW. The Patch Partition layer performs block dimension reduction operation, the Linear Embedding layer performs linear transformation, and the Patch Merging layer is used for down sampling.

[0098] The size-adaptive convolution module first uses ordinary convolutional layers to learn the offset and adjust the weight parameters to enhance the adaptability of convolution to defect morphology features. Then, it outputs the feature map and offset value after adjusting the weights. Finally, it inputs the feature maps of different scales generated in the four stages into the feature fusion network.

[0099] (3) Setting up a feature fusion network:

[0100] The feature fusion network is constructed based on the FPN network. Its input is the overall feature map output by the feature extraction network. By fusing the overall features of the input defect image, the high-level features are upsampled and then laterally connected and fused with the low-level high-resolution features to obtain the predicted feature map. This is to detect small objects that have disappeared in the high-level semantics but still exist in the low-level feature map. The output is the fused feature map after feature fusion.

[0101] (4) Configure the detection head network:

[0102] The detection head network is built based on the TOOD network. Its input is the fused feature map output by the feature fusion network. By classifying and locating the task, it determines whether it is a detection target, and the bounding box regression algorithm determines the boundary coordinates of the detected target.

[0103] The third step is training the power equipment defect target detection model: The model is trained using a power equipment defect image dataset. Within the mmdetection framework, a multi-scale training (MST) approach is employed, using a linearly increasing warmup strategy.

[0104] (1) Training of the feature extraction network: Input the power equipment defect image dataset into the feature extraction network, and use the feature extraction network to train the overall feature map of the power equipment defect images.

[0105] The training of the feature extraction network includes the following steps:

[0106] A1) A patch partitioning and dimensionality reduction operation is performed through the Patch Partition layer, dividing the input substation equipment image into patches. The input image size is H×W×C, where H and W represent the height and width of the image, respectively, and C is the number of channels in the image. The size of each patch is (P,P) = HW / N. The partitioned patches are then linearly projected by the Linear Embedding layer and input into the Swing Transformer Block module for multi-head self-attention calculation to generate a patch of size H×W×C. Feature map F1;

[0107] A2) The above size is The feature map F1 is input into a Patch Merging layer for downsampling operation and then into a Swin Transformer Block module for multi-head self-attention calculation to generate a feature map F2 with a size of

[0108] A3) The size of the feature map F2 is The feature map F2 is input into a Patch Merging layer for downsampling operation and multi-head self-attention calculation to generate a feature map F3 with a size of

[0109] A4) Training of the size-adaptive convolution module:

[0110] A41) The feature map F3 is input into the size-adaptive convolution module, which is first subjected to ordinary convolution with padding of same, i.e., the input and output sizes are unchanged, and the corresponding feature output size is The offset is denoted as offset, which represents the offset of each pixel index in the original feature map.

[0111] wherein the sampling position set G is realized by ΔL n (ΔL n = 1, 2,..., N) to increase the range of the receptive field, N is the number of pixels in the grid, 2C3 represents the offset in the x and y directions, and 1C3 channel represents the adjustment weight value Δm, which is in the range of 0 to 1 after the Sigmoid function.

[0112] A42) The index value of the pixel in the input feature map F3 is added to the offset to obtain the offset position, i.e., the coordinate value of each pixel in F3, and the value of position is limited within the picture size range.

[0113] l0 is an arbitrary position on the output feature map:

[0114]

[0115] Δl n = 1, 2,..., N

[0116] wherein l n n represents the pre-specified offset of the nth position; ω(l n ) is the weight information of the sampling position; G = {(-1, -1), (-1, 0),..., (1, 0), (1, 1)} is the sampling position set; Δl n n is the learnable offset of the nth position; Δm n is the adjustment weight parameter; x(l0+l n + Δl​​n ) and y(l0) represent the features at an arbitrary position l0 from the input feature map x and the output feature map y, respectively;

[0117] A43) Since the sampling points are at irregular offsets l n +Δl n are performed, and the offset Δl n is a floating point number, and accurate pixel values cannot be obtained. In order to obtain accurate pixel values and perform backpropagation, a bilinear interpolation method is used to obtain the pixel corresponding to the coordinates;

[0118] Take a coordinate (a, b), and convert it into four integers floor(a), ceil(a), floor(b), and ceil(b), where the floor() and ceil() operations are to take a and b down or up to the nearest integer, obtaining the four coordinates (floor(a), floor(b)), (floor(a), ceil(b)), (ceil(a), floor(b)), and (ceil(a), ceil(b));

[0119] Each of the four coordinates corresponds to a pixel value in F3, and the pixel value at an arbitrary position x(l) is calculated by bilinear interpolation:

[0120]

[0121] where l = l0+ l n +Δl n represents an arbitrary position after offset; m is an enumeration of the overall spatial position coordinates in the feature map; x(m) is the pixel value of the four adjacent integer coordinates around the feature map F3; H() is the weight corresponding to the four integer coordinates, and it is two-dimensional, consisting of two one-dimensional kernels:

[0122] H(m, l) = g(m x , l x ) · g(m y , l y ) (7)

[0123] where (m x , l x ), (m y , l y ) are the coordinate values in the x and y offset directions, respectively;

[0124] g(a, b) = max(0, 1 - |a - b|) limits the distance between the interpolation point and the four adjacent integer coordinates to no more than 1 pixel, where a and b are coordinate values;

[0125] After obtaining all the pixels of position, the weight value parameter is adjusted by Δm, and finally the size of The new feature map F'3 is input to the next stage;

[0126] A44) The feature map F'3 with the above size is input to the Patch Merging layer for downsampling operation and then multi-head self-attention calculation is performed in the Swin Transformer Block module to generate a feature map F4 with the size of The feature map F4 learns the high-level semantic information of the image by the size adaptive convolution module, so that the sampling position is more consistent with the shape and size of the object itself, rather than the fixed geometric structure sampling, which is more conducive to the extraction of defect features.

[0127] Among them, the core module Swin Transformer Block contains multi-head attention for shifting windows, and the attention calculation of each head in multi-head attention is as follows:

[0128]

[0129] Where Q, K, and V are query, key, and value matrices, respectively, K T is the transpose matrix of K; d is the vector dimension, B is the bias matrix, and SoftMax is a multi-classification activation function.

[0130] A5) The stage1, stage2, stage3, and stage4 layers of the Swin-Transformer backbone network represent the features of each scale in the overall feature map, and the power equipment defect detection image is processed by data augmentation. By adding Gaussian and salt and pepper noise and adjusting the image brightness, the enhanced data is used as the input of the feature extraction network, and the overall feature map is extracted and input to the next stage feature fusion network.

[0131] (2) Training of the feature fusion network: the overall feature map extracted by the feature extraction network is input into the bottom-up hierarchical structure of the feature fusion network to obtain F1, F2, F3, and F4 feature maps, whose width and height are 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively, and the dimension is d.

[0132] In the top-down process, the new feature map and the lower layer feature map are kept consistent in scale through scale change.

[0133] In the length and width direction, the width and height of the lower layer feature map are pulled to the same size by upsampling method; in the depth direction, the depth of the upper layer feature map is compressed to the same depth as the lower layer feature map by a 1x1 convolution, so that the new feature map and the original lower layer feature map have the same scale; then add each corresponding element of the new feature map and the original lower layer feature map to realize the fusion of the upper layer feature and the lower layer feature.

[0134] (3) Training of the detection head network:

[0135] The detection head network first takes the fused feature map as input, uses N consecutive conv layers with activation function to calculate the task interaction feature F k inter :

[0136]

[0137] where F fpn is the feature map output by the fusion network, conv k and δ represent the kth convolution layer and relu activation function respectively;

[0138] After obtaining the task interaction feature, it is input into the TAP module, and then the feature is flattened through the fully connected layer and the sigmoid activation function to obtain the weight of each task interaction feature. The multiplication operation is performed to obtain the feature of the classification or positioning task

[0139]

[0140] where ω k is the kth element of the layer attention, which is calculated from the task interaction feature and can capture the dependency between layers:

[0141]

[0142] where, and are fully connected layers, and σ is a sigmoid activation function. F inter is obtained by using average pooling operation f inter ;

[0143] Finally, the classification or positioning result Z task is obtained:

[0144] Z task = conv2(δ(conv1(F task ))) (4)

[0145] where F task is conv1 is a 1x1 convolution for dimension reduction, followed by a sigmoid activation function to convert Z task into dense classification scores or object bounding boxes

[0146] In order to obtain more accurate results,

[0147] The training of the power equipment defect target detection model further includes an explicit alignment operation for obtaining better classification performance and positioning accuracy; the explicit alignment operation includes the following steps:

[0148] (1) Obtain an alignment classification task Q align from a spatial probability map N

[0149]

[0150] wherein the dense classification scores are calculated from the interaction features and are used to learn the consistency of the spatial position; a positioning task prediction B align is obtained from a spatial deviation, which is implemented by bilinear interpolation:

[0151] B align (i,j,c) = B(i+P(i,j,2xc),j+P(i,j,2xc+1),c) (10)

[0152] wherein the object bounding boxes are spatial deviation maps calculated from the interaction features and used to adjust the predicted bounding boxes of each position; (i,j,c) represents the (i,j)th spatial position on the cth channel in the tensor, and since the offset of each channel is learned independently, it means that each object's boundary has its own offset.

[0153] (2) The alignment mapping N and P are automatically learned from the interaction feature stack:

[0154] N = σ(conv2(δ(conv1(F inter )))) (11)

[0155] P = conv4(δ(conv3(F inter ))) (12)

[0156] conv1 and conv3 are 1x1 convolutions for dimension reduction; F inter is the task interaction feature; δ represents the relu activation function; and σ is the sigmoid activation function.

[0157] The classification loss function QualityFocalLoss is used to adjust the parameters in the network, and finally the accurate defect target in the substation equipment image is output, and the loss QFL(σ) formula is as follows:

[0158] QFL(σ)=-a i *|x-σ| β *[(1-x)log(1-σ)+xlog(σ)] (13)

[0159] Wherein, a i is balanced positive and negative samples, sigma is a predicted value, x is a continuous label, and beta is a proportional factor used to control the weight decay speed;

[0160] CIOUloss is used as a regression loss function, and the loss Loss CIOU is as follows:

[0161]

[0162] Wherein, d represents the minimum circumscribed rectangle surrounding the predicted frame and the real frame, c, c gt respectively represent the center points of the predicted frame and the real frame, rho represents the Euclidean distance between the two center points, and beta and w are the aspect ratio.

[0163] From Figure 3 It can be seen that the method of the application can accurately detect the corrosion area. From Table 1, it can be seen that the method of the application has obvious AP value gain compared with the original SwinT and other detection methods.

[0164] Table 1 Comparison of AP values of detection algorithms

[0165]

[0166] The basic principles, main features and advantages of the application are shown and described above. Those skilled in the art should understand that the application is not limited by the above examples, and the above examples and descriptions in the specification are only the principles of the application. Without departing from the spirit and scope of the application, various changes and improvements can be made to the application, and these changes and improvements all fall within the scope of the claimed application. The scope of protection claimed by the application is defined by the appended claims and their equivalents.

Claims

1. A power equipment corrosion defect image detection method based on SwinT and size adaptive convolution, characterized in that, The method comprises the following steps: 11) Acquisition of power equipment defect image samples: acquire power equipment defect images of different categories in different scenes and pre-process to form a power equipment defect image dataset; 12) Construction of a power equipment defect target detection model: constructing a power equipment defect target detection model based on SwinT and size adaptive convolution; The construction of the power equipment defect target detection model comprises the following steps: 121) Setting the power equipment defect target detection model to include a feature extraction network, a feature fusion network, and a detection head network; 122) Setting the feature extraction network to be constructed based on Swin-Transformer and a size adaptive convolution module, Wherein, the Swin-Transformer is composed of a Patch Partition layer, a Linear Embedding layer, a SwinTransformer Block layer, and a Patch Merging layer, and the size adaptive convolution is composed of a convolutional neural network, and the size adaptive convolution module is inserted at stages 3 and 4 of SwinT to further extract features of objects of different scales and shapes; the input is an enhanced power equipment defect image, and the output is an overall feature map of the power equipment defect image; The input of the Patch Partition layer is an image of a given size HxW, the Patch Partition layer performs block dimension reduction operation, the Linear Embedding layer performs linear transformation, and the Patch Merging layer is used for downsampling; The size adaptive convolution module first uses a normal convolution layer to learn offset shift and adjust weight parameters to enhance the adaptability of convolution to defect morphology features, then outputs the feature map and offset value after adjusting the weight, and finally inputs the different scale feature maps generated in the four stages into the feature fusion network; 123) Setting the feature fusion network: The feature fusion network is set to be constructed based on the FPN network, the input of which is the overall feature map output by the feature extraction network, the overall features of the input defect image are fused, the high-level features are upsampled and horizontally connected with the low-resolution features to obtain a prediction feature map, so as to detect small objects that disappear in high-level semantics but still exist in low-level feature maps, and the output is a fusion feature map after feature fusion; 124) Setting the detection head network: The detection head network is set to be constructed based on the TOOD network, the input of which is the fusion feature map output by the feature fusion network, the classification and positioning of the task are determined to determine whether it is a detection target, and the boundary coordinates of the measured target are determined through a boundary box regression algorithm; 13) Training of the power equipment defect target detection model: training the power equipment defect target detection model using the power equipment defect image dataset; 14) Acquisition of a power equipment defect image to be detected and pre-processing; 15) Power equipment image defect detection: input the power equipment defect image to be detected into the trained power equipment defect target detection model to obtain the target classification and positioning result in the power equipment defect image, and detect the rust area.

2. The method according to claim 1, wherein, The training of the power equipment defect target detection model includes the following steps: 21) Training of the feature extraction network: input the power equipment defect image dataset into the feature extraction network, and use the feature extraction network to train the overall feature map of the power equipment defect image; 22) Training of the feature fusion network: input the overall feature map extracted by the feature extraction network into the bottom-up hierarchical structure of the feature fusion network to obtain F1, F2, F3, and F4 feature maps, which have a width and height of 1 / 4, 1 / 8, 1 / 16, and 1 / 32 of the original image, respectively, and a dimension of d; In the top-down process, the new feature map and the lower layer feature map are kept consistent in scale through scale change; In the length and width directions, the width and height of the lower layer feature map are pulled to the same size by upsampling; in the depth direction, the depth of the upper layer feature map is compressed to the same depth as the lower layer feature map through a 1x1 convolution, so that the new feature map and the original lower layer feature map have the same scale; then the corresponding elements in the new feature map and the original lower layer feature map are added to realize the fusion of the upper layer feature and the lower layer feature; 23) Training of the detection head network: The head network first takes the fused feature map as input, uses N consecutive conv layers with activation functions to calculate the task interaction feature F k inter : where F fpn is the feature map output by the fusion network, conv k and δ represent the kth convolutional layer and the relu activation function, respectively After the task interaction features are obtained, the task interaction features are input into a TAP module, and then the features are flattened through a full connection layer and a sigmoid activation function to obtain the weight of each task interaction feature, and a multiplication operation is performed to obtain the features of the classification or positioning task where ωkis the k-th element of the layer attention, k is the k-th element of the layer attention, It is calculated from the task interaction features and can capture the dependencies between layers: wherein, and is a fully connected layer, and σ is a sigmoid activation function. inter The average pooling operation is used to obtain f inter ; Finally, the result Z of the classification or localization is obtained task : Z task = conv2(δ(conv1(F task ))) (4) Among them, F task for The connection features are used, and conv1 is a 1×1 convolution for dimensionality reduction. Then, the sigmoid activation function is used to transform Z. task Convert to dense classification scores or object bounding box 3. The method according to claim 2, wherein, The training of the feature extraction network includes the following steps: 31) The input substation equipment image is divided into patches by the Patch Partition layer for dimensionality reduction operation, and the input image size is HxWxC, where H and W represent the height and width of the image, respectively, and C is the channel number of the image. The size of each patch (P, P) = HW / N. The divided patch is linearly projected by the Linear Embedding layer and input into the Swin Transformer Block module for multi-head self-attention calculation to generate a feature map F1 with a size of F1; 32) the size of the above The feature map F1 is input into the Patch Merging layer for downsampling operation and then into the Swin Transformer Block module for multi-head self-attention calculation to generate a feature map F2 with a size of the feature map F2; 33) the size of the above The feature map F2 is input to the Patch Merging layer to generate a feature map F3 with a size of the feature map F3; 34) Training of the size adaptive convolution module: 341) The feature map F3 is input into the size adaptive convolution module, which is first subjected to ordinary convolution with the convolution padding being same, i.e. the input and output sizes are not changed, and the corresponding feature output size is The offset is denoted as offset, which represents the offset of each pixel index in the original feature map. Wherein, the sampling position set G is offset by ΔL n , ΔL n = 1, 2,..., N, to increase the receptive field range, N is the number of pixels in the grid, 2C3 represents the offset in the x and y directions, 1C3 channel represents the adjustment weight value Δm, and the range is between 0 and 1 after the Sigmoid function; 342) Add the index value of the pixel in the input feature map F3 to offset to obtain the offset position, that is, the coordinate value of each pixel in f3, and limit the value of position within the picture size range; l0 is any position on the output feature map: wherein, l n respectively represent the pre-designated offset amount at the nth position; ω(l n ) is the weight information of the sampling position; G = {(-1, -1), (-1, 0),..., (1, 0), (1, 1)} is a set of sampling positions; Δi n is a learnable offset for the nth position; Δm n is an adjustment weight parameter; x(l0+ l n + Δl n ) and y(l0) represent features at an arbitrary position l0from the input feature map x and the output feature map y, respectively. 343)Since the sampling points are after the irregular offset l n +Δl n is carried out, and the offset amount Δl n is a floating-point number, accurate pixel values cannot be obtained. In order to obtain accurate pixel values and carry out back propagation, a bilinear interpolation method is adopted to obtain the pixel corresponding to the coordinates. Take a coordinate (a, b) and convert it to floor(a), ceil(a), floor(b), and ceil(b) four integers, where floor() and ceil() operations are to take a, b down or up to get (floor(a), (floor(b)), (floor(a), ceil(b)), (ceil(a), floor(b)), and (ceil(a), ceil(b)) four coordinates; Each of the four coordinates corresponds to a pixel value in F3, and the pixel value x(l) at any position is calculated by bilinear interpolation: where l = lo + l n + Δl n denotes an arbitrary position after offset; m is the enumeration of the overall spatial position coordinates in the feature map; x(m) is the pixel value of the four neighboring integer coordinates around the feature map F3; H() is the weight corresponding to the four integer coordinates, and it is two-dimensional, consisting of two one-dimensional kernels: H(m, 1) = g(m x ,l x ) · g(m y ,l y )(7) wherein (m x ,l x ) and (m y ,l y ) are coordinate values in x and y offset directions, respectively; g(a, b) = max(0, 1 - |a - b|) limits the distance between the interpolation point and the four nearest integer coordinates to no more than 1 pixel, a and b being coordinate values. After all the pixels of position are obtained, the weight value parameter is adjusted by Δm, and finally the size of The new feature map F3' is input to the next stage. 344) The feature map F3' of the above size is input to the Patch Merging layer for downsampling operation and then to the Swin Transformer Block module for multi-head self-attention calculation to generate a feature map F4 of size The size-adaptive convolution module learns the high-level semantic information of the image, so that the sampling position is more consistent with the shape and size of the object itself, rather than fixed geometric structure sampling, which is more conducive to defect feature extraction. In the core module Swin Transformer Block, there is a multi-head attention for shifting windows, and the attention calculation of each head in the multi-head attention is as follows: Wherein, Q, K, V are query, key, value matrix respectively, K T is the transpose matrix of K; d is the vector dimension, B is the bias matrix, and SoftMax is the multi-classification activation function. 35) The stage1, stage2, stage3 and stage4 layers of the Swin-Transformer backbone network are used to represent the features of each scale in the overall feature map, and the power equipment defect detection image is subjected to data enhancement processing, and the enhanced data is used as the input of the feature extraction network by adding Gaussian noise, salt and pepper noise and adjusting the image brightness, and the overall feature map is input into the next stage feature fusion network.

4. The method according to claim 2, wherein, The training of the power equipment defect target detection model also includes an explicit alignment operation for obtaining better classification performance and positioning accuracy; the explicit alignment operation includes the following steps: 41) Obtain alignment classification task Q using spatial probability map N align : where the dense classification score is computed from the interaction features to learn the degree of consistency in spatial locations; the spatial bias is obtained to predict B align by bilinear interpolation: B align (i,j,c) = B(i + P(i,j,2 x c),j + P(i,j,2 x c + l),c) (10) where the object bounding box is the spatial offset map, computed from the interaction features to adjust the predicted bounding box at each location; (i,j,c) denotes the (i,j)th spatial location on the cth channel of the tensor, since the offset for each channel is learned independently, it means that each object's bounding box has its own offset; 42) Aligning the mapping N and P from the interactive feature stack for automatic learning: N = σ(conv2(δ(conv1(F inter )))) (11) P = conv4 (δ(conv3 (F inter ))) (12) conv1, conv3 are 1x1 convolution for dimension reduction operation; F inter is the task interaction feature; δ represents the relu activation function; σ is the sigmoid activation function; A classification loss function QualityFocalLoss is used to adjust the parameters in the network, and finally the accurate defect target in the substation equipment image is output, and the loss QFL(σ) formula is as follows: QFL(σ) = -a i *|x-σ| β *[(1-x)log(1-σ)+xlog(σ)] (13) where a i is the balance of positive and negative samples, σ is the predicted value, x is the continuous label, and β is a scale factor to control the weight decay rate. The CIOUloss is used as the regression loss function, and the loss Loss CIOU is as follows: where d denotes the minimum circumscribed rectangle surrounding the predicted and ground truth boxes, c, c gt denote the center points of the predicted and ground truth boxes, respectively, p denotes the Euclidean distance between the two center points, and b and w are the aspect ratios.

Images