A method for augmenting image data of transmission line insulators based on a dual attention mechanism

Abstract

This invention discloses a method for augmenting image data of transmission line insulators based on a dual attention mechanism. This method belongs to the field of computer vision technology. Based on existing insulator samples, this invention implements a method for augmenting image data of transmission line insulators based on a parallel attention mechanism, focusing on the transformation of insulators within the same category. Compared with traditional techniques, its advantages are: 1) This invention combines channel attention and self-attention mechanisms to accurately control the main target to be transformed. Even when the main target is occluded, accurate transformation can still be performed, greatly improving the accuracy of the transformation. 2) This invention utilizes Cycle-GAN networks and GAN network algorithms to simultaneously address the data scarcity problem of normal and defective insulators. Once the model training is complete, minimal data processing is required; mutual transformation of different categories of insulators is performed on existing data to generate new data from existing data.

Description

Technical Field

[0001] This invention belongs to the field of computer vision technology and relates to an image generation algorithm that uses an attention mechanism. Background Technology

[0002] The condition of insulators in power transmission environments is crucial for ensuring transmission safety. Traditional methods of monitoring insulator conditions are excessively resource-intensive. Using deep learning for insulator condition monitoring requires a large number of real-world insulator defect samples as training data for the deep learning model. Obtaining a large number of real-world insulator image samples is a major challenge. Currently, there are two methods for acquiring this dataset: one is manual on-site collection, which is inefficient, slow, and inherently dangerous due to the complex environment of transmission lines and the short timeframe of defective insulator samples within the transmission environment, significantly limiting the number of samples collected. The other method involves photographing suspended insulators from multiple angles. While this method can acquire more samples, it severely limits the diversity of sample types. Infinitely photographing and sampling a limited number of insulators fails to improve the richness of background information, exacerbates data redundancy, and hinders subsequent deep learning model training. Even if a deep learning model is trained, its generalization ability will be limited, and the accuracy of detecting insulator samples in real-world transmission environments will not meet application-level requirements. Summary of the Invention

[0003] The purpose of this invention is to provide a data generation method for insulator sample datasets in power grid transmission line environments. This method can expand the dataset by converting between different types of samples based on existing datasets.

[0004] To achieve the above-mentioned tasks and overcome the shortcomings of traditional generation methods, this invention provides a method for augmenting transmission line insulator image data based on a dual attention mechanism. This method includes:

[0005] Step 1: Preprocessing the initial dataset;

[0006] Based on manually collected samples, the samples were cropped and classified according to insulator type: composite, ceramic, and glass. After classification and rotation, a dataset of the same resolution was created. Based on the created dataset, a dataset of insulator samples with defects was labeled. Each dataset contains at least 2,000 different samples, and this dataset is labeled as dataset two.

[0007] Step 2. Establish a lossless insulator conversion network;

[0008] The lossless insulator conversion network includes: a first attention module, a generator Gy, a discriminator Dy, a generator Gx, a second attention module, and a discriminator Dx;

[0009] The conversion method is as follows:

[0010] Step 2.1: The lossless insulator image is used as the input of the first attention module. The output of the first attention module and the lossless insulator image X are used together as the input of the generator Gy. The generator Gy generates image Y. After image Y is judged by the discriminator Dy, if it meets the requirements, it is output as an augmented image.

[0011] Step 2.2: Image Y is used as the input of the second attention module. The output of the second attention module and image Y are used together as the input of generator Gx. Generator Gx generates image X. After image X is judged by discriminator Dx, if it meets the requirements, it is output as an augmented image.

[0012] Step 2.3: Treat image X as a lossless insulator image, and return to step 2.1:

[0013] Perform the above steps 2.1 to 2.3 on all non-destructive insulator images until the required number of augmented images are generated;

[0014] Step 3: Establish a defective insulator conversion network;

[0015] The defective insulator transformation network includes: an attention module, a generator Gy, a generator compensateate, a discriminator D1, and a discriminator Dy;

[0016] The conversion method is as follows:

[0017] The image of the defective insulator is input to the attention module. The output of the attention module and the image of the defective insulator are used as the input of the generator Gy. The output of the attention module and the output of the generator Gy are used as the input of the generator compensate. The generator compensate generates an image. The generated image is passed through the discriminator D1 and the discriminator Dy respectively. If either discriminator considers the generated image to meet the requirements, it outputs the augmented image.

[0018] The attention module is processed as follows:

[0019] Figure 5 As shown, the same vector x∈R containing image information is used. c×w×hThe inputs are fed into different feature transformers f(x), g(x), and h(x). The attention region is calculated using f(x), g(x), and h(x). C represents the number of channels in the vector, and w×h represents the vector dimension. All three transformation modules are 1×1 convolutions, differing only in the number of channels in each path. In the transformers, these are referred to as query, key, and value, respectively. Because the convolution operation sets parameters for stride and kernel size, indexing three convolutions can reduce the number of image channels. Previously, networks typically added an activation function at the end of this process to introduce more non-linear transformations, thereby enhancing the neural network's ability to express non-linear distributions. The mathematical expression for the three-channel transformation is shown in the formula.

[0020] All three are weight matrices that need to be learned and updated.

[0021] The output value f(x) of channel one i Transpose the output and compare it with the output value g(x) of channel two. j Multiply them, as shown in the following formula:

[0022] s ij =f(x) i ) T g(x j The results obtained are s ij After softmax normalization

[0023] Figure 6 In the channel attention module, the processing method for each channel is as follows: X∈R H×W×C ,U∈R H×W×C F tr It is a transformation structure; the learning set of the convolutional kernel is represented as V = (v1v2v3…v…). c ), v c This represents the set of all parameters of the c-th convolutional kernel, and the final output is represented by U = (u1u2u3…u ... c ) indicates that u c The calculation is as follows:

[0024] In the formula, * represents the convolution operation. This represents the convolution kernel with s channels and a dimension of 2, x sThis represents the s-th input. After convolution on the channels, the input is finally added to the initial input vector. The feature space relationship learned in this way is a joint expression that simultaneously includes the channel feature relationship and the spatial relationship learned by the convolution kernel. This abstracts the hybrid relationship, enabling a model to directly learn the channel feature relationship in the entire sample. Finally, after transformation, the feature of interest is obtained. The feature is stored in the form of H×W×C, or it can be converted into a visual form through decoding.

[0025] Due to the limitation of convolution kernel size, a larger convolution kernel will result in a quadratic increase in the number of parameters. Therefore, the convolution kernel should generally not be set too large. This also leads to the convolution kernel operation always being performed within a local range. As a result, it is difficult for U to obtain a wider range of information to learn the relationship between channels. Channel attention mechanisms generally combine compression operations to obtain an encoded global feature on the previous channel. Through global average pooling of the convolutional layer, the feature information is compressed into a single channel descriptor. The detailed calculation method is shown in the following formula:

[0026]

[0027] The above formula represents feature compression, using a real number to represent a two-dimensional feature channel. This real number reflects the global receptive field. The number of input channels and the dimension of the output features are matched, so that the feature channels correspond exactly to the global distribution. The initial feature input dimension is H×W×C, where H represents height, W represents width, and C represents the number of channels. The input is first compressed, and the dimension is transformed to 1×1×C. Convolution is used to compress the H×W dimension features into one dimension. After the features become one-dimensional, the information contained in this dimension is the global information of the previous H×W dimension features, and the receptive area is greatly expanded.

[0028] Even after compression, the data features still need to utilize the information they aggregate to incentivize the subsequent learning process in order to fully fit the channel dependencies. After obtaining 1-dimensional C-channel features through compression, a fully connected layer is used to learn the weights of each channel to determine their influence. Once the weights of each channel are obtained, they are applied to the corresponding paths of the input features, as shown in the following formula.

[0029]

[0030] σ represents the activation function, typically the sigmoid activation function. These two values ​​represent the weight matrix parameters of the two preceding connected layers, with r representing the scaling parameter, which takes a value of 8. The purpose of r is to reduce the number of channels in the model; a value that is too large will increase the computational cost. The weight parameters of the activation function can be seen as the proportion of importance of the corresponding channels.

[0031] The compressed features in the channel are represented as follows After fully connected layer operations W1×Z and W2×Z, the output dimension remains 1×1×c. Finally, the sigmoid function is used to obtain s. After fusing information from each channel through the two fully connected layers, the feature map information s is obtained. Then, the original U is processed, u... c Let s represent a two-dimensional matrix. c The corresponding activation value of the learned channel is equivalent to the activation value of u. c The values ​​of the matrix are all multiplied by s c For the information that is needed, the output is set to 0 for the regions that are not of interest. This corresponds to algorithm F. scale As shown in the following formula.

[0032] X c =F scale (u c s c )=u c s c .

[0033] This invention, based on existing insulator samples, focuses on the transformation of insulators within the same category and implements a parallel attention mechanism-based image data augmentation method for transmission line insulators. Compared with traditional techniques, its advantages are: 1) This invention combines channel attention and self-attention mechanisms to accurately control the main target to be transformed, and can accurately transform even when the main target is occluded, greatly improving the accuracy of the transformation. 2) This invention utilizes Cycle-GAN networks and GAN network algorithms to simultaneously address the data scarcity problem of normal and defective insulators. Once the model training is complete, minimal data processing is required; different categories of insulators are mutually transformed based on existing data, generating data from data. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the overall model training process of the present invention.

[0035] Figure 2 This is a structural diagram of the lossless insulator conversion network model of the present invention;

[0036] Figure 3 This is a structural diagram of the defect insulator generation network model of the present invention;

[0037] Figure 4 This is a structural diagram of the parallel dual attention mechanism of the present invention;

[0038] Figure 5 The intention behind the self-attention mechanism structure;

[0039] Figure 6This is a schematic diagram of the channel attention module structure;

[0040] Figure 7 A schematic diagram of the structure for converting a glass insulator to a ceramic insulator;

[0041] Figure 8 A schematic diagram of the structure of a ceramic insulator being converted to a glass insulator;

[0042] Figure 9 A schematic diagram of a glass insulator converted into a composite insulator;

[0043] Figure 10 This is a schematic diagram of a composite insulator-to-glass insulator structure. Implementation

[0044] Step 1. Initial Dataset Preprocessing: Based on previously manually collected samples, the data is cropped and classified according to insulator type (composite, ceramic, glass) to create Dataset 1 with the same resolution. Using this dataset, a dataset of defective insulator samples is generated using PS (Photoprocessor). Each dataset contains at least 2000 different samples; this dataset is labeled Dataset 2.

[0045] Step 2. Network Construction: 1) Build two networks, namely, an attention generation network that uses a channel attention module and a self-attention module in combination; 2) Build two generation networks, using convolutional neural networks and skip-layer structures to build an image generation network; 3) Build two discriminator networks, using pure convolution to judge the input information.

[0046] Step 3. Model Training: Select 1000 images of each of the two different types of insulators as the training set, and use the Cycle-GAN training method to perform local transfer between the two types of insulators.

[0047] Step 4. Network Reconstruction: 1) Deconstruct the original Cycle-GAN structure, retaining the trained generator, discriminator, and attention generation model; 2) Build new discriminator and generator models; 3) Insert the two models into the discriminator and generator positions of the Cycle-GAN model to form a new network model with a traditional GAN ​​network architecture.

[0048] Step 5. Model Training: Use the defective insulator dataset 2 as the training set for this training, and perform local generation transformation based on the region learned by the attention module so that the input conforms to the data feature distribution of the given dataset 2 after transformation.

[0049] Step one involves detailed steps including:

[0050] Step 1-1: Manually classify the sample types into composite insulators, ceramic insulators, and glass insulators. Perform the first augmentation by rotation. When cropping the image, store it in 512*512 pixel format to be suitable for use in other network models.

[0051] Step two includes the following detailed steps:

[0052] Step 2-1: The generator's input image is an occluded image formed by superimposing the attention map extracted by the attention module and the original samples. The output image is then superimposed with the original samples occluded in the opposite direction of the attention map. This method can ensure the maximum preservation of background information during the conversion process.

[0053] In step 2-2, during the discriminator construction, a fully convolutional discriminator was used to distinguish each local part of the generated samples in order to output the authenticity of the images in batches.

[0054] Steps 2-3 involve the attention module, which combines channel attention and self-attention. The self-attention module uses three convolutional operations to obtain three feature matrices with different weights: query, key, and value. In image processing, these three matrices represent extracted features in a certain dimension. The specific calculation methods for query and key features are as follows:

[0055]

[0056] Essentially, it calculates the degree of mutual influence between pixels, β. i,j This represents the correlation weight coefficient of the pixels in the i-th and j-th regions.

[0057] The self-attention module can be represented as:

[0058]

[0059] The channel attention module is implemented through compression and activation. After converting the image into a feature matrix, it is compressed into a 1D c-channel vector matrix. Then, the matrix is ​​learned back to the original data dimension through the weights of the fully connected layer. The calculation is expressed as follows:

[0060]

[0061] σ represents the activation function operation. In this method, the sigmoid activation function is used, and w represents the two weight matrices.

[0062] Step 3 involves the following detailed operations: For data A, which is lacking in the existing dataset, and data B, which is relatively abundant, select 1000 images of each type of data. These images will be used for training the A→B transformation and for the same sample input to the generator during the B→A transformation process. The discriminator will also use samples from the opposite side as input to update its parameters.

[0063] Step four involves detailed steps, including:

[0064] Step 5-1: Disassemble the network modules trained in the first stage, retain the original generator, discriminator and attention modules, and select the corresponding trained modules to build the network to be used in the next step for the next task.

[0065] Step 5-2: Add a secondary generator after the generator module. Add a skip layer structure to the generator network structure to increase gradient propagation capability and avoid ineffective parameter learning. Add a discriminator at the discriminator position. The discriminator adopts an encoder-decoder structure. The fully convolutional autoencoder structure can have better constraint ability in adjusting the overall realism of the training samples.

[0066] Step 5 details the following steps: Use the defective insulator dataset 2 as the real sample input for the training process to train the new GAN network model. Under the joint constraints of the discriminator trained in the first stage and the newly added discriminator, the parameters of the secondary generator are updated and trained. Different discriminators and generators can be selected according to the required insulator type.

[0067] Figure 7 The results demonstrate the effect of converting glass insulators to ceramic insulators using network input data. After 80 epochs of training with a batch size of 1, the conversion performance is quite good. Figure 7 The left column shows the initial input samples, the middle black and white layer represents the insulator distribution area learned by the attention module, and the right image is the image processed by the transformation network. After 80 epochs of learning, the network model accurately controls the insulator distribution area during sample transformation and completely preserves the background area of ​​the sample. Even when processing samples with complex backgrounds, Figure 7 The second and third rows of samples contain complex backgrounds common to power transmission lines, such as iron towers. The designed attention module can accurately find the shape features of glass insulators in both data distributions and exclude similar features such as iron frames. The generator can also accurately transform the target based on the attention distribution map.

[0068] Figure 8 The image shows the effect of converting ceramic insulators into glass insulators. The color gamut characteristics are well converted. However, the glossiness of glass insulators is a detail that current network models cannot generate. For samples with strong reflectivity, Figure 8 In the first row of samples, the light and shadow details of glass insulators are difficult to generate with existing models. Regarding the attention module's learning, while it can accurately locate the shape features of ceramic insulators, with few samples, when the insulator edge has high similarity to the background edge, the recognition area can spread to the background adjacent to the insulator. Given the current conversion effect, the number of glass insulators far exceeds that of ceramic insulators. Converting the data from glass insulators to ceramic insulators could further expand the number of ceramic insulators.

[0069] Figure 9 In the training process of converting glass into insulators, the insulator samples can achieve accurate conversion of specific targets. The attention is somewhat concentrated on the background area. Analyzing the data samples, the possible reason is that the background features of the two datasets are too similar, which makes the attention network identify the background area as the main feature during the learning process. Another reason is the pixel count of the image samples of composite insulators. The weight of background area recognition can be reduced by appropriately increasing the number of samples with different background styles.

[0070] according to Figure 10 Analysis of the conversion test from composite insulator to glass insulator shows that when there is a large area of ​​background similar to the color gamut and structure of the insulator sample, the attention network will also identify the background as an insulator. The color gamuts of the two are too similar, and the background also has similar hierarchical features to the insulator. This situation will not occur in other scenarios.

[0071] Overall, this invention is highly effective in the main feature of insulator conversion. It can expand the number of samples through conversion, and a single training model can simultaneously convert between two types of insulators, which is highly efficient. The training time for 1000 samples to convert is within 3 hours, and the training cost is low.

Claims

1. A method for augmenting image data of transmission line insulators based on a dual attention mechanism, the method comprising: Step 1: Preprocessing the initial dataset; Based on manually collected samples, the samples were cropped and classified according to insulator type: composite, ceramic, and glass. After classification and rotation, a dataset of the same resolution was created. Based on the created dataset, a dataset of insulator samples with defects was labeled. Each dataset contains at least 2,000 different samples, and this dataset is labeled as dataset two. Step 2. Establish a lossless insulator conversion network; The lossless insulator conversion network includes: a first attention module, a generator Gy, a discriminator Dy, a generator Gx, a second attention module, and a discriminator Dx; The conversion method is as follows: Step 2.1: The lossless insulator image is used as the input of the first attention module. The output of the first attention module and the lossless insulator image X are used together as the input of the generator Gy. The generator Gy generates image Y. After image Y is judged by the discriminator Dy, if it meets the requirements, it is output as an augmented image. Step 2.2: Image Y is used as the input of the second attention module. The output of the second attention module and image Y are used together as the input of generator Gx. Generator Gx generates image X. After image X is judged by discriminator Dx, if it meets the requirements, it is output as an augmented image. Step 2.3: Treat image X as a lossless insulator image, and return to step 2.1: Perform the above steps 2.1 to 2.3 on all non-destructive insulator images until the required number of augmented images are generated; Step 3: Establish a defective insulator conversion network; The defective insulator transformation network includes: an attention module, a generator Gy, a generator compensateate, a discriminator D1, and a discriminator Dy; The conversion method is as follows: The image of the defective insulator is input to the attention module. The output of the attention module and the image of the defective insulator are used as the input of the generator Gy. The output of the attention module and the output of the generator Gy are used as the input of the generator compensate. The generator compensate generates an image. The generated image is passed through the discriminator D1 and the discriminator Dy respectively. If either discriminator considers the generated image to meet the requirements, it outputs the augmented image. The attention module is processed as follows: the same vector containing image information into different feature transformers f(x), g(x) and h(x), and the regions focused on are calculated by f(x), g(x) and h(x), C represents the number of channels of the vector, and w x h represents the dimension of the vector; the three conversion modules are all 1 x 1 convolution, the difference lies in that the number of channels of the three paths is inconsistent, and in the transformer, they are called query, key and value respectively; The mathematical expression for the three-channel transformation is shown in the formula; ； wherein , all of which are weight matrices that need to be learned, where ; The output value of channel one Transpose the output and compare it with the output value of channel two. Multiplication is shown in the following formula: ； Results After softmax normalization, an attention vector can be obtained; The processing method for each channel is as follows: , F tr It is a transformation structure; the learning set of the convolutional kernel is represented as , This represents the set of all parameters of the c-th convolutional kernel, and the final output is... express, The calculation is as follows: ； In the formula, * represents the convolution operation. , , ; This represents the convolution kernel for the s-channel, with a dimension of 2. Let represent the s-th input; after convolution on the channel, the input is finally added to the initial input vector; the feature information is compressed into a single channel descriptor, and the detailed calculation method is shown in the following formula: ， ； The above formula represents feature compression, using a real number to represent a two-dimensional feature channel. This real number reflects the global receptive field. The number of input channels and the dimension of the output features are matched, so that the feature channels correspond exactly to the global distribution. The initial feature input dimension is H×W×C, where H represents height, W represents width, and C represents the number of channels. The input is first compressed, and the dimension is transformed to 1×1×C. Convolution is used to compress the H×W dimension features into one dimension. After the features become one-dimensional, the information contained in this dimension is the global information of the previous H×W dimension features, and the receptive area is greatly expanded. After obtaining the 1D C-channel features through compression, a fully connected layer is used to learn the weights of each channel to determine their influence. The weights of each channel are then applied to the corresponding paths of the input features, as shown in the following formula. ； The activation function is typically represented by the sigmoid activation function. , where and represent the weight matrix parameters of the two preceding connected layers, and r represents the scaling parameter; The compressed features in the channel are represented as follows After operation through the fully connected layer and The output dimension remains 1×1×c, and finally, after passing through the sigmoid function, s is obtained. After fusing the information from each channel in the two fully connected layers, the feature map information s is obtained, and then the original U is processed. Represent a two-dimensional matrix, The activation values ​​of the corresponding channels learned are equivalent to... The values ​​of the matrix are all multiplied by For areas of interest, the necessary information is retained, while the output is set to 0 for areas of no interest. This corresponds to the algorithm. As shown in the following formula; 。

Images