A power transformation equipment defect image data expansion and data cleaning method

Abstract

The application provides a power transformation equipment defect image data expansion and data cleaning method, comprising: training a style transfer network by using a preset data set to obtain trained network weight parameters; traversing original power transformation equipment defect images and weak light scene images, and expanding defect image data according to the original power transformation equipment defect images and weak light scene images by using the style transfer network loaded with the network weight parameters to obtain low-light power transformation equipment defect image data, and verifying the style transfer effect; filtering low-quality generated images in the low-light power transformation equipment defect image data, and screening generated images similar to the distribution of real image set samples by using a classification model based on an adversarial verification method to obtain a final expanded low-light power transformation equipment defect image set. The application helps to solve the problem of weak generalization of a deep learning model in a weak light scene.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, in particular to a power transformation equipment defect image data expansion and data cleaning method. BACKGROUND

[0002] With the acceleration of the construction pace of new power systems, the operation and maintenance mode of substations begins to transition from unattended to intelligent. With the help of deep learning methods such as target detection and semantic segmentation, existing power transformation equipment defect images can be centrally processed and analyzed, which can greatly shorten the inspection cycle and improve the efficiency of equipment defect processing. A deep learning model with high accuracy and strong generalization needs a large number of representative labeled samples. However, existing power transformation equipment defect images are often collected in good lighting conditions and high visibility environments, which leads to poor generalization of the trained deep learning model in the morning, dusk, night, sandstorm, heavy rain, snow, and heavy fog scenes, and cannot adapt to the automatic inspection needs in weak light and adverse weather scenes. At the same time, it is expensive to collect a large number of defect images with weather, lighting, and other complex interference changes in actual substations, and a small number of samples cannot support the training and quantitative evaluation of deep learning models.

[0003] Therefore, how to rely on parallel vision technology to generate low-light power transformation equipment defect images using limited power transformation equipment defect images and various weak light scene images, and ensure the consistency of sample distribution to improve the generalization of deep learning models, is a problem that needs to be solved. SUMMARY

[0004] In order to overcome the shortcomings of the prior art, the purpose of the present application is to provide a power transformation equipment defect image data expansion and data cleaning method.

[0005] To achieve the above purpose, the present application provides the following solutions:

[0006] A power transformation equipment defect image data expansion and data cleaning method, comprising:

[0007] Training the style transfer network using a preset data set to obtain trained network weight parameters;

[0008] Iterating through original power transformation equipment defect images and weak light scene images, and using the style transfer network loaded with the network weight parameters to expand defect image data according to the original power transformation equipment defect images and weak light scene images, to obtain low-light power transformation equipment defect image data, and verify the style transfer effect;

[0009] Filtering out low-quality generated images in the low-light power transformation equipment defect image data, and using a classification model to screen out generated images similar to the sample distribution of the real image set relying on the adversarial verification method, to obtain a final expanded low-light power transformation equipment defect image set.

[0010] Preferably, the style transfer network is trained with a preset data set to obtain trained network weight parameters, including:

[0011] The MS COCO 2017 data set is packaged into a fixed batch size image data set, and the images are resized, randomly cropped and normalized for preprocessing to obtain training data;

[0012] The training data is input into the style transfer network for multiple rounds of training, and the weight parameters of the encoding and decoding blocks of the style transfer network are gradually fine-tuned to obtain trained network weight parameters; the encoding and decoding blocks include an encoding network and a decoding network; the training process of the encoding and decoding blocks at each stage is as follows: input features are obtained through the encoding network to obtain encoding features, and the input features are restored through the decoding network, and then the content reconstruction loss, image smoothing loss, content-aware loss and decoding block inversion loss between the input features and output features are calculated, and loss iteration, gradient backpropagation and parameter updating are completed until the set training rounds are reached, and then the weight parameters of the encoding and decoding blocks are saved.

[0013] Preferably, the original transformer equipment defect images and low-light scene images are traversed, and the style transfer network loaded with the network weight parameters is used to expand the defect image data according to the original transformer equipment defect images and low-light scene images to obtain low-light transformer equipment defect image data, and verify the style transfer effect, including:

[0014] An original content image is randomly selected from a transformer equipment defect image set, and a style image is randomly selected from a low-light scene image set S, feature extraction of the content image and the style image is completed using the trained encoding network to obtain a content encoding feature set and a style encoding feature set, and a high-frequency skip connection module is used to realize high-frequency component decomposition of the content image encoding feature to obtain a high-frequency residual component;

[0015] The content encoding features and the style encoding features of the first three stages are aligned in channel distribution by channel correlation-adaptive instance normalization, and are subjected to channel dimension compression and spatial dimension expansion operations by a decoding network, and after dimension transformation, the high-frequency residual component is fused to supplement spatial information to obtain content feature input for the next stage of style transfer; the fourth stage of style transfer takes the style encoding feature as input, aligns the content style feature in spatial distribution by spatial correlation-adaptive instance normalization, and restores the channel and spatial dimensions of the input image using the decoding network to obtain a stylized content image;

[0016] Calculate the structural similarity between the original content image and the stylized content image, the natural image quality evaluation value of the stylized content image, and the average style transfer time of a single image, and take the average value of the indicators as the final style transfer evaluation indicator.

[0017] Preferably, low-quality generated images in the low-light substation defect image data are filtered out, and a classification model is used to screen out generated images similar to the sample distribution of the real image set relying on the adversarial verification method, to obtain a final expanded low-light substation defect image set, comprising:

[0018] Thresholds of the structural similarity and the natural image quality evaluation value are set, and generated images of the low-light substation defect image data below the thresholds are deleted, and the perceptual hash value of the generated images is calculated, and generated images with the same perceptual hash value are deleted to obtain preliminary expanded images;

[0019] The same number of images as the real image set are randomly extracted from the preliminary expanded image set, and classification labels are made, and the real images are mixed and randomly divided into a training set and a verification set, and the adversarial verification model is trained;

[0020] The preliminary expanded image set is used as a test set, the trained adversarial verification model is used to complete the classification of the generated image set, and images with a predicted label of 1 are screened out until the generated image set is traversed, and a judgment threshold is set, and generated images below the threshold are filtered out to obtain a final expanded low-light substation defect image set conforming to the sample distribution of the real image.

[0021] According to the specific embodiments provided by the present application, the following technical effects are disclosed:

[0022] The present application provides a substation defect image data expansion and data cleaning method, comprising: training a style transfer network using a preset data set to obtain trained network weight parameters; traversing original substation defect images and weak light scene images, and using the style transfer network loaded with the network weight parameters to expand defect image data according to the original substation defect images and weak light scene images to obtain low-light substation defect image data, and verifying the style transfer effect; filtering out low-quality generated images in the low-light substation defect image data, and using a classification model to screen out generated images similar to the sample distribution of the real image set relying on the adversarial verification method to obtain a final expanded low-light substation defect image set. The present application helps to solve the problem of weak generalization of deep learning models in weak light scenes. BRIEF DESCRIPTION OF DRAWINGS

[0023] In order to make the technical solutions in the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description only constitute some embodiments of the present application, and other drawings can be obtained by those skilled in the art without any creative effort on the basis of these drawings.

[0024] Figure 1 The method flowchart provided for the embodiments of the present application is as shown in

[0025] Figure 2 The flowchart provided for the embodiments of the present application is as shown in

[0026] Figure 3 The multi-stage training strategy schematic diagram provided for the embodiments of the present application is as shown in

[0027] Figure 4 The data expansion schematic diagram provided for the embodiments of the present application is as shown in

[0028] Figure 5 The joint space-channel correlation adaptive instance normalization structure schematic diagram provided for the embodiments of the present application is as shown in

[0029] Figure 6 The effect comparison diagram of the method and the baseline style transfer method provided for the embodiments of the present application is as shown in DETAILED DESCRIPTION

[0030] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments only constitute some embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without any creative effort belong to the protection scope of the present application.

[0031] The purpose of the present application is to provide a power transformation equipment defect image data expansion and data cleaning method, which helps to solve the problem of weak generalization of deep learning model in weak light scene.

[0032] In order to make the above-mentioned purposes, characteristics and advantages of the present application more obvious and easy to understand, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0033] Figure 1 The method flowchart provided for the embodiments of the present application is as shown in Figure 1 The present application provides a power transformation equipment defect image data expansion and data cleaning method, which comprises:

[0034] Step 100: training the style transfer network by using a preset data set to obtain trained network weight parameters;

[0035] Step 200: Traverse the original power transformation equipment defect image and the weak light scene image, and use the style transfer network loaded with the weight parameters to expand the defect image data according to the original power transformation equipment defect image and the weak light scene image, obtain the low-light power transformation equipment defect image data, and verify the style transfer effect;

[0036] Step 300: Filter out the low-quality generated images in the low-light power transformation equipment defect image data, and use a classification model to screen out generated images similar to the sample distribution of the real image set relying on the adversarial verification method, to obtain the final expanded low-light power transformation equipment defect image set.

[0037] Optionally, in the embodiment, scene features are extracted from substation images in weak light scenes such as early morning, dusk and night, and severe weather scenes such as sand, rainstorm, snow and heavy fog, and the low-light scene is transferred to the power transformation equipment defect image through an adaptive instance normalization method of a joint space-channel feature distribution compensation mechanism, to generate a low-light defect image and clean the generated image set through an adversarial verification model, to ensure consistency with the original image sample distribution, so as to realize high-quality expansion of the low-light power transformation equipment defect image data.

[0038] The entire process of the power transformation equipment defect image data expansion and data cleaning method can be represented as Figure 2 .

[0039] Step 1: Style transfer network training; complete forward propagation, loss calculation and back propagation of the style transfer network by using the MS COCO 2017 dataset, and save the trained model weight parameters. Specifically, the style transfer network adopts a symmetrical encoding-decoding structure, the encoding network is divided into 4 independent encoding blocks by the maximum pooling layer, the decoding network is a flip-symmetrical structure of VGG16, also divided into 4 independent decoding blocks by the maximum pooling layer. It should be noted that the convolutional layers in the encoding-decoding blocks all use reflective padding to avoid the generation of structural artifacts. The style transfer network adopts a multi-stage training strategy, each stage carries out strengthening training in the aspects of perception and reconstruction for a single encoding-decoding block. The schematic diagram is shown in Figure 3 . The training steps are as follows:

[0040] S1: Randomly select an image; randomly select a batch of images from the MS COCO 2017 dataset, adjust the size of the batch of images to 512x512, and randomly crop 256x256 images to be converted into input features In the embodiment, the batch size N is 16, the channel number C is 3, the width H and the height W are 256.

[0041] S2: Multi-stage style transfer network training; In the first stage of encoder-decoder network training, the input feature I passes through the encoder network φ1(·) to obtain the encoded feature φ1(I), and then passes through the decoder network... Restore input features Then, the content reconstruction loss and image smoothing loss between the input and output features are calculated, and loss iteration, gradient backpropagation, and parameter updates are performed until the set number of training epochs are reached. Finally, the weight parameters of the encoder-decoder network are saved. In this example, the number of training epochs is set to 10; the encoder network is loaded with weights pre-trained on ImageNet, while the decoder network is not loaded with weights. Total loss Content Reconstruction Loss L cc The calculation formula is as follows:

[0042]

[0043] In equation (1), and The content reconstruction loss and image smoothing loss are both represented by mean squared error loss. In equation (2), I' ij These are the element values ​​at position (x,y) of the l-th feature map of the input feature I and the output feature I′, respectively.

[0044] Then, the nth stage of the encoder-decoder network training is performed; the input feature I passes through the nth encoder network φ. n (·) Obtain the encoded feature φ n (I), and through the nth decoding network Restore input features Then, the content reconstruction loss, image smoothing loss, content-aware loss, and decoder block inversion loss between input and output features are calculated. Loss iteration, gradient backpropagation, and parameter updates are performed until the set number of training epochs are reached. Finally, the weight parameters of the encoder-decoder network are saved. In this example, the training epochs are set to 10, and the encoder-decoder network blocks are loaded with the weight parameters saved from N-1 training iterations. Total loss. Content-perceived loss Decoding Inversion Loss The calculation formula is as follows:

[0045]

[0046] In equation (4), the content reconstruction loss Image smoothing loss Content-perceived loss and decoding inversion loss Mean squared error loss is used in all cases. In equation (4), The input features are processed through the nth encoding network φ n (I) output characteristics, output feature the nth encoding network φ n output feature of (I). In formula (5), the (n-1)th encoding network φ n-1 output feature of (I), output feature of the nth decoding network.

[0047] Step 2: low-light power equipment defect image data augmentation; traverse the original power equipment defect image and the substation weak light scene image, use the style transfer network loaded with the weight to complete the defect image data augmentation, and verify the style transfer effect. Specifically, first, select the power equipment defect image and the substation weak light scene image to construct the content image set C and the style image set S, respectively. In this example, the selected power equipment defect is oil leakage, and the substation scene image selects three types of weak light scenes in the morning, dusk and night, and four types of adverse weather scenes such as sand, rain, snow and heavy fog. The image augmentation schematic diagram is shown in Figure 4 , which includes the following steps:

[0048] S3: randomly select content and style images and extract content and style features; randomly select a content image c from the set C and a style image s from the set S, and use the trained encoding network φ4(·) to complete feature extraction of the content image and the style image, to obtain the content encoding feature and the style encoding feature set wherein represents the nth layer encoding block of the fourth encoding network, and then the high-low frequency component decomposition of the content image encoding feature is realized by using the high-frequency skip connection (HSC) module to obtain the high-frequency residual component of the nth encoding block The calculation method of the high-frequency residual component

[0049]

[0050] In formula (6), (·) up represents the up-sampling operation realized by bilinear interpolation, and Avgpool(·) represents the average pooling operation.

[0051] S4: multi-stage style transfer; the content encoding feature and the style encoding feature Content style features are aligned across channel distributions using Channel Correlation-Adaptive Instance Normalization (C-AdaIN), and then decoded using blocks. Channel compression and spatial expansion operations are performed. Channel compression is implemented using 3×3 convolution, and spatial expansion is implemented using bilinear interpolation. After dimensional transformation, these operations are combined with high-frequency residual components. The fusion process supplements spatial information, thereby obtaining the content feature input for style transfer in the (m+1)th stage.

[0052] The fourth stage of style transfer is Style coding feature φ 4-1 (s) is the input, and Spatial correlation-Adaptive Instance Normalization (S-AdaIN) is used to align the content style features in spatial distribution. Then, the decoding block is used. By restoring the channels and spatial dimensions of the input image, a stylized content image is obtained. A schematic diagram of the C-AdaIN and S-AdaIN structures is shown below. Figure 5 As shown, the conversion process of C-AdaIN between content encoding feature φ(c) and style encoding feature φ(s) is as follows:

[0053] c s =c cor ×AdaIN(φ(c),φ(s)) (7)

[0054] c cor =c att ×sigmoid(c att ×(s att ) T (8)

[0055]

[0056] In equation (7), c s c represents the output feature map after style transfer. cor The cross-correlation matrix representing the channel feature distributions between content image coding features and style image coding features is shown in Equation (8), where AdaIN(·) represents the adaptive instance normalization operation, as shown in Equation (9). In Equation (8), c att With s attrespectively represent the channel weight between the content image coding feature and the style image feature, which is obtained by global adaptive pooling along the channel direction. sigmoid(·) represents the normalization operation using the sigmoid function. T represents the dimension transposition operation. In equation (9), Norm() represents the normalization operation along the channel direction, σ(·) represents the variance calculation operation, and u(·) represents the mean calculation operation. C-AdaIN aligns the statistical feature distribution of the content image and the style image while aligning the correlation relationship in the channel dimension of the two images, so as to reduce the feature transfer error.

[0057] wherein the conversion process of S-AdaIN on the content coding feature φ(c) and the style coding feature φ(s) is as follows:

[0058] S-AdaIN(x,y)=σ(s cor ×σ(φ(s)))×Norm(φ(c))+u(s cor ×u(φ(s))) (10)

[0059] s cor =Sigmoid(Norm(φ(s))×(Norm(φ(c))) T ) (11)

[0060] In equation (10), σ(·) represents the variance calculation operation, and u(·) represents the mean calculation operation. Norm() represents the normalization operation along the channel direction, and S cor represents the cross-correlation matrix of the spatial feature distribution between the content image coding feature and the style image coding feature. The calculation is shown in equation (11), wherein sigmoid(·) represents the normalization operation using the sigmoid function. T represents the dimension transposition operation.

[0061] S5: sample generation and index calculation; after traversing all the style image sets, the structural similarity (SSIM) between the content image and the stylized content image, the natural image quality evaluator (NIQE) of the stylized content image, and the average style transfer time of a single image are calculated, and the average value of the indexes is taken as the final style transfer evaluation index. Among them, the larger the average SSIM, the closer the structure of the transferred image and the content image, and the smaller the average NIQE value, the closer to the natural real image. In this embodiment, 10 oil leakage images of the transformer equipment and 8 scene images of the transformer station (2 night scene images) are randomly selected, and a total of 80 oil leakage images of the transformer equipment under low light are generated. Figure 6The effect comparison chart of the patent method and the baseline method PhotoWCT2 is given, the core component of PhotoWCT2 adopts Whiten-Color Transform (WCT), and the core component of the patent method is jointSpatial-Channel feature distributions Adaptive Instance Normalization (CS-AdaIN) adopting a joint spatial-channel feature distribution compensation mechanism. Table 1 gives the comparison results of image transfer quality and image transfer speed quantitative indicators of different style transfer network images, wherein the first row is the baseline method, and the second row is the patent method, which shows that the image transfer speed of the patent method is faster than that of the baseline method, and the image transfer quality is better than that of the baseline method under different resolutions.

[0062] Table 1

[0063]

[0064] Step three: low-light variable device defect image data cleaning; specifically, first, filter out low-index images and highly similar images to obtain a preliminary expanded low-light variable device defect image set. Then, classification labels are made for the real variable device defect image set and the generated low-light variable device defect image set, and the same number of samples as the real variable device defect image set are sequentially selected from the generated low-light variable device defect image set, the training set and the validation set are divided, and the training of the real image and the generated image classification model is completed. Then, the generated low-light variable device image set is used as the test set, and the generated images with sample distribution similar to the real image set are selected until the entire generated image set is traversed. The generated images with sample distribution similar to the real image sample set under different model test conditions are counted and selected, and the final expanded low-light variable device defect image set is obtained.

[0065] S6: low-quality image filtering; for the generated low-light variable device defect image set, set the SSIM and NIQE thresholds, delete the generated images below the threshold, and calculate the perceptual hash value of the generated images by using the perceptual hash function. For the generated images with the same perceptual hash value, delete them. In this example, taking the leakage oil image data cleaning as an example, the NIQE average value of 10 real variable device defect images is used as the threshold, and the SSIM threshold is set to 0.75. After filtering out low-quality generated images, a preliminary expanded leakage oil image is obtained.

[0066] S7: Adversarial verification model training; randomly extract the same number of images from the generated image set as the real image set, and make classification labels, mix with real images, randomly divide the training set and the verification set, and complete the training of the adversarial verification model. In this embodiment, 10 images are sequentially extracted from the generated leakage oil image set, and the label value is 0. After mixing with 10 real image leakage oil sets with label value 1, the training set and the verification set are divided, and the proportion is 7:3. The adversarial verification model selects the classification network ResNet50.

[0067] S8: Sample screening; using the generated image set as the test set, using the trained adversarial verification model to complete the classification of the generated image set, screening out the images with predicted label 1 until the generated image set is traversed, that is, the generated images similar to the real image sample distribution, counting the number of times each generated image is classified as a real image sample by different samples, setting a judgment threshold, and filtering out the generated images below the threshold, thereby obtaining the generated images conforming to the real image sample distribution. In this embodiment, the adversarial model is trained 6 times by combining different generated images, the judgment threshold is set to 3, and the final expanded low-light leakage oil image of the transformer equipment is obtained.

[0068] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple, and the relevant parts can be referred to the method part.

[0069] The principles and implementation modes of the present application are described by applying specific examples in this paper. The above embodiment description is only used to help understand the method of the present application and its core idea; at the same time, for those skilled in the art, according to the idea of the present application, the specific implementation mode and application range will be changed. In conclusion, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A power transformation equipment defect image data augmentation and data cleaning method, characterized in that, The method comprises the following steps: training the style transfer network by using a preset data set to obtain trained network weight parameters; traversing original transformer equipment defect images and weak light scene images, and using the style transfer network loaded with the network weight parameters to expand defect image data according to the original transformer equipment defect images and the weak light scene images, to obtain low-light transformer equipment defect image data, and to verify the style transfer effect; filtering low-quality generated images in the low-light transformer equipment defect image data, and using a classification model to screen out generated images similar to the sample distribution of the real image set by relying on the adversarial verification method, to obtain a final expanded low-light transformer equipment defect image set; training the style transfer network by using a preset data set to obtain trained network weight parameters, comprising: packing the MS COCO 2017 data set into an image data set with a fixed batch size, and performing size adjustment, random cropping and normalization preprocessing on the images to obtain training data; inputting the training data into the style transfer network for multi-round training, and gradually fine-tuning the weight parameters of the encoding and decoding blocks of the style transfer network to obtain trained network weight parameters; the encoding and decoding blocks include an encoding network and a decoding network; the training process of the encoding and decoding blocks at each stage is as follows: input features pass through the encoding network to obtain encoded features, and the input features are restored through the decoding network, then the content reconstruction loss, image smoothing loss, content-aware loss and decoding block inversion loss between the input features and output features are calculated, and loss iteration, gradient backpropagation and parameter updating are completed until the set training rounds are reached, and then the weight parameters of the encoding and decoding blocks are saved; traversing original transformer equipment defect images and weak light scene images, and using the style transfer network loaded with the network weight parameters to expand defect image data according to the original transformer equipment defect images and the weak light scene images, to obtain low-light transformer equipment defect image data, and to verify the style transfer effect, comprising: randomly selecting an original content image from the transformer equipment defect image set and a style image from the low-light scene image set S, using the trained encoding network to complete feature extraction of the content image and the style image to obtain content encoding features and style encoding features, and using a high-frequency skip connection module to realize high-frequency and low-frequency component decomposition of the content image encoding features to obtain high-frequency residual components; the content encoding features and the style encoding features of the first three stages are aligned in channel distribution by using channel correlation-adaptive instance normalization, and are subjected to channel dimension compression and spatial dimension expansion operations by using a decoding network, and after dimension transformation, the high-frequency residual components are fused to supplement spatial information, to obtain content features input for style transfer in the next stage; the fourth stage of style transfer takes the style encoding features as input, aligns the content features in spatial distribution by using spatial correlation-adaptive instance normalization, and restores the channel and spatial dimensions of the input image by using a decoding network to obtain a stylized content image. Calculate the structural similarity between the original content image and the stylized content image, the natural image quality evaluation value of the stylized content image, and the average style transfer time of a single image, and take the average value of the indicators as the final style transfer evaluation indicator. 2.The method of claim 1, wherein, The low-quality generated images in the low-light power transformation equipment defect image data are filtered out, and the generated images similar to the distribution of the real image set samples are screened out by using a classification model relying on the adversarial verification method, to obtain the final expanded low-light power transformation equipment defect image set, including: Thresholds of the structural similarity and the natural image quality evaluation value are set, the generated images of the low-light power transformation equipment defect image data below the thresholds are deleted, and the perceptual hash values of the generated images are calculated, the generated images with the same perceptual hash values are deleted, to obtain the preliminary expanded images; The same number of images as the real image set are randomly extracted from the preliminary expanded image set, and classification labels are made, the real images and the images are mixed, and the training set and the verification set are randomly divided, and the training of the adversarial verification model is completed; The preliminary expanded image set is taken as the test set, the trained adversarial verification model is used to complete the classification of the generated image set, the images with the predicted label of 1 are screened out until the generated image set is traversed, a judgment threshold is set, and the generated images below the threshold are filtered out, to obtain the final expanded low-light power transformation equipment defect image set conforming to the distribution of the real image samples.

Images