A Power Image Defect Detection Method Based on Dynamic Activation Heatmap

Abstract

This invention discloses a power image defect detection method based on dynamic activation heatmaps, comprising the following steps: 1. Collection and preprocessing of normal / defective power images, dividing them into training, validation, and test sets; 2. Mapping power image samples to a deep feature space using a CNN-based network; 3. Selecting a target layer and generating a coarse activation heatmap; 4. Normalizing the coarse activation heatmap and downsampling it to the same size as the deep feature map in step 2 to obtain an equivalent soft mask; 5. Superimposing the equivalent soft mask onto the deep feature map, and then inputting it into a global average pooling layer and a fully connected network layer for classification; 6. Iterating model training, dynamically updating the equivalent soft mask, and obtaining the final activation heatmap; 7. Performing defect category identification on the test set, bilinearly upsampling the final activation heatmap to the same size as the input image, superimposing it with the original image to obtain the accurate location of the defect in the image.

Description

Technical Field

[0001] This invention belongs to the field of machine learning and image classification technology, specifically relating to a method for detecting defects in power images based on dynamic activation heatmaps. Background Technology

[0002] Various defects in power equipment are a significant factor affecting the stable operation of the power grid. If these defects are not taken seriously, they may escalate into safety accidents, causing substantial losses to production and daily life. Therefore, rapid and accurate equipment defect detection is crucial for improving power grid reliability, power supply quality, and operation and maintenance levels. Traditional detection methods mostly rely on manual inspections, which require significant manpower and time, and also suffer from problems such as long cycles and low efficiency.

[0003] In recent years, with the maturation of drone technology in power transmission and transformation inspection, utilizing drones to inspect visible light images and deep learning-based image recognition technology for power equipment inspection is an inevitable development trend for the foreseeable future. This inspection method is characterized by high flexibility and low risk, and can significantly improve the efficiency of power equipment defect detection. Chinese patent No. 202110684066.7, "A Multifunctional Intelligent Power Image Analysis Device Based on AI Deep Learning," specifically discloses a device including functional modules such as an image acquisition module, an image noise reduction module, an image preprocessing module, a region growing module, a grayscale mapping contrast enhancement module, a target image recognition module, an image feature extraction and recognition module, an equipment recognition module, and a comparison analysis module. This device can improve the contrast of power images while effectively protecting edge and detail information, enhancing the acquired images, improving image quality, and increasing the accuracy of power image recognition. However, the diversity of shooting time, angle, and distance during drone or robot inspections increases the difficulty of detecting equipment image defects, making existing deep learning-based power image defect detection methods less accurate in identifying defects in small or hidden targets. At the same time, existing feature visualization methods mainly focus on internal analysis of the network during the testing phase and do not explore the dynamic interpretability of the feature space under the model training state. Summary of the Invention

[0004] This invention provides a power image defect detection method based on dynamic activation heatmaps to solve the problems of existing power image defect detection methods lacking dynamic interpretability of the feature space under model training and having low recognition accuracy for small and hidden target defects.

[0005] The technical solution of the present invention is as follows:

[0006] A method for detecting defects in power images based on dynamic activation heatmaps includes the following steps:

[0007] S1. Collect normal and defect images of power equipment, preprocess all images, and divide them into training, validation and test sets.

[0008] S2. Construct a feature extraction network to map the training set samples of the power images obtained in S1 to the deep feature space and obtain a deep feature map.

[0009] S3. In the feature extraction network built in S2, select a target layer and use CAM technology to calculate the weight contribution distribution in the spatial dimension of the target layer to generate a coarse activation heatmap.

[0010] S4. Normalize the coarse activation heatmap generated in S3 and downsample it to the same size as the depth feature map in step S2 to obtain an equivalent soft mask map.

[0011] S5. The equivalent soft mask map in S4 is superimposed on the depth feature map in S2 to enhance the effective information of defect association and filter the redundant information. Then, it is input into the global average pooling layer and the fully connected network layer for classification.

[0012] S6. Perform model training iterations, repeating S2 to S5, dynamically updating the equivalent soft mask map, selecting the best model and obtaining the final activation heatmap.

[0013] S7. On the test set divided by S1, defect category identification is performed. The test set samples are input into the best model in step S7 to obtain the defect classification result prediction. The final activation heatmap in S6 is bilinearly upsampled to the same size as the input image and superimposed on the power equipment sample image to obtain the accurate location information of the defects in the image.

[0014] Furthermore, in step S1, the data preprocessing of the image involves standardizing the graphic. Where μ is the pixel mean, x is the image pixel matrix, σ is the standard deviation, and N is the number of pixels in the image.

[0015] Furthermore, the feature extraction network constructed in step S2 is a classic CNN structure network.

[0016] Furthermore, the cam technology in step S3 is grad-cam and its derivative technologies.

[0017] Furthermore, the specific steps for generating the coarse activation heatmap using CAM technology in step S3 are as follows:

[0018] S31. Select the target layer in the original image according to the proportion of the sample defect size;

[0019] S32. Assuming the class label of training set sample k is c, let the model perform forward propagation and gradient backpropagation. Record the activation value of the target layer during forward propagation and the gradient information of class c during backpropagation to the target layer, and obtain the neuron channel weight vector. , Among them, w tar h tar These represent the length and width of the target layer, respectively, and the score. c The model predicts the score of training set sample k as class c. Let be the pixel value of the training set sample in the i-th row and j-th column of the target layer;

[0020] S33. Based on the activation values ​​of the target layer and the neuron channel weight vectors Linear fusion of features along the channel dimension is performed on the target layer. in, This is the activation map on the l-th channel; The contribution of the l-th channel;

[0021] S34. Considering only pixels that have a positive impact on the category, use the ReLU function to process the heatmap and generate a coarse activation heatmap.

[0022] Furthermore, the normalization operation of the coarse activation heatmap in step S4 includes the following steps:

[0023] S41. Assume that for the element in the i-th row and j-th column of the training set sample k, Based on the min-max normalization criterion, the element in the i-th row and j-th column of the normalized heatmap is: Where max(Cam) coarse ), min(Cam coarse The values ​​are the maximum and minimum values ​​of all elements in the rough activation heatmap, respectively. eps is the minimum value to prevent overflow, set to 10. e -6;

[0024] S42, and downsample to the same size as the depth feature map fea in step 2 to obtain an equivalent soft mask map, specifically as follows: Where, size cam size fea Cam′ coarse The length (or width) and depth feature map fea are given, r is the downsampling factor, and avgpooling is the average pooling operation.

[0025] Furthermore, in step S4, the equivalent soft mask image is superimposed on the depth feature map, that is, the enhanced feature is obtained by performing element-wise multiplication between the equivalent soft mask image and the depth feature map. in, This is an element-wise multiplication operation; the result is input into a global average pooling layer and a fully connected network layer for classification to obtain the prediction result pred; specifically, pred = Soft max(FC(GAP(fea')), where FC is a fully connected layer operation and GAP is a global average pooling operation.

[0026] The present invention has the following beneficial effects:

[0027] 1. First, the most important operations in image classification methods based on convolutional networks are convolution and pooling. However, due to their inherent limitations, convolution in existing technologies does not consider computational complexity from the perspective of information theory and sparse representation, resulting in high computational costs. Therefore, information filtering methods need to be introduced. Pooling, in order to reduce image size, performs forced downsampling, leading to a high information loss rate and a tendency to miss small-scale targets. Second, existing image classification methods can only determine the category of defects. Their structural limitations prevent them from locating defects in the original image. While Cam technology, as a feature visualization method, is currently mainly used in image classification to visualize features of trained models, it is a static, one-time operation. Therefore, this invention, for the first time, integrates category activation heatmaps (CAM) into each round of training of the classification model. Based on the defect recognition correlation, it enhances the deep feature map with task-related information and filters redundant information, assisting the training process and providing dynamic interpretability of the feature space during training. It also improves the model's accuracy in recognizing difficult samples.

[0028] 2. The present invention proposes an equivalent soft mask map, which can enhance the task-related information and filter redundant information of the depth feature map according to the defect identification correlation. It is more suitable for defect mining of small and hidden targets, enhances the usability of visible light images of UAV inspection, and improves the identification accuracy of power equipment defects. Attached Figure Description

[0029] Figure 1 This is a flowchart of the power image defect detection method based on dynamic activation heatmap in this invention. Detailed Implementation

[0030] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments.

[0031] See Figure 1A power image defect detection method based on dynamic activation heatmaps integrates category activation heatmaps into each round of training of the classification model. It enhances task-related information and filters redundant information from the deep feature map based on the defect recognition correlation, assisting the training process and providing dynamic interpretability of the feature space during training. This also improves the model's accuracy in identifying difficult samples. The detection method specifically includes the following steps:

[0032] S1. Collect normal and defect images of power equipment, preprocess all images, and divide them into training, validation, and test sets. The preprocessing mainly involves standardizing the images. Where μ is the pixel mean, x is the image pixel matrix, σ is the standard deviation, and N is the number of pixels in the image;

[0033] S2. Construct a feature extraction network. The feature extraction network is a CNN-based network, such as AlexNet, VGG, ResNet, InceptionNet, or other classic CNN structures. Map the training set samples of the power images obtained in S1 to the deep feature space to obtain the deep feature map fea. In this embodiment, VGG-16 is used as an example, but the technical solution of this invention is not limited to VGG-16. The specific network structure is as follows: ① Two convolutional layers plus one max-pooling layer → ② Two convolutional layers plus one max-pooling layer → ③ Three convolutional layers plus one max-pooling layer → ④ Three convolutional layers plus one pooling layer → ⑤ Three convolutional layers plus one pooling layer. All of these... All convolutional layers use two-dimensional convolution; the network parameters are set as follows: ① Two convolutional layers have a kernel size of 3×3 and a kernel count of 64; ② Two convolutional layers have a kernel size of 3×3 and a kernel count of 128; ③ Three convolutional layers have a kernel size of 3×3 and a kernel count of 256; ④ Three convolutional layers have a kernel size of 3×3 and a kernel count of 512; ⑤ Three convolutional layers have a kernel size of 3×3 and a kernel count of 512; the downsampling factor of all max pooling layers in the network structure is 2×2, and the stride is [2,2]; the normal and defective images of power equipment are processed by the above feature extraction network to obtain a depth feature map fea with a shape of (512, 7, 7);

[0034] S3. In the feature extraction network built in S2, select a target layer and use the Cam technique to calculate the weight contribution distribution in the spatial dimension of the target layer, generating a coarse activation heatmap (Cam). coarse ; CAM technology refers to grad-cam and its derivative technologies, which may include grad-cam, xgrad-cam or layer-cam, etc. In this embodiment, grad-cam is used as an example, but the technical solution of the present invention is not limited to grad-cam;

[0035] S4. The rough activation heatmap Cam generated after S3... coarse Perform normalization and downsample to the same size as the depth feature map fea in step S2 to obtain the equivalent soft mask map Cov;

[0036] S5. Superimpose the equivalent soft mask image Cov from S4 onto the depth feature map fea from S2. That is, perform element-wise multiplication of the equivalent soft mask image Cov and the depth feature map fea to obtain the enhanced feature fea′. in, The element-wise multiplication operation is used to enhance the effective information of defect association and filter redundant information. Then, it is input into the global average pooling layer and the fully connected network layer for classification to obtain the prediction result pred, specifically pred = Soft max(FC(GAP(fea')), where FC is the fully connected layer operation and GAP is the global average pooling operation.

[0037] S6. Perform model training iterations, repeating S2 to S5, and then integrate the CAM category activation heatmap into each round of training of the classification model. During each round of training, dynamically update and record the equivalent soft mask map Cov generated in S4 based on the current model parameters; at the same time, select the round with the best performance on the validation set metrics, record the model parameters for that round, and obtain the optimal model mod el. best The rough activation heatmap of this round is used as the final activation heatmap Cam. final ;

[0038] S7. On the test set partitioned in S1, identify the defect category and input the test set samples into the optimal model mod el in step S7. best In step S5, the test set data is used to repeat steps S1-S5 to obtain the defect classification result prediction; the final activation heatmap Cam in S6 is then used. final The image is bilinearly upsampled to the same size as the input image and then overlaid with the power equipment sample image to obtain accurate location information of the defect in the image.

[0039] In this embodiment, a coarse activation heatmap (Cam) is generated using grad-cam. coarse The specific steps are as follows:

[0040] S31. Unlike existing CAM techniques that select the last convolutional layer of the feature extraction network as the target layer, this method uses the target layer as the target layer. targetThe selection depends on the proportion of the sample defect size in the original image; considering the proportion of the defect target in the original image, the last convolutional layer of the fourth structure of VGG-16 with a shape of (512, 14, 14) is selected.

[0041] S32. Assuming the class label of training set sample k is c, let the model perform forward propagation and gradient backpropagation, and record the target layer during forward propagation. target The activation values ​​and model scores for category c are backpropagated to the target layer. target Gradient information is used to obtain the neuron channel weight vector.

[0042] Among them, w tar h tar These represent the length and width of the target layer, respectively, and the score. c The model predicts the score of training set sample k as class c. Let be the pixel value of the training set sample in the i-th row and j-th column of the target layer;

[0043] S33, Based on the target layer (Layer) target Activation values ​​and neuron channel weight vectors Linear fusion of features along the channel dimension is performed on the target layer. in, This is the activation map on the l-th channel; The contribution of the l-th channel;

[0044] S34. Considering only pixels that have a positive impact on the category, use the ReLU function to process the heatmap and generate a coarse activation heatmap.

[0045] The normalization operation of the coarse activation heatmap in step S4 includes the following steps:

[0046] S41. Assume that for the element in the i-th row and j-th column of the training set sample k, Based on the min-max normalization criterion, the normalized heatmap Cam′ coarse The element in the i-th row and j-th column is

[0047] Where max(Cam) coarse ), min(Cam coarse ( ) are rough activation heatmaps Cam coarse The maximum and minimum values ​​of all elements in the set, where eps is the minimum value to prevent overflow, set to 10. e -6;

[0048] S42, and downsample to the same size as the depth feature map fea in step 2 to obtain an equivalent soft mask map Cov, specifically as follows: Where, size cam size fea Cam′ coarse The length (or width) and depth feature map fea are given, r is the downsampling factor, and avgpooling is the average pooling operation.

[0049] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for detecting defects in power images based on dynamic activation heatmaps, characterized in that, The category activation heatmap is integrated into each round of training of the classification model. Based on the defect recognition correlation, the deep feature map is enhanced with task-related information and redundant information is filtered to assist the training process. It provides dynamic interpretability of the feature space in the training state and is suitable for defect mining and recognition of small and hidden targets. Includes the following steps: S1. Collect normal and defective images of power equipment, preprocess all images, and divide them into training, validation and test sets. S2. Construct a feature extraction network to map the training set samples of the power images obtained in S1 to the deep feature space and obtain a deep feature map. S3. In the feature extraction network built in S2, select a target layer and use CAM technology to calculate the weight contribution distribution in the spatial dimension of the target layer to generate a coarse activation heatmap. S4. Normalize the coarse activation heatmap generated in S3 and downsample it to the same size as the depth feature map in step S2 to obtain an equivalent soft mask map. S5. The equivalent soft mask map in S4 is superimposed on the depth feature map in S2 to enhance the effective information of defect association and filter the redundant information. Then, it is input into the global average pooling layer and the fully connected network layer for classification. S6. Perform model training iterations, repeating S2 to S5, dynamically updating the equivalent soft mask map, selecting the best model and obtaining the final activation heatmap. S7. On the test set divided by S1, perform defect category identification, input the test set samples into the best model in step S6, and obtain the defect classification result prediction; the final activation heatmap in S6 is bilinearly upsampled to the same size as the input image, and superimposed on the power equipment sample image to obtain the accurate location information of the defects in the image.

2. The method for detecting power image defects based on dynamic activation heatmaps according to claim 1, characterized in that: In step S1, the data preprocessing of the image involves standardizing the graphic. , ,in, The average pixel value. The image pixel matrix For standard deviation, This represents the number of pixels in the image.

3. The power image defect detection method based on dynamic activation heatmap according to claim 1, characterized in that: The feature extraction network built in step S2 is a classic CNN structure network.

4. The power image defect detection method based on dynamic activation heatmap according to claim 1, characterized in that: In step S3, the cam technology is grad-cam and its derivative technologies.

5. The power image defect detection method based on dynamic activation heatmap according to claim 4, characterized in that: The specific steps for generating the coarse activation heatmap using CAM technology in step S3 are as follows: S31. Select the target layer in the original image according to the proportion of the sample defect size; S32, Assuming the training set samples The category label is The model is then subjected to forward propagation and gradient backpropagation. The activation values ​​of the target layer during forward propagation are recorded. The model score for each category is backpropagated to the gradient information of the target layer to obtain the neuron channel weight vector. , ,in, , These are the length and width of the target layer, respectively. For the model, the training set samples Predicted as Category score; Let be the pixel value of the training set sample in the i-th row and j-th column of the target layer; S33. Based on the activation values ​​of the target layer and the neuron channel weight vectors Linear fusion of features along the channel dimension is performed on the target layer. ,in, For the first Activation map on each channel; For the first The contribution of each channel; S34. Considering only pixels that have a positive impact on the category, use the ReLU function to process the heatmap and generate a coarse activation heatmap. .

6. The power image defect detection method based on dynamic activation heatmap according to claim 1, characterized in that: The normalization operation of the coarse activation heatmap in step S4 includes the following steps: S41. Assume that for the training set samples The element in the i-th row and j-th column is Based on the min-max normalization criterion, the element in the i-th row and j-th column of the normalized heatmap is... ,in, , These represent the maximum and minimum values ​​of all elements in the coarsely activated heatmap, respectively. To prevent overflow, the minimum value is set to 10. -6 ; S42, and downsample to the same size as the depth feature map in step 2 to obtain an equivalent soft mask map, specifically as follows: ,in, , These represent the length or width of the normalized heatmap and the length or width of the depth feature map, respectively. This is the downsampling factor. This is for average pooling operations.

7. The method for detecting power image defects based on dynamic activation heatmaps according to claim 1, characterized in that: In step S4, the equivalent soft mask image is superimposed on the depth feature map, that is, the enhanced feature is obtained by performing element-wise multiplication between the equivalent soft mask image and the depth feature map. ,in, This is an element-wise multiplication operation; the result is input into a global average pooling layer and a fully connected network layer for classification to obtain the prediction result. Specifically ,in, For fully connected layer operations, This is a global average pooling operation.

Images