Insulator anomaly detection method and system based on infrared image and visible light image

Abstract

The application relates to an insulator anomaly detection method and system based on infrared images and visible light images. The application extracts the features of infrared images and visible light images of insulators through infrared image feature encoders and visible light image feature encoders; cross-attention mechanisms are used to model the correlation between the infrared features and the visible light features related to insulator anomalies; two self-attention mechanisms are respectively used to model the context of the information related to insulator anomalies in the infrared features and the visible light features; the output of each layer of self-attention mechanism and the output of the cross-attention mechanism are combined through element-by-element addition, and then input into the next layer of cross-attention mechanism; the output of each layer of self-attention mechanism is input into the next layer of self-attention mechanism; a semantic segmentation module for insulator anomaly identification and insulator anomaly factor classification performs semantic segmentation based on four groups of attention feature maps to obtain insulator anomalies and reasons.

Description

Technical Field

[0001] This invention relates to the field of insulator condition detection technology, and in particular to an insulator anomaly detection method and system based on infrared and visible light images. Background Technology

[0002] Insulators are indispensable components in overhead lines, serving the dual purpose of preventing grounding and supporting the conductors. Due to manufacturing defects, porcelain insulators may contain microcracks and small pores. As insulators operate under energized conditions for extended periods, they are also affected by conductor pressure, temperature changes, pollution, weather variations, and strong electric fields. When these factors deteriorate to a certain extent, their mechanical strength and insulation performance decrease, leading to defects.

[0003] Infrared thermography is a practical and convenient method for insulator inspection. By detecting localized temperature increases caused by partial discharge and leakage current, it can visually identify signs of insulator defects, allowing for timely intervention and preventing accidents caused by these defects. Infrared imaging technology is characterized by high efficiency, safety and reliability, accurate judgment, intuitive images, long detection distance, fast detection speed, non-contact detection, and immunity to electromagnetic interference. Poor internal material quality and contamination of insulators can lead to low or zero temperature readings, causing abnormal heating and ultimately, faults. A normal insulator thermogram shows a higher temperature on the line side and minimal temperature difference between adjacent insulators. When insulators exhibit low or zero temperature readings, their heating power is lower than normal insulators; in the case of zero temperature readings, the heating power is close to zero. Regardless of the cause of low or zero temperature readings, the insulator temperature is lower than normal. While current technology can identify the anomaly, it cannot pinpoint the cause. Summary of the Invention

[0004] To solve the above-mentioned technical problems, or at least partially solve them, the present invention provides an insulator anomaly detection method and system based on infrared and visible light images.

[0005] In a first aspect, the present invention provides an insulator anomaly detection method based on infrared and visible light images, comprising:

[0006] Infrared images of insulators in power transmission lines were collected using infrared and visible light cameras. and visible light images ;

[0007] Infrared images are extracted using an infrared image feature encoder and a visible light image feature encoder based on a residual convolutional network, respectively. and visible light images The characteristics were used to obtain infrared features. and visible light characteristics ;

[0008] Infrared features extracted by infrared image feature encoders and visible light image feature encoders and visible light characteristics The correlation between infrared and visible light features and insulator anomalies is modeled through a cross-attention mechanism. Two self-attention mechanisms, operating in parallel with the cross-attention mechanism, are used to perform contextual modeling of the information related to insulator anomalies in the infrared and visible light features, respectively. The outputs of each layer of infrared and visible light self-attention mechanisms and the cross-attention mechanism are combined element-wise and then input into the next layer of cross-attention mechanism. The outputs of each layer of infrared and visible light self-attention mechanisms are correspondingly input into the next layer of infrared and visible light self-attention mechanisms.

[0009] The four sets of attention feature maps obtained through self-attention and cross-attention mechanisms are fused and then input into the semantic segmentation module for insulator anomaly identification and insulator anomaly factor classification. Semantic segmentation is performed to obtain insulator anomalies and their causes.

[0010] Furthermore, the infrared image feature encoder and the visible light image feature encoder have the same structure, including: a tool for extracting infrared images. and visible light images The convolutional layer contains shallow features, followed by several residual convolutional blocks stacked in series. The outputs of each residual convolutional block are concatenated along the channel dimension.

[0011] Furthermore, each residual convolutional block contains a series of sequentially connected convolutional layers, ReLU activation functions, and more convolutional layers and ReLU activation functions, along with a skip chain running in parallel with these sequentially connected convolutional layers, ReLU activation functions, and more convolutional layers and ReLU activation functions. The input of the residual convolutional block is processed by the convolutional layers, ReLU activation functions, and more convolutional layers and ReLU activation functions, and then added to the input itself via the skip chain. The result of the addition is then processed by the ReLU activation function and output.

[0012] Furthermore, the mathematical representation of the information related to insulator anomalies between infrared and visible light features modeled by the cross-attention mechanism in each layer is as follows:

[0013] The first layer's cross-attention mechanism uses a fully connected layer to process infrared features. and visible light characteristics Alternatively, the cross-attention feature maps output by the previous layer's cross-attention mechanism can be mapped to queries, keys, and values ​​respectively:

[0014] ,

[0015] ,

[0016] in, The first Infrared features modeled by layer cross-attention mechanism Or the infrared cross-attention feature map output by the previous layer cross-attention mechanism. Queries, keys, and values; They are respectively the first Visible light features modeled by layer cross-attention mechanism Or the visible light cross-attention feature map output by the previous layer cross-attention mechanism. Queries, keys, and values;

[0017] Then, by combining the contextual association of infrared features and the values ​​of visible light features, the visible light cross-attention feature map modeled by the current layer's cross-attention mechanism is obtained. By combining the contextual association of visible light features and the values ​​of infrared features, the infrared cross-attention feature map for modeling the cross-attention mechanism of the current layer is obtained. :

[0018] ,

[0019] ;

[0020] in, Characterizing the first Contextual association of infrared features extracted by layer cross-attention mechanism Characterizing the first Contextual association of visible light features extracted by the layer cross-attention mechanism and Infrared features and visible light characteristics The height and width.

[0021] Furthermore, a set of self-attention mechanisms uses fully connected layers to integrate infrared features. Alternatively, the feature map of the infrared features output by the previous self-attention mechanism can be mapped into a query, key, and value:

[0022] ,

[0023] Another set of self-attention mechanisms will pass through fully connected layers to display visible light features. Alternatively, the feature map of visible light features output by the previous layer's self-attention mechanism can be mapped into a query, key, and value:

[0024] ,

[0025] in, The first Infrared features modeled by layer self-attention mechanism or Feature map of infrared features output by layer self-attention mechanism Queries, keys, and values; The first Visible light features modeled by layer self-attention mechanism or Feature map of visible light features output by layer self-attention mechanism Queries, keys, and values;

[0026] Then, by combining the contextual association of infrared features and the values ​​of infrared features, we obtain the infrared self-attention feature map; by combining the contextual association of visible light features and the values ​​of visible light features, we obtain the visible light self-attention feature map.

[0027]

[0028] ;

[0029] in, Characterizing the first Contextual association of infrared features modeled by layer self-attention mechanism. Characterizing the first Contextual association of visible light features modeled by layer self-attention mechanism. and Infrared features and visible light characteristics Height and width;

[0030] Self-attention fuses the self-attention feature map before and after self-attention through skip chains.

[0031] Furthermore, the semantic segmentation module adopts the Unet semantic segmentation model.

[0032] Furthermore, the infrared image feature encoder and the visible light image feature encoder are trained using a GAN approach, including:

[0033] The training data consisted of infrared and visible light images of insulators. During training, the infrared image feature encoder and the visible light image feature encoder were each configured with a decoder based on a residual convolutional network with identical structures. The decoder structure was symmetrical to the encoder structure. A discriminant network was created based on a multilayer perceptron.

[0034] The infrared and visible light images used for training are input into the infrared image feature encoder and the visible light image feature encoder, respectively.

[0035] The two decoders are based on infrared features respectively. and visible light characteristics The infrared and visible light images are reconstructed; the input infrared image and the original infrared image, and the input visible light image and the original visible light image are provided to the discrimination network, which then outputs whether the image is true or false.

[0036] The absolute value of the difference between the true and false probabilities of the discriminant network output, the sum of the cross-entropy between the input infrared image and the original infrared image, and the cross-entropy between the input visible light image and the original visible light image are used as the loss function. The parameters of the infrared image feature encoder, the visible light image feature encoder, the two decoders, and the discriminant network are updated using the gradient descent algorithm to minimize the loss function.

[0037] Furthermore, the parameters of the infrared image feature encoder and the visible light image feature encoder are frozen, and the parameters of the cross-attention, self-attention, and semantic segmentation modules are trained, including:

[0038] Infrared and visible light images of insulators jointly annotated with anomaly area masks and anomaly cause labels;

[0039] The infrared and visible light images used for training are input into the infrared image feature encoder and the visible light image feature encoder, respectively, to obtain infrared features. and visible light characteristics ;

[0040] Infrared features extracted by infrared image feature encoders and visible light image feature encoders and visible light characteristics A cross-attention mechanism is used to model the correlation between infrared and visible light features and insulator anomalies. Two self-attention mechanisms, operating in parallel with the cross-attention mechanism, are used to perform contextual modeling on the information related to insulator anomalies in both infrared and visible light features, resulting in four sets of attention feature maps. These four sets of attention feature maps include: an infrared self-attention feature map. Visible light self-attention feature map Infrared cross-attention feature map Cross-attention feature map with visible light ;

[0041] Four sets of attention feature maps are obtained through self-attention and cross-attention mechanisms and then fused. These maps are then input into a semantic segmentation module for insulator anomaly identification and insulator anomaly factor classification. Semantic segmentation is performed, and the semantic segmentation module outputs a predicted anomaly mask and a predicted anomaly cause label corresponding to the predicted anomaly mask. The total loss function is the sum of the cross-entropy loss between the predicted anomaly cause label and the actual anomaly cause label, and the IOU loss between the predicted anomaly mask and the actual anomaly mask. The parameters of the cross-attention, self-attention, and semantic segmentation modules are trained using a gradient descent algorithm to minimize the total loss function.

[0042] Secondly, the present invention provides an insulator anomaly detection device based on infrared images, comprising: at least one processing unit, wherein the processing unit, storage unit and acquisition unit are connected via a bus unit, the storage unit stores a computer program, and when the computer program is executed by the processing unit, it implements the insulator anomaly detection method based on infrared images and visible light images as described above.

[0043] Thirdly, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the insulator anomaly detection method based on infrared and visible light images.

[0044] The technical solutions provided in the embodiments of the present invention have the following advantages compared with the prior art:

[0045] This application extracts features from the infrared and visible light images of insulators using an infrared image feature encoder and a visible light image feature encoder; models the correlation between infrared and visible light features and insulator anomalies using a cross-attention mechanism; performs contextual modeling on the information related to insulator anomalies in the infrared and visible light features using two self-attention mechanisms respectively; combines the outputs of each self-attention mechanism and the cross-attention mechanism through element-wise addition, and then inputs them into the next layer of cross-attention mechanism; the output of each self-attention mechanism is input into the next layer of self-attention mechanism; a semantic segmentation module for insulator anomaly identification and insulator anomaly factor classification performs semantic segmentation based on four sets of attention feature maps to obtain insulator anomalies and their causes.

[0046] In the subsequent analysis of the semantic segmentation module, the infrared self-attention feature map can be used for the detection of insulator anomalies on both the surface and deep layers, but it cannot distinguish between the two. However, the visible light self-attention feature map can support the detection of insulator anomalies caused by surface contamination. The infrared cross-attention feature map and the visible light cross-attention feature map can help filter out insulator anomalies caused by surface contamination. In this way, the cooperation of all self-attention feature maps can better distinguish the causes of insulator anomalies that result in low or zero values.

[0047] In this application, the self-attention parallel to cross-attention provides both the infrared self-attention feature map and the visible light self-attention feature map necessary for subsequent analysis. Furthermore, at each layer, the infrared self-attention feature map and the visible light self-attention feature map are combined with the output of cross-attention. This avoids the loss of correlation between the visible light features and infrared features, which are inputs to cross-attention, and the insulator anomaly as they are passed through self-attention. At the same time, the parallel design of cross-attention and self-attention ensures the analysis speed of the model. Attached Figure Description

[0048] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0049] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0050] Figure 1 A flowchart of an insulator anomaly detection method based on infrared and visible light images provided in an embodiment of the present invention;

[0051] Figure 2 An architecture diagram of the model provided in the embodiments of the present invention;

[0052] Figure 3 A schematic diagram of an infrared image feature encoder and a visible light image feature encoder provided in an embodiment of the present invention;

[0053] Figure 4 This is a schematic diagram of an insulator anomaly detection device based on infrared images, provided in an embodiment of the present invention. Detailed Implementation

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0055] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0056] Example 1

[0057] like Figure 1As shown, the present invention provides a method for detecting insulator anomalies based on infrared and visible light images, comprising:

[0058] Infrared images of insulators in power transmission lines were collected using infrared and visible light cameras. and visible light images ;

[0059] Infrared images are extracted using an infrared image feature encoder and a visible light image feature encoder based on a residual convolutional network, respectively. and visible light images The characteristics were used to obtain infrared features. and visible light characteristics .

[0060] In specific implementation, the infrared image feature encoder and the visible light image feature encoder have the same structure but different parameters, such as... Figure 3 As shown, the structure includes: for extracting infrared images and visible light images The system consists of convolutional layers for shallow features, followed by several cascaded residual convolutional blocks. The outputs of each residual convolutional block are concatenated along the channel dimension. Each residual convolutional block contains a series of sequentially connected convolutional layers, ReLU activation functions, and more convolutional layers, along with a skip chain running parallel to these sequentially connected layers. The input to the residual convolutional block, after being processed by the convolutional layers, ReLU activation functions, and more convolutional layers, is added to the input itself via the skip chain and fused. The result is then processed by the ReLU activation function before being output.

[0061] Infrared image feature encoders and visible light image feature encoders extract infrared images respectively. and visible light images The mathematical expression for the characteristics is as follows:

[0062] ,

[0063] ,

[0064] ,

[0065] in, Convolutional processing for extracting shallow features from infrared images using an infrared image feature encoder. This is the first residual convolutional block of the infrared image feature encoder. For the LeakyReLU activation function, The infrared feature map extracted from the output of the first convolutional layer is used as the first residual convolutional block of the infrared image feature encoder. For the first Layer residual convolution block, For infrared image feature encoders Infrared feature maps extracted from layer residual convolutional blocks. For infrared image feature encoders Infrared feature maps extracted from layer residual convolutional blocks. This represents the total number of layers in the residual convolutional blocks of the infrared image feature encoder. This is a concatenation function.

[0066] ,

[0067] ,

[0068] ,

[0069] in, Convolutional methods for extracting shallow features from visible light images in a visible light image feature encoder. This is the first residual convolutional block of the visible light image feature encoder. The visible light feature map extracted from the output of the first convolutional layer is the residual convolutional block of the visible light image feature encoder. For the first Layer residual convolution block, For visible light image feature encoder Visible light feature maps extracted from layer residual convolutional blocks. For visible light image feature encoder Visible light feature maps extracted from layer residual convolutional blocks. This represents the total number of layers in the residual convolutional blocks of the visible light image feature encoder.

[0070] Infrared features extracted by infrared image feature encoders and visible light image feature encoders and visible light characteristics This study models the correlation between infrared and visible light features and their association with insulator anomalies using a cross-attention mechanism. Simultaneously, two self-attention mechanisms, operating in parallel with the cross-attention mechanism, are used to perform contextual modeling of information related to insulator anomalies in both infrared and visible light features. Self-attention and cross-attention mechanisms support cascading and stacking. Figure 2 As shown, the outputs of the infrared and visible light self-attention mechanisms and the cross-attention mechanism of each layer are combined through element-wise addition and then input into the cross-attention mechanism of the next layer; the outputs of the infrared and visible light self-attention mechanisms of each layer are correspondingly input into the infrared and visible light self-attention mechanisms of the next layer.

[0071] In the specific implementation process, the mathematical representation of the information related to insulator anomalies between infrared features and visible light features modeled by the cross-attention mechanism in each layer is as follows:

[0072] The first layer's cross-attention mechanism uses a fully connected layer to process infrared features. and visible light characteristics Alternatively, the cross-attention feature maps output by the previous layer's cross-attention mechanism can be mapped to queries, keys, and values ​​respectively:

[0073] ,

[0074] ,

[0075] in, The first Infrared features modeled by layer cross-attention mechanism Or the infrared cross-attention feature map output by the previous layer cross-attention mechanism. Queries, keys, and values; They are respectively the first Visible light features modeled by layer cross-attention mechanism Or the visible light cross-attention feature map output by the previous layer cross-attention mechanism. Queries, keys, and values;

[0076] Then, by combining the contextual association of infrared features and the values ​​of visible light features, the visible light cross-attention feature map modeled by the current layer's cross-attention mechanism is obtained. By combining the contextual association of visible light features and the values ​​of infrared features, the infrared cross-attention feature map for modeling the cross-attention mechanism of the current layer is obtained. :

[0077]

[0078] ;

[0079] in, Characterizing the first Contextual association of infrared features extracted by layer cross-attention mechanism Characterizing the first Contextual association of visible light features extracted by the layer cross-attention mechanism and Infrared features and visible light characteristics The height and width.

[0080] A set of self-attention mechanisms uses fully connected layers to apply infrared features Alternatively, the feature map of the infrared features output by the previous self-attention mechanism can be mapped into a query, key, and value:

[0081] ,

[0082] Another set of self-attention mechanisms will pass through fully connected layers to display visible light features. Alternatively, the feature map of visible light features output by the previous layer's self-attention mechanism can be mapped into a query, key, and value:

[0083] ,

[0084] in, The first Infrared features modeled by layer self-attention mechanism or Feature map of infrared features output by layer self-attention mechanism Queries, keys, and values; The first Visible light features modeled by layer self-attention mechanism or Feature map of visible light features output by layer self-attention mechanism Queries, keys, and values;

[0085] Then, by combining the contextual association of infrared features and the values ​​of infrared features, we obtain the infrared self-attention feature map; by combining the contextual association of visible light features and the values ​​of visible light features, we obtain the visible light self-attention feature map.

[0086]

[0087] ;

[0088] in, Characterizing the first Contextual association of infrared features modeled by layer self-attention mechanism. Characterizing the first Contextual association of visible light features modeled by layer self-attention mechanism. and Infrared features and visible light characteristics The height and width. Self-attention fuses the self-attention feature map before and after self-attention through a skip chain; for the first layer of self-attention of infrared features: For the remaining layers of self-attention for infrared features: For the first layer of self-attention of visible light features For the remaining layers of self-attention for visible light features: .

[0089] Four sets of attention feature maps were ultimately obtained through self-attention and cross-attention mechanisms, including: infrared self-attention feature map. Visible light self-attention feature map Infrared cross-attention feature map Cross-attention feature map with visible light The first two contain insulator anomaly information with infrared characteristics and insulator anomaly information with visible light characteristics, respectively. The latter two both contain insulator anomaly information with complementary cross-features of infrared and visible light characteristics.

[0090] Thus, in the subsequent analysis process of the semantic segmentation module, the infrared self-attention feature map It can be used for both surface and deep insulator anomaly detection, but cannot distinguish between the two, while visible light self-attention feature maps... Capable of supporting the detection of insulator anomalies caused by surface contamination, using infrared cross-attention feature maps. Cross-attention feature map with visible light It supports the screening of insulator anomalies caused by surface contamination. In this way, with the cooperation of all self-attention feature maps, it can better distinguish the causes of insulator anomalies that result in low or zero values.

[0091] In this application, the self-attention parallel to cross-attention provides both the infrared self-attention feature map and the visible light self-attention feature map necessary for subsequent analysis. Furthermore, at each layer, the infrared self-attention feature map and the visible light self-attention feature map are combined with the output of cross-attention. This avoids the loss of correlation between the visible light features and infrared features, which are inputs to cross-attention, and the insulator anomaly as they are passed through self-attention. At the same time, the parallel design of cross-attention and self-attention ensures the analysis speed of the model.

[0092] The four sets of attention feature maps are fused using a fusion module. The fusion module stitches and fuses the four sets of attention feature maps along the channel dimension.

[0093] The fusion module then inputs the fused four sets of attention feature maps into the semantic segmentation module, which is used for insulator anomaly identification and insulator anomaly factor classification, to perform semantic segmentation and obtain the insulator anomalies and their causes. The semantic segmentation module shown employs the Unet semantic segmentation model. The Unet semantic segmentation model is a popular convolutional neural network (CNN) architecture. The Unet semantic segmentation model has a U-shaped network structure, containing an encoder and a decoder. The encoder consists of convolutional layers and downsampling layers, used to extract features from the attention feature maps. As the number of layers increases, the feature size gradually decreases, but the number of feature channels increases. The decoder consists of upsampling layers and convolutional layers, used to restore the original size of the features. Simultaneously with upsampling, feature maps of the same resolution in the decoder are concatenated with the feature maps in the decoder through skip connections, thereby preserving positional information.

[0094] To achieve the above objectives, the model of this application is trained, and the training process includes:

[0095] The first stage involves training the infrared image feature encoder and the visible light image feature encoder: The training process for the infrared image feature encoder and the visible light image feature encoder adopts a GAN approach, namely:

[0096] The training data consisted of infrared and visible light images of insulators. During training, the infrared image feature encoder and the visible light image feature encoder were each configured with a decoder based on a residual convolutional network with identical structures. The decoder structure was symmetrical to the encoder structure. A discriminant network was created based on a multilayer perceptron.

[0097] The infrared and visible light images used for training are input into the infrared image feature encoder and the visible light image feature encoder, respectively.

[0098] The two decoders are based on infrared features respectively. and visible light characteristics The infrared and visible light images are reconstructed; the input infrared image and the original infrared image, and the input visible light image and the original visible light image are provided to the discrimination network, which then outputs whether the image is true or false.

[0099] The absolute value of the difference between the true and false probabilities of the discriminant network output, the sum of the cross-entropy between the input infrared image and the original infrared image, and the cross-entropy between the input visible light image and the original visible light image are used as the loss function. The parameters of the infrared image feature encoder, the visible light image feature encoder, the two decoders, and the discriminant network are updated using the gradient descent algorithm to minimize the loss function.

[0100] In the second stage, the parameters of the infrared image feature encoder and the visible light image feature encoder are frozen, and the parameters of the cross-attention, self-attention and semantic segmentation modules are trained.

[0101] The training data consists of infrared and visible light images of insulators, labeled with both anomaly region masks and anomaly cause labels.

[0102] The infrared and visible light images used for training are input into the infrared image feature encoder and the visible light image feature encoder, respectively, to obtain infrared features. and visible light characteristics .

[0103] Infrared features extracted by infrared image feature encoders and visible light image feature encoders and visible light characteristics A cross-attention mechanism is used to model the correlation between infrared and visible light features and insulator anomalies. Two self-attention mechanisms, operating in parallel with the cross-attention mechanism, are used to perform contextual modeling on the information related to insulator anomalies in both infrared and visible light features, resulting in four sets of attention feature maps. These four sets of attention feature maps include: an infrared self-attention feature map. Visible light self-attention feature map Infrared cross-attention feature map Cross-attention feature map with visible light .

[0104] Four sets of attention feature maps, obtained through self-attention and cross-attention mechanisms, are fused and input into a semantic segmentation module for insulator anomaly identification and insulator anomaly factor classification. Semantic segmentation is then performed, and the module outputs a predicted anomaly mask and a corresponding predicted anomaly cause label. The total loss function is the sum of the cross-entropy loss between the predicted and actual anomaly cause labels and the IOU loss between the predicted and actual anomaly masks. Gradient descent is used to train the parameters of the cross-attention, self-attention, and semantic segmentation modules to minimize this total loss function.

[0105] Example 2

[0106] See Figure 4As shown, this embodiment of the invention provides an insulator anomaly detection device based on infrared images, comprising: at least one processing unit, wherein the processing unit, storage unit, and acquisition unit are connected via a bus unit. The storage unit, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as the software program, computer-executable program, and module corresponding to the insulator anomaly detection method based on infrared and visible light images in this embodiment of the invention, and can also be used to store data acquired by the acquisition unit. The processing unit implements the aforementioned insulator anomaly detection method based on infrared and visible light images by running the software program, computer-executable program, and module stored in the storage unit, including:

[0107] Infrared images of insulators in power transmission lines were collected using infrared and visible light cameras. and visible light images ;

[0108] Infrared images are extracted using an infrared image feature encoder and a visible light image feature encoder based on a residual convolutional network, respectively. and visible light images The characteristics were used to obtain infrared features. and visible light characteristics ;

[0109] Infrared features extracted by infrared image feature encoders and visible light image feature encoders and visible light characteristics The correlation between infrared and visible light features and insulator anomalies is modeled through a cross-attention mechanism. Two self-attention mechanisms, operating in parallel with the cross-attention mechanism, are used to perform contextual modeling of the information related to insulator anomalies in the infrared and visible light features, respectively. The outputs of each layer of infrared and visible light self-attention mechanisms and the cross-attention mechanism are combined element-wise and then input into the next layer of cross-attention mechanism. The outputs of each layer of infrared and visible light self-attention mechanisms are correspondingly input into the next layer of infrared and visible light self-attention mechanisms.

[0110] The four sets of attention feature maps obtained through self-attention and cross-attention mechanisms are fused and then input into the semantic segmentation module for insulator anomaly identification and insulator anomaly factor classification. Semantic segmentation is performed to obtain insulator anomalies and their causes.

[0111] Of course, the computer program stored in the storage unit of the device for implementing an insulator anomaly detection method based on infrared and visible light images provided in the embodiments of the present invention is not limited to the method operation described above, but can also execute related operations in the insulator anomaly detection method based on infrared and visible light images provided in any embodiment of the present invention.

[0112] Example 3

[0113] This invention provides a computer-readable storage medium storing a computer program. When executed, the computer program implements the insulator anomaly detection method based on infrared and visible light images, comprising:

[0114] Infrared images of insulators in power transmission lines were collected using infrared and visible light cameras. and visible light images ;

[0115] Infrared images are extracted using an infrared image feature encoder and a visible light image feature encoder based on a residual convolutional network, respectively. and visible light images The characteristics were used to obtain infrared features. and visible light characteristics ;

[0116] Infrared features extracted by infrared image feature encoders and visible light image feature encoders and visible light characteristics The correlation between infrared and visible light features and insulator anomalies is modeled through a cross-attention mechanism. Two self-attention mechanisms, operating in parallel with the cross-attention mechanism, are used to perform contextual modeling of the information related to insulator anomalies in the infrared and visible light features, respectively. The outputs of each layer of infrared and visible light self-attention mechanisms and the cross-attention mechanism are combined element-wise and then input into the next layer of cross-attention mechanism. The outputs of each layer of infrared and visible light self-attention mechanisms are correspondingly input into the next layer of infrared and visible light self-attention mechanisms.

[0117] The four sets of attention feature maps obtained through self-attention and cross-attention mechanisms are fused and then input into the semantic segmentation module for insulator anomaly identification and insulator anomaly factor classification. Semantic segmentation is performed to obtain insulator anomalies and their causes.

[0118] The computer-readable storage medium provided in this embodiment of the invention stores a computer program that is not limited to the method operation described above, but can also execute related operations in the insulator anomaly detection method based on infrared and visible light images provided in any embodiment of the invention.

[0119] In the embodiments provided by this invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, structures, or units, and may be electrical, mechanical, or other forms.

[0120] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0121] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0122] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. An insulator anomaly detection method based on infrared images and visible light images, characterized by, Comprise: Acquiring infrared images of insulators in a power transmission line by an infrared camera and a visible light camera and visible light images ; Features of the infrared image and the visible light image are obtained by respectively extracting features of the infrared image and the visible light image through an infrared image feature encoder and a visible light image feature encoder based on a residual convolutional network ​​​​ infrared features extracted by the infrared image feature encoder and the visible light image feature encoder and visible light features correlations between the infrared features and the visible light features related to the insulator anomaly are modeled through a cross-attention mechanism; Wherein, the mathematical representation of each layer of the cross attention mechanism modeling the information related to the insulator abnormality between the infrared features and the visible light features is as follows: The cross-attention mechanism maps the infrared feature and the visible light feature or the cross-attention feature map output by the last cross-attention mechanism into query, key and value respectively through a fully connected layer: ， ， wherein, respectively, the first layer cross-attention mechanism modeled infrared feature or the infrared cross-attention feature map output by the previous layer cross-attention mechanism query, key and value; respectively, the first layer cross-attention mechanism modeled visible light feature or the visible light cross-attention feature map output by the previous layer cross-attention mechanism query, key and value; The visible light cross-attention feature map is obtained by combining the context association of the infrared feature and the value of the visible light feature The infrared cross-attention feature map is obtained by combining the context association of the visible light feature and the value of the infrared feature : ， ； wherein, characterizing the first contextual association of the infrared features extracted by the layer-cross attention mechanism, characterizing the first contextual association of the visible light features extracted by the layer-cross attention mechanism, and respectively the height and the width of the infrared features and the visible light features ​ Through two self-attention mechanisms in parallel with the cross attention mechanism, the information related to the insulator abnormality in the infrared features and the visible light features is respectively modeled in the context; wherein a set of self-attention mechanisms maps the infrared feature or the feature map of the infrared feature output by the last self-attention mechanism into queries, keys and values: ， Another set of self-attention mechanisms pass the visible light features through a fully connected layer or a feature map of the visible light features output by the previous layer of self-attention mechanisms into queries, keys, and values: ， wherein, are the first infrared features modeled by the layer-wise self-attention mechanism or feature maps of the infrared features output by the layer-wise self-attention mechanism query, key and value of are the first visible light features modeled by the layer-wise self-attention mechanism or feature maps of the visible light features output by the layer-wise self-attention mechanism query, key and value of Then, the infrared self-attention feature map is obtained by combining the context association of the infrared features and the value of the infrared features, and the visible light self-attention feature map is obtained by combining the context association of the visible light features and the value of the visible light features: ； wherein, characterizing the first contextual association of the infrared features modeled by the layer self-attention mechanism, characterizing the first contextual association of the visible light features modeled by the layer self-attention mechanism, and respectively the height and the width of the infrared features and the visible light features . The self-attention fuses the self-attention feature map before the self-attention and the self-attention feature map after the self-attention through a skip chain; The output of each layer of the infrared feature and the visible light self-attention mechanism and the output of the cross attention mechanism are combined by element-wise addition, and then input to the next layer of the cross attention mechanism; The output of each layer of the infrared feature and the visible light self-attention mechanism corresponds to the input to the next layer of the infrared feature and the visible light self-attention mechanism; Through the self-attention and the cross attention mechanism, finally four groups of attention feature maps are obtained, which are fused and input to the semantic segmentation module for insulator abnormality recognition and insulator abnormality factor classification, semantic segmentation is performed to obtain the insulator abnormality and the abnormal reason; Wherein, the training of the infrared image feature encoder and the visible light image feature encoder adopts the GAN mode, the parameters of the infrared image feature encoder and the visible light image feature encoder are frozen, and the parameters of the cross attention, self-attention and semantic segmentation module are trained.

2. The insulator anomaly detection method based on infrared images and visible light images according to claim 1, characterized by, The infrared image feature encoder and the visible light image feature encoder are consistent in structure and include a convolutional layer for extracting shallow features of the infrared image and the visible light image The convolutional layer is followed by a plurality of residual convolutional blocks stacked in series, and the outputs of each layer of residual convolutional blocks are spliced in the channel dimension.

3. The method for detecting insulator abnormality based on infrared image and visible light image according to claim 2, characterized in that, Each residual convolution block contains a convolution layer, a ReLU activation function, a convolution layer and a ReLU activation function connected in turn, a skip chain parallel to the convolution layer, the ReLU activation function, the convolution layer and the ReLU activation function, and the input of the residual convolution block is fused by adding the input itself through the convolution layer, the ReLU activation function, the convolution layer and the ReLU activation function, and the addition result is output after the ReLU activation function processing.

4. The method for detecting insulator abnormality based on infrared image and visible light image according to claim 1, characterized in that, The semantic segmentation module adopts the Unet semantic segmentation model.

5. The method for detecting insulator abnormality based on infrared image and visible light image according to claim 1, characterized in that, The training of the infrared image feature encoder and the visible light image feature encoder adopts the GAN mode, including: The training data is the infrared image and the visible light image of the insulator, and when training, the infrared image feature encoder and the visible light image feature encoder are respectively configured with a decoder based on the residual convolution network with the same structure, and the decoder structure is symmetrical to the encoder structure; a discrimination network is created based on a multilayer perception; The infrared image and the visible light image used for training are input to the infrared image feature encoder and the visible light image feature encoder; Two decoders are based on infrared features and visible light features restore infrared images and visible light images; input the input infrared image and the restored infrared image, the input visible light image and the restored visible light image to the discrimination network, and the discrimination network outputs true or false; The absolute value of the difference between the true and false probabilities output by the discrimination network, the cross entropy between the input infrared image and the restored infrared image, and the cross entropy between the input visible light image and the restored visible light image are input as a loss function, and the parameters of the infrared image feature encoder and the visible light image feature encoder, two decoders and the discrimination network are updated through a gradient descent algorithm to minimize the loss function.

6. The method for detecting insulator abnormality based on infrared image and visible light image according to claim 1, characterized in that, The parameters of the infrared image feature encoder and the visible light image feature encoder are frozen, and the parameters of the cross attention, self-attention and semantic segmentation module are trained, including: The insulator infrared image and the visible light image are jointly labeled by an abnormal region mask and an abnormal reason label; The infrared image and the visible light image for training are input into an infrared image feature encoder and a visible light image feature encoder to obtain infrared features and visible light features ; infrared features extracted by the infrared image feature encoder and the visible light image feature encoder and visible light features , the correlation between the infrared features and the visible light features related to the insulator abnormality is modeled through a cross-attention mechanism; information related to the insulator abnormality in the infrared features and the visible light features is respectively context-modeled through two self-attention mechanisms parallel to the cross-attention mechanism to obtain four groups of attention feature maps, the four groups of attention feature maps including: infrared self-attention feature maps , visible light self-attention feature maps , infrared cross-attention feature maps , and visible light cross-attention feature maps ; The four groups of attention feature maps are fused and input into a semantic segmentation module for insulator abnormality identification and insulator abnormality factor classification through self-attention and cross-attention mechanisms, semantic segmentation is performed, the semantic segmentation module outputs a predicted abnormal mask and a predicted abnormal reason label corresponding to the predicted abnormal mask; a cross-entropy loss of the predicted abnormal reason label and an actual abnormal reason label, an IOU loss between the predicted abnormal mask and an actual abnormal mask are taken as a total loss function, and parameters of the cross-attention, the self-attention and the semantic segmentation module are trained through a gradient descent algorithm to minimize the total loss function.

7. An insulator abnormality detection apparatus based on an infrared image, characterized by The method comprises the following steps: At least one processing unit is connected to a bus unit, and the processing unit and a storage unit are interconnected through the bus unit, wherein the storage unit stores a computer program, and the computer program is executed by the processing unit to realize the insulator abnormality detection method based on the infrared image and the visible light image according to any one of claims 1-6.

8. A computer-readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to realize the insulator abnormality detection method based on the infrared image and the visible light image according to any one of claims 1-6.

Images