Insulator anomaly detection method and system based on infrared image and visible light image

Abstract

The invention relates to an insulator anomaly detection method and system based on an infrared image and a visible light image. The method comprises the following steps: extracting features of an infrared image and a visible light image of an insulator through an infrared image feature encoder and a visible light image feature encoder; modeling correlation between the infrared features and the visible light features and insulator abnormity through a cross attention mechanism; context modeling is carried out on information related to insulator abnormity in the infrared features and the visible light features through two self-attention mechanisms; the output of each layer of self-attention mechanism and the output of the cross attention mechanism are combined through element-by-element addition and then are input to the next layer of cross attention mechanism; the output of the self-attention mechanism of each layer is correspondingly input to the self-attention mechanism of the next layer; and a semantic segmentation module used for performing insulator anomaly identification and insulator anomaly factor classification performs semantic segmentation based on the four groups of attention feature maps to obtain insulator anomaly and reasons.

Description

Technical Field

[0001] The present invention relates to the technical field of insulator state detection, and in particular to an insulator abnormality detection method and system based on infrared images and visible light images. Background Art

[0002] Insulators are indispensable components in overhead lines. They play a dual role in preventing current from grounding and supporting conductors. Due to some process defects in the manufacturing process, there will be micro cracks and small pores inside the porcelain insulators. Insulators are operated with power for a long time and are affected by conductor pressure, temperature changes, dirt, weather changes, strong electric fields, etc. When they deteriorate to a certain extent, their mechanical strength and insulation performance will decrease, resulting in defects.

[0003] Infrared thermal imaging technology is a practical and convenient way to detect insulators. It can intuitively find signs of insulator defects by detecting local temperature rise caused by local discharge and leakage current, so as to take corresponding measures in time to avoid accidents caused by insulator defects. Infrared imaging technology has the characteristics of high efficiency, safety and reliability, accurate judgment, intuitive image, long detection distance, fast detection speed, non-contact detection, and no electromagnetic interference. Poor internal material and dirt of insulators will lead to low and zero values ​​of insulators, causing abnormal heating and thus causing failures. The characteristics of normal insulator thermal images are that the temperature on the line side is higher, and the temperature rise between adjacent insulators is very small; when low-value and zero-value insulators deteriorate, their heating power is smaller than that of normal insulators, and the heating power is close to zero in the case of zero value. Regardless of the cause of the low and zero values, the insulator temperature is lower than that of normal insulators. The existing technology can identify the abnormality, but cannot identify the cause of the abnormality. Summary of the invention

[0004] In order to solve the above technical problem or at least partially solve the above technical problem, the present invention provides an insulator abnormality detection method and system based on infrared images and visible light images.

[0005] In a first aspect, the present invention provides an insulator abnormality detection method based on infrared images and visible light images, comprising: The infrared image of the insulator in the transmission line is collected by using an infrared camera and a visible light camera. and visible light images ; The infrared image feature encoder and the visible light image feature encoder based on the residual convolutional network are used to extract the infrared image feature. and visible light images The characteristics of infrared and visible light characteristics ; For infrared features extracted by infrared image feature encoder and visible light image feature encoder and visible light characteristics , the association between infrared features and visible light features related to insulator abnormalities is modeled through a cross-attention mechanism; the context-related information of insulator abnormalities in infrared features and visible light features is modeled through two self-attention mechanisms in parallel with the cross-attention mechanism; the output of each layer of infrared features and visible light self-attention mechanism and the output of the cross-attention mechanism are combined by element-by-element addition and then input into the next layer of cross-attention mechanism; the output of each layer of infrared features and visible light self-attention mechanism is input into the infrared features and visible light self-attention mechanism of the next layer accordingly; Through the self-attention and cross-attention mechanisms, four sets of attention feature maps are finally obtained and fused, and then input into the semantic segmentation module for insulator abnormality identification and insulator abnormality factor classification. Semantic segmentation is performed to obtain insulator abnormalities and abnormal causes.

[0006] Furthermore, the infrared image feature encoder and the visible light image feature encoder have the same structure, including: and visible light images The convolution layer of shallow features is followed by several residual convolution blocks stacked in series, and the output of each residual convolution block is concatenated in the channel dimension.

[0007] Furthermore, each residual convolution block includes a convolution layer, a ReLU activation function, a convolution layer and a ReLU activation function connected in sequence, and a skip chain in parallel with the convolution layer, the ReLU activation function, the convolution layer and the ReLU activation function connected in sequence. The input of the residual convolution block is processed by the convolution layer, the ReLU activation function, the convolution layer and the ReLU activation function, and then fused with the input itself by skip chaining, and the addition result is processed by the ReLU activation function and output.

[0008] Furthermore, the mathematical representation of the information related to insulator anomalies between infrared features and visible light features modeled by the cross-attention mechanism at each layer is as follows: The cross attention mechanism in the first layer uses the fully connected layer to transform the infrared features and visible light characteristics Or the cross-attention feature map output by the previous layer of cross-attention mechanism is mapped into query, key and value respectively: , , in, Respectively Infrared features modeled by layer-cross attention mechanism Or the infrared cross-attention feature map output by the previous cross-attention mechanism Query, keys and values ​​of ; Respectively for Visible light features modeled by layer-cross attention mechanism Or the visible light cross attention feature map output by the previous cross attention mechanism Query, keys and values ​​of ; Then, the contextual association of the infrared feature and the value of the visible light feature are combined to obtain the visible light cross-attention feature map modeled by the cross-attention mechanism of the current layer. , combining the contextual association of visible light features and the value of infrared features to obtain the infrared cross-attention feature map modeled by the cross-attention mechanism of the current layer : , ; in, Characterization The contextual association of infrared features extracted by the layer-cross attention mechanism, Characterization The contextual association of visible light features extracted by the layer-cross attention mechanism, and Infrared characteristics and visible light characteristics height and width.

[0009] Furthermore, a set of self-attention mechanisms integrates infrared features through fully connected layers. Or the feature map of infrared features output by the previous self-attention mechanism is mapped into queries, keys, and values: , Another group of self-attention mechanisms uses the fully connected layer to convert visible light features Or the feature map of the visible light features output by the previous self-attention mechanism is mapped into queries, keys, and values: , in, Respectively Infrared features modeled by layer self-attention mechanism or Feature map of infrared features output by the layer self-attention mechanism Query, keys and values ​​of ; Respectively Visible light features modeled by layer self-attention mechanism or Feature map of visible light features output by the layer self-attention mechanism The query, key and value of Then, the infrared self-attention feature map is obtained by combining the context association of the infrared feature and the value of the infrared feature, and the visible light self-attention feature map is obtained by combining the context association of the visible light feature and the value of the visible light feature:

[0010] ; in, Characterization The contextual association of infrared features modeled by the layer self-attention mechanism, Characterization The contextual association of visible light features modeled by the layer self-attention mechanism, and Infrared characteristics and visible light characteristics height and width; Self-attention fuses the self-attention feature map before self-attention with the self-attention feature map after self-attention through skip chaining.

[0011] Furthermore, the semantic segmentation module adopts the Unet semantic segmentation model.

[0012] Furthermore, the training of the infrared image feature encoder and the visible light image feature encoder adopts the GAN method, including: The training data are infrared images and visible light images of insulators. During training, the infrared image feature encoder and the visible light image feature encoder are respectively configured with decoders based on residual convolutional networks with the same structure. The decoder structure is symmetrical with the encoder structure. A discriminant network is created based on a multi-layer perceptron. Inputting the infrared image and the visible light image for training into the infrared image feature encoder and the visible light image feature encoder; The two decoders are based on infrared features and visible light characteristics Restore the infrared image and the visible light image; provide the input infrared image and the original infrared image, the input visible light image and the original visible light image to the discriminant network, and the discriminant network outputs true or false; The absolute value of the difference between the true and false probabilities output by the discriminant network, the sum of the cross entropy between the input infrared image and the original infrared image, and the cross entropy between the input visible light image and the original visible light image are taken as the loss function. The parameters of the infrared image feature encoder and the visible light image feature encoder, the two decoders and the discriminant network are updated by the gradient descent algorithm to minimize the loss function.

[0013] Furthermore, the parameters of the infrared image feature encoder and the visible light image feature encoder are frozen, and the parameters of the cross-attention, self-attention and semantic segmentation modules are trained, including: Insulator infrared and visible light images annotated with abnormal area masks and abnormal cause labels; The infrared image and visible light image used for training are input into the infrared image feature encoder and the visible light image feature encoder to obtain the infrared feature and visible light characteristics ; For infrared features extracted by infrared image feature encoder and visible light image feature encoder and visible light characteristics , the association between infrared features and visible light features related to insulator abnormalities is modeled through the cross-attention mechanism; the information related to insulator abnormalities in infrared features and visible light features is context-modeled through two self-attention mechanisms in parallel with the cross-attention mechanism to obtain four sets of attention feature maps, which include: infrared self-attention feature map , Visible light self-attention feature map , infrared cross attention feature map and visible light cross attention feature map ; The four sets of attention feature maps finally obtained through the self-attention and cross-attention mechanisms are fused and input into the semantic segmentation module for insulator abnormality identification and insulator abnormality factor classification for semantic segmentation. The semantic segmentation module outputs the predicted abnormality mask and the predicted abnormality cause label corresponding to the predicted abnormality mask; the cross entropy loss of the predicted abnormality cause label and the actual abnormality cause label, and the sum of the IOU loss between the predicted abnormality mask and the actual abnormality mask are used as the total loss function, and the parameters of the cross-attention, self-attention and semantic segmentation modules are trained by the gradient descent algorithm to minimize the total loss function.

[0014] In a second aspect, the present invention provides an insulator abnormality detection device based on infrared images, comprising: at least one processing unit, the processing unit, a storage unit and an acquisition unit are connected via a bus unit, the storage unit stores a computer program, and when the computer program is executed by the processing unit, the insulator abnormality detection method based on infrared images and visible light images as described is implemented.

[0015] In a third aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the insulator abnormality detection method based on infrared images and visible light images is implemented.

[0016] The above technical solution provided by the embodiment of the present invention has the following advantages compared with the prior art: The present application extracts features of infrared images and visible light images of insulators through an infrared image feature encoder and a visible light image feature encoder; models the association between infrared features and visible light features related to insulator abnormalities through a cross-attention mechanism; contextually models the information related to insulator abnormalities in infrared features and visible light features through two self-attention mechanisms respectively; the output of each layer of the self-attention mechanism and the output of the cross-attention mechanism are combined by element-by-element addition, and then input into the cross-attention mechanism of the next layer; the output of each layer of the self-attention mechanism corresponds to the input of the self-attention mechanism of the next layer; the semantic segmentation module for insulator abnormality identification and insulator abnormality factor classification performs semantic segmentation based on four sets of attention feature maps to obtain insulator abnormalities and causes.

[0017] In the subsequent analysis process of the semantic segmentation module, the infrared self-attention feature map can be used for surface and deep insulator anomaly detection, but it cannot distinguish between the two. However, the visible light self-attention feature map can support the detection of insulator anomalies caused by insulator surface contamination. The infrared cross-attention feature map and the visible light cross-attention feature map support the screening of insulator anomalies caused by surface contamination. In this way, through the coordination of all self-attention feature maps, it is possible to better distinguish the causes of insulator anomalies that cause low or zero values.

[0018] In the present application, the self-attention in parallel with the cross-attention not only provides the infrared self-attention feature map and the visible light self-attention feature map necessary for the subsequent analysis, but also combines the infrared self-attention feature map and the visible light self-attention feature map with the output of the cross-attention at each layer to avoid the disappearance of the correlation between the visible light features and the infrared features related to the insulator abnormality as the cross-attention input is transmitted with the self-attention. At the same time, the parallel design of cross-attention and self-attention ensures the analysis speed of the model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.

[0020] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 A flowchart of an insulator abnormality detection method based on infrared images and visible light images provided by an embodiment of the present invention; Figure 2 An architectural diagram of a model provided for an embodiment of the present invention; Figure 3 A schematic diagram of an infrared image feature encoder and a visible light image feature encoder provided by an embodiment of the present invention; Figure 4 A schematic diagram of an insulator abnormality detection device based on infrared images provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0023] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0024] Example 1 like Figure 1 As shown, the technology of the present invention implements an insulator abnormality detection method based on infrared images and visible light images, including: The infrared image of the insulator in the transmission line is collected by using an infrared camera and a visible light camera. and visible light images ; The infrared image feature encoder and the visible light image feature encoder based on the residual convolutional network are used to extract the infrared image feature. and visible light images The characteristics of infrared and visible light characteristics .

[0025] In the specific implementation process, the infrared image feature encoder and the visible light image feature encoder have the same structure but different parameters, such as Figure 3 As shown, the structure includes: for extracting infrared images and visible light images The convolution layer of shallow features is followed by several residual convolution blocks stacked in series. The output of each residual convolution block is spliced ​​in the channel dimension. Each residual convolution block contains a sequentially connected convolution layer, ReLU activation function, convolution layer and ReLU activation function, and a jump chain in parallel with the sequentially connected convolution layer, ReLU activation function, convolution layer and ReLU activation function. The input of the residual convolution block is processed by the convolution layer, ReLU activation function, convolution layer and ReLU activation function, and then added and fused with the input itself through jump chaining. The addition result is processed by the ReLU activation function and then output.

[0026] The infrared image feature encoder and the visible light image feature encoder extract the infrared image and visible light images The mathematical expression of the characteristic is as follows: , , , in, The convolution of the infrared image feature encoder is used to extract the shallow features of the infrared image. is the first layer residual convolution block of the infrared image feature encoder, is the leakyrelu activation function, The first residual convolution block of the infrared image feature encoder is based on the infrared feature map extracted from the output of the first convolution layer. For the layer residual convolutional blocks, The infrared image feature encoder Infrared feature map extracted by layer residual convolution block, The infrared image feature encoder Infrared feature map extracted by layer residual convolution block, is the total number of residual convolution blocks in the infrared image feature encoder, is the concatenation function.

[0027] , , , in, It is the convolution for extracting shallow features of visible light images in the visible light image feature encoder. is the first layer residual convolution block of the visible light image feature encoder, The first residual convolution block of the visible light image feature encoder is based on the visible light feature map extracted from the output of the first convolution layer. For the layer residual convolutional blocks, The feature encoder for visible light images The visible light feature map extracted by the layer residual convolution block, The feature encoder for visible light images The visible light feature map extracted by the layer residual convolution block, is the total number of residual convolution blocks in the visible light image feature encoder.

[0028] For infrared features extracted by infrared image feature encoder and visible light image feature encoder and visible light characteristics , the association between infrared features and visible light features related to insulator anomalies is modeled through a cross-attention mechanism. At the same time, two self-attention mechanisms in parallel with the cross-attention mechanism are used to model the context of information related to insulator anomalies in infrared features and visible light features. Self-attention and cross-attention support cascade stacking. Figure 2 As shown, the output of each layer of infrared features and visible light self-attention mechanism and the output of the cross attention mechanism are combined by element-by-element addition, and then input into the cross attention mechanism of the next layer; the output of each layer of infrared features and visible light self-attention mechanism corresponds to the input of the infrared features and visible light self-attention mechanism of the next layer.

[0029] In the specific implementation process, the mathematical representation of the information related to insulator anomalies between infrared features and visible light features modeled by the cross-attention mechanism at each layer is as follows: The cross attention mechanism in the first layer uses the fully connected layer to transform the infrared features and visible light characteristics Or the cross-attention feature map output by the previous layer of cross-attention mechanism is mapped into query, key and value respectively: , , in, Respectively Infrared features modeled by layer-cross attention mechanism Or the infrared cross-attention feature map output by the previous cross-attention mechanism Query, keys and values ​​of ; Respectively for Visible light features modeled by layer-cross attention mechanism Or the visible light cross attention feature map output by the previous cross attention mechanism Query, keys and values ​​of ; Then, the contextual association of the infrared feature and the value of the visible light feature are combined to obtain the visible light cross-attention feature map modeled by the cross-attention mechanism of the current layer. , combining the contextual association of visible light features and the value of infrared features to obtain the infrared cross-attention feature map modeled by the cross-attention mechanism of the current layer :

[0030] ; in, Characterization The contextual association of infrared features extracted by the layer-cross attention mechanism, Characterization The contextual association of visible light features extracted by the layer-cross attention mechanism, and Infrared characteristics and visible light characteristics height and width.

[0031] A set of self-attention mechanisms integrates infrared features through a fully connected layer Or the feature map of infrared features output by the previous self-attention mechanism is mapped into queries, keys, and values: , Another group of self-attention mechanisms uses the fully connected layer to convert visible light features Or the feature map of the visible light features output by the previous self-attention mechanism is mapped into queries, keys, and values: , in, Respectively Infrared features modeled by layer self-attention mechanism or Feature map of infrared features output by the layer self-attention mechanism Query, keys and values ​​of ; Respectively Visible light features modeled by layer self-attention mechanism or Feature map of visible light features output by the layer self-attention mechanism The query, key and value of Then, the infrared self-attention feature map is obtained by combining the context association of the infrared feature and the value of the infrared feature, and the visible light self-attention feature map is obtained by combining the context association of the visible light feature and the value of the visible light feature:

[0032] ; in, Characterization The contextual association of infrared features modeled by the layer self-attention mechanism, Characterization The contextual association of visible light features modeled by the layer self-attention mechanism, and Infrared characteristics and visible light characteristics The height and width of the self-attention feature map before self-attention are fused with the self-attention feature map after self-attention through skip chaining; for the first layer of self-attention of infrared features: , for the remaining layers of infrared features self-attention: ; The first layer of self-attention for visible light features , for the remaining layers of self-attention of visible light features: .

[0033] Through the self-attention and cross-attention mechanisms, four sets of attention feature maps are finally obtained, including: infrared self-attention feature map , Visible light self-attention feature map , infrared cross attention feature map and visible light cross attention feature map The first two contain insulator anomaly information of infrared characteristics and insulator anomaly information of visible light characteristics respectively, and the latter two contain insulator anomaly information of cross-complementary characteristics of infrared characteristics and visible light characteristics.

[0034] In this way, in the subsequent analysis process of the semantic segmentation module, the infrared self-attention feature map It can be used for surface and deep insulator anomaly detection, but cannot distinguish between the two. However, the visible light self-attention feature map Able to support insulator anomaly detection caused by insulator surface contamination, infrared cross-attention feature map and visible light cross attention feature map It supports screening out insulator anomalies caused by surface contamination. In this way, through the coordination of all self-attention feature maps, it supports better differentiation of the causes of insulator anomalies that cause low or zero values.

[0035] In the present application, the self-attention in parallel with the cross-attention not only provides the infrared self-attention feature map and the visible light self-attention feature map necessary for the subsequent analysis, but also combines the infrared self-attention feature map and the visible light self-attention feature map with the output of the cross-attention at each layer to avoid the disappearance of the correlation between the visible light features and the infrared features related to the insulator abnormality as the cross-attention input is transmitted with the self-attention. At the same time, the parallel design of cross-attention and self-attention ensures the analysis speed of the model.

[0036] The four sets of attention feature maps are fused through a fusion module. The fusion module concatenates and fuses the four sets of attention feature maps in the channel dimension.

[0037] Then, the fusion module inputs the fused four sets of attention feature maps into the semantic segmentation module for insulator anomaly identification and insulator anomaly factor classification to perform semantic segmentation and obtain insulator anomalies and anomaly causes. The semantic segmentation module shown uses the Unet semantic segmentation model. The Unet semantic segmentation model is a popular convolutional neural network (CNN) architecture. The network structure of the Unet semantic segmentation model is "U"-shaped and includes an encoder and a decoder; the encoder consists of a convolutional layer and a downsampling layer, which is used to extract the features of the attention feature map. As the number of layers increases, the size of the feature gradually decreases, but the number of feature channels increases. The decoder consists of an upsampling layer and a convolutional layer, which is used to restore the original size of the feature. While upsampling, the feature map of the same resolution in the decoder is spliced ​​with the feature map of the decoder through a jump connection, thereby retaining the position information.

[0038] In order to achieve the above purpose, the model of this application is trained, and the training process includes: In the first stage, the infrared image feature encoder and the visible light image feature encoder are trained: The training process of the infrared image feature encoder and the visible light image feature encoder adopts the GAN method, namely: The training data are infrared images and visible light images of insulators. During training, the infrared image feature encoder and the visible light image feature encoder are respectively configured with decoders based on residual convolutional networks with the same structure. The decoder structure is symmetrical with the encoder structure. A discriminant network is created based on a multi-layer perceptron. Inputting the infrared image and the visible light image for training into the infrared image feature encoder and the visible light image feature encoder; The two decoders are based on infrared features and visible light characteristics Restore the infrared image and the visible light image; provide the input infrared image and the original infrared image, the input visible light image and the original visible light image to the discriminant network, and the discriminant network outputs true or false; The absolute value of the difference between the true and false probabilities output by the discriminant network, the sum of the cross entropy between the input infrared image and the original infrared image, and the cross entropy between the input visible light image and the original visible light image are taken as the loss function. The parameters of the infrared image feature encoder and the visible light image feature encoder, the two decoders and the discriminant network are updated by the gradient descent algorithm to minimize the loss function.

[0039] In the second stage, the parameters of the infrared image feature encoder and the visible light image feature encoder are frozen, and the parameters of the cross-attention, self-attention and semantic segmentation modules are trained.

[0040] The training data are infrared images and visible light images of insulators annotated with abnormal area masks and abnormal cause labels.

[0041] The infrared image and visible light image used for training are input into the infrared image feature encoder and the visible light image feature encoder to obtain the infrared feature and visible light characteristics .

[0042] For infrared features extracted by infrared image feature encoder and visible light image feature encoder and visible light characteristics , the association between infrared features and visible light features related to insulator abnormalities is modeled through the cross-attention mechanism; the information related to insulator abnormalities in infrared features and visible light features is context-modeled through two self-attention mechanisms in parallel with the cross-attention mechanism to obtain four sets of attention feature maps, which include: infrared self-attention feature map , Visible light self-attention feature map , infrared cross attention feature map and visible light cross attention feature map .

[0043] The four sets of attention feature maps are finally obtained through the self-attention and cross-attention mechanisms and fused and input into the semantic segmentation module for insulator anomaly identification and insulator anomaly factor classification for semantic segmentation. The semantic segmentation module outputs the predicted anomaly mask and the predicted anomaly cause label corresponding to the predicted anomaly mask. The sum of the cross entropy loss between the predicted anomaly cause label and the actual anomaly cause label and the IOU loss between the predicted anomaly mask and the actual anomaly mask is used as the total loss function. The parameters of the cross-attention, self-attention and semantic segmentation modules are trained through the gradient descent algorithm to minimize the total loss function.

[0044] Example 2 See also Figure 4 As shown, an embodiment of the present invention provides an insulator abnormality detection device based on infrared images, comprising: at least one processing unit, wherein the processing unit, the storage unit and the acquisition unit are connected via a bus unit, and the storage unit, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as the software programs, computer executable programs and modules corresponding to an insulator abnormality detection method based on infrared images and visible light images in an embodiment of the present invention, which can be used to store data collected by the acquisition unit. The processing unit implements the above-mentioned insulator abnormality detection method based on infrared images and visible light images by running the software programs, computer executable programs and modules stored in the storage unit, comprising: The infrared image of the insulator in the transmission line is collected by using an infrared camera and a visible light camera. and visible light images ; The infrared image feature encoder and the visible light image feature encoder based on the residual convolutional network are used to extract the infrared image feature. and visible light images The characteristics of infrared and visible light characteristics ; For infrared features extracted by infrared image feature encoder and visible light image feature encoder and visible light characteristics , the association between infrared features and visible light features related to insulator abnormalities is modeled through a cross-attention mechanism; the context-related information of insulator abnormalities in infrared features and visible light features is modeled through two self-attention mechanisms in parallel with the cross-attention mechanism; the output of each layer of infrared features and visible light self-attention mechanism and the output of the cross-attention mechanism are combined by element-by-element addition and then input into the next layer of cross-attention mechanism; the output of each layer of infrared features and visible light self-attention mechanism is input into the infrared features and visible light self-attention mechanism of the next layer accordingly; Through the self-attention and cross-attention mechanisms, four sets of attention feature maps are finally obtained and fused, and then input into the semantic segmentation module for insulator abnormality identification and insulator abnormality factor classification. Semantic segmentation is performed to obtain insulator abnormalities and abnormal causes.

[0045] Of course, the computer program stored in the storage unit of the device for implementing the insulator abnormality detection method based on infrared images and visible light images provided in an embodiment of the present invention is not limited to the method operations described above, and can also execute related operations in the insulator abnormality detection method based on infrared images and visible light images provided in any embodiment of the present invention.

[0046] Example 3 An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed, the insulator abnormality detection method based on infrared images and visible light images is implemented, including: The infrared image of the insulator in the transmission line is collected by using an infrared camera and a visible light camera. and visible light images ; The infrared image feature encoder and the visible light image feature encoder based on the residual convolutional network are used to extract the infrared image feature. and visible light images The characteristics of infrared and visible light characteristics ; For infrared features extracted by infrared image feature encoder and visible light image feature encoder and visible light characteristics , the association between infrared features and visible light features related to insulator abnormalities is modeled through a cross-attention mechanism; the context-related information of insulator abnormalities in infrared features and visible light features is modeled through two self-attention mechanisms in parallel with the cross-attention mechanism; the output of each layer of infrared features and visible light self-attention mechanism and the output of the cross-attention mechanism are combined by element-by-element addition and then input into the next layer of cross-attention mechanism; the output of each layer of infrared features and visible light self-attention mechanism is input into the infrared features and visible light self-attention mechanism of the next layer accordingly; Through the self-attention and cross-attention mechanisms, four sets of attention feature maps are finally obtained and fused, and then input into the semantic segmentation module for insulator abnormality identification and insulator abnormality factor classification. Semantic segmentation is performed to obtain insulator abnormalities and abnormal causes.

[0047] A computer-readable storage medium provided in an embodiment of the present invention stores a computer program which is not limited to the method operations described above, but can also execute related operations in an insulator abnormality detection method based on infrared images and visible light images provided in any embodiment of the present invention.

[0048] In the embodiments provided by the present invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, structures or units, which can be electrical, mechanical or other forms.

[0049] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0050] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0051] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A method for detecting insulator abnormality based on infrared images and visible light images, characterized in that: include: The infrared image of the insulator in the transmission line is collected by using an infrared camera and a visible light camera. and visible light images ; The infrared image feature encoder and the visible light image feature encoder based on the residual convolutional network are used to extract the infrared image feature. and visible light images The characteristics of infrared and visible light characteristics ; For infrared features extracted by infrared image feature encoder and visible light image feature encoder and visible light characteristics , the association between infrared features and visible light features related to insulator abnormalities is modeled through a cross-attention mechanism; the context-related information of insulator abnormalities in infrared features and visible light features is modeled through two self-attention mechanisms in parallel with the cross-attention mechanism; the output of each layer of infrared features and visible light self-attention mechanism and the output of the cross-attention mechanism are combined by element-by-element addition and then input into the next layer of cross-attention mechanism; the output of each layer of infrared features and visible light self-attention mechanism is input into the infrared features and visible light self-attention mechanism of the next layer accordingly; Through the self-attention and cross-attention mechanisms, four sets of attention feature maps are finally obtained and fused, and then input into the semantic segmentation module for insulator abnormality identification and insulator abnormality factor classification. Semantic segmentation is performed to obtain insulator abnormalities and abnormal causes.

2. The insulator abnormality detection method based on infrared image and visible light image according to claim 1 is characterized in that: The infrared image feature encoder and the visible light image feature encoder have the same structure, including: and visible light images The convolution layer of shallow features is followed by several residual convolution blocks stacked in series, and the output of each residual convolution block is concatenated in the channel dimension.

3. The insulator abnormality detection method based on infrared images and visible light images according to claim 2 is characterized in that: Each residual convolution block contains a convolution layer, a ReLU activation function, a convolution layer and a ReLU activation function connected in sequence, and a skip chain in parallel with the convolution layer, the ReLU activation function, the convolution layer and the ReLU activation function connected in sequence. The input of the residual convolution block is processed by the convolution layer, the ReLU activation function, the convolution layer and the ReLU activation function, and then added and fused with the input itself through skip chaining. The addition result is processed by the ReLU activation function and then output.

4. The insulator abnormality detection method based on infrared image and visible light image according to claim 1 is characterized in that: The mathematical representation of the cross-attention mechanism in each layer to model the information related to insulator anomalies between infrared features and visible light features is as follows: The cross attention mechanism in the first layer uses the fully connected layer to transform the infrared features and visible light characteristics Or the cross-attention feature map output by the previous layer of cross-attention mechanism is mapped into query, key and value respectively: ， ， in, Respectively Infrared features modeled by layer-cross attention mechanism Or the infrared cross-attention feature map output by the previous cross-attention mechanism Query, keys and values ​​of ; Respectively for Visible light features modeled by layer-cross attention mechanism Or the visible light cross attention feature map output by the previous cross attention mechanism Query, keys and values ​​of ; Then, the contextual association of the infrared feature and the value of the visible light feature are combined to obtain the visible light cross-attention feature map modeled by the cross-attention mechanism of the current layer. , combining the contextual association of visible light features and the value of infrared features to obtain the infrared cross-attention feature map modeled by the cross-attention mechanism of the current layer : ， ； in, Characterization The contextual association of infrared features extracted by the layer-cross attention mechanism, Characterization The contextual association of visible light features extracted by the layer-cross attention mechanism, and Infrared characteristics and visible light characteristics height and width.

5. The insulator abnormality detection method based on infrared image and visible light image according to claim 1, characterized in that: A set of self-attention mechanisms integrates infrared features through a fully connected layer Or the feature map of infrared features output by the previous self-attention mechanism is mapped into queries, keys, and values: ， Another group of self-attention mechanisms uses the fully connected layer to convert visible light features Or the feature map of the visible light features output by the previous self-attention mechanism is mapped into queries, keys, and values: ， in, Respectively Infrared features modeled by layer self-attention mechanism or Feature map of infrared features output by the layer self-attention mechanism Query, keys and values ​​of ; Respectively Visible light features modeled by layer self-attention mechanism or Feature map of visible light features output by the layer self-attention mechanism The query, key and value of Then, the infrared self-attention feature map is obtained by combining the context association of the infrared feature and the value of the infrared feature, and the visible light self-attention feature map is obtained by combining the context association of the visible light feature and the value of the visible light feature: ； in, Characterization The contextual association of infrared features modeled by the layer self-attention mechanism, Characterization The contextual association of visible light features modeled by the layer self-attention mechanism, and Infrared characteristics and visible light characteristics height and width; Self-attention fuses the self-attention feature map before self-attention with the self-attention feature map after self-attention through skip chaining.

6. The insulator abnormality detection method based on infrared image and visible light image according to claim 1 is characterized in that: The semantic segmentation module uses the Unet semantic segmentation model.

7. The insulator abnormality detection method based on infrared image and visible light image according to claim 1 is characterized in that: The training of infrared image feature encoder and visible light image feature encoder adopts GAN method, including: The training data are infrared images and visible light images of insulators. During training, the infrared image feature encoder and the visible light image feature encoder are respectively configured with decoders based on residual convolutional networks with the same structure. The decoder structure is symmetrical with the encoder structure. A discriminant network is created based on a multi-layer perceptron. Inputting the infrared image and the visible light image for training into the infrared image feature encoder and the visible light image feature encoder; The two decoders are based on infrared features and visible light characteristics Restore the infrared image and the visible light image; provide the input infrared image and the original infrared image, the input visible light image and the original visible light image to the discriminant network, and the discriminant network outputs true or false; The absolute value of the difference between the true and false probabilities output by the discriminant network, the sum of the cross entropy between the input infrared image and the original infrared image, and the cross entropy between the input visible light image and the original visible light image are taken as the loss function. The parameters of the infrared image feature encoder and the visible light image feature encoder, the two decoders and the discriminant network are updated by the gradient descent algorithm to minimize the loss function.

8. The insulator abnormality detection method based on infrared image and visible light image according to claim 1, characterized in that: Freeze the parameters of the infrared image feature encoder and the visible light image feature encoder, and train the parameters of the cross attention, self-attention, and semantic segmentation modules, including: Insulator infrared and visible light images annotated with abnormal area masks and abnormal cause labels; The infrared image and visible light image used for training are input into the infrared image feature encoder and the visible light image feature encoder to obtain the infrared feature and visible light characteristics ; For infrared features extracted by infrared image feature encoder and visible light image feature encoder and visible light characteristics , the association between infrared features and visible light features related to insulator abnormalities is modeled through the cross-attention mechanism; the information related to insulator abnormalities in infrared features and visible light features is context-modeled through two self-attention mechanisms in parallel with the cross-attention mechanism to obtain four sets of attention feature maps, which include: infrared self-attention feature map , Visible light self-attention feature map , infrared cross attention feature map and visible light cross attention feature map ; The four sets of attention feature maps finally obtained through the self-attention and cross-attention mechanisms are fused and input into the semantic segmentation module for insulator abnormality identification and insulator abnormality factor classification for semantic segmentation. The semantic segmentation module outputs the predicted abnormality mask and the predicted abnormality cause label corresponding to the predicted abnormality mask; the cross entropy loss of the predicted abnormality cause label and the actual abnormality cause label, and the sum of the IOU loss between the predicted abnormality mask and the actual abnormality mask are used as the total loss function, and the parameters of the cross-attention, self-attention and semantic segmentation modules are trained by the gradient descent algorithm to minimize the total loss function.

9. An insulator abnormality detection device based on infrared images, characterized in that: include: At least one processing unit, the processing unit and the storage unit are interconnected through a bus unit, the storage unit stores a computer program, and when the computer program is executed by the processing unit, the insulator abnormality detection method based on infrared images and visible light images as described in any one of claims 1-8 is implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the insulator abnormality detection method based on infrared images and visible light images as described in any one of claims 1 to 8 is implemented.

Images