Power equipment segmentation and abnormal heating detection method and device based on different-source image feature fusion

Abstract

The invention discloses a power equipment segmentation and abnormal heating detection method and device based on heterogeneous image feature fusion. Comprising the following steps: acquiring two registration images of power equipment in different modes; feature extraction is carried out, low-level features and high-level features of the two images are extracted in multiple levels, and the features of different modal images under the corresponding levels are fused; generating a high-level semantic segmentation feature map by iteratively decoding and aggregating the three high-level fusion feature maps, and obtaining a corresponding semantic segmentation result image and an edge detection result image; a multi-level feature fusion and attention enhancement module is introduced, and the understanding and analysis ability of the image is optimized; and a binary cross entropy loss function is adopted to supervise a semantic segmentation result and a semantic segmentation truth value of the two processed images, and fine features of an abnormal heating area are captured. The method can greatly improve the precision and reliability of power equipment segmentation and abnormal heating detection, reduces the false detection rate, and has high practicability and wide application prospects.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and in particular to a method and device for segmenting and detecting abnormal heating of electric power equipment based on heterogeneous image feature fusion. Background Art

[0002] With the rapid development of modern power systems, the status monitoring and fault diagnosis of power equipment have become key tasks to ensure the safe and stable operation of power systems. Once a power equipment fails, especially abnormal heating, it may cause equipment damage or even cause large-scale power outages. Therefore, accurately detecting abnormal heating of power equipment is of great significance for preventing potential failures and ensuring system safety.

[0003] At present, the detection of abnormal heating of power equipment mainly relies on infrared thermal imaging technology. Infrared thermal imaging can intuitively display the temperature distribution of the equipment and quickly identify high-temperature areas. However, infrared images are greatly affected by external environmental conditions such as temperature, humidity, emissivity and other factors, which may cause errors in thermal imaging results. In addition, power equipment has complex shapes, and there are occlusions and similar backgrounds between devices. It is difficult to achieve accurate segmentation and fault detection of equipment by relying solely on infrared images. In response to these problems, researchers have gradually tried to introduce other imaging methods, such as visible light images, to combine multi-source images for information fusion and improve the accuracy of abnormal heating detection of substation equipment.

[0004] In recent years, deep learning-based image segmentation and target detection methods have made significant progress in the field of computer vision. In power equipment monitoring, some studies use convolutional neural networks (CNNs) to segment infrared images and detect temperature. For example, classic models such as Mask-RCNN can generate accurate segmentation masks, identify power equipment areas, and analyze their temperature distribution, but these methods have high computational complexity and poor real-time performance. Other studies, such as CenterNet, have improved the temperature detection accuracy of equipment by extracting multi-scale features and performing rotation angle regression, but there are still limitations in equipment edge recognition and abnormal heating area detection.

[0005] In order to overcome the shortcomings of the above methods, heterogeneous feature fusion methods based on multimodal data have gradually attracted attention. Multimodal feature fusion compensates for the problem of insufficient single modal information by fusing the features of infrared images and visible light images. This type of method uses infrared images to provide temperature information of the device, while visible light images can more clearly display the boundaries and structures of the device. The combination of the two helps to improve the accuracy of segmentation and abnormal fever detection. However, since simple feature splicing or element-by-element operations cannot fully utilize the information complementarity between modalities, it may lead to information loss or increased computational complexity. Therefore, how to effectively fuse the features of these two different modalities remains a technical difficulty. Summary of the invention

[0006] The present invention provides a method and device for segmenting and detecting abnormal heating of power equipment based on heterogeneous image feature fusion, which solves the problems of information loss, inaccurate equipment boundary identification and low abnormal heating detection accuracy in the process of infrared image and visible light image fusion in the prior art, and also solves the problem that infrared thermal imaging technology is greatly affected by factors such as ambient temperature and surface emissivity, and is prone to misjudgment. The present invention fuses the features of infrared images and visible light images to make up for the deficiency of single modal information, and can achieve accurate segmentation of power equipment and reliably detect abnormal heating areas of equipment in complex environments.

[0007] Technical solution: The present invention discloses a method for segmenting and detecting abnormal heating of electric equipment based on heterogeneous image feature fusion, comprising the following steps:

[0008] Step (1) photographing at the same time and from the same direction to obtain two registered images of the power equipment in different modes, namely, an infrared image and a visible light image;

[0009] Step (2) inputting the two captured images into a multi-level feature extraction network model and a multi-level feature fusion model, wherein the multi-level feature extraction network model extracts low-level features and high-level features of the two images at multiple levels, and the multi-level feature fusion model fuses the features of the two images at the corresponding level to output a multi-level fusion feature map;

[0010] Step (3) The multi-level fusion feature map fused by the multi-level feature fusion model is used as the input of the semantic segmentation decoder and the edge detection decoder. The semantic segmentation decoder and the edge detection decoder share weight parameters. The semantic information in the high-level features is used to suppress the interference and noise in the low-level feature map, and the edge detection feature map and the high-level semantic segmentation feature map are obtained. After channel processing of the edge detection feature map and the high-level semantic segmentation feature map, the edge detection result image and the semantic segmentation result image are determined.

[0011] Step (4) combines the semantic segmentation result image with the visible light image in step 1 to obtain a visible light foreground image, combines the semantic segmentation result image with the infrared image in step 1 to obtain an infrared foreground image, introduces a multi-level feature fusion and attention enhancement module, extracts features from the infrared foreground image and the visible light foreground image, performs feature fusion on the infrared foreground image and the visible light foreground image, fuses cross-modal information, and adds a spatial attention enhancement module for processing;

[0012] Step (5) uses a binary cross entropy loss function to supervise the semantic segmentation results and semantic segmentation truth values ​​of the infrared image and the visible light image, capture the subtle features of the abnormal heating area, obtain the residual thermal map, and finally determine the abnormal heating area.

[0013] Furthermore, the multi-level feature extraction network model includes two symmetric encoders based on the ResNet architecture, which are used to extract multi-level features of infrared images and visible light images respectively. Each symmetric encoder based on the ResNet architecture includes encoders 1 to 5. A dense block is added to each level of the encoder of the backbone network ResNet and the last fully connected layer is removed. The encoder is used as a network encoder for feature extraction. The multi-level features of infrared and visible light images are extracted through the convolutional layers that go deeper and deeper.

[0014] Set the two images to be acquired and They are sent to the trained multi-level feature extraction network model respectively, and then the i-th level encoder En i Extracted image features T i =En i (I ir ),R i =En i (I vs ), i∈{1,2,3,4,5}, and obtain the image features T at the corresponding level i With R i , where R i is the multi-level feature of the visible light image, T i is the multi-level feature of the infrared image. When i∈{1,2}, T1, T2, R1, R2 are low-level features, and T3, T4, T5, R3, R4, R5 are high-level features.

[0015] Further, the multi-level feature fusion model includes a low-level feature fusion model and a high-level feature fusion model;

[0016] The low-level feature fusion model processes the image features T1, T2, R1, and R2. First, the multi-level features T1 and T2 of the infrared image are processed by the maximum pooling operation of the channel dimension. Then, a 5×5 convolution kernel is used to convolve the infrared image features to obtain the processed infrared image features T1' and T2'. The processed infrared image features T1' and T2' are used to enhance the visible light image features R1 and R2 to obtain the preliminary fusion results f1 and f2. The preliminary fusion results f1 and f2 are used as weight matrices to calculate the weighted feature maps f1' and f2' of the visible light image features and the preliminary fusion feature maps by element-by-element multiplication. The processed infrared image features T1' and T2' are used to enhance the weighted fusion feature maps f1' and f2' to generate enhanced weighted feature maps. Finally, the channel attention mechanism is used to enhance the fusion of the enhanced weighted feature maps and output the fused feature maps F1 and F2;

[0017] The advanced feature fusion model processes the image features T3, T4, T5, R3, R4, and R5, and fuses the image features T3, T4, and T5 of encoder 3, encoder 4, and encoder 5 respectively. i and R i , i∈{3,4,5}, the preliminary fusion features of each level of encoder are obtained, and the preliminary fusion features of each level of encoder are input into the dense block to enhance the expressiveness of the features and alleviate the problem of gradient disappearance; dense semantic blocks are designed to further enhance the receptive field of the feature map and transmit global semantic information; in each layer of the dense semantic block, the convolution processing is replaced by a hole convolution, and the infrared image features, visible light image features and dense semantic block fusion features are further fused through jump connections, and the dense semantic blocks are used again for feature enhancement to obtain the output F3, F4, and F5 of high-level feature fusion.

[0018] Furthermore, the specific operation of the low-level feature fusion model is as follows:

[0019] (2.1) After the maximum pooling and convolution operations, the processed infrared image features are obtained:

[0020] T i ′=Conv5(Maxpool(T i )) Formula (1)

[0021] Among them, T i ′ is the infrared image feature after maximum pooling and 5×5 convolution, which can be used as auxiliary guidance information to enhance the visible light image features, i∈{1,2};

[0022] (2.2) Use the Sigmoid function to transform the pixels of the feature map element by element to obtain the preliminary fusion results of the visible light image features and the infrared image features:

[0023]

[0024] in, represents the element-by-element summation, σ(·) represents the element-by-element transformation of the pixels of the feature map using the Sigmoid function, and f i represents the preliminary fusion result of visible light image features and infrared image features, i∈{1,2};

[0025] (2.3) f i As a weight matrix, the weighted feature map f of the visible light image feature and the preliminary fusion feature map is calculated by element-by-element multiplication. i ′:

[0026]

[0027] in, Indicates element-wise multiplication of the feature map, f i ′ represents the weighted fusion feature map, which represents the correlation between the visible light image and the infrared image, i∈{1,2};

[0028] (2.4) Use the processed infrared image features to enhance the weighted fusion feature map to generate an enhanced weighted feature map

[0029]

[0030] (2.5) Use channel attention mechanism to weight feature map Enhance and output the fusion feature map F i , i∈{1,2}:

[0031]

[0032] Among them, MLP(·) is a multi-layer perceptron.

[0033] Furthermore, the specific operations of the advanced feature fusion model are:

[0034] (2.6) Fuse the features of the two encoders corresponding to the two symmetric encoders based on the ResNet architecture to obtain the preliminary fusion feature f i , i∈{3,4,5}:

[0035]

[0036] (2.7) The initial fusion feature f i Input into dense blocks to enhance the expressiveness of features and alleviate the problem of gradient disappearance:

[0037]

[0038] Among them, Cat(·) represents splicing along the channel dimension, D(·) represents four densely connected CBR modules, and the CBR module consists of convolution processing, normalization and ReLU activation function. The convolution processing adopts the method of dilated convolution. The sizes of the two groups of Conv3 convolution kernels are both 3×3. The first group of Conv3 convolution processing does not modify the number of feature map channels, and the second group of Conv3 convolution processing modifies the number of output feature map channels to 64.

[0039] (2.8) The specific operations of dense semantic blocks are:

[0040]

[0041] Among them, the dense semantic block fusion feature is Conv1(·) modifies the number of channels to 16, Conv3(·) modifies the number of output feature map channels to 64, D′(·) represents four densely connected CBR modules, and the expansion rates of the four groups of dilated convolutions are set to 1, 3, 5, and 7 respectively;

[0042] (2.9) The infrared image features, visible light image features and dense semantic block fusion features are further fused through skip connections to obtain the output of high-level feature fusion:

[0043]

[0044] Among them, F i , i∈{3,4,5} represents the output of high-level feature fusion.

[0045] Furthermore, in the step (3), the semantic segmentation decoder and the edge detection decoder both include decoder 1, decoder 2 and decoder 3, the semantic segmentation decoder decodes the fused features to generate a high-level semantic segmentation feature map, and the edge detection decoder decodes the fused features to generate an edge detection feature map. The specific operations are:

[0046] Step 3.1: Generate high-level semantic segmentation feature maps. The specific steps are as follows:

[0047] The semantic segmentation decoder aggregates three high-level fusion feature maps H = {F3, F4, F5} through iterative decoding to generate a high-level semantic segmentation feature map

[0048]

[0049] Among them, De1, De2, and De3 represent three decoder operations;

[0050] Step 3.2: Generate edge detection feature map. The specific operations are as follows:

[0051] By summing up the elements, we can integrate the edge detection decoder output feature map M e ′, low-level feature fusion output F2 and F1, the integrated feature map is directly combined with the feature map M output by the semantic segmentation encoder s Multiply element by element, and use the semantic information in the high-level features to suppress the interference and noise that may exist in the low-level feature map to obtain the final edge detection feature map

[0052]

[0053] Among them, F up Represents an upsampling operation;

[0054] Step 3.3: After performing channel processing on the edge detection feature map and the high-level semantic segmentation feature map, the edge detection result image and the semantic segmentation result image are determined. The specific operations are as follows:

[0055] Use a convolution block with a 1×1 kernel size and upsampling to modify the edge detection feature map M. e and semantic segmentation feature map M s The number of channels and resolution are restored to the spatial resolution size of the original image to obtain the corresponding edge detection result image and semantic segmentation result image.

[0056] Furthermore, in step (4), a segmentation mask method is used to combine the semantic segmentation result image with the visible light image in step 1 to obtain a visible light foreground image, and the semantic segmentation result image is combined with the infrared image in step 1 to obtain an infrared foreground image; the specific operation of the multi-level feature fusion and attention enhancement module is as follows:

[0057] Step 4.1: Use the multi-level feature extraction network model in step 2 to extract the multi-level features of the infrared foreground image and the visible light foreground image. and Where i∈{1,2,3,4,5}, and an asymmetric feature fusion is used to fusion the infrared features Enhanced, based on multi-level features and Calculate the weight activation matrix;

[0058]

[0059] Among them, Conv1(·) represents the convolution processing with a convolution kernel size of 1×1. The output is the weight activation matrix W of the two channels i ∈R (Hi)×(Wi)×2 ;

[0060] Step 4.2, weight activation matrix W i As a learnable parameter, it is separated along the channel and multiplied element-by-element with the infrared feature and the visible light feature respectively:

[0061]

[0062] Step 4.3: In the cross-modal multi-level feature fusion part, the enhanced T i 'As input, perform convolution, normalization and ReLU activation function processing respectively:

[0063]

[0064] Among them, BN(·) represents batch normalization processing, δ(·) represents the ReLU activation function, is the intermediate feature after processing;

[0065] Step 4.4: Adjust the size of the intermediate features at different levels through upsampling and uniformly adjust them to Respectively The width and height of the three sets of features are expanded to 2, 4 and 8 times of the original, and then the four-level features are fused by splicing along the channel dimension, as shown below:

[0066]

[0067] Among them, Up2, Up4, and Up8 respectively represent enlarging the feature map by 2 times, 4 times, and 8 times;

[0068] Step 4.5: The spatial attention enhancement part enhances the feature map through a learnable weight activation matrix, as shown below:

[0069]

[0070] Among them, F f Represents the fused feature map;

[0071] Step 4.6: In the feature reconstruction stage, three sets of cascaded decoders are used to restore the size of the feature map. The input features of the first-level decoder are upsampled to ensure the consistency of the feature map resolution in the subsequent decoding process, as shown below:

[0072]

[0073]

[0074]

[0075] in, Respectively represent the decoding feature maps generated by the three decoders, It is also the final output feature map.

[0076] Furthermore, the specific operations of step (5) are as follows:

[0077] (5.1) The binary cross entropy loss function is used to supervise the semantic segmentation result S and the semantic segmentation truth value GT, as shown below:

[0078]

[0079] Among them, S i GT i It represents the segmentation result during training and the true value of the segmentation result. i takes values ​​from the 1st pixel to the H×Wth pixel. H and W are the height and width of the feature map respectively, and H×W represents the total number of pixels in the feature map.

[0080] (5.2) Based on the residual information loss function of positive and negative samples, the edge contour is optimized, and the mean square error MSE and structural similarity metric SSIM are used to supervise the detection result S and sample residual R, as shown below:

[0081]

[0082]

[0083] Among them, R i represents the sample residual of the i-th pixel, i ranges from the 1st pixel to the H×Wth pixel, μ S and μ R represents the mean of the two images of the fever detection result S during training and the true value R of the residual result, that is, the average pixel intensity of the image, σ S and σ R Represents the standard deviation of the two images, representing the contrast of the images, σ SR It represents the covariance between two images, reflecting the structural similarity of the images. C1 and C2 are constants used to maintain the stability of the calculation and avoid the situation where the denominator is zero.

[0084] (5.3) Define the residual information loss function L rs , in residual information supervised learning, more attention is paid to the difference of pixel information:

[0085] L rs (S,R)=αL mse (S,R)+(1-α)L ssim (S,R) Formula (23)

[0086] Among them, α is the weight coefficient, which is set to 0.8, increasing the loss of pixel information L mse Weight;

[0087] (5.4) The final network supervision loss function is composed of the addition of the binary cross entropy loss function and the residual information loss function, as shown below:

[0088] L SOD =L bce +L rs Formula (23).

[0089] The present invention also discloses a device for segmenting and detecting abnormal heating of electric equipment based on heterogeneous image feature fusion, comprising a camera, a processor and a machine-readable storage medium;

[0090] The camera is used to collect images of the appearance of the substation and its surroundings, and send the images to the processor;

[0091] The machine-readable storage medium is used to store machine-executable instructions that can be executed by the processor;

[0092] The processor is used to be driven by the machine executable instructions to implement the above-mentioned power equipment segmentation and abnormal heating detection method based on heterogeneous image feature fusion.

[0093] Beneficial effects:

[0094] 1. The present invention improves the accuracy of power equipment segmentation: Through multi-level feature extraction and cross-modal fusion network models, the present invention fully utilizes the advantages of infrared images and visible light images to make up for the problem of insufficient information in single-modal images. The cross-modal fusion module can extract and fuse the features of images of different modalities, enhance the recognition accuracy of device boundaries, and make device segmentation in complex environments more accurate. By fusing high-level semantic features with edge detection features, it effectively suppresses background noise and interference and accurately locates the edges of power equipment.

[0095] 2. Improve the accuracy of abnormal fever detection: This invention uses multi-level feature fusion and attention enhancement to perform weighted fusion of multi-level features of infrared and visible light images, which helps the model to more accurately capture the details of abnormal fever areas. The attention mechanism can adaptively focus on key areas in the image, thereby significantly improving the detection accuracy of abnormal fever areas.

[0096] 3. Adapt to equipment monitoring in complex scenes: The present invention aims at the complex scenes and occlusion problems in power equipment monitoring. By fusing multimodal image features, the robustness and adaptability of the system in different environments are improved. It can ensure high detection accuracy and segmentation effect in different environments and is suitable for a wide range of practical application scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0097] Figure 1 It is a flow chart of the method for segmenting and detecting abnormal heating of electric power equipment based on heterogeneous image feature fusion according to the present invention;

[0098] Figure 2 This is a network architecture diagram of the multi-level feature extraction model of the present invention;

[0099] Figure 3 This is a network architecture diagram of the multi-level feature fusion module of the present invention;

[0100] Figure 4 The network architecture diagram of the semantic segmentation decoder and edge detection decoder supervised by edge information of the present invention;

[0101] Figure 5 This is a network architecture diagram of the multi-level feature fusion and attention enhancement module of the present invention. DETAILED DESCRIPTION

[0102] The following is a clear and complete description of the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0103] Example 1: Please refer to Figure 1 The present invention provides a method for segmenting and detecting abnormal heating of electric equipment based on heterogeneous image feature fusion, comprising the following steps:

[0104] Step 1. Take two images of the power equipment in different modes at the same time and from the same direction.

[0105] Step 2. Input the two captured images into a preset image feature extraction network model, extract low-level features and high-level features of the two images at multiple levels, fuse the features of different modal images at the corresponding levels, and enhance the adaptability to power equipment in complex scenes. The preset image feature extraction network model may include: a multi-level feature extraction network model and a multi-level feature fusion model.

[0106] Step 21: Input the two acquired images into a multi-level feature extraction network model to extract multi-level features of the two images. The specific steps are as follows:

[0107] See also Figure 2 , the multi-level feature extraction network model can be the RN feature extraction network model; RN contains two symmetric encoders based on the ResNet architecture, which are used to extract multi-level features of infrared and visible light images respectively. These features are divided into low-level features R1, R2, T1, T2 and high-level features R3, R4, R5, T3, T4, T5 according to the depth of the encoder. In order to reduce the computational complexity, dense blocks are added to each level of the encoder of the backbone network ResNet and the last fully connected layer is removed, and it is used as a network encoder for feature extraction. The encoder effectively extracts multi-level features of infrared and visible light images through progressively deeper convolutional layers, providing richer and more accurate feature representations for subsequent semantic segmentation tasks.

[0108] The two images acquired and The image features T extracted by the i-th level encoder Eni are respectively sent to the trained RN feature extraction network model. i =En i (I ir ),R i =En i (I vs ), and obtain the image features T at the corresponding leveli With R i .

[0109] Step 22: Fuse the multi-level features of the two images. The specific steps are as follows:

[0110] See also Figure 3 The multi-level feature fusion model includes a low-level feature fusion model and a high-level feature fusion model. The low-level feature fusion model is an algorithm for fusing shallow network features. First, the infrared feature map is processed by the maximum pooling operation of the channel dimension to achieve dimensionality reduction and feature extraction. Then, a 5×5 convolution kernel is used to convolve the infrared image features to deeply extract feature information, and then the processed image features are used. i ′ for image feature R i Make enhancements.

[0111] Specifically, the specific steps for fusing the low-level features of two images (only involving i∈{1,2}) are:

[0112] (2a) After the maximum pooling and convolution operations, the processed infrared image features are obtained:

[0113] T i ′=Conv5(Maxpool(T i )) Formula (1)

[0114] Among them, T i ′ is the infrared image feature after maximum pooling and 5×5 convolution, which can be used as auxiliary guidance information to enhance the visible light image features, i∈{1,2}.

[0115] (2b) Use the Sigmoid function to transform the pixels of the feature map element by element to obtain the preliminary fusion result of the visible light image features and the infrared image features:

[0116]

[0117] in, represents the element-by-element summation, and σ(·) represents the transformation of the pixels of the feature map element-by-element using the Sigmoid function. i Represents the preliminary fusion result of visible light image features and infrared image features.

[0118] (2c) f i As a weight matrix, the weighted feature map f of the visible light image feature and the preliminary fusion feature map is calculated by element-by-element multiplication. i ′, which is calculated as shown in formula (3):

[0119]

[0120] f i ′ represents the weighted fusion feature map, which represents the correlation between the visible light image and the infrared image.

[0121] (2d) Use the processed infrared image features to enhance the weighted fusion feature map to generate an enhanced weighted feature map

[0122]

[0123] (2e) The channel attention mechanism is used to enhance the fused feature map and output the fused feature map F i :

[0124]

[0125] Among them, MLP(·) is a multi-layer perceptron.

[0126] Specifically, high-level feature fusion (i∈{3,4,5}) mainly processes the image features deep in the network to understand the overall structure and category relationship of the scene. Correspondingly, the specific steps for high-level feature fusion of two images are:

[0127] (2f) Fuse the features of the two encoders to obtain the preliminary fusion feature f i :

[0128]

[0129] Among them, i∈{3,4,5}.

[0130] (2g) The initial fusion feature f i Input into dense blocks to enhance the expressiveness of features and alleviate the problem of gradient disappearance:

[0131]

[0132] Among them, Cat(·) indicates splicing along the channel dimension. D(·) represents four densely connected CBR modules, which are composed of convolution processing, normalization and ReLU activation function, and the convolution processing adopts the method of dilated convolution. The size of the two groups of Conv3 convolution kernels is 3×3. The first group of Conv3 convolution processing does not modify the number of feature map channels, and the second group of Conv3 convolution processing modifies the number of output feature map channels to 64 to reduce the number of parameters.

[0133] (2h) Dense semantic blocks are designed to further enhance the receptive field of feature maps and transmit global semantic information. Specifically, the dense semantic block draws on the densely connected architecture design in the dense block, and the network can directly obtain the feature maps of all previous layers at each layer. In each layer, the convolution process is replaced by a dilated convolution. The dilated convolution inserts zero values ​​at intervals in the convolution kernel to achieve expansion, increasing the receptive field without increasing the number of convolution kernel parameters. The dilated convolution is effective in semantic segmentation tasks and can capture a wider range of contextual information while maintaining the resolution of the feature map. In addition, the expansion rates of the four groups of dilated convolutions are set to 1, 3, 5, and 7, respectively:

[0134]

[0135] Among them, the dense semantic block fusion feature is Conv1(·) takes the input feature map The number of channels is changed to 16.

[0136] Conv3(·) modifies the number of output feature map channels to 64. D′(·) represents four densely connected CBR modules.

[0137] (2i) The infrared image features, visible light image features and dense semantic block fusion features are further fused through skip connections, and the dense semantic blocks are used again for feature enhancement to obtain the output of high-level feature fusion:

[0138]

[0139] Among them, F i Represents the output of high-level feature fusion. Conv1(·) is used to adjust the number of channels of the feature map to 64 to keep the number of channels consistent.

[0140] Step 3. The multi-level fusion feature map after fusion by the multi-level feature fusion model is used as the input of the semantic segmentation decoder and the edge detection decoder. The semantic segmentation decoder and the edge detection decoder share weight parameters, see Figure 4 , so that the semantic segmentation task can implicitly use the edge features learned by the edge detection decoder. According to the semantic information of the foreground target, the noise and interference of the background in the low-level fusion features are suppressed, so as to achieve more accurate edge detection of the foreground target. The semantic segmentation decoder and the edge detection decoder both include decoder 1, decoder 2 and decoder 3. The semantic segmentation decoder decodes the fusion features to generate a high-level semantic segmentation feature map, and the edge detection decoder decodes the fusion features to generate an edge detection feature map. The specific operations are as follows:

[0141] Step 31: Generate a high-level semantic segmentation feature map. The specific steps are as follows:

[0142] (3a) (Decoder working principle) The decoder processes the input feature map and obtains the corresponding output

[0143]

[0144] in, and are the decoder output and input respectively. Conv3 represents three cascaded convolution blocks with a kernel size of 3×3. up Represents an upsampling operation.

[0145] (3b) The semantic segmentation decoder generates a high-level semantic segmentation feature map M by iteratively decoding and aggregating three high-level fusion feature maps H = {F3, F4, F5} s :

[0146]

[0147] Among them, De i represents the i-th decoder.

[0148] Step 32: Generate an edge detection feature map, specifically:

[0149] (3c) By summing up the elements, we can integrate the edge detection decoder output feature map M e ′, low-level feature fusion output F2 and F1. The integrated feature map is directly combined with the feature map M output by the semantic segmentation encoder s Multiply element by element, and suppress the possible interference and noise in the low-level feature map through the semantic information in the high-level features to obtain the final edge detection feature map M e :

[0150]

[0151] Step 33, after performing channel processing on the edge detection feature map and the high-level semantic segmentation feature map, an edge detection result image and a semantic segmentation result image are determined. The specific operation is as follows:

[0152] (3d) Use a convolution block with a 1×1 kernel size and upsampling to modify the edge detection feature map M. e and semantic segmentation feature map M s The number of channels and resolution are restored to the spatial resolution size of the original image to obtain the corresponding edge detection result image and semantic segmentation result image.

[0153] Step 4: Combine the semantic segmentation result image with the visible light image in step 1 to obtain a visible light foreground image, combine the semantic segmentation result image with the infrared image in step 1 to obtain an infrared foreground image, introduce a multi-level feature fusion and attention enhancement module, extract features from the infrared foreground image and the visible light foreground image, perform feature fusion, fuse cross-modal information, and add a spatial attention enhancement module for processing, including:

[0154] (4a) Use the two ResNet-based encoders in step 2 to extract multi-level features of the infrared foreground image and the visible light foreground image respectively. and Where i∈{1,2,3,4,5}. In the multi-level feature extraction process, asymmetric feature fusion is used to extract infrared features. Enhanced, based on multi-level features and Calculate the weight activation matrix;

[0155]

[0156] Among them, Conv1(·) represents the convolution processing with a convolution kernel size of 1×1. The output is the weight activation matrix W of the two channels i ∈R (H / i)×(W / i)×2 .

[0157] (4b) Weight activation matrix W i As a learnable parameter, it is separated along the channel and multiplied element-wise with the infrared feature and the visible light feature respectively;

[0158]

[0159] (4c) In the cross-modal multi-level feature fusion part, the enhanced As input, convolution, normalization and ReLU activation function are performed respectively;

[0160]

[0161] Among them, BN(·) represents batch normalization processing. δ(·) represents the ReLU activation function. is the intermediate feature after processing.

[0162] (4d) According to formula (16), the sizes of the intermediate features at different levels are adjusted by upsampling and uniformly adjusted to Therefore, respectively The width and height of the three sets of features are expanded to 2, 4, and 8 times of the original. Then, the four-level features are fused by splicing along the channel dimension. The details are as follows:

[0163]

[0164] Among them, Up i (·) indicates that the feature map is magnified i times.

[0165] (4e) The spatial attention enhancement part enhances the feature map through a learnable weight activation matrix, so that the network can adjust the weights of different spatial positions. The details are as follows:

[0166]

[0167] Among them, F f Represents the fused feature map.

[0168] (4f) In the feature reconstruction stage, three sets of cascaded decoders are used to restore the size of the feature map. Among them, the input features of the first-level decoder are upsampled to ensure the consistency of the feature map resolution in the subsequent decoding process. The details are as follows:

[0169]

[0170]

[0171]

[0172] in, Respectively represent the decoding feature maps generated by the three decoders, It is also the final output feature map.

[0173] Step 5. Strengthen edge information and improve the accuracy and clarity of detection results through jump connections of low-level features of visible light. Use binary cross entropy loss function to supervise the semantic segmentation results and true values ​​of infrared images and visible light images, capture subtle features of abnormal heating areas, and solve the problem of blurred boundaries of abnormal heating area detection results, including:

[0174] Step 51: To address the problem of blurred boundaries in the abnormal heating area detection results, on the one hand, low-level features of visible light are reused through jump connections to enhance edge information and improve the accuracy and clarity of the detection results, and supervised learning is performed through true value results.

[0175] (5a) The binary cross entropy loss function is used to supervise the semantic segmentation result S and the semantic segmentation truth value GT. The details are as follows:

[0176]

[0177] Among them, H and W are the height and width of the feature map respectively, and H×W represents the total number of pixels in the feature map.

[0178] (5b) A residual information loss function based on positive and negative samples is proposed to optimize the edge contour. Specifically, the mean square error (MSE) and structural similarity metric (SSIM) are used to supervise the detection result S and sample residual R. The details are as follows:

[0179]

[0180]

[0181] Among them, μ S and μ R Represents the mean of the two images, that is, the average pixel intensity of the image. S and σ R Represents the standard deviation of the two images, representing the contrast of the images. SR Represents the covariance between two images, reflecting the structural similarity of the images. C is a small constant used to maintain calculation stability and avoid the situation where the denominator is zero. The details are as follows:

[0182] (5c) Define the residual information loss function L rs , in residual information supervised learning, more attention is paid to the differences in pixel information.

[0183] L rs (S,R)=αL mse (S,R)+(1-α)L ssim (S,R) Formula (24)

[0184] Among them, α is set to 0.8, increasing the loss of pixel information L mse Weight.

[0185] (5d) The final network supervision loss function is composed of the binary cross entropy loss function and the residual information loss function, as shown below:

[0186] L SOD =L bce +L rs Formula (25)

[0187] In addition, the embodiment of the present application also provides a power equipment segmentation and abnormal heating detection device based on heterogeneous image feature fusion, including a camera, a processor and a machine-readable storage medium;

[0188] A camera, used to collect images of the appearance of the substation and its surroundings, and send the images to the processor;

[0189] a machine-readable storage medium for storing machine-executable instructions that can be executed by the processor;

[0190] A processor is used to be driven by the machine executable instructions to implement the above-mentioned power equipment segmentation and abnormal heating detection method based on heterogeneous image feature fusion.

[0191] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0192] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0193] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0194] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0195] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for power equipment segmentation and abnormal heating detection based on heterogeneous image feature fusion, characterized in that: The following steps are involved: Step (1) photographing at the same time and from the same direction to obtain two registered images of the power equipment in different modes, namely, an infrared image and a visible light image; Step (2) inputting the two captured images into a multi-level feature extraction network model and a multi-level feature fusion model, wherein the multi-level feature extraction network model extracts low-level features and high-level features of the two images at multiple levels, and the multi-level feature fusion model fuses the features of the two images at the corresponding level to output a multi-level fusion feature map; Step (3) The multi-level fusion feature map fused by the multi-level feature fusion model is used as the input of the semantic segmentation decoder and the edge detection decoder. The semantic segmentation decoder and the edge detection decoder share weight parameters. The semantic information in the high-level features is used to suppress the interference and noise in the low-level feature map, and the edge detection feature map and the high-level semantic segmentation feature map are obtained. After channel processing of the edge detection feature map and the high-level semantic segmentation feature map, the edge detection result image and the semantic segmentation result image are determined. Step (4) combines the semantic segmentation result image with the visible light image in step 1 to obtain a visible light foreground image, combines the semantic segmentation result image with the infrared image in step 1 to obtain an infrared foreground image, introduces a multi-level feature fusion and attention enhancement module, extracts features from the infrared foreground image and the visible light foreground image, performs feature fusion on the infrared foreground image and the visible light foreground image, fuses cross-modal information, and adds a spatial attention enhancement module for processing; Step (5) uses a binary cross entropy loss function to supervise the semantic segmentation results and semantic segmentation truth values ​​of the infrared image and the visible light image, capture the subtle features of the abnormal heating area, and finally determine the abnormal heating area.

2. According to claim 1, a method for power equipment segmentation and abnormal heating detection based on heterogeneous image feature fusion is characterized by: The multi-level feature extraction network model includes two symmetric encoders based on the ResNet architecture, which are used to extract multi-level features of infrared images and visible light images respectively. Each symmetric encoder based on the ResNet architecture includes encoders 1 to 5. A dense block is added to each level of the encoder of the backbone network ResNet and the last fully connected layer is removed. The encoder is used as a network encoder for feature extraction. The multi-level features of infrared and visible light images are extracted through the convolutional layers that go deeper and deeper. Set the two images to be acquired and They are sent to the trained multi-level feature extraction network model respectively, and then the i-th level encoder En i Extracted image features T i =En i (I ir ),R i =En i (I vs ), i∈{1,2,3,4,5}, and obtain the image features T at the corresponding level i With R i , where R i is the multi-level feature of the visible light image, T i is the multi-level feature of the infrared image. When i∈{1,2}, T1, T2, R1, R2 are low-level features, and T3, T4, T5, R3, R4, R5 are high-level features.

3. The method for segmenting and detecting abnormal heating of electric power equipment based on heterogeneous image feature fusion according to claim 2 is characterized in that: The multi-level feature fusion model includes a low-level feature fusion model and a high-level feature fusion model; The low-level feature fusion model processes the image features T1, T2, R1, and R2. First, the multi-level features T1 and T2 of the infrared image are processed by the maximum pooling operation of the channel dimension. Then, a 5×5 convolution kernel is used to convolve the infrared image features to obtain the processed infrared image features T1' and T2'. The processed infrared image features T1' and T2' are used to enhance the visible light image features R1 and R2 to obtain the preliminary fusion results f1 and f2. The preliminary fusion results f1 and f2 are used as weight matrices to calculate the weighted feature maps f1' and f2' of the visible light image features and the preliminary fusion feature maps by element-by-element multiplication. The processed infrared image features T1' and T2' are used to enhance the weighted fusion feature maps f1' and f2' to generate enhanced weighted feature maps. Finally, the channel attention mechanism is used to enhance the fusion of the enhanced weighted feature maps and output the fused feature maps F1 and F2; The advanced feature fusion model processes the image features T3, T4, T5, R3, R4, and R5, and fuses the image features T3, T4, and T5 of encoder 3, encoder 4, and encoder 5 respectively. i and R i , i∈{3,4,5}, the preliminary fusion features of each level of encoder are obtained, and the preliminary fusion features of each level of encoder are input into the dense block to enhance the expressiveness of the features and alleviate the problem of gradient disappearance; dense semantic blocks are designed to further enhance the receptive field of the feature map and transmit global semantic information; in each layer of the dense semantic block, the convolution processing is replaced by a hole convolution, and the infrared image features, visible light image features and dense semantic block fusion features are further fused through jump connections, and the dense semantic blocks are used again for feature enhancement to obtain the output F3, F4, and F5 of high-level feature fusion.

4. The method for segmenting and detecting abnormal heating of electric power equipment based on heterogeneous image feature fusion according to claim 3 is characterized in that: The specific operation of the low-level feature fusion model is as follows: (2.1) After the maximum pooling and convolution operations, the processed infrared image features are obtained: T i ′=Conv5(Maxpool(T i )) Formula(1) Among them, T i ′ is the infrared image feature after maximum pooling and 5×5 convolution, which can be used as auxiliary guidance information to enhance the visible light image features, i∈{1,2}; (2.2) Use the Sigmoid function to transform the pixels of the feature map element by element to obtain the preliminary fusion results of the visible light image features and the infrared image features: in, represents the element-by-element summation, σ(·) represents the element-by-element transformation of the pixels of the feature map using the Sigmoid function, and f i represents the preliminary fusion result of visible light image features and infrared image features, i∈{1,2}; (2.3) f i As a weight matrix, the weighted feature map f of the visible light image feature and the preliminary fusion feature map is calculated by element-by-element multiplication. i ′: in, Indicates element-wise multiplication of the feature map, f i ′ represents the weighted fusion feature map, which represents the correlation between the visible light image and the infrared image, i∈{1,2}; (2.4) Use the processed infrared image features to enhance the weighted fusion feature map to generate an enhanced weighted feature map (2.5) Use channel attention mechanism to weight feature map Enhance and output the fusion feature map F i , i∈{1,2}: Among them, MLP(·) is a multi-layer perceptron.

5. The method for segmenting and detecting abnormal heating of electric power equipment based on heterogeneous image feature fusion according to claim 3 is characterized in that: The specific operations of the advanced feature fusion model are: (2.6) Fuse the features of the two encoders corresponding to the two symmetric encoders based on the ResNet architecture to obtain the preliminary fusion feature f i , i∈{3,4,5}: (2.7) The initial fusion feature f i Input into dense blocks to enhance the expressiveness of features and alleviate the problem of gradient disappearance: Among them, Cat(·) represents splicing along the channel dimension, D(·) represents four densely connected CBR modules, and the CBR module consists of convolution processing, normalization and ReLU activation function. The convolution processing adopts the method of dilated convolution. The sizes of the two groups of Conv3 convolution kernels are both 3×3. The first group of Conv3 convolution processing does not modify the number of feature map channels, and the second group of Conv3 convolution processing modifies the number of output feature map channels to 64. (2.8) The specific operations of dense semantic blocks are: Among them, the dense semantic block fusion feature is Conv1(·) modifies the number of channels to 16, Conv3(·) modifies the number of output feature map channels to 64, D′(·) represents four densely connected CBR modules, and the expansion rates of the four groups of dilated convolutions are set to 1, 3, 5, and 7 respectively; (2.9) The infrared image features, visible light image features and dense semantic block fusion features are further fused through skip connections to obtain the output of high-level feature fusion: Among them, F i , i∈{3,4,5} represents the output of high-level feature fusion.

6. The method for segmenting and detecting abnormal heating of electric power equipment based on heterogeneous image feature fusion according to claim 3 is characterized in that: In the step (3), the semantic segmentation decoder and the edge detection decoder both include decoder 1, decoder 2 and decoder 3. The semantic segmentation decoder decodes the fused features to generate a high-level semantic segmentation feature map, and the edge detection decoder decodes the fused features to generate an edge detection feature map. The specific operations are: Step 3.1: Generate high-level semantic segmentation feature maps. The specific steps are as follows: The semantic segmentation decoder aggregates three high-level fusion feature maps H = {F3, F4, F5} through iterative decoding to generate a high-level semantic segmentation feature map Among them, De1, De2, and De3 represent three decoder operations; Step 3.2: Generate edge detection feature map. The specific operations are as follows: By summing up the elements, we can integrate the edge detection decoder output feature map M e ′, low-level feature fusion output F2 and F1, the integrated feature map is directly combined with the feature map M output by the semantic segmentation encoder s Multiply element by element, and use the semantic information in the high-level features to suppress the interference and noise that may exist in the low-level feature map to obtain the final edge detection feature map Among them, F up Represents an upsampling operation; Step 3.3: After performing channel processing on the edge detection feature map and the high-level semantic segmentation feature map, the edge detection result image and the semantic segmentation result image are determined. The specific operations are as follows: Use a convolution block with a 1×1 kernel size and upsampling to modify the edge detection feature map M. e and semantic segmentation feature map M s The number of channels and resolution are restored to the spatial resolution size of the original image to obtain the corresponding edge detection result image and semantic segmentation result image.

7. The method for segmenting and detecting abnormal heating of electric power equipment based on heterogeneous image feature fusion according to claim 1, characterized in that: In the step (4), a segmentation mask method is used to combine the semantic segmentation result image with the visible light image in step 1 to obtain a visible light foreground image, and the semantic segmentation result image is combined with the infrared image in step 1 to obtain an infrared foreground image; the specific operation of the multi-level feature fusion and attention enhancement module is as follows: Step 4.1: Use the multi-level feature extraction network model in step 2 to extract the multi-level features of the infrared foreground image and the visible light foreground image. and Where i∈{1,2,3,4,5}, and an asymmetric feature fusion is used to fusion the infrared features Enhanced, based on multi-level features and Calculate the weight activation matrix; Among them, Conv1(·) represents the convolution processing with a convolution kernel size of 1×1. The output is the weight activation matrix W of the two channels i ∈R (H / i)×(W / i)×2 ; Step 4.2: Weight activation matrix W i As a learnable parameter, it is separated along the channel and multiplied element-by-element with the infrared feature and the visible light feature respectively: Step 4.3: In the cross-modal multi-level feature fusion part, the enhanced As input, convolution, normalization and ReLU activation function are performed respectively: Among them, BN(·) represents batch normalization processing, δ(·) represents the ReLU activation function, is the intermediate feature after processing; Step 4.4: Adjust the size of the intermediate features at different levels through upsampling and uniformly adjust them to Respectively The width and height of the three sets of features are expanded to 2, 4 and 8 times of the original, and then the four-level features are fused by splicing along the channel dimension, as shown below: Among them, Up2, Up4, and Up8 respectively represent enlarging the feature map by 2 times, 4 times, and 8 times; Step 4.5: The spatial attention enhancement part enhances the feature map through a learnable weight activation matrix, as shown below: Among them, F f Represents the fused feature map; Step 4.6: In the feature reconstruction stage, three sets of cascaded decoders are used to restore the size of the feature map. The input features of the first-level decoder are upsampled to ensure the consistency of the feature map resolution in the subsequent decoding process, as shown below: in, Respectively represent the decoding feature maps generated by the three decoders, It is also the final output feature map.

8. A method for segmenting and detecting abnormal heating of electric equipment based on heterogeneous image feature fusion according to claim 1, characterized in that: The specific operation of step (5) is as follows: (5.1) The binary cross entropy loss function is used to supervise the semantic segmentation result S and the semantic segmentation truth value GT, as shown below: Among them, S i GT i It represents the segmentation result during training and the true value of the segmentation result. i takes values ​​from the 1st pixel to the H×Wth pixel. H and W are the height and width of the feature map respectively, and H×W represents the total number of pixels in the feature map. (5.2) Based on the residual information loss function of positive and negative samples, the edge contour is optimized, and the mean square error MSE and structural similarity metric SSIM are used to supervise the detection result S and sample residual R, as shown below: Among them, R i represents the sample residual of the i-th pixel, i ranges from the 1st pixel to the H×Wth pixel, μ S and μ R represents the mean of the two images of the fever detection result S during training and the true value R of the residual result, that is, the average pixel intensity of the image, σ S and σ R Represents the standard deviation of the two images, representing the contrast of the images, σ SR It represents the covariance between two images, reflecting the structural similarity of the images. C1 and C2 are constants used to maintain the stability of the calculation and avoid the situation where the denominator is zero. (5.3) Define the residual information loss function L rs , in residual information supervised learning, more attention is paid to the difference of pixel information: L rs (S, R) = αL mse (S, R) + (1 - α)L ssim (S, R) Formula (23) Among them, α is the weight coefficient, which is set to 0.8, increasing the loss of pixel information L mse Weight; (5.4) The final network supervision loss function is composed of the addition of the binary cross entropy loss function and the residual information loss function, as shown below: L SOD =L bce +L rs Formula (23).

9. A device for segmenting and detecting abnormal heating of electric equipment based on heterogeneous image feature fusion, characterized in that: including a camera, a processor, and a machine-readable storage medium; The camera is used to collect images of the appearance of the substation and its surroundings, and send the images to the processor; The machine-readable storage medium is used to store machine-executable instructions that can be executed by the processor; The processor is used to be driven by the machine executable instructions to implement the power equipment segmentation and abnormal heating detection method based on heterogeneous image feature fusion as described in any one of claims 1 to 8.

Images