A power equipment segmentation and abnormal heating detection method and device based on heterogeneous image feature fusion

Abstract

The application discloses a power equipment segmentation and abnormal heating detection method and device based on heterogeneous image feature fusion. The method comprises the following steps: acquiring two registered images of power equipment under different modalities; extracting features and extracting low-level features and high-level features of two images in multiple levels, fusing features of different modalities under corresponding levels; generating a high-level semantic segmentation feature map through iterative decoding and aggregation of three high-level fusion feature maps, obtaining a corresponding semantic segmentation result image and an edge detection result image; introducing a multi-level feature fusion and attention enhancement module to optimize the understanding and analysis capability of the image; using a binary cross-entropy loss function to supervise the semantic segmentation result and the semantic segmentation true value of the two processed images, and capturing the subtle features of the abnormal heating area. The application can greatly improve the precision and reliability of power equipment segmentation and abnormal heating detection, reduce the false detection rate, and has high practicality and wide application prospect.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image processing, and particularly relates to a power equipment segmentation and abnormal heating detection method and device based on heterogeneous image feature fusion. BACKGROUND

[0002] With the rapid development of modern power systems, the state monitoring and fault diagnosis of power equipment have become key tasks to ensure the safe and stable operation of power systems. Once a fault occurs in power equipment, especially abnormal heating, it may cause equipment damage and even trigger large-scale power outages. Therefore, accurately detecting the abnormal heating of power equipment is of great significance for preventing potential faults and ensuring system safety.

[0003] Currently, the detection of abnormal heating of power equipment mainly relies on infrared thermal imaging technology. Infrared thermal imaging can directly display the temperature distribution of equipment and quickly identify high-temperature areas. However, infrared images are greatly affected by external environmental conditions such as temperature, humidity, and emissivity, which may cause errors in thermal imaging results. In addition, power equipment has complex shapes, and there are occlusions and similar backgrounds between equipment, making it difficult to achieve accurate segmentation and fault detection of equipment relying solely on infrared images. To address these issues, researchers have gradually attempted to introduce other imaging methods, such as visible light images, to combine multi-source images for information fusion and improve the accuracy of abnormal heating detection of power equipment.

[0004] In recent years, image segmentation and target detection methods based on deep learning have made significant progress in the field of computer vision. In the monitoring of power equipment, some studies use convolutional neural networks (CNN) to segment and detect temperatures of infrared images. For example, classic models such as Mask-RCNN can generate accurate segmentation masks, identify power equipment regions, and analyze their temperature distribution, but these methods have high computational complexity and poor real-time performance. Other studies such as CenterNet models improve the accuracy of temperature detection of equipment by extracting multi-scale features and performing rotation angle regression, but there are still limitations in the identification of equipment edges and the detection of abnormal heating areas.

[0005] To overcome the shortcomings of the above methods, heterogeneous feature fusion methods based on multi-modal data have gradually attracted attention. Multi-modal feature fusion combines the features of infrared images and visible light images to make up for the lack of information from a single modality. This type of method provides temperature information from infrared images, while visible light images can more clearly display the boundaries and structures of equipment, and the combination of the two helps to improve segmentation accuracy and the accuracy of abnormal heating detection. However, simple feature splicing or element-by-element operations cannot fully utilize the information complementarity between modalities, which may result in information loss or increased computational complexity. Therefore, how to effectively fuse features of these two different modalities remains a technical difficulty. SUMMARY

[0006] The application provides a power equipment segmentation and abnormal heating detection method and device based on heterogeneous image feature fusion, which solves the problems of information loss, inaccurate equipment boundary identification and low abnormal heating detection precision in the fusion process of infrared images and visible light images in the prior art, and also solves the problem that the infrared thermal imaging technology is greatly affected by environmental temperature, surface emissivity and other factors, and is prone to misjudgment. The application fuses the features of infrared images and visible light images, makes up for the deficiency of single modal information, can realize accurate segmentation of power equipment, and reliably detects the abnormal heating area of the equipment in a complex environment.

[0007] Technical scheme: The application discloses a power equipment segmentation and abnormal heating detection method based on heterogeneous image feature fusion, comprising the following steps:

[0008] Step (1) photographing at the same time and from the same direction to obtain two registered images of power equipment under different modalities, i.e. infrared images and visible light images;

[0009] Step (2) inputting the two photographed images into a multi-level feature extraction network model and a multi-level feature fusion model, the multi-level feature extraction network model extracts low-level features and high-level features of the two images in multiple levels, and the multi-level feature fusion model fuses the features of the two images in the corresponding levels to output multi-level fusion feature maps;

[0010] Step (3) the multi-level fusion feature maps fused by the multi-level feature fusion model are used as the input of a semantic segmentation decoder and an edge detection decoder, the semantic segmentation decoder and the edge detection decoder share weight parameters, the semantic information in the high-level features is used to suppress the interference and noise existing in the low-level feature maps, edge detection feature maps and high-level semantic segmentation feature maps are obtained, and after channel processing of the edge detection feature maps and the high-level semantic segmentation feature maps, an edge detection result image and a semantic segmentation result image are determined;

[0011] Step (4) combining the semantic segmentation result image with the visible light image in step 1 to obtain a visible light foreground image, combining the semantic segmentation result image with the infrared image in step 1 to obtain an infrared foreground image, introducing a multi-level feature fusion and attention enhancement module, performing feature extraction on the infrared foreground image and the visible light foreground image, and then performing feature fusion, and adding a spatial attention enhancement module for processing;

[0012] Step (5) using a binary cross-entropy loss function to supervise the semantic segmentation results of the infrared image and the visible light image and the semantic segmentation true value, capturing the subtle features of the abnormal heating area, obtaining a residual heat map, and finally determining the area of abnormal heating.

[0013] Further, the multi-level feature extraction network model comprises two symmetrical encoders based on the ResNet architecture, respectively used for extracting multi-level features of the infrared image and the visible light image, each symmetrical encoder based on the ResNet architecture comprises encoders 1-5, a dense block is added in each level of the encoder of the backbone network ResNet and the last fully connected layer is removed, which is used as a network encoder for feature extraction, and multi-level features of the infrared and visible light images are extracted through the convolution layer in each level.

[0014] The two images obtained are set to be sent into the trained multi-level feature extraction network model respectively, and the image features T and are extracted through the i-level encoder En i , R i = En i (I ir ), R i = En i (I vs ), i∈{1,2,3,4,5}, to obtain the image features T i and R i of the corresponding level, wherein R i is the multi-level feature of the visible light image, T i is the multi-level feature of the infrared image, and T1, T2, R1 and R2 are low-level features when i∈{1,2}, and T3, T4, T5, R3, R4 and R5 are high-level features.

[0015] Further, the multi-level feature fusion model comprises a low-level feature fusion model and a high-level feature fusion model.

[0016] The low-level feature fusion model processes the image features T1, T2, R1 and R2, first processes the multi-level features T1 and T2 of the infrared image through the maximum pooling operation in the channel dimension, then uses a 5×5 convolution kernel to perform convolution processing on the infrared image features to obtain the processed infrared image features T1' and T2', uses the processed infrared image features T1' and T2' to enhance the visible light image features R1 and R2 to obtain the preliminary fusion results f1 and f2, uses the preliminary fusion results f1 and f2 as a weight matrix, and calculates the weighted feature maps f1' and f2' of the visible light image features and the preliminary fusion feature maps through element-by-element multiplication, uses the processed infrared image features T1' and T2' to enhance the weighted fusion feature maps f1' and f2', and generates the enhanced weighted feature maps Finally, the enhanced weighted feature maps are enhanced and fused through the channel attention mechanism, and the fusion feature maps F1 and F2 are output.

[0017] The high-level feature fusion model processes image features T3, T4, T5, R3, R4, R5, respectively fuses image features T i and R i , i∈{3,4,5}, to obtain the preliminary fusion features of each level encoder, inputs the preliminary fusion features of each level encoder into the dense block to enhance the expression ability of the features and relieve the problem of gradient disappearance; the dense semantic block is designed to further enhance the receptive field of the feature map and transmit global semantic information; in each layer, the convolution processing is replaced by the dilated convolution, the infrared image features, the visible light image features and the fusion features of the dense semantic block are further fused through the skip connection mode, the dense semantic block is used again for feature enhancement, and the output F3, F4, F5 of the high-level feature fusion is obtained.

[0018] Further, the low-level feature fusion model specifically operates as follows:

[0019] (2.1) After the max-pooling and convolution operations, the processed infrared image features are obtained:

[0020] T i ′=Conv5(Maxpool(T i )) Formula (1)

[0021] Wherein, T i ′ is the infrared image feature processed by the max-pooling and 5*5 convolution, which can be used as auxiliary guidance information to enhance the visible light image feature, i∈{1,2};

[0022] (2.2) The pixels of the feature map are converted element by element using the Sigmoid function to obtain the preliminary fusion results of the visible light image features and the infrared image features:

[0023]

[0024] Wherein, represents element-wise summation, σ(·) represents conversion of pixels of the feature map element by element using the Sigmoid function, f i represents the preliminary fusion results of the visible light image features and the infrared image features, i∈{1,2};

[0025] (2.3) f i is used as a weight matrix to calculate the weighted feature map f i ′ of the visible light image features and the preliminary fusion feature map by element-wise multiplication:

[0026]

[0027] Wherein, f represents element-wise multiplication of the feature map. i ′ represents the weighted fusion feature map, which characterizes the correlation between the visible light image and the infrared image, i∈{1,2};

[0028] (2.4) The weighted fusion feature map is enhanced using the processed infrared image features to generate an enhanced weighted feature map.

[0029]

[0030] (2.5) Employing a channel attention mechanism to weighted feature maps Enhancement is performed, and the fused feature map F is output. i , i∈{1,2}:

[0031]

[0032] MLP(·) is a multilayer perceptron.

[0033] Furthermore, the specific operation of the advanced feature fusion model is as follows:

[0034] (2.6) Fuse the features of the two encoders corresponding to the two ResNet-based symmetric encoders to obtain the initial fused features f. i , i∈{3,4,5}:

[0035]

[0036] (2.7) Initially merge features f i Inputting into dense blocks enhances the expressive power of features and alleviates the gradient vanishing problem:

[0037]

[0038] Where Cat(·) represents concatenation along the channel dimension, and D(·) represents four densely connected CBR modules. The CBR module consists of convolution processing, normalization, and ReLU activation function. The convolution processing adopts dilated convolution. The kernel size of both Conv3 convolutions is 3×3. The first Conv3 convolution does not modify the number of channels in the feature map, while the second Conv3 convolution modifies the number of channels in the output feature map to 64.

[0039] (2.8) The specific operations for dense semantic blocks are as follows:

[0040]

[0041] Among them, the dense semantic block fusion feature is Conv1(·) modifies the number of channels to 16, Conv3(·) modifies the number of output feature map channels to 64, D'(·) represents four densely connected CBR modules, and the expansion rates of the four sets of atrous convolutions are set to 1, 3, 5 and 7 respectively;

[0042] (2.9) The infrared image features, the visible light image features and the dense semantic block fusion features are further fused by a skip connection mode to obtain high-level feature fusion output:

[0043]

[0044] wherein F i , i∈{3,4,5} represents the high-level feature fusion output.

[0045] Further, in the step (3), the semantic segmentation decoder and the edge detection decoder each include a decoder 1, a decoder 2 and a decoder 3, the semantic segmentation decoder decodes the fusion features to generate a high-level semantic segmentation feature map, and the edge detection decoder decodes the fusion features to generate an edge detection feature map, and the specific operation is as follows:

[0046] Step 3.1, generating a high-level semantic segmentation feature map, and the specific steps are as follows:

[0047] The semantic segmentation decoder aggregates three high-level fusion feature maps H={F3,F4,F5} through iterative decoding to generate a high-level semantic segmentation feature map

[0048]

[0049] wherein De1, De2 and De3 represent 3 decoder operations;

[0050] Step 3.2, generating an edge detection feature map, and the specific operation is as follows:

[0051] The edge detection decoder output feature map M e is integrated by element-wise summation, and the integrated feature map is directly multiplied with the feature map M s output by the semantic segmentation encoder to obtain the final edge detection feature map

[0052]

[0053] wherein F up represents an up-sampling operation;

[0054] Step 3.3, after channel processing of the edge detection feature map and the high-level semantic segmentation feature map, the edge detection result image and the semantic segmentation result image are determined, and the specific operation is as follows:

[0055] The channel number and resolution of the edge detection feature map M e and the semantic segmentation feature map M s are modified using a convolution block with a 1x1 kernel size and up-sampling processing, and the spatial resolution size of the original image is restored to obtain the corresponding edge detection result image and semantic segmentation result image.

[0056] Further, in step (4), the semantic segmentation result image is combined with the visible light image in step 1 to obtain a visible light foreground image, and the semantic segmentation result image is combined with the infrared image in step 1 to obtain an infrared foreground image using a segmentation mask method; the specific operation of the multi-level feature fusion and attention enhancement module is as follows:

[0057] Step 4.1, the multi-level features of the infrared foreground image and the visible light foreground image are extracted using the multi-level feature extraction network model in step 2 and where i∈{1,2,3,4,5}, and the infrared feature is enhanced using asymmetric feature fusion, and the weight activation matrix is calculated according to the multi-level features and

[0058]

[0059] where Conv1(·) represents a 1x1 convolution kernel size convolution processing, and outputs a two-channel weight activation matrix W i ∈R (Hi)×(Wi)×2 ;

[0060] Step 4.2, the weight activation matrix W i is separated along the channel and multiplied element by element with the infrared feature and the visible light feature respectively:

[0061]

[0062] Step 4.3, in the cross-modal multi-level feature fusion part, the enhanced T i ' is input, and convolution, normalization and ReLU activation function processing are performed respectively:

[0063]

[0064] where BN(·) represents batch normalization processing, and δ(·) represents ReLU activation function, ​the processed intermediate feature;

[0065] Step 4.4, the size of the intermediate feature in different levels is adjusted by upsampling processing, and is uniformly adjusted to respectively, the width and height of the three groups of features are expanded to 2 times, 4 times and 8 times of the original, and then the four-level features are fused by concatenating along the channel dimension, as follows:

[0066]

[0067] wherein, Up2, Up4 and Up8 respectively represent that the feature map is enlarged by 2 times, 4 times and 8 times;

[0068] Step 4.5, the spatial attention enhancement part enhances the feature map by a learnable weight activation matrix, as follows:

[0069]

[0070] wherein, F f represents the fused feature map;

[0071] Step 4.6, in the feature reconstruction stage, three groups of cascaded decoders are used to recover the size of the feature map, wherein the input feature of the first level decoder is processed by upsampling to ensure the consistency of the resolution of the feature map in the subsequent decoding process, as follows:

[0072]

[0073]

[0074]

[0075] wherein, respectively represent the decoding feature maps generated by the three decoders, which are also the final output feature maps.

[0076] Further, the step (5) is specifically operated as follows:

[0077] (5.1) using binary cross-entropy loss function to supervise the semantic segmentation result S and the semantic segmentation true value GT, as follows:

[0078]

[0079] wherein, S i , GT i represent the segmentation result and the true value of the segmentation result during training, i takes values from the 1st pixel to the HxWth pixel, H and W are respectively the height and width of the feature map, and HxW represents the total number of pixels of the feature map.​

[0080] (5.2) Based on the residual information loss function of positive and negative samples, the edge contour is optimized, and the mean square error (MSE) and the structural similarity measurement (SSIM) are used to supervise the detection result S and the sample residual R, as shown in the following formula:

[0081]

[0082]

[0083] Wherein, R i The sample residual of the i-th pixel, i takes the value from the first pixel to the HxW-th pixel, μ S And μ R Indicate the mean value of the two images of the heat detection result S and the residual result R during training, that is, the average pixel intensity of the image, σ S And σ R Indicate the standard deviation of the two images, which represents the contrast of the image, σ SR Indicates the covariance between the two images, which reflects the structural similarity of the image, C1 and C2 are constants to maintain the stability of the calculation and avoid the case that the denominator is zero.

[0084] (5.3) Define the residual information loss function L rs , which pays more attention to the difference of pixel information in the residual information supervised learning:

[0085] L rs (S,R)=αL mse (S,R)+(1-α)L ssim (S,R) formula (23)

[0086] Wherein, alpha is a weight coefficient, which is set to 0.8, and the loss L mse Weight of pixel information is increased.

[0087] (5.4) The final network supervision loss function is composed of binary cross entropy loss function and residual information loss function, as shown in the following formula:

[0088] L SOD =L bce +L rs Formula (23).

[0089] The application also discloses a power equipment segmentation and abnormal heat detection device based on heterogeneous image feature fusion, which comprises a camera, a processor and a machine readable storage medium.

[0090] The camera is used for collecting the images of the appearance of the power transformation equipment and the surrounding environment, and sending the images to the processor.

[0091] the machine readable storage medium, for storing machine executable instructions capable of being executed by the processor;

[0092] the processor, caused by the machine executable instructions, to implement the above-mentioned power equipment segmentation and abnormal heating detection method based on heterogeneous image feature fusion.

[0093] Advantages:

[0094] 1. The present application improves the accuracy of power equipment segmentation: through multi-level feature extraction and cross-modal fusion network model, the present application makes full use of the advantages of infrared image and visible light image, and makes up for the problem of insufficient information of single modal image. The cross-modal fusion module can extract and fuse the features of different modal images, enhance the recognition accuracy of the device boundary, make the device segmentation more accurate in complex environment, and effectively suppress the background noise and interference by fusing high-level semantic features and edge detection features, accurately locate the edge of power equipment.

[0095] 2. Improve the accuracy of abnormal heating detection: the present application can weight and fuse the multi-level features of infrared and visible light images through multi-level feature fusion and attention enhancement, which helps the model to capture the details of the abnormal heating area more accurately. The attention mechanism can adaptively focus on the key areas in the image, thereby significantly improving the detection accuracy of the abnormal heating area.

[0096] 3. Adapt to device monitoring in complex scenes: the present application is aimed at the complex scenes and occlusion problems in power equipment monitoring, and improves the robustness and adaptability of the system in different environments by fusing multi-modal image features, which can guarantee high detection accuracy and segmentation effect in different environments, and is suitable for a wide range of practical application scenarios. BRIEF DESCRIPTION OF DRAWINGS

[0097] Figure 1 is a flowchart of the power equipment segmentation and abnormal heating detection method based on heterogeneous image feature fusion of the present application;

[0098] Figure 2 is a multi-level feature extraction model network architecture diagram of the present application;

[0099] Figure 3 is a multi-level feature fusion module network architecture diagram of the present application;

[0100] Figure 4 is a semantic segmentation decoder and edge detection decoder network architecture diagram of the present application supervised by edge information;

[0101] Figure 5 is a network architecture diagram of the multi-level feature fusion and attention enhancement module of the present application. DETAILED DESCRIPTION

[0102] The technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work fall within the scope of the present application.

[0103] Embodiment 1: Please refer to Figure 1 The present application provides a power equipment segmentation and abnormal heating detection method based on heterogeneous image feature fusion, comprising the following steps:

[0104] Step 1. Take two images of the power equipment under different modalities at the same time from the same direction.

[0105] Step 2. Input the two images taken into a preset image feature extraction network model, extract low-level features and high-level features of the two images in multiple levels, and fuse the features of the images under different modalities at the corresponding levels to enhance the adaptability to the power equipment in complex scenes. The preset image feature extraction network model can include a multi-level feature extraction network model and a multi-level feature fusion model.

[0106] Step 21, input the two images obtained into a multi-level feature extraction network model to extract multi-level features of the two images, and the specific steps are as follows:

[0107] Referring to Figure 2 The multi-level feature extraction network model can be an RN feature extraction network model; RN contains two symmetrical encoders based on ResNet architecture for extracting multi-level features of infrared and visible light images. These features are divided into low-level features R1, R2, T1, T2 and high-level features R3, R4, R5, T3, T4, T5 according to the depth of the encoder. In order to reduce the computational complexity, a dense block is added in each level of the encoder of the backbone network ResNet and the last fully connected layer is removed, which is used as a network encoder for feature extraction. The encoder effectively extracts multi-level features of infrared and visible light images through convolution layers at each level, providing more accurate feature representation for subsequent semantic segmentation tasks.

[0108] The two images obtained and are respectively sent into the trained RN feature extraction network model, and the image features T i = En i (I ir ), R i = En i (I vs ) are extracted by the i-th level encoder Eni, and the image features Ti with R i .

[0109] Step 22, fuse the multi-level features of the two images, the specific steps are as follows:

[0110] Referring to Figure 3 , the multi-level feature fusion model includes a low-level feature fusion model and a high-level feature fusion model. The low-level feature fusion model is an algorithm for fusing network shallow features. First, the infrared feature map is processed by a channel dimension maximum pooling operation to realize dimension reduction and feature extraction. Then, a 5x5 convolution kernel is used to convolve the infrared image features to deeply extract feature information, and then the processed image features T i ′ are enhanced. i

[0111] Specifically, the specific steps of low-level feature fusion of the two images (only involving i∈{1, 2}) are as follows:

[0112] (2a) After the maximum pooling and convolution operation, the processed infrared image features are obtained:

[0113] T i ′=Conv5(Maxpool(T i )) Formula (1)

[0114] Wherein, T i ′ is the infrared image feature processed by maximum pooling and 5x5 convolution, which can be used as auxiliary guidance information to enhance the visible light image feature, i∈{1, 2}.

[0115] (2b) Use the Sigmoid function to convert the pixels of the feature map element by element to obtain the preliminary fusion result of the visible light image feature and the infrared image feature:

[0116]

[0117] Wherein, represents element-wise summation, and σ(·) represents converting the pixels of the feature map element by element using the Sigmoid function. f i represents the preliminary fusion result of the visible light image feature and the infrared image feature.

[0118] (2c) Take f i as a weight matrix, and calculate the weighted feature map f i ′ of the visible light image feature and the preliminary fusion feature map by element-wise multiplication, and the calculation method is shown in formula (3):

[0119]

[0120] ​f i represents a weighted fusion feature map, representing the correlation between the visible light image and the infrared image.

[0121] (2d) The processed infrared image features are used to enhance the weighted fusion feature map to generate an enhanced weighted feature map

[0122]

[0123] (2e) The channel attention mechanism is adopted to enhance the fusion feature map, and the output fusion feature map F is output i :

[0124]

[0125] wherein MLP(·) is a multi-layer perceptron.

[0126] Specifically, the high-level feature fusion (i∈{3,4,5}) mainly processes the image features in the deep layer of the network, which is used to understand the overall structure and category relationship of the scene. Correspondingly, the specific steps of high-level feature fusion of the two images are as follows:

[0127] (2f) The features of the two encoders are fused to obtain the preliminary fusion feature f i :

[0128]

[0129] wherein i∈{3,4,5}.

[0130] (2g) The preliminary fusion feature f i is input into the dense block to enhance the expression ability of the feature and alleviate the problem of gradient vanishing:

[0131]

[0132] wherein Cat(·) represents concatenation along the channel dimension. D(·) represents four densely connected CBR modules, which are composed of convolution processing, normalization and ReLU activation function, wherein the convolution processing adopts the way of dilated convolution. The size of the two groups of Conv3 convolution kernels is 3×3, the first group of Conv3 convolution processing does not modify the number of feature map channels, and the second group of Conv3 convolution processing modifies the number of output feature map channels to 64 to reduce the parameter amount.

[0133] (2h) The dense semantic block is designed to further enhance the receptive field of the feature map and pass the global semantic information. Specifically, the dense semantic block draws on the architecture design of dense connections in the dense block, and the network can directly obtain the feature maps of all previous layers at each layer. In each layer, the convolution processing is replaced by the atrous convolution. The atrous convolution inserts zero values in the middle of the convolution kernel to achieve expansion, which increases the receptive field without increasing the parameter amount of the convolution kernel. Atrous convolution has a significant effect in the semantic segmentation task, which can capture more extensive context information while maintaining the resolution of the feature map. In addition, the expansion rates of the four groups of atrous convolution are set to 1, 3, 5, and 7, respectively:

[0134]

[0135] wherein the dense semantic block fusion feature is Conv1(·) modifies the input feature map to 16.

[0136] Conv3(·) modifies the output feature map channel number to 64. D'(·) represents four CBR modules connected densely.

[0137] (2i) The infrared image features, visible light image features, and dense semantic block fusion features are further fused through a skip connection method, and a dense semantic block is used again for feature enhancement to obtain the output of high-level feature fusion:

[0138]

[0139] wherein F i represents the output of high-level feature fusion, and Conv1(·) is used to adjust the channel number of the feature map to 64, keeping the channel number consistent.

[0140] Step 3. The multi-level fusion feature map after the multi-level feature fusion model fusion is used as the input of the semantic segmentation decoder and the edge detection decoder, and the semantic segmentation decoder and the edge detection decoder share the weight parameters, as shown in Figure 4 so that the semantic segmentation task can implicitly utilize the edge features learned by the edge detection decoder. According to the semantic information of the foreground target, the noise and interference of the background in the low-level fusion feature are suppressed, thereby realizing more accurate edge detection of the foreground target. The semantic segmentation decoder and the edge detection decoder both include decoder 1, decoder 2, and decoder 3. The semantic segmentation decoder decodes the fusion feature to generate a high-level semantic segmentation feature map, and the edge detection decoder decodes the fusion feature to generate an edge detection feature map. The specific operations are as follows:

[0141] Step 31, generating a high-level semantic segmentation feature map, the specific steps are as follows:

[0142] (3a) (Working principle of decoder) The decoder processes the input feature map to obtain the corresponding output

[0143]

[0144] wherein, and are the decoder output and input, respectively. Conv3 represents three cascaded convolution blocks with a 3x3 kernel size. F up represents an up-sampling operation.

[0145] (3b) The semantic segmentation decoder can generate high-level semantic segmentation feature map M s by iteratively decoding and aggregating three high-level fusion feature maps H = {F3, F4, F5}.

[0146]

[0147] wherein, De i represents the i-th decoder.

[0148] Step 32, generate an edge detection feature map, specifically:

[0149] (3c) By element-wise summation, integrate the edge detection decoder output feature map M e ', low-level feature fusion output F2 and F1. The integrated feature map is directly multiplied with the feature map M s output by the semantic segmentation encoder, to suppress the interference and noise that may exist in the low-level feature map through the semantic information in the high-level feature, to obtain the final edge detection feature map M e .

[0150]

[0151] Step 33, after channel processing of the edge detection feature map and the high-level semantic segmentation feature map, determine the edge detection result image and the semantic segmentation result image, specifically:

[0152] (3d) Use a convolution block with a 1x1 kernel size and up-sampling processing to modify the channel number and resolution of the edge detection feature map M e and the semantic segmentation feature map M s , and restore them to the spatial resolution size of the original image, to obtain the corresponding edge detection result image and semantic segmentation result image.

[0153] Step 4, combine the semantic segmentation result image with the visible light image in step 1 to obtain a visible light foreground image, and combine the semantic segmentation result image with the infrared image in step 1 to obtain an infrared foreground image, introduce a multi-level feature fusion and attention enhancement module, perform feature extraction on the infrared foreground image and the visible light foreground image, and then perform feature fusion, fuse the cross-modal information and add a spatial attention enhancement module for processing, including:

[0154] (4a) Extract the multi-level features of the infrared foreground image and the visible light foreground image using two ResNet-based encoders in step 2 and where i ∈ {1, 2, 3, 4, 5}. In the multi-level feature extraction process, asymmetric feature fusion is used to enhance the infrared features , and the weight activation matrix is calculated according to the multi-level features and .

[0155]

[0156] where Conv1(·) represents a 1x1 convolution kernel size convolution process, and is output as a two-channel weight activation matrix W i ∈ R (H / i)×(W / i)×2 .

[0157] (4b) The weight activation matrix W i is taken as a learnable parameter, which is separated along the channel and multiplied element-wise with the infrared features and the visible light features, respectively.

[0158]

[0159] (4c) In the cross-modal multi-level feature fusion part, the enhanced is taken as input, and convolution, normalization and ReLU activation function processing are performed respectively.

[0160]

[0161] where BN(·) represents batch normalization processing. δ(·) represents the ReLU activation function. is the processed intermediate feature.

[0162] (4d) According to formula (16), the size of the intermediate feature in different levels is adjusted by upsampling processing, and is uniformly adjusted to Therefore, the width and height of three groups of features are expanded to twice, four times and eight times of the original, respectively. Then, through channel dimension splicing, the four-level features are fused. The specific process is as follows:

[0163]

[0164] wherein, Up i denotes that the feature map is enlarged i times.

[0165] (4e) The spatial attention enhancement part enhances the feature map through a learnable weight activation matrix, so that the network can adjust the weight of different spatial positions. The specific expression is as follows:

[0166]

[0167] wherein, F f denotes the fusion feature map.

[0168] (4f) Three groups of cascaded decoders are used in the feature reconstruction stage to restore the size of the feature map. The input feature of the first level decoder is up-sampled to ensure the consistency of the resolution of the feature map in the subsequent decoding process. The specific expression is as follows:

[0169]

[0170]

[0171]

[0172] wherein, denote the decoding feature maps generated by the three decoders respectively, is also the final output feature map.

[0173] Step 5. The edge information is strengthened and the accuracy and clarity of the detection result are improved through the skip connection of the visible light low-level feature. The binary cross-entropy loss function is used to supervise the semantic segmentation result and the semantic segmentation ground truth of the infrared image and the visible light image, capture the subtle features of the abnormal heating area, and solve the problem of fuzzy boundary of the abnormal heating area detection result, including:

[0174] Step 51, for the problem of fuzzy boundary of the abnormal heating area detection result, on the one hand, the visible light low-level feature is reused through the skip connection to strengthen the edge information and improve the accuracy and clarity of the detection result, and supervised learning is performed through the ground truth result.

[0175] (5a) The binary cross-entropy loss function is used to supervise the semantic segmentation result S and the semantic segmentation ground truth GT. The specific expression is as follows:

[0176]

[0177] wherein, H and W are the height and width of the feature map respectively, and HxW represents the total number of pixels of the feature map.

[0178] (5b) A residual information loss function based on positive and negative samples is proposed to optimize the edge contour. Specifically, the mean square error (MSE) and the structural similarity measure (SSIM) are used to supervise the detection result S and the sample residual R. Specifically, as follows:

[0179]

[0180]

[0181] wherein μ S and μ R represent the mean values of two images, i.e. the average pixel intensity of the image. σ S and σ R represent the standard deviations of two images, representing the contrast of the image. σ SR represents the covariance between the two images, reflecting the structural similarity of the image. C is a constant with a small value, used to maintain the stability of the calculation and avoid the case of zero denominator. Specifically, as follows:

[0182] (5c) The residual information loss function L rs is defined

[0183] L rs (S,R)=αL mse (S,R)+(1-α)L ssim (S,R) formula (24)

[0184] wherein α is set to 0.8, increasing the loss L mse weight of pixel information.

[0185] (5d) The final network supervision loss function is composed of the binary cross entropy loss function and the residual information loss function, as follows:

[0186] L SOD =L bce +L rs formula (25)

[0187] In addition, the application embodiment also provides an electric power equipment segmentation and abnormal heating detection device based on heterogeneous image feature fusion, comprising a camera, a processor and a machine readable storage medium;

[0188] The camera is used to collect the images of the appearance of the power transformation equipment and the surrounding environment, and send the images to the processor.

[0189] The machine readable storage medium is used to store machine executable instructions that can be executed by the processor.

[0190] A processor, configured to be enabled by machine-executable instructions to implement the above-described method for power equipment segmentation and abnormal heating detection based on heterogeneous image feature fusion.

[0191] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0192] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart... Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0193] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0194] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0195] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A power equipment segmentation and abnormal heating detection method based on heterogeneous image feature fusion, characterized in that, The method comprises the following steps: Step (1) two registered images of power equipment in different modalities, i.e. infrared images and visible light images, are obtained by shooting at the same time from the same direction; Step (2) the two images are input into a multi-level feature extraction network model and a multi-level feature fusion model, the multi-level feature extraction network model extracts low-level features and high-level features of the two images in multiple levels, and the multi-level feature fusion model fuses the features of the two images in the corresponding levels and outputs multi-level fusion feature maps; Step (3) the multi-level fusion feature maps fused by the multi-level feature fusion model are used as inputs of a semantic segmentation decoder and an edge detection decoder, the semantic segmentation decoder and the edge detection decoder share weight parameters, the semantic information in the high-level features is used to suppress the interference and noise existing in the low-level feature maps, and an edge detection feature map and a high-level semantic segmentation feature map are obtained, and the edge detection feature map and the high-level semantic segmentation feature map are processed in channels to determine an edge detection result image and a semantic segmentation result image; Step (4) the semantic segmentation result image is combined with the visible light image in step 1 to obtain a visible light foreground image, and the semantic segmentation result image is combined with the infrared image in step 1 to obtain an infrared foreground image, a multi-level feature fusion and attention enhancement module is introduced, feature extraction is performed on the infrared foreground image and the visible light foreground image, and then feature fusion is performed; In step (4), a segmentation mask method is used to combine the semantic segmentation result image with the visible light image in step 1 to obtain a visible light foreground image, and combine the semantic segmentation result image with the infrared image in step 1 to obtain an infrared foreground image; the specific operation of the multi-level feature fusion and attention enhancement module is as follows: Step 4.1, the multi-level feature extraction network model in step 2 is used to extract the multi-level features of the infrared foreground image and the visible light foreground image respectively and wherein and the asymmetric feature fusion is used to enhance the infrared features , and the weight activation matrix is calculated according to the multi-level features and ​ Equation (12); wherein, represents a convolution process of a 1x1 kernel size, and the output is a two-channel weight activation matrix ; Step 4.2, Weight activation matrix As a learnable parameter, it is element-wise multiplied with the infrared feature and the visible light feature respectively after being separated along the channel: Equation (13); Step 4.3, in the cross-modal multi-level feature fusion part, the enhanced As input, respectively, convolution, normalization and ReLU activation function processing: Equation (14); wherein, denotes a batch normalization process, denotes a ReLU activation function, is the processed intermediate feature; Step 4.4, adjust the size of the intermediate features in different levels by up-sampling processing, uniformly adjust to , respectively, expand the width and height of the three groups of features to 2 times, 4 times and 8 times of the original, and then fuse the four-level features by concatenating along the channel dimension, as shown below: times of the original, and then fuse the four-level features by concatenating along the channel dimension, as shown below: Equation (15); wherein, , , respectively represent upsampling the feature map by 2, 4 and 8 times, respectively. Step 4.5, the spatial attention enhancement part enhances the feature map through a learnable weight activation matrix, and the specific operation is as follows: Equation (16); wherein, denotes a fused feature map; Step 4.6, three groups of cascaded decoders are used to restore the size of the feature map in the feature reconstruction stage, wherein the input feature of the first level decoder is subjected to up-sampling processing to ensure the consistency of the resolution of the feature map in the subsequent decoding process, and the specific operation is as follows: Equation (17); (Formula 18); (Formula 19); wherein, respectively represent three decoded feature maps generated by three decoders, is also the last output feature map; Step (5) a binary cross-entropy loss function is used to supervise the semantic segmentation result of the infrared image and the visible light image and the semantic segmentation true value, capture the subtle features of the abnormal heating area, and finally determine the area of abnormal heating; (5.1) Supervise the semantic segmentation result with binary cross-entropy loss function and semantic segmentation ground truth GT, as shown below: and semantic segmentation ground truth GT, as shown below: Equation (20); wherein, , denote the segmentation result at training time, the ground truth of the segmentation result, i takes values from the 1st pixel to the i-th pixel, and are the height and width of the feature map, respectively, denotes the total number of pixels of the feature map; (5.2) Based on the positive and negative sample residual information loss function, the edge contour is optimized, and the mean square error MSE and structural similarity measurement SSIM are used to detect the results and sample residual Supervision is carried out, as shown below: Equation (21); Equation (22); wherein, represents the sample residual of the i-th pixel, i takes value from the 1st pixel to the Nth pixel, , and represents the mean of the two images of the heat detection result S and the true value R of the residual result during training, i.e. the image average pixel intensity, and represents the standard deviation of the two images, representing the contrast of the image, represents the covariance between the two images, reflecting the structural similarity of the image, , is a constant, used to maintain the stability of the calculation, to avoid the case that the denominator is zero; (5.3) Define the residual information loss function More attention is paid to the difference of pixel information in residual information supervised learning: Equation (23); wherein, is a weight coefficient, set to 0.8, increasing the loss of pixel information weight; (5.4) the final network supervision loss function is composed of a binary cross-entropy loss function and a residual information loss function, and the specific operation is as follows: Equation (24).

2. The power equipment segmentation and abnormal heating detection method based on heterogeneous image feature fusion according to claim 1, characterized in that: The multi-level feature extraction network model comprises two symmetrical encoders based on the ResNet architecture, which are used to extract multi-level features of the infrared image and the visible light image respectively, each symmetrical encoder based on the ResNet architecture comprises encoders 1-5, a dense block is added in each level of the encoder of the backbone network ResNet and the last fully connected layer is removed, which is used as a network encoder for feature extraction, and multi-level features of the infrared and visible light images are extracted through the convolution layers in multiple levels; The two images to be acquired are set and respectively into a trained multi-level feature extraction network model, and the image features extracted by the first level encoder corresponding level image features and , wherein is the multi-level feature of the visible light image, is the multi-level feature of the infrared image, , , is a low-level feature, , is a high-level feature.​​ 3. The power equipment segmentation and abnormal heating detection method based on heterogeneous image feature fusion according to claim 2, characterized in that: The multi-level feature fusion model comprises a low-level feature fusion model and a high-level feature fusion model; Low-level feature fusion model processes image features , , first processes multi-level features of the infrared image through a maximum pooling operation in the channel dimension , then uses a 5x5 convolution kernel to convolve the infrared image features to obtain processed infrared image features , uses the processed infrared image features to enhance the visible light image features to obtain a preliminary fusion result ; takes the preliminary fusion result as a weight matrix, calculates a weighted feature map of the visible light image features and the preliminary fusion feature map by element-by-element multiplication ; uses the processed infrared image features to enhance the weighted fusion feature map , generates an enhanced weighted feature map , and finally uses a channel attention mechanism to enhance and fuse the enhanced weighted feature map to output a fusion feature map ; The advanced feature fusion model for image features , The image features from encoders 3, 4, and 5 are processed and fused respectively. , The initial fusion features of each encoder level are obtained and input into dense blocks to enhance the expressive power of the features and alleviate the gradient vanishing problem. Dense semantic blocks are designed to further enhance the receptive field of the feature maps and convey global semantic information. In each layer of the dense semantic block, convolution processing is replaced with dilated convolution, and infrared image features, visible light image features, and dense semantic block fusion features are further fused through skip connections. The dense semantic blocks are used again for feature enhancement to obtain the output of advanced feature fusion. .

4. The power equipment segmentation and abnormal heating detection method based on heterogeneous image feature fusion according to claim 3, characterized in that: The low-level feature fusion model specifically operates as follows: (2.1) After the max-pooling and convolution operations, the processed infrared image features are obtained: Formula (1); wherein, is the infrared image feature processed by max-pooling and 5x5 convolution, which can be used as auxiliary guidance information to enhance the visible light image feature, ; (2.2) The pixels of the feature map are converted element by element using the Sigmoid function to obtain the preliminary fusion results of the visible light image features and the infrared image features: Formula (2); wherein, denotes element-wise summation, denotes element-wise conversion of pixels of the feature map using a Sigmoid function, denotes a preliminary fusion result of the visible light image features and the infrared image features, ; (2.3) to As the weight matrix, the weighted feature map of the visible light image feature and the preliminary fusion feature map is calculated by element-by-element multiplication : Formula (3); wherein, represents element-wise multiplication on the feature map, represents weighted fusion of the feature map, representing the correlation between the visible light image and the infrared image, ; (2.4) using the processed infrared image features to enhance the weighted fused feature map to generate an enhanced weighted feature map : Formula (4); (2.5) adopting a channel attention mechanism on the weighted feature map enhanced, outputting a fused feature map , : Equation (5); wherein, is a multi-layer perceptron.

5. The power equipment segmentation and abnormal heating detection method based on heterogeneous image feature fusion according to claim 3, characterized in that: The specific operation of the high-level feature fusion model is as follows: (2.6) fusing features of two branches of the two symmetric encoders based on ResNet architecture to obtain preliminary fused features , : Equation (6); (2.7) The preliminary fusion features are input into the dense block to enhance the expression ability of the features and alleviate the problem of gradient disappearance: Equation (7); wherein, denotes concatenation along the channel dimension, denotes four densely connected CBR modules, which are composed of convolution processing, normalization and ReLU activation function, wherein the convolution processing adopts the way of dilated convolution, two groups of convolution kernels are 3x3 in size, the first group of convolution processing does not modify the number of feature map channels, the second group of convolution processing modifies the number of output feature map channels to 64; (2.8) The specific operation of the dense semantic block is as follows: Equation (8); wherein the dense semantic block fusion feature is , The number of channels is modified to 16, The number of output feature map channels is modified to 64, indicates four densely connected CBR modules, and the expansion rates of the four groups of cavity convolutions are set to 1, 3, 5 and 7 respectively; (2.9) The infrared image features, the visible light image features and the dense semantic block fusion features are further fused through the skip connection mode to obtain the output of the high-level feature fusion: Equation (9); wherein , denotes the output of the high-level feature fusion.

6. The power equipment segmentation and abnormal heating detection method based on heterogeneous image feature fusion according to claim 3, characterized in that: In step (3), the semantic segmentation decoder and the edge detection decoder each include a decoder 1, a decoder 2 and a decoder 3, the semantic segmentation decoder decodes the fusion features to generate a high-level semantic segmentation feature map, and the edge detection decoder decodes the fusion features to generate an edge detection feature map, and the specific operation is as follows: Step 3.1, generating a high-level semantic segmentation feature map, and the specific steps are as follows: The semantic segmentation decoder aggregates three high-level fused feature maps by iterative decoding , to generate a high-level semantic segmentation feature map : Formula (10); wherein , , represents 3 decoder operations; Step 3.2, generating an edge detection feature map, and the specific operation is as follows: The edge detection decoder output feature map is integrated by element-wise summation , low-level feature fusion output and The integrated feature map is directly multiplied with the feature map output by the semantic segmentation encoder By the semantic information in the high-level feature, the interference and noise possibly existing in the low-level feature map are suppressed, and the final edge detection feature map is obtained : Equation (11); wherein denotes an up-sampling operation; Step 3.3, after the channel processing of the edge detection feature map and the high-level semantic segmentation feature map, determining an edge detection result image and a semantic segmentation result image, and the specific operation is as follows: The edge detection feature map is modified using a convolution block with a 1x1 kernel size and an up-sampling process and the channel number and resolution of the semantic segmentation feature map are restored to the spatial resolution size of the original image to obtain a corresponding edge detection result image and a semantic segmentation result image. ​ 7. A power equipment segmentation and abnormal heating detection device based on heterogeneous image feature fusion, characterized in that, The camera, the processor and the machine readable storage medium are included. The camera is configured to collect images of the appearance of the power equipment and the surrounding environment and send the images to the processor. The machine readable storage medium is configured to store machine executable instructions that can be executed by the processor. The processor is configured to be prompted by the machine executable instructions to implement the power equipment segmentation and abnormal heating detection method based on the fusion of heterogeneous image features according to any one of claims 1 to 6.

Images