A power equipment state monitoring method based on RGB-T interactive feature fusion

Abstract

The application discloses a power equipment state monitoring method based on RGB-T interactive feature fusion, comprising the following steps: S1, data acquisition and preprocessing; S2, constructing a power equipment state monitoring network model; S3, model training, comprising the following steps: sending a training image into the power equipment state monitoring network model, applying a side output module to each layer of the decoder, and amplifying the output image of the side output module to the same resolution as the original image through an up-sampling function; calculating the loss by using a binary cross-entropy loss function, and optimizing the parameters and weights in the network by back-propagation loss; stopping when the training is performed to a preset round or ending the model training when the validation set accuracy no longer decreases in the last 10 rounds in the training process; S4, model application and evaluation, deploying the trained model on a host for monitoring power equipment, collecting a visible light image and an infrared thermal imaging image of the equipment to be monitored by a handheld terminal, and running the model to obtain a prediction result.

Description

Technical Field

[0001] This application belongs to the field of power equipment condition monitoring technology, and in particular relates to a power equipment condition monitoring method based on RGB-T interactive feature fusion. Background Technology

[0002] Condition monitoring of power equipment is a crucial link in the safe, economical, and stable operation of power systems. Traditional methods include manual inspections and periodic preventive maintenance and testing, which play an important role in ensuring the safe operation of power equipment. However, with the development of sensing and computer technologies, power equipment condition monitoring methods are evolving towards automation and intelligence.

[0003] Visible light images provide detailed morphological and color information of equipment under visible light, while infrared images reveal the temperature distribution on the equipment surface. Combining these two types of images allows for more comprehensive monitoring of the status of electrical equipment. Infrared thermal imaging technology can detect thermal defects in equipment during operation, which are often early signals of potential faults. By analyzing the temperature difference on the equipment surface, overheated areas can be identified and located in a timely manner, thereby preventing faults. Furthermore, infrared images are not limited by lighting conditions and can operate at night or in inclement weather, providing a more flexible monitoring method. Simultaneously, the introduction of visible light images allows the system to better identify and differentiate equipment and its surrounding environment, which is crucial for improving the accuracy of fault detection and reducing false alarms. This multimodal image feature fusion method not only improves the reliability of detection but also provides richer information for fault diagnosis and subsequent maintenance.

[0004] However, existing methods have not fully utilized the complementarity of visible light and infrared thermal imaging modalities when handling information interaction between these two modalities, especially in the interaction between different feature levels, which limits further improvement in model performance. Specifically, during the downsampling process of convolutional neural networks, while semantic information is enhanced, the details contained in the feature maps are lost, meaning that the information contained in each level of features has its own characteristics. However, many existing models use the same fusion strategy at different levels, which is not conducive to fully utilizing the feature advantages of each level, nor to information transfer between multiple levels. In addition, the images of power equipment acquired in practical applications are often affected by noise interference and insufficient contrast, and the robustness of the models needs to be improved. Therefore, in order to improve the accuracy and robustness of RGB-T semantic segmentation in power equipment defect detection, existing methods need to be improved, especially in terms of multimodal feature fusion, information interaction, and interference suppression. Summary of the Invention

[0005] The purpose of this application is to overcome the problems of the prior art by disclosing a power equipment condition monitoring method based on RGB-T interactive feature fusion, so as to solve the problem of insufficient information interaction between visible light images and infrared images, enhance the model's ability to suppress interference, and improve the model's accuracy and robustness.

[0006] The objective of this application is achieved through the following technical solution:

[0007] A power equipment condition monitoring method based on RGB-T interactive feature fusion, the power equipment condition monitoring method comprising:

[0008] S1: Data acquisition and preprocessing, completing the acquisition of visible light images and infrared thermal imaging images, and preprocessing the acquired data;

[0009] S2: Construct a power equipment condition monitoring network model.

[0010] The power equipment condition monitoring network model includes: two encoders, two attention mechanism modules (DEA), three interactive multimodal feature complementarity modules (IMFC), a multi-level global context fusion module (MGCF), and a decoder;

[0011] Two encoders are configured to extract five layers of features from visible light and infrared images, respectively. The features output from the first two layers of the encoders are input to two attention mechanism modules (DEA) to suppress interference information and enhance the model's ability to recognize target boundaries.

[0012] The features output from the last three layers of the encoder are respectively input into three interactive multimodal feature complementarity modules (IMFC) to obtain the complementary relationship between visible light and infrared features;

[0013] The multi-level global context fusion module MGCF is configured to receive the output features of two attention mechanism modules DEA and three interactive multimodal feature complementation modules IMFC, and fuse global context information from each level to enhance semantic features.

[0014] The decoder is configured to progressively upsample multiple features output by the multi-level global context fusion module MGCF and output a prediction structure;

[0015] S3: Model training, the model training process includes:

[0016] The training images are fed into the power equipment condition monitoring network model, and the side output modules are applied to each layer of the decoder. The output images of the side output modules are then upsampled to the same resolution as the original images using an upsampling function.

[0017] The loss is calculated using the binary cross-entropy loss function, and the parameters and weights in the network are optimized by backpropagation loss. The training stops when the preset number of rounds is reached or ends when the accuracy of the validation set no longer decreases within 10 consecutive rounds during the training process.

[0018] S4: Model application and evaluation. The trained model is deployed on the host computer for monitoring power equipment. The handheld device collects visible light images and infrared thermal images of the equipment to be monitored, and runs the model to perform inference to obtain prediction results.

[0019] According to a preferred embodiment, step S1, the image preprocessing process includes:

[0020] Image registration is performed between visible light images and infrared thermal images to align them spatially.

[0021] According to a preferred embodiment, in step S1, the image preprocessing process further includes: normalizing the images of the two modalities respectively.

[0022] According to a preferred embodiment, in step S1, the image preprocessing process further includes scaling the acquired image to a size of 640×480.

[0023] According to a preferred embodiment, in step S2, a pre-trained convolutional neural network MobileNet-V2 is used as the encoder.

[0024] According to a preferred embodiment, the attention mechanism module DEA receives the features of the first two encoded layers' output and processes them as follows:

[0025] The visible light features of the i-th level and infrared thermal imaging features After cascading along the channel direction, the joint features are obtained through depthwise separable convolution. Where i = 1, 2;

[0026] Visible light features of the i-th level and infrared thermal imaging features Global max pooling and 5×5 2D convolution are performed separately to extract global salient features, yielding the following results. and

[0027] Will Adding together yields the features

[0028] Features Features are obtained by cascading along the channel direction and then passing through the channel attention module.

[0029] feature The output features of the attention mechanism module DEA are obtained after depthwise separable convolution.

[0030] According to a preferred embodiment, the Feature Input Interactive Multimodal Feature Complementary Module (IMFC) receives the feature inputs from the last three layers of the encoder and performs the following processing:

[0031] The visible light features of the i-th level and infrared thermal imaging features After cascading along the channel direction, the resulting data undergoes depthwise separable convolution to obtain preliminary fused joint features. Where i = 3, 4, 5;

[0032] Visible light features of the i-th level and infrared thermal imaging features Perform 3×3 two-dimensional convolutions respectively to obtain the results. and Where i = 3, 4, 5;

[0033] Bundle As the query value for the multi-head attention module, The key and value values ​​of the multi-head attention module are combined using an 8-head attention module to obtain... Accordingly, and Obtained by combining 8 attention modules

[0034] By injecting complementary information from visible and infrared modes into the initial fusion features of the two. In the middle, the obtained and The output features of the IMFC module are obtained after concatenation and depthwise separable convolution.

[0035] According to a preferred embodiment, the multi-level global context fusion module MGCF receives two output features from the DEA module. and the three output features of the IMFC module And the following operations are used to fuse multi-level global contextual information to enhance semantic features, including:

[0036] Features and Sampling respectively Size, features Upsampling Dimensions;

[0037] Will After concatenation along the channel direction, depthwise separable convolution is performed to obtain F. DC ;

[0038] Will F DC Cascaded along the channel direction to obtain F DI ;

[0039] F DI Depth-separable convolution is performed to obtain Another F DI Perform another depthwise separable convolution and transpose to obtain

[0040] Will and Perform matrix multiplication to obtain F DM ;

[0041] For F DM After applying the ReLU activation function, a 1x1 convolution is performed to obtain F. R ;

[0042] For F DM Applying the Sigmoid activation function yields F S ;

[0043] F R With F S After addition, perform a 1x1 convolution to obtain F. RSC ;

[0044] F RSC With F DC Performing the Hadamard product (element-wise multiplication) yields the output feature F of the MGCF module. MGCF .

[0045] According to a preferred embodiment, in step S3, a side output module is applied to each layer of the decoder, defined as:

[0046]

[0047] Where δ(·) represents the ReLU activation function, and BN(·) represents batch normalization. De represents a 1×1 convolution with c output channels. i This represents the output of each layer of the decoder, i = 1, 2, 3, 4;

[0048] The loss is calculated using the binary cross-entropy loss function, defined as:

[0049]

[0050] Where L i O i The corresponding binary cross-entropy loss.

[0051] According to a preferred embodiment, step S4 further includes: comparing the prediction results with the actual values ​​given by professional staff, and calculating various evaluation indicators, including accuracy and overlap.

[0052] The aforementioned main solution and its various further alternative solutions can be freely combined to form multiple solutions, all of which are solutions that can be adopted and are claimed in this application. Those skilled in the art, after understanding the solution of this application, will realize that there are many combinations based on the prior art and common general knowledge, all of which are technical solutions to be protected in this application, and will not be exhaustively listed here.

[0053] The beneficial effects of this application are:

[0054] 1. Multimodal information complementarity mechanism: By constructing an interactive multimodal feature complementarity module, the complementary relationship between multimodal features was effectively explored, especially the long-term dependency relationship in the spatial and channel dimensions.

[0055] 2. Multi-level global context information fusion mechanism: By constructing a multi-level global context fusion module, global context information can be captured and global guidance can be provided for feature interaction, which helps to improve feature representation and enhance segmentation performance.

[0056] 3. Interference suppression mechanism: By introducing an attention mechanism module to enhance details, boundary information and binary masks can be better utilized, thereby improving the detail accuracy of the segmentation results. Attached Figure Description

[0057] Figure 1 This is a schematic diagram of the overall structure of the power equipment condition monitoring network of this application.

[0058] Figure 2 This is a schematic diagram of the DEA module in the power equipment condition monitoring network of this application.

[0059] Figure 3 This is a schematic diagram of the IMFC module in the power equipment condition monitoring network of this application.

[0060] Figure 4 This is a schematic diagram of the MGCF module in the power equipment condition monitoring network of this application. Detailed Implementation

[0061] The following specific examples illustrate the implementation of this application. Those skilled in the art can easily understand other advantages and effects of this application from the content disclosed in this specification. This application can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of this application. It should be noted that, unless otherwise specified, the following embodiments and features in the embodiments can be combined with each other.

[0062] Example

[0063] refer to Figures 1 to 4 As shown, this application discloses a power equipment condition monitoring method based on RGB-T interactive feature fusion, the power equipment condition monitoring method including the following steps.

[0064] Step S1: Data Acquisition and Preprocessing. This step involves acquiring visible light and infrared thermal images and preprocessing the acquired data. The specific preprocessing steps are as follows:

[0065] (1) Perform image registration between the visible light image and the infrared thermal imaging image to make them spatially aligned;

[0066] (2) Normalize the images of the two modalities respectively;

[0067] (3) Scale the image to a size of 640×480.

[0068] Step S2: Construct a power equipment condition monitoring network model.

[0069] The power equipment condition monitoring network model includes: two encoders, two attention mechanism modules (DEA), three interactive multimodal feature complementation modules (IMFC), a multi-level global context fusion module (MGCF), and a decoder. Furthermore, a pre-trained convolutional neural network, MobileNet-V2, is used as the encoder.

[0070] Preferably, the two encoders are configured to extract five layers of multi-scale features from the visible light image and the infrared image, respectively, and the features output from the first two layers of the encoder are respectively input to two attention mechanism modules (DEA) to suppress interference information and enhance the model's ability to recognize target boundaries.

[0071] Specifically: the attention mechanism module DEA receives the features of the first two encoded layers and processes them as follows:

[0072] (1) The visible light features of the i-th level and infrared thermal imaging features After cascading along the channel direction, the joint features are obtained through depthwise separable convolution. Where i = 1, 2;

[0073] (2) Visible light characteristics of the i-th level and infrared thermal imaging features Global max pooling and 5×5 2D convolution are performed separately to extract global salient features, yielding the following results. and

[0074] (3) Adding together yields the features

[0075] (4) Features Features are obtained by cascading along the channel direction and then passing through the channel attention module.

[0076] (5) Features The output features of the attention mechanism module DEA are obtained after depthwise separable convolution.

[0077] Preferably, three interactive multimodal feature complementarity modules (IMFCs) receive features from the last three layers of the encoder output and are used to obtain the complementary relationship between visible light and infrared features. Specifically:

[0078] (1) The visible light features of the i-th level and infrared thermal imaging features After cascading along the channel direction, the resulting data undergoes depthwise separable convolution to obtain preliminary fused joint features. Where i = 3, 4, 5;

[0079] (2) Visible light characteristics of the i-th level and infrared thermal imaging features Perform 3×3 two-dimensional convolutions respectively to obtain the results. and Where i = 3, 4, 5;

[0080] (3) Put As the query value for the multi-head attention module, The key and value values ​​of the multi-head attention module are combined using an 8-head attention module to obtain...

[0081] (4) Similar to (3), and Obtained by combining 8 attention modules

[0082] (5) Inject complementary information of visible and infrared modes into the initial fusion features of the two modes through (3) and (4). In the middle, the obtained and The output features of the IMFC module are obtained after concatenation and depthwise separable convolution. i = 3, 4, 5.

[0083] Preferably, the multi-level global context fusion module MGCF is configured to receive the output features of two attention mechanism modules DEA and three interactive multimodal feature complementation modules IMFC, and fuse global context information from each level to enhance semantic features.

[0084] Specifically, the multi-level global context fusion module MGCF receives two output features from the DEA module. and the three output features of the IMFC module And the following operations are used to fuse multi-level global contextual information to enhance semantic features, including:

[0085] (1) Features and Sampling respectively Size, features Upsampling Dimensions;

[0086] (2) After concatenation along the channel direction, depthwise separable convolution is performed to obtain F. DC ;

[0087] (3) F DC Cascaded along the channel direction to obtain F DI ;

[0088] (4)F DI Depth-separable convolution is performed to obtain Another F DI Perform another depthwise separable convolution and transpose to obtain

[0089]

[0090] (5) and Perform matrix multiplication to obtain F DM ;

[0091] (6) For F DM After applying the ReLU activation function, a 1x1 convolution is performed to obtain F. R ;

[0092] (7) For F DMApplying the Sigmoid activation function yields F S ;

[0093] (8) F R With F S After addition, perform a 1x1 convolution to obtain F. RSC ;

[0094] (9) F RSC With F DC Performing the Hadamard product (element-wise multiplication) yields the output feature F of the MGCF module. MGCF .

[0095] Preferably, the decoder is configured to progressively upsample multiple features output by the multi-level global context fusion module MGCF and output a prediction structure. Specifically, the side output of the last layer of the decoder is binarized to obtain the prediction map.

[0096] Step S3: Model training, the model training process includes:

[0097] (1) The training images are fed into the power equipment condition monitoring network model, and the encoder, deep feature extraction module DEA, information multi-stream fusion module IMFC, multi-scale feature fusion module MGCF and decoder are used.

[0098] (2) Apply side output modules to each layer of the decoder, as defined below:

[0099]

[0100] Where δ(·) represents the ReLU activation function, and BN(·) represents batch normalization. De represents a 1×1 convolution with c output channels. i This represents the output of each layer of the decoder, i = 1, 2, 3, 4;

[0101] (3) Upsample the output image of the side output module to the same resolution as the original image using an upsampling function;

[0102] (4) The loss is calculated using the binary cross-entropy loss function, defined as follows:

[0103]

[0104] Where L i O i The corresponding binary cross-entropy loss;

[0105] (5) The parameters and weights in the network are optimized using backpropagation loss. The initial learning rate of the Adam optimizer is set to 5 × 10⁻⁶. -4 .

[0106] (6) Stop training when it reaches 200 rounds or when the accuracy of the validation set no longer decreases within 10 consecutive rounds during training.

[0107] Step S4: Model application and evaluation, including:

[0108] (1) The trained model is deployed on the host of the power equipment monitoring. The handheld device collects visible light images and infrared thermal images of the equipment to be monitored, and runs the model to perform inference to obtain prediction results.

[0109] (2) Compare the prediction results with the actual values ​​given by professional staff, and calculate various evaluation indicators, including Acc (accuracy) and IoU (overlap).

[0110] This application effectively explores the complementary relationships between multimodal features, particularly long-term dependencies in the spatial and channel dimensions, by constructing an interactive multimodal feature complementarity module. By constructing a hierarchical global context fusion module, global contextual information can be captured, providing global guidance for feature interactions, which helps improve feature representation and enhance segmentation performance. Introducing a detail-enhancing attention mechanism module allows for better utilization of boundary information and binary masks, improving the detail accuracy of the segmentation results.

[0111] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A power equipment state monitoring method based on RGB-T interactive feature fusion, characterized in that, The power equipment state monitoring method comprises: S1: data acquisition and preprocessing, completing the acquisition of visible light images and infrared thermal imaging images, and preprocessing the collected data; S2: constructing a power equipment state monitoring network model, The power equipment state monitoring network model comprises: two encoders, two attention mechanism modules DEA, three interactive multi-modal feature complementary modules IMFC, a multi-level global context fusion module MGCF and a decoder; The two encoders are configured to extract five layers of features from the visible light images and the infrared images respectively, and the features output by the first two layers of the encoders are input into the two attention mechanism modules DEA respectively, for suppressing interference information and enhancing the recognition ability of the model to the target boundary; The features output by the last three layers of the encoders are input into the three interactive multi-modal feature complementary modules IMFC respectively, for obtaining the complementary relationship between the visible light and infrared features; The multi-level global context fusion module MGCF is configured to receive the features output by the two attention mechanism modules DEA and the three interactive multi-modal feature complementary modules IMFC, fuse the global context information of each level to enhance the semantic features; The decoder is configured to gradually up-sample the multiple features output by the multi-level global context fusion module MGCF, and output a predicted structure; S3: model training, the model training process comprises: sending the training images into the power equipment state monitoring network model, applying a side output module to each layer of the decoder, and enlarging the output images of the side output module to the same resolution as the original images through an up-sampling function; using a binary cross-entropy loss function to calculate the loss, and optimizing the parameters and weights in the network through back propagation of the loss, stopping when the training reaches a preset round or ending the model training when the validation set accuracy no longer decreases in 10 consecutive rounds during the training process; S4: model application and evaluation, deploying the trained model on a host for monitoring power equipment, collecting visible light images and infrared thermal imaging images of the equipment to be monitored by a handheld device, running the model to obtain a prediction result.

2. The power equipment condition monitoring method according to claim 1, wherein, In step S1, the preprocessing of the images comprises: aligning the visible light images and the infrared thermal imaging images in space by image registration.

3. The power equipment state monitoring method according to claim 1, characterized in that, In step S1, the preprocessing of the images comprises: The pre-processing process further comprises normalizing the images of the two modalities separately.

4. The power equipment condition monitoring method according to claim 1, wherein, In step S1, the preprocessing of the images further comprises: scaling the collected images to a size of 640x480.

5. The power equipment condition monitoring method according to claim 1, wherein, In step S2, a pre-trained convolutional neural network MobileNet-V2 is used as the encoder.

6. The power equipment condition monitoring method of claim 1, wherein, The attention mechanism module DEA receives the features output by the first two layers of the encoder and processes them as follows: Visible light features of the i-th level and infrared thermal imaging features After cascading along the channel direction, the joint features are obtained after depth separable convolution processing where i = 1, 2; visible light features of the i-th level and infrared thermal imaging features global max pooling and 5x5 two-dimensional convolution are performed respectively to extract global prominent features, and and will be added to the features ​ features features Features Output features of the attention mechanism module DEA after a deep separable convolution 7. The power equipment condition monitoring method of claim 6, wherein, The interactive multi-modal feature complementary module IMFC receives the features input by the last three layers of the encoder and processes them as follows: Visible light features of the i-th level and infrared thermal imaging features After cascading along the channel direction and deep separable convolution processing, a preliminary fused joint feature is obtained where i = 3, 4, 5; visible light features of the i-th level and infrared thermal imaging features are respectively performed 3x3 two-dimensional convolutions to obtain and where i = 3, 4, 5; put query values of the multi-head attention module, key values and value values of the multi-head attention module are combined with an 8-head attention module to obtain Correspondingly, the and combined with an 8-head attention module to obtain By injecting complementary information of visible light and infrared modalities into the preliminary fusion feature of both The obtained and After cascading and deep separable convolution processing, the output feature of the IMFC module is obtained 8. The power equipment condition monitoring method of claim 7, wherein, The multi-level global context fusion module MGCF receives two output features of the DEA module and three output features of the IMFC module and fuses the multi-level global context information to enhance the semantic features by: The features and are down-sampled to the size of respectively, the features are up-sampled to the size of ; Will After cascading in the channel direction, a depth separable convolution is performed to obtain F DC ; Will F DC Cascade along the channel direction, get F DI ; F DI performing a depthwise separable convolution to obtain another F DI performing a depthwise separable convolution and transposing to obtain DM ;​​​ F DM ReLU activation function is applied after 1 * 1 convolution, and F R ; F DM Applying a sigmoid activation function gives F S ; F R is added to F S and a 1*1 convolution is performed to obtain F RSC ; F RSC with F DC performing Hadamard product to get the output feature F MGCF of the MGCF module.

9. The power equipment condition monitoring method according to claim 7, wherein, In step S3, a side output module is applied to each layer of the decoder, which is defined as: where δ(·) denotes the ReLU activation function, BN(·) denotes batch normalization, denotes a 1x1 convolution with c output channels, De i denotes the output of each layer of the decoder, i = 1, 2, 3, 4; The binary cross-entropy loss function is used to calculate the loss, which is defined as: where L i represents O i The corresponding binary cross-entropy loss.

10. The power equipment condition monitoring method of claim 1, wherein, Step S4 further comprises: (2) comparing the prediction result with the true value given by professional staff, calculating evaluation indexes, and the evaluation indexes include accuracy and overlap degree.

Images