Power equipment state monitoring method based on RGB-T interactive feature fusion

Abstract

The invention discloses a power equipment state monitoring method based on RGB-T interactive feature fusion. The method comprises the steps of S1, data acquisition and preprocessing; s2, constructing a power equipment state monitoring network model; s3, model training: sending a training image into a power equipment state monitoring network model, applying a side output module to each layer of a decoder, and amplifying an output image of the side output module to have the same resolution as an original image through an up-sampling function; performing loss calculation by using a binary cross entropy loss function, optimizing parameters and weights in the network through back propagation loss, and stopping training when the training is performed to a preset round or ending model training when the accuracy of a verification set is not reduced any more within continuous 10 rounds in the training process; and S4, model application and evaluation: deploying the trained model on a host for monitoring power equipment, collecting a visible light image and an infrared thermal imaging image of to-be-monitored equipment by handheld terminal equipment, and operating the model for reasoning to obtain a prediction result.

Description

Technical Field

[0001] The present application belongs to the technical field of power equipment status detection, and in particular, relates to a power equipment status monitoring method based on RGB-T interactive feature fusion. Background Art

[0002] Power equipment status detection is a key link in the safe, economical and stable operation of the power system. Traditional methods include manual inspections and regular preventive maintenance and testing, which play an important role in ensuring the safe operation of power equipment. However, with the development of sensor technology and computer technology, power equipment status monitoring methods are moving towards automation and intelligence.

[0003] Visible light images provide detailed morphological and color information of the equipment under visible light, while infrared images can reveal the temperature distribution on the surface of the equipment. Combining these two images can more comprehensively monitor the status of power equipment. Infrared thermal imaging technology can detect thermal defects of equipment during operation, which are often early signs of potential equipment failures. By analyzing the temperature difference on the surface of the equipment, overheating areas can be discovered and located in time to prevent failures. In addition, infrared images are not limited by lighting conditions and can work at night or in bad weather, providing a more flexible monitoring method. At the same time, the introduction of visible light images enables the system to better identify and distinguish equipment and its surroundings, which is crucial to improving the accuracy of fault detection and reducing false alarms. This method of multimodal image feature fusion not only improves the reliability of detection, but also provides richer information for fault diagnosis and subsequent maintenance work.

[0004] However, existing methods have not been able to fully utilize the complementarity of visible light and infrared thermal imaging modalities when dealing with information interaction between the two modalities, especially in the interaction between different feature levels, which limits the further improvement of model performance. Specifically, in the process of downsampling of convolutional neural networks, as semantic information is enhanced, the details contained in the feature map are lost, that is, the information contained in the features of each level has its own characteristics. However, many existing models use the same fusion strategy at different levels, which is not conducive to fully utilizing the feature advantages of each level, nor is it conducive to information transmission between multiple levels. In addition, the power equipment images collected in practical applications are often affected by noise interference and insufficient contrast, and the model robustness needs to be improved. Therefore, in order to improve the accuracy and robustness of RGB-T semantic segmentation in power equipment defect detection, it is necessary to improve the existing methods, especially in multimodal feature fusion, information interaction and interference suppression. Summary of the invention

[0005] The purpose of this application is to overcome the problems of the prior art and disclose a method for monitoring the state of electric power equipment based on RGB-T interactive feature fusion to solve the problem of insufficient information interaction between visible light images and infrared images, enhance the interference suppression ability of the model, and improve the accuracy and robustness of the model.

[0006] The purpose of this application is achieved through the following technical solutions:

[0007] A method for monitoring the state of electric equipment based on RGB-T interactive feature fusion, the method comprising:

[0008] S1: Data acquisition and preprocessing, complete the acquisition of visible light images and infrared thermal imaging images, and preprocess the acquired data;

[0009] S2: Construct a network model for power equipment status monitoring.

[0010] The power equipment status monitoring network model includes: two encoders, two attention mechanism modules DEA, three interactive multimodal feature complementary modules IMFC, a multi-level global context fusion module MGCF and a decoder;

[0011] The two encoders are configured to extract five layers of features from visible light images and infrared images respectively, and the features output by the first two layers of the encoders are respectively input into two attention mechanism modules DEA to suppress interference information and enhance the model's ability to recognize target boundaries;

[0012] The features output by the last three layers of the encoder are respectively input into three interactive multimodal feature complementation modules IMFC to obtain the complementary relationship between visible light and infrared features;

[0013] The multi-level global context fusion module MGCF is configured to receive output features of two attention mechanism modules DEA and three interactive multimodal feature complementation modules IMFC, and fuse the global context information of each level to enhance the semantic features;

[0014] The decoder is configured to gradually upsample multiple features output by the multi-level global context fusion module MGCF and output a prediction structure;

[0015] S3: Model training, the model training process includes:

[0016] The training image is sent to the power equipment status monitoring network model, and the side output module is applied to each layer of the decoder. The output image of the side output module is enlarged to the same resolution as the original image through the upsampling function;

[0017] The loss is calculated using the binary cross entropy loss function, and the parameters and weights in the network are optimized by back-propagating the loss. The training stops when the preset round is reached or when the accuracy of the validation set does not decrease within 10 consecutive rounds during the training process;

[0018] S4: Model application and evaluation: deploy the trained model on the host computer for monitoring power equipment. The handheld device collects visible light images and infrared thermal imaging images of the equipment to be monitored, and runs the model for inference to obtain prediction results.

[0019] According to a preferred embodiment, in step S1, the image preprocessing process includes:

[0020] The visible light image and the infrared thermal imaging image are registered to align the visible light image and the infrared thermal imaging image in space.

[0021] According to a preferred embodiment, in step S1, the image preprocessing process further includes: normalizing the images of the two modalities respectively.

[0022] According to a preferred embodiment, in step S1, the image preprocessing process further includes: scaling the acquired image to a size of 640×480.

[0023] According to a preferred embodiment, in step S2, a pre-trained convolutional neural network MobileNet-V2 is used as an encoder.

[0024] According to a preferred embodiment, the attention mechanism module DEA receives the features of the output of the first two layers of encoding and performs the following processing:

[0025] The visible light feature of the i-th level and infrared thermal imaging features After cascading along the channel direction and undergoing depth-separable convolution processing, the joint feature is obtained Where i = 1, 2;

[0026] For the visible light characteristics of the i-th level and infrared thermal imaging features Global maximum pooling and 5×5 two-dimensional convolution are performed to extract global salient features, respectively. and

[0027] Will Add up the features

[0028] The characteristics After cascading along the channel direction, the features are obtained through the channel attention module

[0029] feature After the depth-separable convolution, the output features of the attention mechanism module DEA are obtained

[0030] According to a preferred embodiment, the feature input interactive multimodal feature complementation module IMFC receives the feature input of the last three layers of the encoder and performs the following processing:

[0031] The visible light feature of the i-th level and infrared thermal imaging features After cascading along the channel direction and undergoing depth-separable convolution processing, the preliminary fused joint features are obtained. Where i = 3, 4, 5;

[0032] For the visible light characteristics of the i-th level and infrared thermal imaging features Perform 3×3 two-dimensional convolution respectively, and get and Where i = 3, 4, 5;

[0033] Bundle As the query value of the multi-head attention module, As the key value and value of the multi-head attention module, we combine it with the 8-head attention module to get Accordingly, and Combined with 8 attention modules

[0034] By injecting complementary information of visible light and infrared modalities into the initial fusion features of the two In and After cascading, the output features of the IMFC module are obtained by depth-separable convolution.

[0035] According to a preferred embodiment, the multi-level global context fusion module MGCF receives two output features of the DEA module And the three output characteristics of the IMFC module The following operations are performed to fuse multi-level global context information to enhance semantic features, including:

[0036] The characteristics and Downsampled to Size, features Upsampling to Size;

[0037] Will After cascading along the channel direction, a depth-separable convolution is performed to obtain F DC ;

[0038] Will F DC Cascade along the channel direction to get F DI ;

[0039] F DI Perform depth-wise separable convolution to obtain Another pair of F DI Perform another depth-wise separable convolution and transpose to get

[0040] Will and Perform matrix multiplication to get F DM ;

[0041] F DM After applying the ReLU activation function, a 1*1 convolution is performed to obtain F R ;

[0042] F DM Apply the Sigmoid activation function to get F S ;

[0043] F R With F S After adding, perform 1*1 convolution to get F RSC ;

[0044] F RSC With F DC Perform Hadamard product (element multiplication) to obtain the output feature F of the MGCF module MGCF .

[0045] According to a preferred embodiment, in step S3, a side output module is applied to each layer of the decoder, which is defined as:

[0046]

[0047] Where δ(·) represents the ReLU activation function, BN(·) represents batch normalization, Denotes 1×1 convolution, the number of output channels is c, De i Represents the output of each layer of the decoder, i = 1, 2, 3, 4;

[0048] The loss is calculated using the binary cross entropy loss function, which is defined as:

[0049]

[0050] Where L i Indicates O i The corresponding binary cross entropy loss.

[0051] According to a preferred embodiment, step S4 further includes: comparing the prediction result with the real value given by the professional staff, and calculating various evaluation indicators, wherein the evaluation indicators include accuracy and overlap.

[0052] The aforementioned main scheme of the present application and its further options can be freely combined to form multiple schemes, all of which are schemes that can be adopted and claimed for protection in the present application. After understanding the scheme of the present application, those skilled in the art can understand that there are multiple combinations based on the prior art and common knowledge, all of which are technical schemes to be protected by the present application, and they are not exhaustively listed here.

[0053] Beneficial effects of this application:

[0054] 1. Multimodal information complementarity mechanism: By constructing an interactive multimodal feature complementarity module, the complementary relationship between multimodal features is effectively explored, especially the long-term dependencies in spatial and channel dimensions.

[0055] 2. Multi-level global context information fusion mechanism: By constructing a hierarchical global context fusion module, it is possible to capture global context information and provide global guidance for feature interaction, which helps to improve feature representation and enhance segmentation performance.

[0056] 3. Interference suppression mechanism: By introducing an attention mechanism module for enhancing details, it can better utilize boundary information and binary masks, thereby improving the detail accuracy of the segmentation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a schematic diagram of the overall structure of the power equipment status monitoring network of this application.

[0058] Figure 2 This is a schematic diagram of the structure of the DEA module in the power equipment status monitoring network of this application.

[0059] Figure 3 This is a schematic diagram of the structure of the IMFC module in the power equipment status monitoring network of this application.

[0060] Figure 4 This is a schematic diagram of the structure of the MGCF module in the power equipment status monitoring network of this application. DETAILED DESCRIPTION

[0061] The following describes the embodiments of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. The present application can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0062] Example

[0063] refer to Figures 1 to 4 As shown, the present application discloses a method for monitoring the state of electric equipment based on RGB-T interactive feature fusion, and the method for monitoring the state of electric equipment includes the following steps.

[0064] Step S1: Data collection and preprocessing, complete the collection of visible light images and infrared thermal imaging images, and preprocess the collected data. The specific operations for image preprocessing are as follows:

[0065] (1) Perform image registration on the visible light image and the infrared thermal imaging image so that they are spatially aligned;

[0066] (2) Normalize the images of the two modalities respectively;

[0067] (3) Scale the image to 640×480 size.

[0068] Step S2: Construct a power equipment status monitoring network model.

[0069] The power equipment status monitoring network model includes: two encoders, two attention mechanism modules DEA, three interactive multimodal feature complementary modules IMFC, a multi-level global context fusion module MGCF and a decoder. In addition, a pre-trained convolutional neural network MobileNet-V2 is used as the encoder.

[0070] Preferably, the two encoders are configured to extract five layers of multi-scale features from the visible light image and the infrared image respectively, and the features output by the first two layers of the encoders are respectively input into two attention mechanism modules DEA to suppress interference information and enhance the model's ability to recognize target boundaries.

[0071] Specifically: the attention mechanism module DEA receives the features of the output of the first two layers of encoding and performs the following processing:

[0072] (1) The i-th level visible light feature and infrared thermal imaging features After cascading along the channel direction and undergoing depth-separable convolution processing, the joint feature is obtained Where i = 1, 2;

[0073] (2) Visible light features for the i-th level and infrared thermal imaging features Global maximum pooling and 5×5 two-dimensional convolution are performed to extract global salient features, respectively. and

[0074] (3) Add up the features

[0075] (4) Features After cascading along the channel direction, the features are obtained through the channel attention module

[0076] (5) Features After the depth-separable convolution, the output features of the attention mechanism module DEA are obtained

[0077] Preferably, three interactive multimodal feature complementation modules IMFC receive features output by the last three layers of the encoder to obtain the complementary relationship between visible light and infrared features. Specifically:

[0078] (1) The i-th level visible light feature and infrared thermal imaging features After cascading along the channel direction and undergoing depth-separable convolution processing, the preliminary fused joint features are obtained. Where i = 3, 4, 5;

[0079] (2) Visible light features for the i-th level and infrared thermal imaging features Perform 3×3 two-dimensional convolution respectively, and get and Where i = 3, 4, 5;

[0080] (3) As the query value of the multi-head attention module, As the key value and value of the multi-head attention module, we combine it with the 8-head attention module to get

[0081] (4)Similar to (3), and Combined with 8 attention modules

[0082] (5) Through (3) and (4), the complementary information of visible light and infrared modalities is injected into the initial fusion features of the two. In and After cascading, the output features of the IMFC module are obtained by depth-separable convolution. i=3,4,5.

[0083] Preferably, the multi-level global context fusion module MGCF is configured to receive output features from two attention mechanism modules DEA and three interactive multimodal feature complementation modules IMFC, and fuse the global context information of each level to enhance the semantic features.

[0084] Specifically, the multi-level global context fusion module MGCF receives two output features of the DEA module: And the three output characteristics of the IMFC module The following operations are performed to fuse multi-level global context information to enhance semantic features, including:

[0085] (1) The features and Downsampled to Size, features Upsampling to Size;

[0086] (2) After cascading along the channel direction, a depth-separable convolution is performed to obtain F DC ;

[0087] (3) F DC Cascade along the channel direction to get F DI ;

[0088] (4)F DI Perform depth-wise separable convolution to obtain Another pair of F DI Perform another depth-wise separable convolution and transpose to get

[0089]

[0090] (5) and Perform matrix multiplication to get F DM ;

[0091] (6) For F DM After applying the ReLU activation function, a 1*1 convolution is performed to obtain F R ;

[0092] (7) For F DMApply the Sigmoid activation function to get F S ;

[0093] (8) F R With F S After adding, perform 1*1 convolution to get F RSC ;

[0094] (9) F RSC With F DC Perform Hadamard product (element multiplication) to obtain the output feature F of the MGCF module MGCF .

[0095] Preferably, the decoder is configured to gradually upsample the multiple features output by the multi-level global context fusion module MGCF and output a prediction structure. Specifically, the side output of the last layer of the decoder is binarized to obtain a prediction map.

[0096] Step S3: Model training, the model training process includes:

[0097] (1) The training image is sent to the power equipment condition monitoring network model through the encoder, deep feature extraction module DEA, information multi-stream fusion module IMFC, multi-scale feature fusion module MGCF and decoder;

[0098] (2) Apply a side output module to each layer of the decoder, which is defined as follows:

[0099]

[0100] Where δ(·) represents the ReLU activation function, BN(·) represents batch normalization, Denotes 1×1 convolution, the number of output channels is c, De i Represents the output of each layer of the decoder, i = 1, 2, 3, 4;

[0101] (3) The output image of the side output module is enlarged to the same resolution as the original image through an upsampling function;

[0102] (4) Use the binary cross entropy loss function to calculate the loss, which is defined as follows:

[0103]

[0104] Where L i Indicates O i The corresponding binary cross entropy loss;

[0105] (5) The parameters and weights in the network are optimized by back-propagation loss. The initial learning rate of the Adam optimizer is set to 5×10 -4 .

[0106] (6) The training is stopped when the 200th round is reached or when the accuracy of the validation set does not decrease for 10 consecutive rounds during the training process.

[0107] Step S4: Model application and evaluation, including:

[0108] (1) The trained model is deployed on the host computer that monitors the power equipment. The handheld device collects visible light images and infrared thermal imaging images of the equipment to be monitored, and runs the model for inference to obtain prediction results.

[0109] (2) Compare the prediction results with the true values ​​given by professional staff and calculate various evaluation indicators, including Acc (accuracy) and IoU (interpretation of overlap).

[0110] This application effectively explores the complementary relationship between multimodal features, especially the long-term dependencies in spatial and channel dimensions, by constructing an interactive multimodal feature complementation module. By constructing a hierarchical global context fusion module, it is possible to capture global context information and provide global guidance for feature interaction, which helps to improve feature representation and enhance segmentation performance. By introducing an attention mechanism module for enhancing details, it is possible to better utilize boundary information and binary masks, thereby improving the detail accuracy of the segmentation results.

[0111] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for monitoring the state of electric equipment based on RGB-T interactive feature fusion, characterized in that: The power equipment status monitoring method comprises: S1: Data acquisition and preprocessing, complete the acquisition of visible light images and infrared thermal imaging images, and preprocess the acquired data; S2: Construct a network model for power equipment status monitoring. The power equipment status monitoring network model includes: two encoders, two attention mechanism modules DEA, three interactive multimodal feature complementary modules IMFC, a multi-level global context fusion module MGCF and a decoder; The two encoders are configured to extract five layers of features from visible light images and infrared images respectively, and the features output by the first two layers of the encoders are respectively input into two attention mechanism modules DEA to suppress interference information and enhance the model's ability to recognize target boundaries; The features output by the last three layers of the encoder are respectively input into three interactive multimodal feature complementation modules IMFC to obtain the complementary relationship between visible light and infrared features; The multi-level global context fusion module MGCF is configured to receive output features of two attention mechanism modules DEA and three interactive multimodal feature complementation modules IMFC, and fuse the global context information of each level to enhance the semantic features; The decoder is configured to gradually upsample multiple features output by the multi-level global context fusion module MGCF and output a prediction structure; S3: Model training, the model training process includes: The training image is sent to the power equipment status monitoring network model, and the side output module is applied to each layer of the decoder. The output image of the side output module is enlarged to the same resolution as the original image through the upsampling function; The loss is calculated using the binary cross entropy loss function, and the parameters and weights in the network are optimized by back-propagating the loss. The training stops when the preset round is reached or when the accuracy of the validation set does not decrease within 10 consecutive rounds during the training process; S4: Model application and evaluation: deploy the trained model on the host computer for monitoring power equipment. The handheld device collects visible light images and infrared thermal imaging images of the equipment to be monitored, and runs the model for inference to obtain prediction results.

2. The method for monitoring the state of electric power equipment according to claim 1, characterized in that: In step S1, the image preprocessing process includes: The visible light image and the infrared thermal imaging image are registered to align the visible light image and the infrared thermal imaging image in space.

3. The method for monitoring the state of electric power equipment according to claim 1, characterized in that: In step S1, the image is processed The preprocessing process also includes: normalizing the images of the two modalities respectively.

4. The method for monitoring the state of electric power equipment according to claim 1, characterized in that: In step S1, the image preprocessing process further includes: scaling the captured image to a size of 640×480.

5. The method for monitoring the state of electric power equipment according to claim 1, characterized in that: In step S2, the pre-trained convolutional neural network MobileNet-V2 is used as the encoder.

6. The method for monitoring the state of electric power equipment according to claim 1, characterized in that: The attention mechanism module DEA receives the features of the first two layers of encoding output and performs the following processing: The visible light feature of the i-th level and infrared thermal imaging features After cascading along the channel direction and undergoing depth-separable convolution processing, the joint feature is obtained Where i = 1, 2; For the visible light characteristics of the i-th level and infrared thermal imaging features Global maximum pooling and 5×5 two-dimensional convolution are performed to extract global salient features, respectively. and Will Add up the features The characteristics After cascading along the channel direction, the features are obtained through the channel attention module feature After the depth-separable convolution, the output features of the attention mechanism module DEA are obtained 7. The method for monitoring the state of electric power equipment according to claim 6, characterized in that: The interactive multimodal feature complementation module IMFC receives the feature input of the last three layers of the encoder and performs the following processing: The visible light feature of the i-th level and infrared thermal imaging features After cascading along the channel direction and undergoing depth-separable convolution processing, the preliminary fused joint features are obtained. Where i = 3, 4, 5; For the visible light characteristics of the i-th level and infrared thermal imaging features Perform 3×3 two-dimensional convolution respectively, and get and Where i = 3, 4, 5; Bundle As the query value of the multi-head attention module, As the key value and value of the multi-head attention module, we combine it with the 8-head attention module to get Accordingly, and Combined with 8 attention modules By injecting complementary information of visible light and infrared modalities into the initial fusion features of the two In and After cascading, the output features of the IMFC module are obtained by depth-separable convolution.

8. The method for monitoring the state of electric power equipment according to claim 7, characterized in that: The multi-level global context fusion module MGCF receives two output features of the DEA module. And the three output characteristics of the IMFC module The following operations are performed to fuse multi-level global context information to enhance semantic features, including: The characteristics and Downsampled to Size, features Upsampling to Size; Will After cascading along the channel direction, a depth-separable convolution is performed to obtain F DC ; Will F DC Cascade along the channel direction to get F DI ; F DI Perform depth-wise separable convolution to obtain Another pair of F DI Perform another depth-wise separable convolution and transpose to get Will and Perform matrix multiplication to get F DM ; F DM After applying the ReLU activation function, a 1*1 convolution is performed to obtain F R ; F DM Apply the Sigmoid activation function to get F S ; F R With F S After adding, perform 1*1 convolution to get F RSC ; F RSC With F DC Perform Hadamard product to obtain the output feature F of the MGCF module MGCF .

9. The method for monitoring the state of electric power equipment according to claim 7, characterized in that: In step S3, a side output module is applied to each layer of the decoder, which is defined as: Where δ(·) represents the ReLU activation function, BN(·) represents batch normalization, Denotes 1×1 convolution, the number of output channels is c, De i Represents the output of each layer of the decoder, i = 1, 2, 3, 4; The loss is calculated using the binary cross entropy loss function, which is defined as: Where L i Indicates O i The corresponding binary cross entropy loss.

10. The method for monitoring the state of electric power equipment according to claim 1, characterized in that: Step S4 also includes: (2) comparing the prediction results with the actual values ​​given by professional staff and calculating various evaluation indicators, wherein the evaluation indicators include accuracy and overlap.

Images