Power equipment defect detection system and method based on multi-mode Transform

Abstract

The invention provides a power equipment defect detection system and method based on a multi-mode Transform, and the system comprises a double-flow feature extraction module which is used for carrying out the feature extraction of a collected visible light image and an infrared image of power equipment through employing a double-flow feature extraction network based on YOLOv5, and obtaining a visible light feature pattern and an infrared feature pattern; the multi-modal information interaction module is used for carrying out multi-modal information interaction on the visible light characteristic pattern and the infrared characteristic pattern to obtain a multi-modal fusion characteristic pattern; and the defect classification module is used for carrying out defect detection and classification on the power equipment by adopting a two-stage residual full-connection defect classifier based on the multi-modal fusion feature map to obtain a defect classification result. The method can overcome the influence of environmental factors such as heavy fog, rain, haze and the like, is not limited by illumination conditions, and has relatively high anti-interference capability. In addition, the method can also solve the problems of low resolution and fuzzy texture details of the infrared image.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, specifically relating to a power equipment defect detection system and method based on multimodal Transformer. Background Technology

[0002] With the expansion of my country's power grid and the increase in operating load, defect diagnosis of power equipment such as insulators, tension clamps, and splicing pipes in transmission lines has become increasingly critical. Traditional manual inspections rely on visual observation, which suffers from low efficiency, high missed detection rate, and high safety risks to inspection personnel, making it difficult to meet the needs of smart grids.

[0003] In recent years, with the development of artificial intelligence, intelligent defect diagnosis technologies such as infrared thermal imaging and visible light image recognition and diagnosis have been gradually applied to the defect diagnosis of power equipment. Infrared thermal imaging technology can effectively capture latent defects in power equipment, such as localized temperature rises caused by insulator deterioration and hot spots caused by poor contact of clamps. Visible light image recognition and diagnosis technology can identify defects such as external defects, such as insulator cracks and hardware deformation. However, single-modal detection has inherent limitations: infrared images, due to their low resolution, are difficult to distinguish fine structural defects; visible light images show a significant decrease in detection performance under adverse weather conditions such as rain, fog, and strong light. Existing fusion technologies mostly employ simple pixel-level fusion, which cannot fully exploit the complementary information of the two modes, resulting in low defect diagnosis rates. Furthermore, existing deep learning defect diagnosis models are mostly designed for single modes and do not fully utilize the synergistic effect of cross-modal features, making it difficult to adapt to equipment defect diagnosis under complex operating conditions. Therefore, there is an urgent need for an efficient cross-modal fusion method to improve the reliability and adaptability of power equipment defect diagnosis. Summary of the Invention

[0004] To address the problems existing in the prior art, this invention provides a power equipment defect detection system and method based on multimodal Transformer, which improves the defect diagnosis accuracy of power transmission line equipment.

[0005] To achieve the above objectives, the present invention provides the following solution: A power equipment defect detection system based on multimodal Transformer, comprising: The dual-stream feature extraction module is used to extract features from the acquired visible light and infrared images of power equipment using a YOLOv5-based dual-stream feature extraction network to obtain visible light feature maps and infrared feature maps. A multimodal information interaction module is used to perform multimodal information interaction between the visible light feature map and the infrared feature map to obtain a multimodal fused feature map; The defect classification module is used to detect and classify defects in power equipment based on multimodal fusion feature maps and a two-level residual fully connected defect classifier to obtain defect classification results.

[0006] Preferably, in the dual-stream feature extraction module, the YOLOv5 network includes a Backbone layer and a Neck layer; The backbone layer uses CSPDarknet, whose main branch employs a 3×3 convolutional layer cascade structure to extract spatial features from visible light and infrared images; the secondary branch connects to a dynamic attention module to enhance key features through weight calibration. The Neck layer uses 3×3 and 1×1 convolutional kernels as its core to extract multi-scale and multi-semantic features, and introduces a cross-dimensional feature calibration module for feature calibration.

[0007] Preferably, the dynamic attention module includes a 1×1 convolutional layer, a dynamic weight generator, and a Sigmoid activation function; The 1×1 convolutional layer is used to reduce the dimensionality of the spatial feature map; The dynamic weight generator includes a global average pooling layer and two fully connected layers. The global average pooling layer obtains global statistical information of the spatial feature map. The fully connected network is used to learn the importance mapping relationship of the global statistical information and generate a dynamic weight matrix that is completely consistent with the size of the spatial feature map. The Sigmoid activation function is used to normalize the weight values ​​of the dynamic weight matrix to a preset range and multiply them pixel by pixel with the dimensionality-reduced spatial feature map to enhance key features.

[0008] Preferably, the feature calibration process performed by the cross-dimensional feature calibration module includes: A calibration mechanism is added before feature fusion. The feature distribution is unified through standardization. The mean and variance of the preset high-level features and the preset low-level features are calculated respectively to obtain global statistics. The preset high-level features and preset low-level features are mapped to a reference distribution using a linear transformation formula. Based on the global statistics, the importance of the fused features is evaluated through a two-layer perceptron to obtain the weight coefficients of each level of features, thus completing feature calibration.

[0009] Preferably, the multimodal information interaction module includes: Two-dimensional unfolding unit, used to unfold visible light feature map and infrared feature map into a two-dimensional matrix; The location embedding unit is used to connect each two-dimensional matrix and add location embedding information to the Transformer model; The matrix calculation unit is used to project the input of the Transformer model onto three preset weight matrices and calculate the query matrix, key matrix, and value matrix. The context-aware acquisition unit is used to calculate the attention weight matrix using the scaled dot product between the query matrix and the key matrix, and multiply the attention weight matrix with the value matrix to obtain the context-aware representation of the Transformer model input; A multi-head self-attention unit is used to represent the relationship between different modal subspaces at different locations based on the context-aware representation and employing a multi-head self-attention mechanism. The output acquisition unit is used to calculate the output of the Transformer model based on the relationship between different modal subspaces at different locations, using a two-layer fully connected feedforward network with a GELU activation function in the middle. The feature map acquisition unit is used to convert the output of the Transformer model into visible light feature maps and infrared feature maps after fusing information from different modes by using the inverse operation of the two-dimensional unfolding unit. The feature enhancement unit is used to add the visible light feature map and infrared feature map after fusing different modal information to the original modal branch to obtain the visible light enhanced feature map and infrared enhanced feature map, and then stitch the visible light enhanced feature map and infrared enhanced feature map to obtain the multimodal fusion feature map.

[0010] Preferably, the two-stage residual fully connected defect classifier includes a two-stage residual fully connected layer and a Softmax output layer. The two-stage residual fully connected layer includes three residual fully connected blocks, where the input of the first residual fully connected block and the output of the second residual fully connected block are used together as the input of the third residual fully connected block.

[0011] This invention also provides a method for detecting defects in power equipment based on a multimodal Transformer, wherein the system includes: A YOLOv5-based dual-stream feature extraction network was used to extract features from the acquired visible light and infrared images of power equipment to obtain visible light feature maps and infrared feature maps. Multimodal information interaction is performed on the visible light feature map and the infrared feature map to obtain a multimodal fused feature map; Based on multimodal fusion feature maps, a two-level residual fully connected defect classifier is used to detect and classify defects in power equipment, and obtain defect classification results.

[0012] Preferably, the YOLOv5 network includes a Backbone layer and a Neck layer; The backbone layer uses CSPDarknet, whose main branch employs a 3×3 convolutional layer cascade structure to extract spatial features from visible light and infrared images; the secondary branch connects to a dynamic attention module to enhance key features through weight calibration. The Neck layer uses 3×3 and 1×1 convolutional kernels as its core to extract multi-scale and multi-semantic features, and introduces a cross-dimensional feature calibration module for feature calibration.

[0013] Preferably, the method by which the dynamic attention module strengthens key features through weight calibration includes: The dimensionality of the spatial feature map is reduced by the number of channels, and global statistical information of the spatial feature map is obtained through a global average pooling layer; Learn the importance mapping relationship of global statistical information to generate a dynamic weight matrix that is completely consistent with the size of the spatial feature map; The weight values ​​of the dynamic weight matrix are normalized to a preset range and multiplied pixel by pixel with the dimensionality-reduced spatial feature map to enhance key features.

[0014] Preferably, the method for performing feature calibration includes: A calibration mechanism is added before feature fusion. The feature distribution is unified through standardization. The mean and variance of the preset high-level features and the preset low-level features are calculated respectively to obtain global statistics. The preset high-level features and preset low-level features are mapped to a reference distribution using a linear transformation formula. Based on the global statistics, the importance of the fused features is evaluated through a two-layer perceptron to obtain the weight coefficients of each level of features, thus completing feature calibration.

[0015] Compared with existing technologies, the beneficial effects of this invention are as follows: First, the method employs dual-stream YOLOv5 to extract features from visible light and infrared images of power transmission line equipment, using 3×3 and 1×1 convolutional kernels as the core, combined with CSP and cross-stage connections (achieving bidirectional fusion by processing features at different scales through 3×3 convolution). This reduces computational load while completing multi-scale, multi-semantic feature extraction. Then, features from different stages of the visible light and infrared images are input into a Transformer-based information interaction module. A self-attention mechanism learns the feature information between different modalities and adds it as a supplement to the original modal branch, achieving complementary advantages between visible light and infrared modal features. Finally, a residual fully connected defect classifier is used to classify defects in the power transmission line equipment. This invention can overcome the influence of environmental factors such as fog, rain, and haze and is not limited by lighting conditions, exhibiting strong anti-interference capabilities. Furthermore, this invention can also solve the problem of low resolution and blurred texture details in infrared images. Attached Figure Description

[0016] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram of the power equipment defect detection system based on multimodal Transformer according to an embodiment of the present invention; Figure 2 This is an architecture diagram of the multimodal information interaction module according to an embodiment of the present invention; Figure 3 This is a structural diagram of a two-level residual fully connected defect classifier according to an embodiment of the present invention; Figure 4 This is a structural diagram of the residual fully connected block according to an embodiment of the present invention. Detailed Implementation

[0018] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] The following explanations are provided for some of the related terms used in this embodiment: (1) Visible light defect diagnosis: Use the visible light image of the equipment to observe its appearance, color, brightness, texture details and other high-resolution features to determine whether the equipment is abnormal.

[0021] (2) Infrared defect diagnosis: This technique utilizes the thermal radiation characteristics of objects to obtain temperature distribution images of equipment through infrared sensors, and identifies defects caused by abnormal temperatures.

[0022] (3) Multimodal fusion: Integrate the information of the same scene data collected by different sensors (such as infrared and visible light) to generate a more discriminative image.

[0023] (4) Transformer network: a deep learning architecture based on self-attention mechanism. It was initially applied to natural language processing and is now widely used in computer vision, speech recognition and other fields. Its core abandons the temporal dependence of traditional recurrent neural networks and processes sequential information in parallel through attention mechanism to improve computational efficiency.

[0024] (5) Cross-modal feature interaction: In the deep learning model, the mutual transfer and fusion of different modal features of visible light and infrared images are realized, so that the model can make comprehensive use of the complementary information of the two modalities.

[0025] Example 1 like Figure 1 As shown, a power equipment defect detection system based on multimodal Transformer includes: a dual-stream feature extraction module, a multimodal information interaction module, and a defect classification module.

[0026] The dual-stream feature extraction module is used to extract features from the acquired visible light and infrared images of power equipment using a YOLOv5-based dual-stream feature extraction network, thereby obtaining visible light feature maps and infrared feature maps.

[0027] A further implementation involves a dual-stream feature extraction module where the YOLOv5 network includes a Backbone layer and a Neck layer (FPN+PAN, PANET). The Backbone layer employs CSPDarknet, incorporating a CSP structure, and extracts features through alternating 3×3 and 1×1 convolutional kernels. The 3×3 convolution captures spatial information, while the 1×1 convolution reduces dimensionality. The weight allocation rules of CSPDarknet's static attention mechanism are fixed and cannot be adjusted based on the real-time content of the input feature map. When extracting features from a detected target, if the target is in a complex background or occluded, the static weights are prone to misclassifying background noise as valid features or failing to focus on unoccluded key feature regions, leading to a decrease in feature extraction accuracy. To address this, this invention proposes a dynamic attention module to replace the traditional static attention module.

[0028] The main branch of CSPDarknet employs a cascaded 3×3 convolutional layer structure to extract spatial features from visible light and infrared images. The secondary branch connects to a dynamic attention module to enhance key features through weight calibration. Specifically, the original CSP module's main and secondary branches had overlapping functions, but the improved branch has a clear division of labor. The main branch retains the cascaded 3×3 convolutional layer structure, focusing on extracting basic texture and edge features; the secondary branch connects to the dynamic attention module to enhance key features through weight calibration. Specifically, the main branch performs two 3×3 convolutions on the input features (stride 1, padding 1), maintaining the feature map size unchanged. The network structure designed in this invention includes two main branches: one for extracting feature maps from visible light images and the other for extracting feature maps from infrared images. Both main branches are based on an improved YOLOv5 network structure for feature extraction.

[0029] The secondary branch is first processed by the dynamic attention module, then the number of channels is adjusted to match that of the main branch through a 1×1 convolution, and finally concatenated with the main branch and fused together through a 1×1 convolution for output.

[0030] A further implementation method is that the dynamic attention module includes a 1×1 convolutional layer, a dynamic weight generator, and a Sigmoid activation function; A 1×1 convolutional layer is used to reduce the number of channels in the spatial feature map. In this embodiment, the number of channels in the feature map is reduced to 1 / 4 of the original number through 1×1 convolution, thus reducing the amount of computation.

[0031] The dynamic weight generator consists of a global average pooling layer and two fully connected layers. The global average pooling layer obtains global statistical information of the spatial feature map, and the fully connected network is used to learn the importance mapping relationship of the global statistical information to generate a dynamic weight matrix that is completely consistent with the size of the spatial feature map. The Sigmoid activation function is used to normalize the weight values ​​of the dynamic weight matrix to a preset range [0,1], and multiply them pixel by pixel with the reduced spatial feature map to enhance key features.

[0032] The Neck layer processes features at different scales through 3×3 convolutions to achieve bidirectional fusion. Using 3×3 and 1×1 convolutional kernels as the core, combined with CSP and cross-stage connections, it extracts multi-scale and multi-semantic features while reducing computational cost. A cross-dimensional feature calibration module is introduced for feature calibration. PANet's feature fusion suffers from structural defects: high-level features (such as P5) are rich in semantic information but have low resolution, while low-level features (such as P2) have high resolution but ambiguous semantics, with significant differences in scale and feature distribution. Direct fusion leads to the loss of effective information due to feature distribution mismatch, while also introducing redundant features. Furthermore, fused features cannot meet the needs of targets at different scales; large targets are prone to localization errors due to insufficient low-level details, while small targets are prone to classification errors due to missing high-level semantics. To address these issues, this invention proposes a cross-dimensional feature calibration module.

[0033] A further implementation method involves the cross-dimensional feature calibration module performing feature calibration as follows: A calibration mechanism is added before feature fusion. The feature distribution is standardized, and the mean and variance are calculated for the preset high-level feature P5 and the preset low-level feature P2, respectively, to obtain global statistics (mean and variance). Feature fusion in this step refers to the fusion of infrared image features and visible light features, i.e., multimodal information interaction. Multimodal information interaction involves the fusion and interaction of global information from visible light image features and infrared image features. However, not all global information is useful; therefore, a dynamic weighting mechanism is used, assigning higher weights to important features and lower weights to less important features.

[0034] A linear transformation formula is used to map preset high-level and low-level features to a reference distribution (based on the distribution of the middle layer P3), eliminating distribution differences between different levels of features. Based on global statistics, a two-layer perceptron is used to evaluate the importance of the fused features, obtaining the weight coefficients of each level of features, thus completing feature calibration. Specifically, a two-layer perceptron is used to evaluate the importance of the fused features, with the global statistics of the features as input and the weight coefficients of the three levels of features as output. , , ,satisfy The weight allocation rule is learned through training: when a small target is detected, Automatically increases to 0.4-0.5; when a large target is detected, Increase to 0.4-0.5. After multiplying the weights by the features, the semantic consistency is enhanced by the CSP module (containing 3 residual blocks), and the fused features are output. The linear transformation formula can be expressed as: .

[0035] The multimodal information interaction module is used to perform multimodal information interaction between visible light feature maps and infrared feature maps, fully utilizing the inherent complementary characteristics between different modalities to obtain a multimodal fused feature map. For example... Figure 2 As shown.

[0036] A further implementation method includes a multimodal information interaction module comprising: Two-dimensional unfolding units are used to unfold visible light feature maps and infrared feature maps into two-dimensional matrices; specifically, assuming the visible light feature map... The size is Infrared feature map The size is Each feature map is flattened into a two-dimensional matrix to obtain the visible light features. Figure 2 The size of the dimensional matrix is Infrared features Figure 2 The size of the dimensional matrix is .

[0037] The position embedding unit is used to concatenate each 2D matrix and add position embedding information before inputting it into the Transformer model; specifically, it concatenates the 2D matrices of each modality and adds a position embedding information of size [value missing]. Thus, the input of the Transformer model The size is Location embedding information enables the model to distinguish spatial information between different modalities during training.

[0038] The matrix calculation unit projects the input of the Transformer model onto three predefined weight matrices, calculating the query matrix, key matrix, and value matrix; specifically, the input... The matrix is ​​projected onto three weight matrices. The query matrix, key matrix, and value matrix are calculated using the following formula: in, , , Represents the weight matrix. D represents the dimension of the hidden layer, and C represents the length of the feature map.

[0039] Context-aware acquisition unit, used to utilize query matrix AND key matrix The attention weight matrix is ​​calculated by scaling the dot product between the matrix and the value matrix. Multiplying the attention weight matrix by the value matrix yields the context-aware representation of the Transformer model input. The calculation formula is: , in, This represents a scaling factor used to prevent scaling when the magnitude of the dot product increases. The function gets stuck in a region where the gradient is minimal.

[0040] Multi-head self-attention units (MSIs) are used for context-aware representation, employing a MSI mechanism to represent relationships between different modal subspaces at different locations. Specifically, the MSI mechanism represents multiple complex relationships between different modal subspaces at different locations, and the calculation formula is as follows: in, Indicates the number of heads. express The projection matrix, This represents context awareness after the multi-head attention mechanism is assembled.

[0041] The output acquisition unit is used to compute the output of the Transformer model based on the relationships between different modal subspaces at different locations, using a two-layer fully connected feedforward network with a GELU activation function in between. The calculation formula is: , in, , This represents the input to a fully connected network.

[0042] The feature map acquisition unit is used to convert the output of the Transformer model into visible light feature maps and infrared feature maps after fusing information from different modes by using the inverse operation of the two-dimensional unfolding unit. The feature enhancement unit is used to enhance the visible light feature map after fusing information from different modes. and infrared feature map Add it to the original modality branch (i.e., the main branch feature extraction network) to obtain visible light enhanced feature maps and infrared enhanced feature maps, and then stitch the visible light enhanced feature maps and infrared enhanced feature maps together to obtain a multimodal fusion feature map.

[0043] The defect classification module is used to detect and classify defects in power equipment based on multimodal fusion feature maps and a two-level residual fully connected defect classifier to obtain defect classification results.

[0044] A further implementation involves a two-stage residual fully connected defect classifier, comprising two stages of residual fully connected layers and a Softmax output layer. The two stages of residual fully connected layers each include three residual fully connected blocks, where the input of the first residual fully connected block and the output of the second residual fully connected block are used together as the input of the third residual fully connected block. For example... Figure 3 As shown.

[0045] Each residual fully connected block consists of three hidden layers and one batch normalization (BN) layer. The input of the first hidden layer and the output of the second hidden layer are used as the input of the third hidden layer. The BN layer normalizes the data, ensuring that the output of each residual fully connected block remains within the range [-1, 1], thus resolving the cascading problem between layers and enabling the defect classification model to converge quickly. Figure 4 As shown.

[0046] The output of the first hidden layer can be represented as: In the formula, This represents the output vector of the first hidden layer; Represents the ReLU activation function; This represents the output vector of the first hidden layer after performing a linear operation on the input vector; This represents the weight matrix of the first hidden layer; This represents the input vector of the first hidden layer; This represents the bias vector of the first hidden layer.

[0047] The output of the second hidden layer can be expressed as: In the formula, This represents the output vector of the second hidden layer; This represents the output vector of the second hidden layer after performing a linear operation on the input vector; This represents the weight matrix of the second hidden layer; This represents the bias vector of the second hidden layer.

[0048] The output of the third hidden layer can be represented as: In the formula, This represents the output vector of the third hidden layer; This represents the output vector of the third hidden layer after performing a linear operation on the input vector; This represents the weight matrix of the third hidden layer; This represents the bias vector of the third hidden layer.

[0049] The output of the BN layer can be represented as: In the formula, This represents the output vector of the BN layer after data normalization. This represents the normalization function.

[0050] The Softmax output layer uses the cross-entropy loss function as the model's loss function. During model training, the backpropagation algorithm is used to optimize the weight matrix and bias vector parameters to minimize the loss function of the defect diagnosis model.

[0051] The overall feature extraction process of this invention is as follows: Figure 1 Pn (n=1,2,3,4,5) is obtained by fusing visible light feature n and infrared feature n. The entire architecture contains 3 information interaction modules and 10 convolutional kernels. The size of the convolutional kernels is 3×3. Taking the first information interaction module as an example, the downward arrow indicates that visible light feature 2 and infrared feature 2 are processed through convolutional kernel 3 and then exchanged through the information interaction module to extract infrared features and visible light features, which are then fused and complemented to obtain useful information.

[0052] The present invention addresses the challenges of cross-modal fusion and poor intrinsic complementarity between visible light and infrared images of power equipment during defect diagnosis of power equipment in transmission lines. It can enhance one modality from another and simultaneously fuse intra-modal and inter-modal features to improve the accuracy of defect diagnosis of power equipment in transmission lines.

[0053] Example 2 This invention also provides a method for detecting defects in power equipment based on a multimodal Transformer, using the system described in Embodiment 1, comprising: A YOLOv5-based dual-stream feature extraction network was used to extract features from the acquired visible light and infrared images of power equipment to obtain visible light feature maps and infrared feature maps. Multimodal information interaction is performed between visible light feature maps and infrared feature maps to obtain a multimodal fused feature map; Based on multimodal fusion feature maps, a two-level residual fully connected defect classifier is used to detect and classify defects in power equipment, and obtain defect classification results.

[0054] A further implementation method is that the YOLOv5 network includes a Backbone layer and a Neck layer; The backbone layer uses CSPDarknet, and the main branch of CSPDarknet adopts a 3×3 convolutional layer cascade structure to extract spatial features from visible light and infrared images; the secondary branch is connected to a dynamic attention module to enhance key features through weight calibration. The Neck layer uses 3×3 and 1×1 convolutional kernels as its core to extract multi-scale and multi-semantic features, and introduces a cross-dimensional feature calibration module for feature calibration.

[0055] A further implementation method involves the dynamic attention module strengthening key features through weight calibration, comprising: The dimensionality of the spatial feature map is reduced by the number of channels, and global statistical information of the spatial feature map is obtained through a global average pooling layer; Learn the importance mapping relationship of global statistical information to generate a dynamic weight matrix that is completely consistent with the size of the spatial feature map; The weight values ​​of the dynamic weight matrix are normalized to a preset range and multiplied pixel by pixel with the dimensionality-reduced spatial feature map to enhance key features.

[0056] A further implementation method includes: The method for performing feature calibration includes: A calibration mechanism is added before feature fusion. The feature distribution is unified through standardization. The mean and variance of the preset high-level features and the preset low-level features are calculated respectively to obtain global statistics. The preset high-level features and preset low-level features are mapped to a reference distribution using a linear transformation formula. Based on the global statistics, the importance of the fused features is evaluated through a two-layer perceptron to obtain the weight coefficients of each level of features, thus completing feature calibration.

[0057] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made to the technical solutions of the present invention by those skilled in the art without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A multi-modal Transformer-based power equipment defect detection system, characterized in that, The method comprises the steps of: a dual-flow feature extraction module is used to extract features of visible light images and infrared images of power equipment collected by using a dual-flow feature extraction network based on YOLOv5, to obtain visible light feature maps and infrared feature maps; a multi-modal information interaction module is used to interact multi-modal information of the visible light feature maps and the infrared feature maps, to obtain multi-modal fusion feature maps; a defect classification module is used to detect and classify defects of power equipment based on the multi-modal fusion feature maps by using a two-level residual full connection defect classifier, to obtain a defect classification result.

2. The system of claim 1, wherein, In the dual-flow feature extraction module, the YOLOv5 network comprises a Backbone layer and a Neck layer; the Backbone layer adopts a CSPDarknet, a main branch of the CSPDarknet adopts a 3*3 convolution layer series structure, and is used to extract spatial features of visible light images and infrared images; a sub-branch accesses a dynamic attention module, which is used to strengthen key features through weight calibration; the Neck layer extracts multi-scale and multi-semantics features with 3*3 and 1*1 convolution kernels as cores, and introduces a cross-dimension feature calibration module for feature calibration.

3. The system of claim 2, wherein, The dynamic attention module comprises a 1*1 convolution layer, a dynamic weight generator and a Sigmoid activation function; the 1*1 convolution layer is used to reduce the dimension of a channel number of a spatial feature map; the dynamic weight generator comprises a global average pooling layer and two layers of full connection networks, the global average pooling layer is used to obtain global statistical information of the spatial feature map, and the full connection networks are used to learn an importance mapping relationship of the global statistical information, to generate a dynamic weight matrix consistent with a size of the spatial feature map; the Sigmoid activation function is used to normalize weight values of the dynamic weight matrix to a preset range, and to multiply the normalized weight values with the spatial feature map after dimension reduction pixel by pixel, to strengthen the key features.

4. The system of claim 2, wherein, The process of feature calibration of the cross-dimension feature calibration module comprises the steps of: a calibration mechanism is added before feature fusion, feature distribution is unified through standardization processing, mean values and variances of preset high-level features and preset low-level features are calculated respectively, and global statistical quantities are obtained; preset high-level features and preset low-level features are mapped to a reference distribution by using a linear transformation formula, the importance of fusion features is evaluated by two layers of perception machines based on the global statistical quantities, weight coefficients of each level of features are obtained, and feature calibration is completed.

5. The system of claim 1, wherein, The multi-modal information interaction module comprises: a two-dimensional expansion unit is used to expand the visible light feature maps and the infrared feature maps into two-dimensional matrices; a position embedding unit is used to connect each two-dimensional matrix and add position embedding information, and input a Transformer model; a matrix calculation unit is used to project input of the Transformer model to three preset weight matrices, to calculate a query matrix, a key matrix and a value matrix; a context perception acquisition unit is used to calculate an attention weight matrix by using a scaling dot product between the query matrix and the key matrix, to multiply the attention weight matrix with the value matrix, and to obtain a context perception representation of input of the Transformer model; The multi-head self-attention unit is configured to represent the relationship between different position and different modal subspaces by using a multi-head self-attention mechanism based on the context perception representation. The output acquisition unit is configured to calculate the output of the Transformer model by using a two-layer fully connected feedforward connection network with a GELU activation function based on the relationship between different position and different modal subspaces. The feature map acquisition unit is configured to convert the output of the Transformer model into visible light feature maps and infrared feature maps after fusing different modal information by using an inverse operation of the two-dimensional expansion unit. The feature enhancement unit is configured to add the visible light feature maps and the infrared feature maps after fusing different modal information to the original modal branch to obtain visible light enhanced feature maps and infrared enhanced feature maps, and to splice the visible light enhanced feature maps and the infrared enhanced feature maps to obtain multi-modal fusion feature maps.

6. The system of claim 1, wherein, The two-level residual fully connected defect classifier includes a two-level residual fully connected layer and a Softmax output layer, wherein the two-level residual fully connected layer includes three residual fully connected blocks, and the input of the first residual fully connected block is used as the input of the third residual fully connected block together with the output of the second residual fully connected block.

7. A method for power equipment defect detection based on multi-modal Transformer, applying the system of any one of claims 1-6, characterized in that, The method comprises the following steps: The YOLOv5-based double-flow feature extraction network is used to extract features of the collected visible light images and infrared images of the power equipment to obtain visible light feature maps and infrared feature maps; The visible light feature maps and the infrared feature maps are subjected to multi-modal information interaction to obtain multi-modal fusion feature maps; Based on the multi-modal fusion feature maps, a two-level residual fully connected defect classifier is used to detect and classify defects of the power equipment to obtain a defect classification result.

8. The method of claim 7, wherein, The YOLOv5 network includes a Backbone layer and a Neck layer; The Backbone layer adopts a CSPDarknet, and a main branch of the CSPDarknet adopts a 3×3 convolution layer series structure for extracting spatial features of visible light images and infrared images; A sub-branch accesses a dynamic attention module for strengthening key features through weight calibration; The Neck layer extracts multi-scale and multi-semantics features with 3×3 and 1×1 convolution kernels as cores, and introduces a cross-dimension feature calibration module for feature calibration.

9. The method of claim 8, wherein, The method for strengthening key features through weight calibration of the dynamic attention module comprises the following steps: The channel number of a spatial feature map is reduced, and global statistical information of the spatial feature map is obtained through a global average pooling layer; A dynamic weight matrix completely consistent with the size of the spatial feature map is generated by learning the importance mapping relationship of the global statistical information; The weight values of the dynamic weight matrix are normalized to a preset range, and are multiplied with the reduced spatial feature map pixel by pixel to enhance the key features.

10. The method of claim 8, wherein, The method for feature calibration comprises the following steps: A calibration mechanism is added before feature fusion to unify the feature distribution through standardization processing, and the mean and variance of preset high-level features and preset low-level features are calculated respectively to obtain global statistics; The preset high-level feature and the preset low-level feature are mapped to a reference distribution by using a linear transformation formula; and based on the global statistics, the importance of the fused feature is evaluated by a two-layer perception machine to obtain a weight coefficient of each level feature, thereby completing feature calibration.

Images