Transform architecture-based multi-modal power transmission line icing type identification method

Abstract

The invention discloses a multi-modal power transmission line icing type identification method based on a Transform architecture, and the method comprises the steps: obtaining a power transmission line icing image and corresponding meteorological data, and carrying out the preprocessing; the method comprises the following steps: extracting image features through a Vision Transform model; extracting meteorological features through a meteorological feature extraction module; fusing the image features and the meteorological features through a cross attention mechanism to generate fused features; and outputting an icing type classification result based on the fusion features. According to the method, the identification accuracy and robustness of the icing type of the power transmission line are improved by combining the multi-modal model of the image and the meteorological data.

Description

Technical Field

[0001] This invention relates to a method for identifying icing types of transmission lines, and more particularly to a multimodal method for identifying icing types of transmission lines based on the Transformer architecture. Background Technology

[0002] With the accumulation of massive images from online icing monitoring systems for power transmission lines and the development of machine learning technology, icing type identification in recent years has primarily relied on icing images. Existing methods quantify the differences in grayscale and hue distribution between rime and mixed frost using color histograms, and combine this with Support Vector Machines (SVM) to achieve a classification accuracy of 80%–90%. Alternatively, they fuse texture features from Local Binary Patterns (LBP) and Gray-Level Co-occurrence Matrix (GLCM), and use genetic algorithms to optimize SVM parameters to identify icing types. These methods rely on feature engineering and suffer from high environmental sensitivity and insufficient generalization ability. Other methods integrate micro-meteorological data with the Nearest Neighbor (KNN) algorithm to achieve multi-type icing identification; however, the KNN method is sensitive to data balance, and the recognition rate drops significantly when there are insufficient mixed frost samples. Convolutional neural networks (CNNs) are also used to automatically extract image features and construct an icing classification early warning model for ultra-high voltage transmission lines. Although its classification performance is much better than traditional methods, it requires a large amount of manually labeled data and has a high misclassification rate for icing types with similar textures.

[0003] Although current methods for identifying icing types have evolved from manual experience-based judgment to intelligent recognition based on deep learning, several key issues remain to be addressed. Existing research largely relies on single-source image data to construct classification and recognition models, neglecting the physical constraints of meteorological factors such as temperature, humidity, and wind speed on the icing formation mechanism. This disconnect between mechanistic understanding and visual data features hinders breakthroughs in the accuracy of icing type identification for transmission lines. Therefore, a method for identifying icing types that can integrate image and meteorological data, possesses strong generalization capabilities, and achieves high accuracy is urgently needed. Summary of the Invention

[0004] Purpose of the invention: The purpose of this invention is to provide a method for identifying icing types of multimodal transmission lines based on the Transformer architecture, thereby improving the accuracy of icing type identification for transmission lines.

[0005] Technical solution: The present invention provides a method for identifying icing types of multimodal transmission lines based on the Transformer architecture, comprising:

[0006] Acquire images of icing on power transmission lines and corresponding meteorological data, and perform preprocessing;

[0007] Image features are extracted using the Vision Transformer model;

[0008] Meteorological features are extracted using the meteorological feature extraction module;

[0009] The above image features and meteorological features are fused using a cross-attention mechanism to generate fused features;

[0010] The classification results of icing type are output based on the fusion features.

[0011] Furthermore, the meteorological data includes temperature, humidity, and wind speed; the image features include the shape and texture of the icing.

[0012] Furthermore, the preprocessing includes converting the meteorological data into a two-dimensional numerical vector.

[0013] Furthermore, the meteorological feature extraction module includes a multilayer perceptron, the structure of which includes:

[0014] The first linear layer maps the input meteorological data to 128 dimensions through linear transformation;

[0015] The second linear layer maps the 128-dimensional meteorological features to 768 dimensions, consistent with the dimensions of image features.

[0016] Furthermore, the multilayer perceptron structure also includes an activation function ReLU, two Dropout layers, and a LayerNorm layer; after processing by the first linear layer or the second linear layer, the data are processed by the ReLU activation function and the Dropout layer.

[0017] Furthermore, in the cross-attention mechanism: image features are used as the query, meteorological features are used as the key and value, and fused features are output through multi-head attention calculation.

[0018] Furthermore, the multi-head attention calculation includes: splitting Query, Key, and Value into multiple subspaces, and calculating for each subspace as follows:

[0019] The similarity dot product of Query and Key is performed, and then normalized using the SoftMax function to obtain attention weights. The attention weights are then weighted and summed with the Value to obtain the output of that subspace. The outputs of all subspaces are concatenated together, and the concatenated output is residually connected to the original Query through a Dropout layer. Finally, the output is normalized by the layer.

[0020] Furthermore, the calculation formula for the multi-head attention mechanism is as follows:

[0021]

[0022]

[0023] in, Indicates Query, Indicates the transpose of the key. It is the dimension of the key vector. This represents the attention score. This represents the input value, This represents the output of the attention mechanism.

[0024] The computer device of the present invention includes one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs, when executed by the processors, implement the steps of the multi-mode transmission line icing type identification method based on Transformer architecture.

[0025] The present invention discloses a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of a method for identifying icing types of multimodal transmission lines based on a Transformer architecture.

[0026] Beneficial effects: Compared with the prior art, the present invention has the following significant advantages:

[0027] (1) By introducing a multimodal fusion model based on the Transformer architecture (CADMEF-ViT), this invention overcomes the shortcomings of existing technologies that rely on single-source image data and ignore the physical constraints of meteorological elements, and achieves a significant improvement in the accuracy and robustness of the identification of icing type of transmission lines.

[0028] (2) By dynamically fusing image features with meteorological data such as temperature and humidity through a cross-attention mechanism, the model can make up for the lack of visual information in complex scenarios (such as night or foggy days), reduce misjudgments, and improve the overall accuracy to 97.78%.

[0029] (3) The model of this invention has better interpretability, and the attention weight intuitively reflects the correlation between the image and meteorological data, providing more reliable and refined technical support for power grid disaster prevention and mitigation decision-making. Attached Figure Description

[0030] Figure 1 This is a structural diagram of the CADMEF-ViT multimodal model for meteorological element fusion in this invention.

[0031] Figure 2 This is a structural diagram of the meteorological feature extraction module of the present invention.

[0032] Figure 3 This is a structural diagram of the cross-attention mechanism of the present invention.

[0033] Figure 4 These are diagrams illustrating four types of icing according to embodiments of the present invention.

[0034] Figure 5 This is a comparison chart of the prediction effects of three models: ViT_B_16, MFEM-ViT, and CADMEF-ViT, in embodiments of the present invention. Detailed Implementation

[0035] The technical solution of the present invention will be further described below with reference to the accompanying drawings.

[0036] This invention provides a method for identifying icing types on multimodal transmission lines based on the Transformer architecture, including:

[0037] (1) Obtain images of icing on transmission lines and corresponding meteorological data, and perform preprocessing; the meteorological data includes temperature, humidity, wind speed, etc., and the preprocessing includes converting the meteorological text data into a two-dimensional numerical vector.

[0038] (2) Construct a meteorological element fusion multimodal model driven by cross-attention mechanism (CADMEF-ViT) and extract image features through Vision Transformer (ViT);

[0039] (3) Extract meteorological features through the meteorological feature extraction module;

[0040] The meteorological feature extraction module is a multilayer perceptron structure, consisting of two linear layers, two activation functions, two dropout layers, and one LayerNorm layer. Meteorological text data, converted to two-dimensional numerical vectors, is input into the first linear layer. It is mapped to 128 dimensions through linear transformation, followed by a ReLU activation function to introduce non-linear features, and the dropout layer randomly discards neurons. In the second linear layer, the 128-dimensional meteorological features are mapped to 768 dimensions, consistent with the extracted image feature dimensions. After passing through a ReLU activation function and a dropout layer, the model's expressive power and generalization ability are further enhanced. Finally, layer normalization is used to stabilize the network training process.

[0041] The LayerNorm layer aims to eliminate scale differences between different meteorological variables, preventing a single input variable from dominating cross-modal fusion due to excessive size; stabilize feature distribution, reducing the interference of meteorological condition bias between different samples on training; and normalize the distribution of meteorological features and image features to be more uniform, which is beneficial for multimodal fusion.

[0042] (4) The above image features and meteorological features are fused through the cross-attention mechanism to generate fused features; finally, the icing type classification result is output based on the fused features.

[0043] The Cross Attention mechanism receives image features extracted from the Vision Transformer and meteorological features extracted from the meteorological feature extraction module. Image features include the shape and texture of ice in the image, which serve as the Query; meteorological features represent environmental factors closely related to ice formation, serving as both the Key and Value, and are input into the Multi-Head Attention mechanism.

[0044] Multi-head attention computation involves splitting the Query, Key, and Value into multiple subspaces, and calculating the following for each subspace:

[0045] The similarity dot product of Query and Key is performed, and then normalized using the SoftMax function to obtain attention weights. The attention weights are then weighted and summed with the Value to obtain the output of that subspace. The outputs of all subspaces are concatenated together, and the concatenated output is residually connected to the original Query through a Dropout layer. Finally, the output is normalized by the layer.

[0046] The formula is as follows:

[0047]

[0048]

[0049] in, Indicates Query, Indicates the transpose of the key. It is the dimension of the key vector. This represents the attention score. This represents the input value, This represents the output of the attention mechanism.

[0050] The cross-attention mechanism enables the model to associate visual features of an image with favorable meteorological features. When an image shows rime ice (transparent, dense ice), if the meteorological data also indicates a 0°C and high humidity environment, the model will be more certain that it is rime ice. For icing images where information is incomplete due to factors such as lighting and occlusion, the model gains a more comprehensive and robust understanding of the icing situation, thereby improving the accuracy of identifying icing types on transmission lines. The cross-attention mechanism allows the model to not only rely on the "seen" image but also incorporate "perceived" environmental information for judgment. Furthermore, the attention weights in the cross-attention mechanism can, to some extent, improve the interpretability of the model's decisions.

[0051] The embodiments of the present invention verify the beneficial effects of the method described in the present invention through specific experiments.

[0052] Using local ice-observation camera data provided by Region A and Region B, the "Ice Observation Camera" is a visual detection device for icing on power transmission lines. It mainly consists of a monitoring camera and a high-efficiency AC electric field induction power supply. It also has a built-in sensor module to monitor micro-meteorological elements such as ambient temperature and humidity at a given moment. The "Ice Observation Camera" is installed on the power transmission line conductors to achieve close-range, high-frequency shooting, with a minimum shooting interval of 2 minutes. The monitoring data is transmitted to the central station via 4G. Images captured by the "Ice Observation Camera" are stored in JPG format, and the image names are the power transmission line name plus the shooting time. The dataset includes four different categories: Rime Ice, Glaze Ice, Mixed Ice, and No Ice.

[0053] Accuracy, precision, recall, and F1 score were used as evaluation metrics for the model in this invention. Table 1 shows the comparison of the accuracy achieved by AlexNet, VGG16, ResNet50, MobileNetV3, EfficientNetV2, ViT_B_16, and CADMEF-ViT models on the test set. Experimental results show that the CADMEF-ViT multimodal model integrating meteorological elements in this invention has the highest recognition accuracy of 97.78%, demonstrating that fusing meteorological elements and image features can effectively improve recognition accuracy. Meteorological data, as a physical constraint, can provide crucial supplementary information for visual features, and the cross-attention mechanism is a superior approach to achieving this effective fusion of multimodal information.

[0054] Table 1:

[0055] Model Accuracy CADMEF-ViT(ViT+MFEM+CrossAttention) 97.78% ViT_B_16 95.97% EfficientNetV2 94.49% VGG16 91.74% MobileNetV3 87.82% AlexNet 86.12% ResNet50 83.05%

[0056] To further investigate the role of the meteorological feature extraction module and the cross-attention mechanism module, an ablation experiment was set up to compare the performance of three models on the transmission line icing image dataset with ViT_B_16: ViT (MFEM-ViT) which uses only the meteorological feature extraction module to achieve multimodal feature fusion, and ViT multimodal model (CADMEF-ViT) which fuses meteorological elements under the cross-attention mechanism. The classification performance of the model after fusing meteorological elements was evaluated.

[0057] Table 2 shows the accuracy of ViT_B_16, MFEM-ViT, and CADMEF-ViT on the transmission line icing test set. It is concluded that the ViT multimodal model (CADMEF-ViT) that integrates meteorological elements under the cross-attention mechanism performs the best.

[0058] Table 2:

[0059] Model Accuracy CADMEF-ViT(ViT+MFEM+CrossAttention) 97.78% MFEM-ViT (ViT+MFEM) 97.14% ViT_B_16 95.97%

[0060] Table 3 shows the precision, recall, and F1 score of the three models ViT_B_16, MFEM-ViT, and CADMEF-ViT for each category: Rime Ice, Mixed Ice, Glaze Ice, and No Ice.

[0061] Table 3:

[0062] Rime Ice Mixed Ice Glaze Ice No Ice ViT_B_16 Precision 94.98% 96.77% 93.91% 99.13% Recall 97.93% 89.55% 96.00% 99.13% F1-Score 96.43% 93.02% 94.93% 99.13% MFEM-ViT Precision 98.60% 97.92% 97.38% 97.83% Recall 97.24% 93.53% 99.55% 97.41% F1-Score 97.91% 95.67% 98.45% 97.62% CADMEF-ViT Precision 97.92% 97.45% 95.65% 98.71% Recall 98.26% 95.02% 98.21% 99.13% F1-Score 98.09% 96.22% 96.92% 98.92%

[0063] The MFEM-ViT model, which incorporates meteorological features, and the CADMEF-ViT model, which further employs a multimodal cross-attention mechanism, both significantly improve overall performance compared to the pure visual ViT model. The MFEM-ViT model generally achieves higher F1 scores than ViT_B_16 across all icing categories, with the most significant improvement in recall for rime and mixed rime, increasing the overall accuracy from 95.97% to 97.14%. CADMEF-ViT, through its cross-attention mechanism, also improves recall and F1 scores for mixed rime and non-icing categories, ultimately achieving an overall accuracy of 97.78%. Experiments demonstrate that multimodal models combining image and meteorological data can significantly improve classification performance.

[0064] like Figure 5As shown, in scenes misclassified as rime by ViT_B_16, MFEM-ViT corrected the prediction by using meteorological data input, identifying it as either rime or mixed rime. CADMEF-ViT, on the other hand, further correlated the image with meteorological data, ultimately classifying it more robustly as the correct category. For icy scenes with blurred boundaries, such as rime photographed at night, due to factors like low light and overexposure, samples that are actually rime may appear visually similar to rime. Similarly, images taken on cloudy or foggy days may appear visually similar to rime even though they are not actually icy. Relying solely on visual information may lead to misclassification due to texture or brightness. However, by incorporating meteorological elements (temperature, humidity), the model can comprehensively consider environmental conditions and correct the results.

[0065] Therefore, CADMEF-ViT further strengthens the correlation between images and meteorological information through a multi-head attention mechanism, enabling samples with blurred boundaries (such as weak rime and mixed rime) to be more robustly distinguished, thereby improving the robustness and accuracy of the overall classification and enhancing the accuracy of identifying icing types on transmission lines.

Claims

1. A method for identifying icing types of multimodal transmission lines based on Transformer architecture, characterized in that, include: Acquire images of icing on power transmission lines and corresponding meteorological data, and perform preprocessing; Image features are extracted using the Vision Transformer model; Meteorological features are extracted using the meteorological feature extraction module; The above image features and meteorological features are fused using a cross-attention mechanism to generate fused features; The classification results of icing type are output based on the fusion features.

2. The method for identifying icing types of multimode transmission lines based on Transformer architecture according to claim 1, characterized in that, The meteorological data includes temperature, humidity, and wind speed; the image features include the shape and texture of the ice floes.

3. The method for identifying icing types of multimodal transmission lines based on Transformer architecture according to claim 1, characterized in that, The preprocessing includes converting meteorological data into two-dimensional numerical vectors.

4. The method for identifying icing types of multimode transmission lines based on Transformer architecture according to claim 1, characterized in that, The meteorological feature extraction module includes a multilayer perceptron, the structure of which includes: The first linear layer maps the input meteorological data to 128 dimensions through linear transformation; The second linear layer maps the 128-dimensional meteorological features to 768 dimensions, consistent with the image feature dimensions.

5. The method for identifying icing types of multimode transmission lines based on Transformer architecture according to claim 4, characterized in that, The multilayer perceptron structure also includes the ReLU activation function, two Dropout layers, and a LayerNorm layer; after processing by the first linear layer or the second linear layer, the data are processed by the ReLU activation function and the Dropout layer.

6. The method for identifying icing types of multimode transmission lines based on Transformer architecture according to claim 1, characterized in that, In the cross-attention mechanism, image features are used as the query, and meteorological features are used as the key and value. Through multi-head attention calculation, fused features are output.

7. The method for identifying icing types of multimodal transmission lines based on Transformer architecture according to claim 6, characterized in that, The multi-head attention calculation includes: splitting Query, Key, and Value into multiple subspaces, and calculating for each subspace as follows: The similarity dot product of Query and Key is performed, and then normalized using the SoftMax function to obtain attention weights. The attention weights are then weighted and summed with the Value to obtain the output of that subspace. The outputs of all subspaces are concatenated together, and the concatenated output is residually connected to the original Query through a Dropout layer. Finally, the output is normalized by the layer.

8. The method for identifying icing types of multimodal transmission lines based on Transformer architecture according to claim 7, characterized in that, The calculation formula for the multi-head attention mechanism is as follows: in, Indicates Query, Indicates the transpose of the key. It is the dimension of the key vector. This represents the attention score. This represents the input value, This represents the output of the attention mechanism.

9. A computer device, characterized in that, The method includes one or more processors, a memory, and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the programs, when executed by the processors, implement the steps of a method for identifying icing types of multimodal transmission lines based on a Transformer architecture as described in any one of claims 1-8.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the method for identifying icing types of multimodal transmission lines based on the Transformer architecture as described in any one of claims 1-8.

Images