Converter station transformer fault prediction method and system based on multi-modal fusion

Abstract

The invention provides a converter station transformer fault prediction method and system based on multi-modal fusion, and relates to the technical field of power equipment state monitoring and fault prediction, and the method comprises the steps: collecting the multi-source state data of a converter station transformer; inputting the instrument state image into a multi-scale convolutional neural network to extract a visual feature vector, inputting a space-time matrix formed by sensor time sequence data into a space-time feature extraction network to extract a time sequence feature vector, and inputting fault text data into a semantic coding model to extract a text feature vector; inputting the visual feature vector, the time sequence feature vector and the text feature vector into an evidence generator, and carrying out dynamic weight fusion on the visual feature vector, the time sequence feature vector and the text feature vector by taking total uncertainty as a weight to obtain a comprehensive feature vector; and mapping the comprehensive feature vector into fault probability distribution of a corresponding fault type, and obtaining a fault prediction result corresponding to the converter station transformer. According to the invention, the accuracy of fault prediction of the converter station transformer can be improved.

Description

Technical Field

[0001] This invention relates to the field of power equipment condition monitoring and fault prediction technology, and in particular to a method and system for predicting transformer faults in converter stations based on multimodal fusion. Background Technology

[0002] As a critical node in high-voltage direct current (HVDC) transmission systems, the reliability of converter station transformers directly impacts the safe and stable operation of the power grid. Traditional fault prediction methods primarily rely on single-type data, such as threshold alarms or statistical models based on time-series signals from sensors like vibration, current, voltage, or temperature; manual inspections and image processing based on equipment appearance images; or experience-based judgments and text analysis based on maintenance logs. These single-modal methods have significant limitations: sensor data is susceptible to noise interference and struggles to capture slow degradation trends; visual inspection is easily affected by environmental factors such as lighting and angle and cannot reflect internal state changes; text records rely on human experience, lack objective quantification, and are difficult to process massive amounts of unstructured data.

[0003] Chinese patent CN117809164A discloses a substation equipment fault detection method and system based on multimodal fusion. The method includes acquiring visible light image sequences, thermal infrared image sequences, and sound data of key equipment locations within a set time period while the substation is in operation. The acquired data is preprocessed to obtain multiple sets of preprocessed, one-to-one corresponding visible light meter images, thermal infrared images showing the maximum temperature region, and audio frame data of key equipment locations. Feature extraction is performed on each set of data, extracting meter reading features, maximum temperature region features, and audio frequency features of key equipment locations. The extracted features are then fused using multimodal features and input into a VGG-based convolutional neural network model to output fault detection results. However, compared to other methods, the above solution focuses on detecting existing faults in its fault handling mechanism, prioritizing current state judgment over future trend prediction, resulting in insufficient early warning capabilities and accuracy in predicting potential faults. Therefore, it is essential to provide a multimodal fusion-based transformer fault prediction method and system for converter stations to improve the accuracy of transformer fault prediction. Summary of the Invention

[0004] In view of this, the present invention proposes a method and system for predicting transformer faults in converter stations based on multimodal fusion.

[0005] This invention provides a method for predicting transformer faults in converter stations based on multimodal fusion, the method comprising: Collect multi-source status data of the converter station transformer, wherein the multi-source status data includes instrument status images, sensor time-series data and fault text data; The instrument status image is input into a multi-scale convolutional neural network to extract visual feature vectors; the spatiotemporal matrix composed of the sensor time series data is input into a spatiotemporal feature extraction network to extract time series feature vectors; and the fault text data is input into a semantic coding model to extract text feature vectors. The visual feature vector, the temporal feature vector, and the text feature vector are input into the evidence generator, and the visual feature vector, the temporal feature vector, and the text feature vector are dynamically weighted and fused with overall uncertainty as the weight to obtain a comprehensive feature vector; The comprehensive feature vector is mapped to the fault probability distribution corresponding to the fault type to obtain the fault prediction result corresponding to the converter station transformer.

[0006] Based on the above technical solutions, preferably, the step of inputting the instrument status image into a multi-scale convolutional neural network to extract visual feature vectors specifically includes: The instrument status images corresponding to the transformers in the converter station are subjected to standardized preprocessing. The preprocessed instrument status image is input into a multi-scale convolutional neural network, wherein the multi-scale convolutional neural network includes three parallel convolutional branches, and the three parallel convolutional branches have convolutional kernels of different sizes. The feature maps output by each parallel convolutional branch are concatenated along the channel dimension and then compressed through a max pooling layer to generate a visual feature vector. Each convolutional branch includes a convolutional layer, a batch normalization layer, a ReLU activation function layer, and a max pooling layer connected in sequence.

[0007] Based on the above technical solutions, preferably, the step of inputting the spatiotemporal matrix composed of the sensor time-series data into the spatiotemporal feature extraction network to extract the time-series feature vector specifically includes: The sensor time series data is converted into a two-dimensional spatiotemporal matrix through a sliding window, wherein the sensor time series data includes current time series data, voltage time series data and temperature time series data; The two-dimensional spatiotemporal matrix is ​​input into the spatiotemporal feature extraction network, wherein the spatiotemporal feature extraction network includes a one-dimensional convolutional layer, a batch normalization layer, a ReLU activation function layer, and a max pooling layer; The sensor temporal feature vector is extracted from the hidden state of the last time step in the spatiotemporal feature extraction network.

[0008] More preferably, the step of inputting the fault text data into a semantic coding model to extract text feature vectors specifically includes: The faulty text data is cleaned to obtain transitional faulty text data; The transition fault text data is segmented into words according to the power industry dictionary to obtain a word sequence, and special markers are added to the beginning and end of the word sequence. The word sequence is mapped to a word embedding vector, and a position encoding vector is generated using a sine-cosine position encoding function. The word embedding vector and the position encoding vector are then integrated to form an input representation. The input representation is input into the semantic coding model, and the semantic features of the text in the input representation are extracted by a multi-layer encoder. In the final coding layer, a hidden state with a special label is selected as the text feature vector.

[0009] More preferably, the acquisition of the comprehensive feature vector specifically includes: The visual feature vector, the temporal feature vector, and the text feature vector are input into the evidence generator to generate a non-negative evidence vector corresponding to the visual feature vector, the temporal feature vector, and the text feature vector; Based on the nonnegative evidence vector, the Dirichlet distribution parameters of the corresponding feature vector are derived, and the overall uncertainty corresponding to the Dirichlet distribution parameters is calculated. Using overall uncertainty as the weight, the visual feature vector, the temporal feature vector, and the text feature vector are weighted and summed respectively to obtain a comprehensive feature vector.

[0010] More preferably, the method further includes: If the visual feature vector, the temporal feature vector, and the text feature vector do not meet the preset modal judgment conditions, the overall uncertainty is automatically set to 1, and the weights of the corresponding feature vectors are reset to zero. The preset modal judgment conditions include the instrument status image missing judgment condition, the sensor temporal data missing judgment condition, and the fault text data missing judgment condition.

[0011] More preferably, the fault types include primary fault types and secondary fault types, wherein the secondary fault types are subordinate fault types of primary fault types. The primary fault types include electrical faults, insulation faults, mechanical faults, thermal faults, and auxiliary system faults. The secondary fault types include DC side faults, AC side faults, winding inter-turn faults, arcing faults, poor lead contact, overvoltage breakdown, insulation overheating, insulation oil deterioration, bushing flashover, internal discharge, aging of oil-paper insulation, mechanical wear, loose fasteners, tap changer jamming, abnormal cooling fan, excessive oil pump vibration, local overheating, abnormal overall temperature rise, excessive hot spot temperature, radiator blockage, abnormal cooling system, abnormal oil level, breather failure, and protection malfunction.

[0012] A second aspect of this application provides a converter station transformer fault prediction system based on multimodal fusion. The converter station transformer fault prediction system includes a data acquisition module, a feature fusion module, and a fault prediction module, wherein... The data acquisition module is used to collect multi-source status data of the converter station transformer, wherein the multi-source status data includes instrument status images, sensor time-series data and fault text data. The feature fusion module is used to input the instrument status image into a multi-scale convolutional neural network to extract visual feature vectors, input the spatiotemporal matrix composed of the sensor time series data into a spatiotemporal feature extraction network to extract time series feature vectors, input the fault text data into a semantic coding model to extract text feature vectors, input the visual feature vectors, the time series feature vectors, and the text feature vectors into an evidence generator, and dynamically weight and fuse the visual feature vectors, the time series feature vectors, and the text feature vectors with overall uncertainty as the weight to obtain a comprehensive feature vector; The fault prediction module is used to map the comprehensive feature vector to the fault probability distribution of the corresponding fault type, so as to obtain the fault prediction result corresponding to the converter station transformer.

[0013] A third aspect of this application provides an electronic device including a processor, a memory, a user interface, and a network interface, wherein the memory is used to store instructions, the user interface and the network interface are used to communicate with other devices, and the processor is used to execute the instructions stored in the memory.

[0014] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the steps of a converter station transformer fault prediction method based on multimodal fusion.

[0015] The present invention provides a method and system for predicting transformer faults in converter stations based on multimodal fusion, which has the following advantages over existing technologies: (1) By integrating visual, temporal and textual data, the transformer operating status can be fully perceived from multiple dimensions, making up for the lack of information from a single data source. Furthermore, the cross-validation of multi-source data effectively reduces the false alarm rate caused by single sensor failure or data anomaly. The multi-scale convolutional neural network can capture image features of different granularities, from local details to global patterns. The spatiotemporal feature extraction network fully explores the correlation between sensor data in time and space dimensions, transforms fault text data into computable semantic features, makes full use of historical fault records and expert knowledge, and the dynamic weighting mechanism based on overall uncertainty can automatically adjust the fusion weight according to the confidence level of each modality data, thereby improving the accuracy and reliability of transformer fault prediction in converter stations.

[0016] (2) By constructing a three-branch parallel multi-scale convolutional neural network and combining standardized preprocessing and hierarchical feature learning mechanisms, it is possible to efficiently extract multi-level visual features from local details to global structure in instrument status images, which significantly improves the perception accuracy and feature expression ability of visual information on transformer operation status. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 A flowchart illustrating a converter station transformer fault prediction method based on multimodal fusion provided by the present invention; Figure 2 This is a schematic diagram of the CNN-LSTM hybrid network provided by the present invention; Figure 3 A schematic diagram illustrating the steps for obtaining a comprehensive feature vector provided by this invention; Figure 4 A schematic diagram illustrating the structure of the joint loss function provided by this invention; Figure 5 The structural block diagram of the converter station transformer fault prediction system based on multimodal dynamic fusion provided by the present invention; Figure 6 This invention provides evidence strength diagrams for each modality for five types of faults; Figure 7 The final failure probability distribution diagram provided for this invention; Figure 8 This is a schematic diagram of the structure of the converter station transformer fault prediction system provided by the present invention; Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention.

[0019] Explanation of reference numerals in the attached diagram: 1. Converter station transformer fault prediction system; 11. Data acquisition module; 12. Feature fusion module; 13. Fault prediction module; 2. Electronic equipment; 21. Processor; 22. Communication bus; 23. User interface; 24. Network interface; 25. Memory. Detailed Implementation

[0020] The technical solutions of the present invention will be clearly and completely described below with reference to the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0021] At the transformer operation site in converter stations, fault monitoring and prediction systems typically combine manual inspection with automated data uploading. Manual monitoring takes two forms: one where maintenance personnel read instrument data on-site without a camera pointing directly at the dial; and another where high-definition cameras are used to remotely monitor readings from one or more dials in real time, allowing direct observation from the control room. Automated monitoring directly uploads time-series data of key parameters such as oil level, oil temperature, hydrogen, and acetylene through a sensor network. These two monitoring methods complement each other in practice: inconsistencies in data can be detected promptly, preventing serious consequences from undetected errors. Typical monitoring data includes oil level, oil temperature, and the concentrations of various gases. Abnormal oil levels pose significant risks: excessively high oil levels are usually caused by internal overheating leading to oil expansion; if not addressed promptly, this can cause oil to overflow from the conservator, resulting in environmental pollution or fire risks. Excessively low oil levels affect insulation and cooling, leading to localized overheating and insulation aging. Furthermore, abnormally high oil temperatures and changes in the concentrations of characteristic gases such as hydrogen and acetylene are direct signs of latent faults such as overheating and discharge, thus requiring continuous monitoring and real-time analysis.

[0022] This invention discloses a method for predicting transformer faults in converter stations based on multimodal fusion, with reference to... Figure 1 The steps of this method include S1 to S4.

[0023] Step S1: Collect multi-source status data of the converter station transformer, including instrument status images, sensor timing data, and fault text data.

[0024] Deployed at the transformer site and control center of the converter station, it forms a distributed monitoring and centralized analysis architecture. The field layer includes various sensors, high-definition cameras, and data acquisition units installed on and around the transformer body to collect real-time time-series data on current, voltage, temperature, oil level, hydrogen concentration, and acetylene concentration, as well as images of instrument panels and equipment appearance. The control center layer receives multimodal data uploaded from the field layer and connects to the operation and maintenance database to store historical fault records. Cross-validation is achieved by dynamically comparing sensor readings with instrument data identified from images. If data inconsistencies are found, an anomaly alarm is triggered, prompting operation and maintenance personnel to intervene and verify, thus avoiding missed or false alarms caused by errors in a single data source or equipment failure, and improving the reliability and practicality of the system in actual industrial environments.

[0025] In this step, instrument status images are acquired, and the current, voltage, and temperature time-series signals of the converter station transformer equipment are simultaneously collected in real time through the sensor network. Fault record text is extracted from the operation and maintenance database. The time-series data is converted into a two-dimensional spatiotemporal matrix using sliding window technology to fuse time delay information. The images are standardized and enhanced to eliminate environmental interference. The text is segmented according to the power professional dictionary and start and end marks are added to adapt to the semantic model input.

[0026] Step S2: Input the instrument status image into a multi-scale convolutional neural network to extract visual feature vectors, input the spatiotemporal matrix composed of sensor time series data into a spatiotemporal feature extraction network to extract temporal feature vectors, and input the fault text data into a semantic coding model to extract text feature vectors.

[0027] In this step, the instrument status image is input into the visual feature extraction network, which employs a multi-scale convolutional structure to extract equipment structural anomalies and thermal distribution features in a hierarchical manner, outputting a visual feature vector. The two-dimensional matrix generated from the sensor time-series data is input into the spatiotemporal feature extraction network, which captures local patterns through convolution operations and models long-term dependencies through recurrent memory units to output a feature vector representing the evolution trend of the equipment status. The text data is input into a semantic encoding model based on the Transformer architecture to parse the semantic associations and contextual information in the fault description and generate a text feature vector.

[0028] Image visual feature vectors, sensor temporal feature vectors, and text feature vectors are extracted from the above processing. The visual feature vectors are extracted through a global average pooling layer before the final fully connected layer of a multi-scale convolutional neural network to compress spatial dimensions and retain salient features. The sensor temporal feature vectors are obtained from the hidden states of the final LSTM layer of a spatiotemporal feature extraction network (CNN-LSTM network) to characterize the long-term evolution trend of the device state. The text feature vectors are extracted by extracting the hidden states corresponding to the [CLS] markers in the output layer of a semantic coding model (BERT model) to capture the global semantic information of the fault description.

[0029] In one example, such as Figure 2 As shown, the instrument status images corresponding to the transformers in the converter station are standardized and preprocessed. The preprocessed instrument status images are then input into a multi-scale convolutional neural network, which includes three parallel convolutional branches with convolutional kernels of different sizes. The feature maps output by each parallel convolutional branch are concatenated along the channel dimension and then compressed through a max pooling layer to generate visual feature vectors. Each convolutional branch includes a convolutional layer, a batch normalization layer, a ReLU activation function layer, and a max pooling layer connected in sequence.

[0030] The acquired transformer equipment instrument images from the converter station undergo standardized preprocessing, including scaling to a fixed size, grayscale normalization, or RGB three-channel separation. Histogram equalization or contrast-limited adaptive histogram equalization is used to suppress illumination variations and environmental noise interference. The preprocessed images are then input into a multi-scale convolutional neural network (CNN), which contains three parallel convolutional branches using 3×3, 5×5, and 7×7 kernels to simultaneously extract local detail features and global structural features. Each convolutional branch includes a convolutional layer, a batch normalization layer, a ReLU activation function layer, and a 2×2 max-pooling layer. The output feature maps from each branch are concatenated along the channel dimension and then compressed using a global average pooling layer to generate visual feature vectors. During training, the cross-entropy loss function is used, and L2 weight decay regularization is introduced to prevent overfitting and improve the model's generalization ability to complex field environments.

[0031] In one example, sensor time-series data is converted into a two-dimensional spatiotemporal matrix through a sliding window. The sensor time-series data includes current time-series data, voltage time-series data, and temperature time-series data. The two-dimensional spatiotemporal matrix is ​​then input into a spatiotemporal feature extraction network, which includes a one-dimensional convolutional layer, a batch normalization layer, a ReLU activation function layer, and a max pooling layer. Finally, sensor time-series feature vectors are extracted from the hidden state of the last time step in the spatiotemporal feature extraction network.

[0032] One-dimensional time-series signals acquired by current, voltage, and temperature sensors are converted into a two-dimensional spatiotemporal matrix using a sliding window technique. The window length T is set according to the device response characteristics, and C is the number of sensor channels. This matrix is ​​input into a CNN-LSTM hybrid network. First, local spatiotemporal features are extracted through two one-dimensional convolutional layers. Each convolutional layer is followed by batch normalization, ReLU activation, and max pooling. The convolutional output is flattened and then input into a two-layer stacked LSTM network. Each LSTM layer has 128 units, and the forget gate bias is initialized to 1.0 to promote long-range memory. The LSTM units model long-term dependencies through the following gating mechanism:

[0033]

[0034]

[0035]

[0036]

[0037]

[0038] in, f t Represents the forget gate vector. σ ( ) represents the Sigmoid activation function. x t This represents the data at time t. h t-1 express t The hidden state at time -1 W f This represents the input weight matrix of the forget gate. R f The hidden state weight matrix of the forget gate is represented. b f Indicates the forget gate bias. i t Represents the input gate vector. W i This represents the input weight matrix of the input gate. R i This represents the hidden state weight matrix of the input gate. b i This indicates the input gate bias. Indicates the state of candidate cells. φ ( ) represents the activation function for the candidate cell state. W c The input weight matrix represents the candidate cell state. R c The hidden state weight matrix represents the candidate cell state. b f The symbol represents the candidate cell state bias, · represents the dot product operation, and × represents the vector multiplication. C t Indicates the current cell state. O t Represents the output gate vector. W o This represents the input weight matrix of the output gate. R o This represents the hidden state weight matrix of the output gate. b o This indicates the output gate bias.

[0039] Finally, from the hidden state of the last time step h t Extracting sensor time-series feature vectors Dropout is used to alleviate overfitting during training, and the Adam optimizer is used for parameter updates.

[0040] Furthermore, a spatiotemporal feature extraction network (CNN-LSTM network) is used to extract multi-scale spatiotemporal features from sensor time-series data. Its specific composition and data processing flow are as follows: Sliding window processing module: processes one-dimensional timing signals The sample matrix is ​​converted into a two-dimensional sample matrix through a sliding window with a width of d, where n is the length of the observation time, m is the number of observed features, and d represents the time delay step size. The amount of time delay information can be controlled by adjusting the value of d. CNN Feature Extraction Module: Receives two-dimensional sample input and performs convolution operations. The input for the convolution operation is three-dimensional data. Where w, h, and c represent its width, height, and number of channels, respectively. The output is new 3D data. .in, w * and h * It can be calculated using the following formula:

[0041]

[0042] in, F This represents the size of each convolution kernel, S represents the stride of the convolution, and P is the number of zero values. For each convolution operation, zero values ​​are padded with data. In this way, the transformed data is obtained. .

[0043] CNN-LSTM bridge: Rearranges the 3D feature map output by CNN into 2D sequence data, keeping the time dimension unchanged, flattening the feature dimension, and converting it into the input format required by LSTM.

[0044] LSTM timing modeling module: Employs a two-layer stacked LSTM structure, with each layer containing 128 hidden units. Calculations are performed using a gating mechanism, ultimately starting from the hidden state of the last time step. h t Extract the temporal feature vector of the sensor.

[0045] In this embodiment, by constructing a three-branch parallel multi-scale convolutional neural network and combining standardized preprocessing and hierarchical feature learning mechanisms, it is possible to efficiently extract multi-level visual features from local details to global structure in instrument status images, significantly improving the perception accuracy and feature expression capability of visual information on transformer operating status, and providing a richer and more reliable visual feature foundation for subsequent multimodal fusion.

[0046] In one example, fault text data is cleaned to obtain transitional fault text data; the transitional fault text data is segmented into words according to the power industry dictionary to obtain word sequences, and special markers are added to the beginning and end of the word sequences; the word sequences are mapped to word embedding vectors, and a sine-cosine positional encoding function is used to generate positional encoding vectors. The word embedding vectors and positional encoding vectors are integrated to form the input representation; the input representation is input into the semantic encoding model, and the semantic features of the text in the input representation are extracted through a multi-layer encoder. In the final encoding layer, the hidden states with special markers are selected as text feature vectors.

[0047] Data cleaning was performed on the operation and maintenance fault record text to remove irrelevant characters and stop words; word segmentation was performed based on a power industry dictionary, which includes professional terms in the fields of "insulator flashover", "bushing overheating", "tap switch jamming", "arc discharge", "oil chromatography abnormality", "excitation current abnormality", "cooling system fault", "thyristor breakdown", "DC bus fault" and "protection maloperation"; special markers [CLS] and [SEP] were added to the beginning and end of the segmented sequence to identify the sequence boundaries.

[0048] The word sequence is mapped to a high-dimensional vector through a word embedding layer. Simultaneously, a sine-cosine position encoding function is used to generate the position vector. The word embedding vector and the position encoding vector are added together to form the input representation.

[0049] The input representation is fed into a semantic encoding model based on the Transformer architecture. Text semantic features are extracted through a multi-layer encoder, with each layer containing a self-attention mechanism and a feedforward neural network. Residual connections and layer normalization are used to stabilize the training process. The hidden states corresponding to the [CLS] markers in the final encoding layer are then selected. h [CLS] As text feature vectors A two-stage training strategy of pre-training and fine-tuning is adopted. In the pre-training stage, a masked language model and a next sentence prediction task are used, while in the fine-tuning stage, supervised training is carried out using text data of transformer faults in converter stations.

[0050] In this embodiment, efficient spatiotemporal feature learning of multi-sensor time-series data such as current, voltage, and temperature is achieved through sliding window spatiotemporal matrix transformation and one-dimensional convolutional network architecture. It can simultaneously capture the temporal evolution law and the cooperative relationship between sensors, and form a time-series feature vector containing rich historical information by extracting the hidden state of the last time step. This provides a high-quality time-series feature foundation for transformer fault prediction and significantly improves the utilization efficiency and feature expression capability of time-series data.

[0051] Furthermore, the semantic encoding model based on the Transformer architecture (BERT pre-trained model) specifically processes the following steps: During the input representation construction process, the position encoding is generated using a deterministic function. For position pos and dimension i, the calculation formula is as follows:

[0052]

[0053] Where PE represents the position encoding result of Embeddimg, and pos represents the position of the word in the text. The text dimension is represented by 2i, where 2i indicates an even dimension and 2i+1 indicates an odd dimension. Concatenating the position Embeddimg with the word Embeddimg yields the input vector for the BERT model.

[0054] The semantic encoder consists of L layers of stacked Transformer encoders, each layer containing a multi-head self-attention mechanism and a feedforward neural network. The self-attention mechanism calculates attention weights using the query matrix Q, the key matrix K, and the value matrix V, specifically as follows:

[0055] in, d k The key vector dimension is represented by , Attention represents the output matrix of the attention mechanism, and softmax represents the normalization exponential function. H attention heads are used for parallel computation, and the outputs of each head are concatenated and linearly transformed to obtain the multi-subspace semantic representation. The feedforward neural network consists of two fully connected layers and uses the GELU activation function. Each sub-layer is followed by residual connections and layer normalization.

[0056] In the pre-training stage, a masked language model task is used to randomly mask 15% of the input words and predict their original words. At the same time, the next sentence prediction task is used to learn the relationship between sentences. In the fine-tuning stage, a converter station transformer operation and maintenance fault text dataset is used to optimize the model's ability to understand power field professional terms and semantic patterns. Finally, the [CLS] labeled hidden state output by the last encoder layer is taken as the text feature vector.

[0057] In this embodiment, a semantic understanding system specifically adapted to power fault text is constructed by using technologies such as power professional dictionary word segmentation, word embedding and positional encoding fusion, and multi-layer Transformer encoder. It can accurately capture the semantics of professional terms and contextual relationships in fault descriptions, and extract high-quality text feature vectors containing full-text semantic information by extracting the hidden states of specially marked positions. This significantly improves the semantic understanding accuracy and feature expression capability of fault text information, and provides a reliable text semantic foundation for multimodal fusion prediction.

[0058] Step S3: Input the visual feature vector, temporal feature vector, and text feature vector into the evidence generator, and dynamically weight and fuse the visual feature vector, temporal feature vector, and text feature vector with the overall uncertainty as the weight to obtain the comprehensive feature vector.

[0059] like Figure 3 As shown, the three types of feature vectors are input into the evidence generator to generate non-negative evidence vectors for each modality; the Dirichlet distribution parameters of each modality are derived based on the evidence vectors, and their overall uncertainty is calculated; the time-series, visual, and text feature vectors are weighted and summed using (1-uncertainty) as the weight to generate a comprehensive feature vector; when data for a certain modality is missing, its uncertainty is automatically set to 1 and the weight is set to zero to maintain the robustness of the system.

[0060] This step also includes steps S31 to S33.

[0061] Step S31: Input the visual feature vector, temporal feature vector, and text feature vector into the evidence generator to generate non-negative evidence vectors corresponding to the visual feature vector, temporal feature vector, and text feature vector.

[0062] Step S32: Derive the Dirichlet distribution parameters of the corresponding feature vectors based on the nonnegative evidence vectors, and calculate the overall uncertainty corresponding to the Dirichlet distribution parameters.

[0063] Step S33: Using the overall uncertainty as the weight, the visual feature vector, temporal feature vector, and text feature vector are weighted and summed to obtain the comprehensive feature vector.

[0064] Furthermore, if the visual feature vector, temporal feature vector, and text feature vector do not meet the preset modal judgment conditions, the overall uncertainty is automatically set to 1, and the weights of the corresponding feature vectors are reset to zero. The preset modal judgment conditions include the instrument status image missing judgment condition, the sensor temporal data missing judgment condition, and the fault text data missing judgment condition.

[0065] The criteria for determining missing instrument status images are as follows: Based on image sharpness, a sharpness score can be calculated using a method such as the Laplacian method, and a sharpness threshold can be set. If the sharpness score is lower than the set threshold, the image is considered blurry and features cannot be reliably extracted, and is therefore considered to have missing image modalities. Alternatively, the determination can be based on the reliability of the image content. An object detection model (such as YOLO or Faster R-CNN) can be used to determine whether the image contains key instrument or equipment areas. If the key areas are occluded, cropped, or not captured, the image is considered invalid.

[0066] Sensor time-series data missing determination criteria: The proportion of missing data points within the sliding window can be counted. If it exceeds a set threshold (e.g., 20%), it is determined that the data for that period is missing. If the sensor outputs a constant value, exceeds the range, has a NaN value, or exhibits abnormal jumps, it can be considered that the data is unreliable.

[0067] Fault text data missing determination criteria: can be determined based on text length and content validity. If the text length is too short (e.g., less than 5 words) or does not contain any power industry terminology, it is considered invalid text. It can also be determined based on the integrity of the text structure. If key fields (such as fault type, time, equipment number, etc.) are missing in the text, it can be determined as incomplete.

[0068] Furthermore, the fault types include primary fault types and secondary fault types. Secondary fault types are subordinate fault types of primary fault types. Primary fault types include electrical faults, insulation faults, mechanical faults, thermal faults, and auxiliary system faults. Secondary fault types include DC side faults, AC side faults, winding inter-turn faults, arc faults, poor lead contact, overvoltage breakdown, insulation overheating, insulation oil deterioration, bushing flashover, internal discharge, aging of oil-paper insulation, mechanical wear, loose fasteners, tap changer jamming, abnormal cooling fan, excessive oil pump vibration, local overheating, abnormal overall temperature rise, excessive hot spot temperature, radiator blockage, abnormal cooling system, abnormal oil level, breather failure, and protection malfunction.

[0069] First, faults are classified according to their nature into primary categories: electrical faults, insulation faults, mechanical faults, thermal faults, and auxiliary system faults. Then, they are classified according to specific fault types into secondary categories: electrical faults include DC-side faults, AC-side faults, winding inter-turn faults, arcing faults, poor lead contact, and overvoltage breakdown; insulation faults include insulation overheating, insulating oil deterioration, bushing flashover, internal discharge, and oil-paper insulation aging; mechanical faults include mechanical wear, loose fasteners, tap changer jamming, abnormal cooling fan operation, and excessive oil pump vibration; thermal faults include localized overheating, abnormal overall temperature rise, excessive hot spot temperature, and radiator blockage; and auxiliary system faults include cooling system abnormalities, abnormal oil level, breather failure, and protection malfunction. These fault categories are included in the calculation of the overall uncertainty for each mode.

[0070] Step S4: Map the comprehensive feature vector to the fault probability distribution of the corresponding fault type to obtain the fault prediction result of the converter station transformer.

[0071] In this step, the comprehensive feature vector is mapped to a probability distribution of fault types via a fully connected layer. Fault types include, but are not limited to, DC-side short circuit, insulation overheating, mechanical wear, arc fault, and cooling system anomaly. A joint loss function is used to implement multi-objective optimization, simultaneously optimizing classification error, confidence assessment error, and feature sparsity: minimizing classification error improves fault identification accuracy, constraining confidence assessment bias enhances modal reliability perception, and introducing sparse regularization terms compresses redundant feature dimensions. During inference, if a modality continuously outputs low-confidence predictions, its weight in the fusion is automatically reduced; if there is dimensional redundancy in the feature vector, sparsification is triggered to improve computational efficiency and generalization ability.

[0072] In this embodiment, by fusing data from three different modalities—visual, temporal, and textual—the transformer's operating status can be comprehensively perceived from multiple dimensions, compensating for the information gaps caused by a single data source. Furthermore, the cross-validation of multi-source data effectively reduces the false alarm rate caused by single sensor failures or data anomalies. The multi-scale convolutional neural network can capture image features of different granularities, from local details to global patterns. The spatiotemporal feature extraction network fully explores the correlation between sensor data in time and space dimensions, transforming fault text data into computable semantic features. It makes full use of historical fault records and expert knowledge. At the same time, the dynamic weighting mechanism based on overall uncertainty can automatically adjust the fusion weights according to the confidence level of each modal data, thereby improving the accuracy and reliability of transformer fault prediction in converter stations.

[0073] In one example, such as Figure 4 As shown, this scheme also includes multi-objective optimization using a joint loss function. The specific structure and calculation method of multi-objective optimization using the joint loss function are as follows:

[0074] Joint loss function Cross-entropy loss from multimodal fusion classification Confidence assessment mean square error loss Single-modal classification cross-entropy loss and feature sparsity regularization loss It consists of four weighted parts, expressed as follows:

[0075] in, The cross-entropy loss, representing the classification result after multimodal fusion and the true label, is used to optimize the overall classification accuracy. The expression for the multimodal fusion classification cross-entropy loss is:

[0076] in, N Indicates the number of samples. KIndicates the number of categories. Indicates sample i The true label, Indicates the model predicts samples i Category k The probability of.

[0077] True confidence for each modality With network prediction confidence The mean squared error loss between the two is used to calibrate modal reliability perception. The expression for the mean squared error loss in confidence assessment is:

[0078] in, M Indicates the number of modes and the true confidence level. It is calculated from the matching accuracy between the modality classification result and the real label.

[0079] This is the sum of the cross-entropy losses of each modality-independent classifier, used to ensure the representational ability of each feature extraction network.

[0080] For the feature sparsity regularization term, we adopt L 1-norm constrained eigenvectors v The sparsity of the feature sparsity regularization loss is expressed as follows:

[0081] Feature sparsity regularization loss This is used to compress redundant feature dimensions and improve the model's generalization ability. j For feature vectors v The first in j Each dimension, traversing all dimensions. L 1. Normal form summation.

[0082] To introduce a consistent clustering loss, we ensure that the features of the same sample remain consistent in the semantic space across different modalities:

[0083] Here, softmax represents a mathematical function that takes an arbitrary vector of real numbers and transforms (normalizes) it into a probability distribution. All elements of this probability distribution are between 0 and 1, and the sum of all elements is 1. The overall meaning is based on the first i The first sample mFeatures of each modality, and the probability distribution of fault categories calculated by the model; KL stands for Kullback-Leibler Divergence, which is used here to calculate the difference in "pseudo-classification probabilities" between different modalities: Indicates the first i The sample at the th m Predicted probability distribution under each modality Indicates the first i The sample at the th m Target probability distribution under each modality Indicates from distribution To distribution The Kullback-Leibler divergence, used to measure the difference between the two, quantifies the difference between the two. To approximate The amount of information lost in time; α, β, γ, and δ are all hyperparameters used to balance the weights of confidence assessment loss, single-modality classification loss, and sparse regularization loss in the total loss, respectively.

[0084] This embodiment presents a specific implementation process for a converter station transformer fault prediction method based on multimodal fusion and joint deep learning.

[0085] At the transformer site in the converter station, the system collects real-time operational data such as current, voltage, oil temperature, oil level, and concentrations of various characteristic gases (e.g., hydrogen, acetylene) through a sensor network deployed on the equipment itself. It also captures images of the equipment's instrument panel and external condition via high-definition cameras, replacing or assisting manual on-site meter readings and remote video monitoring. Furthermore, it extracts textual data such as historical fault records and inspection reports from the maintenance database. By simultaneously collecting sensor readings and instrument images, the system lays the foundation for subsequent cross-validation. For example, when the oil temperature data reported by the sensors differs from the instrument readings identified by the image, a data reliability assessment can be triggered.

[0086] Sliding window processing is performed on time-series data to convert it into a two-dimensional spatiotemporal matrix to fuse time delay features; image data is subjected to size normalization, grayscale conversion, and histogram equalization to eliminate environmental interference such as illumination changes and angle shifts; text data is segmented into words according to the constructed power industry dictionary, and special markers [CLS] and [SEP] are added to the beginning and end of the sequence.

[0087] The preprocessed image data is input into a multi-scale CNN network (containing three parallel convolutional branches: 3x3, 5x5, and 7x7; the outputs are concatenated and then passed through a global average pooling layer) to extract visual feature vectors. The temporal data matrix is ​​input into a CNN-LSTM hybrid network, which consists of a one-dimensional CNN layer (32 filters, 5 kernels) and an LSTM layer (128 units). Temporal feature vectors are extracted from the hidden states at the final time step. The text sequence is input into the pre-trained BERT model, and the output corresponding to the last layer [CLS] label is taken as the text feature vector. .

[0088] Please see the appendix Figure 3 First, establish a working memory buffer. M buffer Temporarily store features of each modality M buffer ={ v img , v sensor , v text}. Feature vectors for each modality v m Evidence vectors are generated using a fully connected layer and the Softplus activation function. e m : e m =log(1+exp( W m × v m + b m )) in, W m Indicates the first m Each modality has its own unique, trainable weight matrix, whose main function is to perform linear transformations. b m This represents the bias vector for the m-th mode.

[0089] Vector of Evidence e m Used for parameterizing Dirichlet distribution ,in , indicating the first m The Dirichlet distribution concentration parameter vector for each modality. This distribution represents the confidence level of the classification result for that modality, where... p m Let K represent a K-dimensional random vector, which indicates that only from the th... m Based on data from multiple modalities, what is the true probability that the sample belongs to each fault category? This is an uncertain proposition.

[0090] Overall uncertainty for each modeu m and confidence levels for each category b m,k The calculation is as follows:

[0091]

[0092] in, K Indicates the number of fault categories. b m,k Indicates the first m The modality pair of the first k Confidence level for each fault category e m,k Indicates the first m The modality is the first k The strength of the "evidence" generated by each fault category. Overall uncertainty. u m The lower the value, the more reliable the mode.

[0093] Multimodal evidence is fused using a simplified DS evidence theory. The joint confidence level after fusion is then determined. b k and overall uncertainty u Represented as:

[0094]

[0095] Final fused feature vector v fusion It is obtained by weighting and summing the features of each modality according to their confidence weights:

[0096] If data for a certain modality is missing or has extremely high uncertainty, its weights will be automatically reset to zero, and the system can still make reliable predictions based on the remaining modalities.

[0097] fuse feature vectors v fusio Input the final multimodal classifier (fully connected layer), and output the probability distribution of each type of fault. Model training uses a joint loss function (such as...). Figure 4 (As shown) Perform multi-objective optimization, the loss function is defined as:

[0098] in, This represents the classification cross-entropy loss after multimodal fusion. This represents the mean squared error loss between the predicted and actual values ​​for each modality confidence level. This represents the classification loss for each single modality. Represents the fusion of feature vectors L 1. Sparse regularization term, This indicates the introduction of a consistent clustering loss, where α, β, γ, and δ are all hyperparameters, and all network parameters are updated synchronously through backpropagation.

[0099] Validate model performance using a reserved test set, analyzing fault identification accuracy, recall, and confusion matrix. Adjust model hyperparameters (such as learning rate and loss weights) or data preprocessing strategies based on the validation results, and continuously optimize the model using historical data.

[0100] Please see Figure 5 This embodiment provides a transformer fault prediction framework for converter stations, including a data acquisition and preprocessing module, a multimodal feature extraction module, a reliable dynamic fusion module, and a joint optimization module. The data acquisition and preprocessing module is responsible for acquiring raw data from sensors, cameras, and databases, and completing the aforementioned preprocessing operations.

[0101] The multimodal feature extraction module includes a visual feature extraction submodule (multi-scale CNN), a temporal feature extraction submodule (CNN-LSTM), and a text feature extraction submodule (BERT), which are used to extract high-level features from various modal data.

[0102] Trustworthy dynamic fusion module: includes uncertainty assessment unit, evidence fusion unit and dynamic weighting unit, responsible for calculating weights, fusion features and generating final fault classification results.

[0103] Joint optimization module: responsible for calculating the joint loss function and adjusting the parameters of the entire network through optimization algorithms.

[0104] The modules are connected sequentially and work together to automatically predict transformer faults in the converter station. The structure, parameters, and training methods of the multi-scale CNN network, the CNN-LSTM hybrid network, and the BERT model have been explained in the preceding steps and will not be repeated here. The dynamic fusion strategy... Figure 3 The uncertainty weighting mechanism shown is based on using the performance of single-modal classification as feedback to dynamically adjust its contribution to the final decision.

[0105] The joint loss function unifies multiple optimization objectives, such as classification accuracy, modality reliability perception, and feature simplicity, within a single framework through weighted summation. It then performs end-to-end optimization using gradient descent, thereby comprehensively improving the overall performance of the model.

[0106] In one example, the following data was monitored from a transformer at a converter station: The sensor time-series data includes: oil temperature 0.85 (normal value 0.2-0.7), hydrogen concentration 0.72 (normal value 0.1-0.5), acetylene concentration 0.68 (normal value 0.05-0.3), and oil level 0.35 (normal value 0.4-0.8); the visual feature vector (first 5 dimensions) extracted by multi-scale CNN is: [0.12, 0.45, 0.78, 0.23, 0.91] (high values ​​indicate the detection of obvious heat distribution anomalies); the BERT semantic feature vector (first 5 dimensions) is: [0.67, 0.34, 0.89, 0.56, 0.12].

[0107] Based on the input data above, the evidence generator outputs the following modules regarding the strength of evidence for the five types of faults: Figure 6 As shown. Then, an uncertainty assessment is performed to determine the uncertainty of the time series data. =0.15, uncertainty of visual data =0.25, uncertainty of text data =0.10. Meanwhile, the actual weights of each modality in the final decision, after normalization, are: 0.327 for time series data, 0.288 for visual data, and 0.385 for text data.

[0108] From the above data, we can obtain the following: Figure 7 The final failure probability distribution shown is obtained through Figure 6 and attached Figure 7 The following conclusions can be drawn: insulation overheating (42.3%) is the most likely type of fault; text data provides the strongest evidence for insulation overheating (3.2) and has the lowest uncertainty (0.10); time series data and text data are highly consistent in their judgment of insulation overheating; the system mainly relies on text semantic features (weight 38.5%) and time series features (weight 32.7%).

[0109] Based on the above method, this application discloses a converter station transformer fault prediction system based on multimodal fusion, referencing... Figure 8 The converter station transformer fault prediction system 1 includes a data acquisition module 11, a feature fusion module 12, and a fault prediction module 13, wherein... The data acquisition module 11 is used to acquire multi-source status data of the transformer in the converter station. The multi-source status data includes instrument status images, sensor time-series data and fault text data. The feature fusion module 12 is used to input the instrument status image into a multi-scale convolutional neural network to extract visual feature vectors, input the spatiotemporal matrix composed of sensor time series data into a spatiotemporal feature extraction network to extract time series feature vectors, input fault text data into a semantic coding model to extract text feature vectors, input the visual feature vector, time series feature vector, and text feature vector into an evidence generator, and dynamically weight and fuse the visual feature vector, time series feature vector, and text feature vector with overall uncertainty as the weight to obtain a comprehensive feature vector; The fault prediction module 13 is used to map the comprehensive feature vector to the fault probability distribution of the corresponding fault type in order to obtain the fault prediction result of the converter station transformer.

[0110] In one example, the feature fusion module 12 is used to perform standardized preprocessing on the instrument status image corresponding to the converter station transformer; the preprocessed instrument status image is input into a multi-scale convolutional neural network, wherein the multi-scale convolutional neural network includes three parallel convolutional branches, and the three parallel convolutional branches have convolutional kernels of different sizes; the feature maps output by each parallel convolutional branch are concatenated in the channel dimension, and the spatial dimension is compressed through a max pooling layer to generate a visual feature vector, wherein each convolutional branch includes a convolutional layer, a batch normalization layer, a ReLU activation function layer and a max pooling layer connected in sequence.

[0111] In one example, the feature fusion module 12 is used to convert sensor time-series data into a two-dimensional spatiotemporal matrix through a sliding window, wherein the sensor time-series data includes current time-series data, voltage time-series data, and temperature time-series data; input the two-dimensional spatiotemporal matrix into a spatiotemporal feature extraction network, wherein the spatiotemporal feature extraction network includes a one-dimensional convolutional layer, a batch normalization layer, a ReLU activation function layer, and a max pooling layer; and extract sensor time-series feature vectors from the hidden state of the last time step in the spatiotemporal feature extraction network.

[0112] In one example, the feature fusion module 12 is used to clean the fault text data to obtain transition fault text data; the transition fault text data is segmented into words according to the power professional dictionary to obtain word sequence, and special markers are added to the beginning and end of the word sequence; the word sequence is mapped to word embedding vector, and a sine-cosine position encoding function is used to generate position encoding vector; the word embedding vector and the position encoding vector are integrated to form the input representation; the input representation is input into the semantic encoding model, and the semantic features of the text in the input representation are extracted through a multi-layer encoder, and the hidden state with special markers is selected as the text feature vector in the final encoding layer.

[0113] In one example, the feature fusion module 12 is used to input visual feature vectors, temporal feature vectors, and text feature vectors into the evidence generator to generate non-negative evidence vectors corresponding to the visual feature vectors, temporal feature vectors, and text feature vectors; derive the Dirichlet distribution parameters of the corresponding feature vectors based on the non-negative evidence vectors, and calculate the overall uncertainty corresponding to the Dirichlet distribution parameters; use the overall uncertainty as a weight to perform weighted summation on the visual feature vectors, temporal feature vectors, and text feature vectors respectively to obtain the comprehensive feature vector.

[0114] In one example, the method also includes: If the visual feature vector, temporal feature vector, and text feature vector do not meet the preset modal judgment conditions, the overall uncertainty will be automatically set to 1, and the weights of the corresponding feature vectors will be reset to zero. The preset modal judgment conditions include the instrument status image missing judgment condition, the sensor temporal data missing judgment condition, and the fault text data missing judgment condition.

[0115] In one example, the fault types include primary fault types and secondary fault types. The secondary fault types are the lower-level fault types of the primary fault types. Primary fault types include electrical faults, insulation faults, mechanical faults, thermal faults, and auxiliary system faults. Secondary fault types include DC side faults, AC side faults, winding inter-turn faults, arcing faults, poor lead contact, overvoltage breakdown, insulation overheating, insulation oil deterioration, bushing flashover, internal discharge, aging of oil paper insulation, mechanical wear, loose fasteners, tap changer jamming, abnormal cooling fan, excessive oil pump vibration, local overheating, abnormal overall temperature rise, excessive hot spot temperature, radiator blockage, abnormal cooling system, abnormal oil level, breather failure, and protection malfunction.

[0116] Please see Figure 9 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 9 As shown, the electronic device 2 may include: at least one processor 21, at least one network interface 24, user interface 23, memory 25, and at least one communication bus 22.

[0117] The communication bus 22 is used to enable communication between these components.

[0118] The user interface 23 may include a display screen and a camera. Optionally, the user interface 23 may also include a standard wired interface and a wireless interface.

[0119] The network interface 24 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0120] The processor 21 may include one or more processing cores. The processor 21 connects to various parts of the server using various interfaces and lines, and performs various server functions and processes data by running or executing instructions, programs, code sets, or instruction sets stored in the memory 25, and by calling data stored in the memory 25. Optionally, the processor 21 may be implemented using at least one hardware form of Digital Signal Processing (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may integrate one or a combination of several of the following: Central Processing Unit (CPU), Graphics Processing Unit (GPU), and modem. The CPU primarily handles the operating system, user interface, and applications; the GPU is responsible for rendering and drawing the content required for display; and the modem handles wireless communication. It is understood that the modem may also not be integrated into the processor 21 and may be implemented as a separate chip.

[0121] The memory 25 may include random access memory (RAM) or read-only memory. Optionally, the memory 25 may include non-transitory computer-readable storage medium. The memory 25 can be used to store instructions, programs, code, code sets, or instruction sets. The memory 25 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as touch function, sound playback function, image playback function, etc.), instructions for implementing the above-described method embodiments, etc.; the data storage area may store data involved in the above-described method embodiments, etc. Optionally, the memory 25 may also be at least one storage device located remotely from the aforementioned processor 21. Figure 9 As shown, the memory 25, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and an application program for a converter station transformer fault prediction method based on multimodal fusion.

[0122] exist Figure 9In the electronic device 2 shown, the user interface 23 is mainly used to provide an input interface for the user and obtain the user input data; while the processor 21 can be used to call the application program stored in the memory 25, which is a method for predicting transformer faults in converter stations based on multimodal fusion. When executed by one or more processors, the electronic device executes one or more methods as described in the above embodiments.

[0123] A computer-readable storage medium storing instructions that, when executed by one or more processors, cause a computer to perform one or more methods as described in the embodiments above.

[0124] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, as some steps may be performed in other orders or simultaneously according to this application. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0125] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0126] In the several embodiments provided in this application, it should be understood that the disclosed apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the shown or discussed mutual couplings or direct couplings or communication connections may be through some service interfaces; indirect couplings or communication connections between apparatuses or units may be electrical or other forms.

[0127] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0128] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0129] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage device (CMD). Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned memory includes various media capable of storing program code, such as USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0130] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for predicting transformer faults in converter stations based on multimodal fusion, characterized in that, The method includes: Collect multi-source status data of the converter station transformer, wherein the multi-source status data includes instrument status images, sensor time-series data and fault text data; The instrument status image is input into a multi-scale convolutional neural network to extract visual feature vectors; the spatiotemporal matrix composed of the sensor time series data is input into a spatiotemporal feature extraction network to extract time series feature vectors; and the fault text data is input into a semantic coding model to extract text feature vectors. The visual feature vector, the temporal feature vector, and the text feature vector are input into the evidence generator, and the visual feature vector, the temporal feature vector, and the text feature vector are dynamically weighted and fused with overall uncertainty as the weight to obtain a comprehensive feature vector; The comprehensive feature vector is mapped to the fault probability distribution corresponding to the fault type to obtain the fault prediction result corresponding to the converter station transformer.

2. The method for predicting transformer faults in a converter station based on multimodal fusion as described in claim 1, characterized in that, The step of inputting the instrument status image into a multi-scale convolutional neural network to extract visual feature vectors specifically includes: The instrument status images corresponding to the transformers in the converter station are subjected to standardized preprocessing. The preprocessed instrument status image is input into a multi-scale convolutional neural network, wherein the multi-scale convolutional neural network includes three parallel convolutional branches, and the three parallel convolutional branches have convolutional kernels of different sizes. The feature maps output by each parallel convolutional branch are concatenated along the channel dimension and then compressed through a max pooling layer to generate a visual feature vector. Each convolutional branch includes a convolutional layer, a batch normalization layer, a ReLU activation function layer, and a max pooling layer connected in sequence.

3. The method for predicting transformer faults in converter stations based on multimodal fusion as described in claim 1, characterized in that, The step of inputting the spatiotemporal matrix composed of the sensor time-series data into the spatiotemporal feature extraction network to extract the time-series feature vector specifically includes: The sensor time series data is converted into a two-dimensional spatiotemporal matrix through a sliding window, wherein the sensor time series data includes current time series data, voltage time series data and temperature time series data; The two-dimensional spatiotemporal matrix is ​​input into the spatiotemporal feature extraction network, wherein the spatiotemporal feature extraction network includes a one-dimensional convolutional layer, a batch normalization layer, a ReLU activation function layer, and a max pooling layer; The sensor temporal feature vector is extracted from the hidden state of the last time step in the spatiotemporal feature extraction network.

4. The method for predicting transformer faults in a converter station based on multimodal fusion as described in claim 1, characterized in that, The step of inputting the fault text data into a semantic coding model to extract text feature vectors specifically includes: The faulty text data is cleaned to obtain transitional faulty text data; The transition fault text data is segmented into words according to the power industry dictionary to obtain a word sequence, and special markers are added to the beginning and end of the word sequence. The word sequence is mapped to a word embedding vector, and a position encoding vector is generated using a sine-cosine position encoding function. The word embedding vector and the position encoding vector are then integrated to form an input representation. The input representation is input into the semantic coding model, and the text semantic features in the input representation are extracted by a multi-layer encoder. In the final coding layer, a hidden state with a special label is selected as the text feature vector.

5. The method for predicting transformer faults in converter stations based on multimodal fusion as described in claim 1, characterized in that, The acquisition of the comprehensive feature vector specifically includes: The visual feature vector, the temporal feature vector, and the text feature vector are input into the evidence generator to generate a non-negative evidence vector corresponding to the visual feature vector, the temporal feature vector, and the text feature vector; Based on the nonnegative evidence vector, the Dirichlet distribution parameters of the corresponding feature vector are derived, and the overall uncertainty corresponding to the Dirichlet distribution parameters is calculated. Using overall uncertainty as the weight, the visual feature vector, the temporal feature vector, and the text feature vector are weighted and summed respectively to obtain a comprehensive feature vector.

6. The method for predicting transformer faults in a converter station based on multimodal fusion as described in claim 5, characterized in that, The method further includes: If the visual feature vector, the temporal feature vector, and the text feature vector do not meet the preset modal judgment conditions, the overall uncertainty is automatically set to 1, and the weights of the corresponding feature vectors are reset to zero. The preset modal judgment conditions include the instrument status image missing judgment condition, the sensor temporal data missing judgment condition, and the fault text data missing judgment condition.

7. The method for predicting transformer faults in a converter station based on multimodal fusion as described in claim 1, characterized in that, The fault types include primary fault types and secondary fault types, wherein the secondary fault types are subordinate fault types of primary fault types. The primary fault types include electrical faults, insulation faults, mechanical faults, thermal faults, and auxiliary system faults. The secondary fault types include DC side faults, AC side faults, winding inter-turn faults, arcing faults, poor lead contact, overvoltage breakdown, insulation overheating, insulation oil deterioration, bushing flashover, internal discharge, aging of oil-paper insulation, mechanical wear, loose fasteners, tap changer jamming, abnormal cooling fan, excessive oil pump vibration, local overheating, abnormal overall temperature rise, excessive hot spot temperature, radiator blockage, abnormal cooling system, abnormal oil level, breather failure, and protection malfunction.

8. A transformer fault prediction system for converter stations based on multimodal fusion, characterized in that, The converter station transformer fault prediction system (1) includes a data acquisition module (11), a feature fusion module (12), and a fault prediction module (13), wherein, The data acquisition module (11) is used to acquire multi-source status data of the converter station transformer, wherein the multi-source status data includes instrument status images, sensor timing data and fault text data; The feature fusion module (12) is used to input the instrument status image into a multi-scale convolutional neural network to extract visual feature vectors, input the spatiotemporal matrix composed of the sensor time series data into a spatiotemporal feature extraction network to extract time series feature vectors, input the fault text data into a semantic coding model to extract text feature vectors, input the visual feature vector, the time series feature vector and the text feature vector into an evidence generator, and dynamically weight and fuse the visual feature vector, the time series feature vector and the text feature vector with the overall uncertainty as the weight to obtain a comprehensive feature vector; The fault prediction module (13) is used to map the comprehensive feature vector to the fault probability distribution of the corresponding fault type in order to obtain the fault prediction result corresponding to the converter station transformer.

9. An electronic device, characterized in that, The device includes a processor (21), a memory (25), a user interface (23), and a network interface (24). The memory (25) is used to store instructions. The user interface (23) and the network interface (24) are used to communicate with other devices. The processor (21) is used to execute the instructions stored in the memory (25) to cause the electronic device (2) to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1-7.

Images