Converter valve state monitoring method, device and equipment based on multi-modal data, storage medium and program product

Abstract

The invention relates to the technical field of converter valve state monitoring, provides a converter valve state monitoring method and device based on multi-modal data, equipment, a storage medium and a program product, and can improve the converter valve state monitoring accuracy. The method comprises the following steps: performing noise reduction and time alignment processing on multi-modal initial data of a converter valve to obtain multi-modal target data; through a convolutional neural network and a long-short-term memory network of a converter valve state monitoring model, features of the electrical modal target data and the environmental modal target data are extracted respectively, and electrical modal features and environmental modal features are obtained; processing the electrical modal characteristics and the environmental modal characteristics through a multi-head attention unit of a converter valve state monitoring model to obtain a multi-head attention result; integrating the electrical modal characteristics, the environment modal characteristics and the multi-head attention result to obtain residual connection characteristics; and according to the residual connection characteristics and a state prediction unit of the converter valve state monitoring model, obtaining a converter valve state prediction result.

Description

Technical Field

[0001] The present application relates to the technical field of converter valve status monitoring, and in particular to a converter valve status monitoring method, apparatus, computer equipment, storage medium, and computer program product based on multimodal data. Background Art

[0002] As a key component of the power system, the working status of the converter valve is directly related to the stable operation, power quality and power supply reliability of the power system. Therefore, it is necessary to monitor the status of the converter valve.

[0003] In traditional technology, single-mode data of the converter valve is usually used for state prediction. Single-mode data only reflects information in a single dimension of the converter valve, resulting in limited accuracy of state monitoring. Summary of the Invention

[0004] Based on this, it is necessary to provide a method, device, computer equipment, storage medium and computer program product for monitoring the status of a converter valve based on multimodal data to address the above technical problems.

[0005] The present application provides a method for monitoring the status of a converter valve based on multimodal data, the method comprising:

[0006] Perform noise reduction and time alignment on the multi-modal initial data of the converter valve to obtain multi-modal target data;

[0007] Using a convolutional neural network and a long short-term memory network in a pre-built converter valve state monitoring model, feature extraction processing is performed on the electrical modal target data and the environmental modal target data in the multimodal target data to obtain electrical modal features and environmental modal features;

[0008] Processing the electrical modal features and the environmental modal features through a multi-head attention unit in the converter valve state monitoring model to obtain a multi-head attention result;

[0009] Combining the electrical modal features, environmental modal features and multi-head attention results to obtain residual connection features;

[0010] A state prediction result of the converter valve is obtained according to the residual connection feature and the state prediction unit in the converter valve state monitoring model.

[0011] In one embodiment, noise reduction and time alignment are performed on the multimodal initial data of the converter valve to obtain multimodal target data, including:

[0012] Using a sliding average filter, high-frequency noise is removed from the multi-modal initial data of the converter valve to obtain multi-modal noise-reduced data.

[0013] Performing time alignment processing on the multimodal noise reduction data using a dynamic time warping algorithm to obtain multimodal aligned data;

[0014] Missing values and outlier processing are performed on the multimodal aligned data to obtain multimodal target data.

[0015] In one embodiment, processing missing values and outliers on the multimodal alignment data to obtain multimodal target data includes:

[0016] Processing missing values and outliers on the multimodal aligned data to obtain multimodal processed data;

[0017] The multimodal processed data is scaled uniformly to obtain multimodal target data.

[0018] In one embodiment, the electrical modal features, environmental modal features, and multi-head attention results are integrated to obtain residual connection features, including:

[0019] Adding the electrical modal features, environmental modal features, and multi-head attention results feature by feature;

[0020] According to the result of adding each feature, the residual connection feature is obtained.

[0021] In one embodiment, obtaining a state prediction result of the converter valve according to the residual connection feature and a state prediction unit in the converter valve state monitoring model includes:

[0022] Calculating the mean and variance of the residual connection features in the dimension of the feature vector;

[0023] Normalizing the residual connection features according to the mean and variance to obtain normalized features;

[0024] A state prediction result of the converter valve is obtained according to the normalized features and the state prediction unit in the converter valve state monitoring model.

[0025] In one embodiment, obtaining a state prediction result of the converter valve according to the normalized feature and a state prediction unit in the converter valve state monitoring model includes:

[0026] Scaling and translating the normalized features to obtain fused features;

[0027] The fusion feature is input into the state prediction unit in the converter valve state monitoring model to obtain the state prediction result of the converter valve.

[0028] The present application provides a converter valve state monitoring device based on multimodal data, the device comprising:

[0029] A data preprocessing module is used to perform noise reduction and time alignment on the multi-modal initial data of the converter valve to obtain multi-modal target data;

[0030] a feature extraction module for performing feature extraction processing on the electrical modal target data and the environmental modal target data in the multimodal target data using a convolutional neural network and a long short-term memory network in a pre-built converter valve state monitoring model, respectively, to obtain electrical modal features and environmental modal features;

[0031] an attention processing module, configured to process the electrical modal features and the environmental modal features through a multi-head attention unit in the converter valve state monitoring model to obtain a multi-head attention result;

[0032] A residual connection processing module, configured to synthesize the electrical modal features, the environmental modal features, and the multi-head attention results to obtain residual connection features;

[0033] The result acquisition module is used to obtain the state prediction result of the converter valve according to the residual connection feature and the state prediction unit in the converter valve state monitoring model.

[0034] The present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the above method.

[0035] The present application provides a computer-readable storage medium having a computer program stored thereon, and the computer program is used by a processor to execute the above method.

[0036] The present application provides a computer program product having a computer program stored thereon, wherein the computer program is used by a processor to execute the above method.

[0037] The above-mentioned converter valve state monitoring method, device, computer equipment, storage medium and computer program product based on multimodal data perform noise reduction and time alignment processing on the multimodal initial data of the converter valve to obtain multimodal target data; through the convolutional neural network and long short-term memory network in the pre-constructed converter valve state monitoring model, feature extraction processing is performed on the electrical modal target data and environmental modal target data in the multimodal target data respectively to obtain electrical modal features and environmental modal features; through the multi-head attention unit in the converter valve state monitoring model, the electrical modal features and environmental modal features are processed to obtain multi-head attention results; the electrical modal features, environmental modal features and multi-head attention results are integrated to obtain residual connection features; based on the residual connection features and the state prediction unit in the converter valve state monitoring model, the state prediction result of the converter valve is obtained. The solution provided by the present application is to perform state prediction based on the multimodal data of the converter valve, which can improve the accuracy of state monitoring. In addition, considering that the multimodal data of the converter valve are collected by different sensors, and the collection frequency, accuracy and format of different sensors vary greatly, the present application performs noise reduction and time alignment on the multimodal initial data to obtain multimodal target data, so as to avoid the difference in sensor collection affecting the accuracy of the state prediction result. In addition, the present application uses the convolutional neural network and the long short-term memory network in the converter valve state monitoring model to respectively process the electrical modal target data and the environmental modal target data in the multimodal target data. Feature extraction and processing are performed on the data to obtain electrical modal features and environmental modal features, which can effectively mine the structural information of each modality in the time and space dimensions, and further improve the accuracy of the state prediction results; through the multi-head attention unit in the converter valve state monitoring model, the electrical modal features and environmental modal features are processed, which helps to capture the implicit correlation between modalities and realize deep semantic interaction between different modal data. The residual connection features are obtained by integrating the multi-head attention results, electrical modal features and environmental modal features. Combined with the state prediction unit in the converter valve state monitoring model, a state prediction result of the converter valve with high accuracy can be obtained. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 1 is a flow chart of a method for monitoring a converter valve state based on multimodal data in one embodiment;

[0040] Figure 21 is a schematic diagram of a process for preprocessing multimodal data in one embodiment;

[0041] Figure 3 Schematic diagram of the residual connection process in one embodiment;

[0042] Figure 4 1 is a structural block diagram of a converter valve state monitoring device based on multimodal data in one embodiment;

[0043] Figure 5 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0045] The converter valve status monitoring method based on multimodal data provided in this application can be executed by a computer device, including: Figure 1 Steps shown.

[0046] Step S101 : performing noise reduction and time alignment processing on the multimodal initial data of the converter valve to obtain multimodal target data.

[0047] Multimodal data from converter valves, also known as multi-source data, can include but is not limited to temperature, pressure, voltage, current, vibration, and strain data. Data of different modalities is collected by different sensors. Converter valve multimodal data is not only diverse but also exhibits significant differences in physical meaning, dimensions, sampling frequency, and data distribution. This data heterogeneity hinders the efficient construction of data fusion models and effective data integration.

[0048] This step preprocesses the multimodal data of the converter valve. The preprocessing includes but is not limited to noise reduction and time alignment, which reduces the noise in the multimodal data and aligns the multimodal data in time to obtain multimodal target data.

[0049] In step S102, a convolutional neural network and a long short-term memory network in a pre-built converter valve state monitoring model are used to perform feature extraction processing on the electrical modal target data and the environmental modal target data in the multimodal target data, respectively, to obtain electrical modal features and environmental modal features.

[0050] After obtaining the multimodal target data, the multimodal target data can be classified to obtain electrical modal target data and environmental modal target data. The electrical modal target data includes current data and voltage data, and the environmental modal target data includes temperature data and vibration data.

[0051] In order to effectively mine the structural information of each mode in the time and space dimensions, the converter valve state monitoring model pre-constructed in this application has a dual-branch feature extraction framework. The converter valve state monitoring model includes a convolutional neural network and a long short-term memory network; the electrical modal target data is input into the convolutional neural network to obtain the electrical modal features; the environmental modal features are input into the long short-term memory network to obtain the environmental modal features.

[0052] In step S103, the electrical modal features and the environmental modal features are processed by the multi-head attention unit in the converter valve state monitoring model to obtain a multi-head attention result.

[0053] The electrical modal features obtained in step S102 may include, but are not limited to, current and voltage features. The environmental modal features obtained in step S102 may include, but are not limited to, temperature and vibration features. In this case, the current, voltage, temperature, and vibration features may be input into the multi-head attention unit in the converter valve state monitoring model, and the output of the multi-head attention unit may be used as the multi-head attention result.

[0054] The multi-head attention unit can adopt the cross-modal self-attention mechanism to fully capture the implicit correlation information between the modalities. , voltage characteristics , temperature characteristics and vibration characteristics After inputting the multi-head attention unit, the multi-head attention unit uses the cross-modal self-attention mechanism to achieve deep semantic interaction between different signal sources, thereby obtaining the multi-head attention result; the calculation formula involved in the multi-head attention unit is as follows:

[0055] , ,

[0056] ,

[0057] in, ; is the dimension of the key matrix, which is used to scale the attention score to prevent the dot product result from being too large, causing the gradient of the softmax function (normalized exponential function) to disappear.

[0058] is a learnable weight matrix used to transform the current features Linear transformation to query vector . By current characteristics Weight matrix The query vector obtained by linear transformation.

[0059] Is a learnable weight matrix used to transform the temperature feature Linear transformation into key vector . By temperature characteristics Weight matrix The key vector obtained by linear transformation.

[0060] By temperature characteristics Weight matrix The value vector obtained by linear transformation participates in the attention weight calculation.

[0061] Indicates that according to the query vector and key vector The calculated attention weight is obtained by This formula is calculated to reflect the degree of correlation between different parts of the current characteristics and temperature characteristics, thereby achieving deep semantic interaction between different signal sources.

[0062] In the multi-head attention result, different heads independently calculate the cross-modal attention weights. The specific process is as follows:

[0063] First perform linear transformation: for each modal feature, such as current feature , temperature characteristics Etc., define a set of learnable weight matrices for each head. Assuming there are n heads, for the input features, perform linear transformation on each head separately. For example, for the current feature , through the i-th head 、 、 The three weight matrices are transformed to obtain 、 、 . is the learnable weight matrix of the i-th head, which is used to transform the temperature feature into the current feature. Linear transformation to query vector . is the learnable weight matrix of the i-th head, which is used to transform the temperature feature into the current feature. Linear transformation to key vector . is the learnable weight matrix of the i-th head, which is used to transform the temperature feature into the current feature. Linear transformation to value vector Similarly, for temperature characteristics , perform linear transformation on the i-th head to obtain the corresponding query vector , key vector Sum vector .

[0064] After the linear transformation, the attention weight can be calculated: for the i-th head, the cross-modal attention weight is calculated independently. Taking the interaction between current and temperature modalities as an example, the attention weight is calculated according to the following formula; the formula used is:

[0065] ,

[0066] in, is the key matrix dimension corresponding to the i-th head. This process is similar to the calculation of the single-head attention mechanism, except that each head has its own independent parameter matrix, thus independently focusing on different aspects of the correlation information between different modalities.

[0067] Then perform concatenation and linear transformation: concatenate the results calculated by n heads (such as the attention output matrix calculated by n different heads) to form a new matrix. Then, perform linear transformation on the matrix Transform the concatenated matrix to obtain the feature representation processed by the multi-head attention mechanism (which can be called multi-head attention result) and is used for subsequent fusion feature calculation and other links.

[0068] Step S104: Integrate the electrical modal features, environmental modal features, and multi-head attention results to obtain residual connection features.

[0069] For example, after obtaining the current characteristic , voltage characteristics , temperature characteristics , vibration characteristics And the multi-head attention results After that, the current characteristics can be , voltage characteristics , temperature characteristics , vibration characteristics And the multi-head attention results Perform comprehensive processing (such as addition processing) to obtain residual connection features.

[0070] Step S105 : obtaining a state prediction result of the converter valve according to the residual connection feature and the state prediction unit in the converter valve state monitoring model.

[0071] After obtaining the residual connection features, a state prediction result for the converter valve can be obtained based on the residual connection features and the state prediction unit in the converter valve state monitoring model. The state prediction result can include a fault classification result for the converter valve, which indicates whether the converter valve is faulty. The state prediction result can also include a temperature prediction result for the converter valve, which reflects the temperature of the converter valve.

[0072] In the above-mentioned converter valve state monitoring method based on multimodal data, the multimodal initial data of the converter valve is subjected to noise reduction and time alignment processing to obtain multimodal target data; the electrical modal target data and environmental modal target data in the multimodal target data are respectively subjected to feature extraction processing through the convolutional neural network and long short-term memory network in the pre-constructed converter valve state monitoring model to obtain electrical modal features and environmental modal features; the electrical modal features and environmental modal features are processed through the multi-head attention unit in the converter valve state monitoring model to obtain multi-head attention results; the electrical modal features, environmental modal features and multi-head attention results are integrated to obtain residual connection features; based on the residual connection features and the state prediction unit in the converter valve state monitoring model, the state prediction result of the converter valve is obtained. The solution provided by the present application is to perform state prediction based on the multimodal data of the converter valve, which can improve the accuracy of state monitoring. In addition, considering that the multimodal data of the converter valve are collected by different sensors, and the collection frequency, accuracy and format of different sensors vary greatly, the present application performs noise reduction and time alignment on the multimodal initial data to obtain multimodal target data, so as to avoid the difference in sensor collection affecting the accuracy of the state prediction result. In addition, the present application uses the convolutional neural network and the long short-term memory network in the converter valve state monitoring model to respectively process the electrical modal target data and the environmental modal target data in the multimodal target data. Feature extraction and processing are performed on the data to obtain electrical modal features and environmental modal features, which can effectively mine the structural information of each modality in the time and space dimensions, and further improve the accuracy of the state prediction results; through the multi-head attention unit in the converter valve state monitoring model, the electrical modal features and environmental modal features are processed, which helps to capture the implicit correlation between modalities and realize deep semantic interaction between different modal data. The residual connection features are obtained by integrating the multi-head attention results, electrical modal features and environmental modal features. Combined with the state prediction unit in the converter valve state monitoring model, a state prediction result of the converter valve with high accuracy can be obtained.

[0073] In one embodiment, the multimodal initial data of the converter valve is subjected to noise reduction and time alignment processing to obtain multimodal target data, including Figure 2 The steps shown are:

[0074] In step S201, a sliding average filter is used to remove high-frequency noise from the multimodal initial data of the converter valve to obtain multimodal noise-reduced data. In step S202, a dynamic time warping algorithm is used to time-align the multimodal noise-reduced data to obtain multimodal aligned data. In step S203, missing value and outlier processing is performed on the multimodal aligned data to obtain multimodal target data.

[0075] After obtaining the multi-modal initial data of the converter valve, noise reduction processing can be performed on the multi-modal initial data.

[0076] Noise reduction involves the following aspects: Using a sliding average filter to process the multimodal initial data, removing high-frequency noise and enhancing the continuity and recognizability of the multimodal data; For wildly fluctuating signals in the multimodal initial data, a median filter or wavelet threshold denoising algorithm can be used in conjunction with the multimodal initial data to remove these wildly fluctuating signals and improve outlier robustness. The result of denoising the multimodal initial data is called multimodal denoised data.

[0077] Considering the different sampling frequencies and activation times of the sensors in each modality, a dynamic time warping algorithm can be used to time-align the multimodal noise reduction data. This ensures the temporal synchronization between the modal data and lays the foundation for subsequent fusion. The result of time-aligning the multimodal noise reduction data is called multimodal aligned data.

[0078] Due to sensor damage, communication delays, etc., some modal data may have missing data points, sudden changes, or drift. This embodiment handles missing values and outliers in multimodal alignment data. The specific steps include:

[0079] (1) Short-term missing repair: Use interpolation (linear, nearest neighbor or KNN) within the sliding window for fast filling; the full name of KNN is "K-Nearest Neighbor" in English, and "K nearest neighbor algorithm" in Chinese.

[0080] (2) Long-term missing mark processing: Use the mask matrix to participate in subsequent model training and adapt the missing perception mechanism in the Transformer model.

[0081] (3) Anomaly detection: Through the joint discrimination based on statistical methods (such as IQR, Z-score) and machine learning models (such as IsolationForest), outliers that deviate from the global trend are screened out and replaced or discarded; the full name of IQR in English is "Inter-Quartile Range", which is "interquartile range" in Chinese; Z-score in Chinese is "Z score"; IsolationForest in Chinese is "isolation forest", which is a machine learning algorithm used for anomaly detection.

[0082] After processing missing values and outliers in the multimodal alignment data, the multimodal target data can be obtained.

[0083] After acquiring the multimodal initial data, this embodiment can perform data cleaning, such as noise reduction, using a sliding average filter to remove high-frequency noise. Subsequently, time alignment and normalization can be performed to align the data collected by sensors of different frequencies through dynamic time warping.

[0084] In one embodiment, missing values and outliers are processed on the multimodal alignment data to obtain multimodal target data, including:

[0085] The multimodal alignment data is processed for missing values and outliers to obtain multimodal processed data; the multimodal processed data is scaled to obtain multimodal target data.

[0086] After processing missing values and outliers in the multimodal aligned data, the resulting data can be called multimodal processed data. Normalization can be performed on the multimodal processed data to eliminate dimensional differences in physical quantities and unify the features of each modality to a comparable scale. This facilitates efficient model training and stable gradient propagation during training, and helps improve the accuracy of predictions during inference. Normalization can be performed using the Z-score or Min-Max normalization methods. The normalized multimodal processed data is used as the multimodal target data.

[0087] In this embodiment, after missing values and outliers are processed for the multimodal aligned data, scale unification is also performed to unify the features of each modality to a comparable scale. This facilitates efficient model training and stable gradient propagation during training, and helps improve the accuracy of state prediction results during inference.

[0088] In one embodiment, the electrical modal features, environmental modal features and multi-head attention results are integrated to obtain residual connection features, including Figure 3 The steps shown are:

[0089] In step S301, electrical modal features, environmental modal features, and multi-head attention results are added feature by feature; in step S302, residual connection features are obtained based on the result of feature-by-feature addition.

[0090] For example, after obtaining the current characteristic , voltage characteristics , temperature characteristics , vibration characteristics And the multi-head attention results After that, the current characteristics can be , voltage characteristics , temperature characteristics , vibration characteristics And the multi-head attention results By adding each feature, the residual connection feature obtained can be expressed as: .

[0091] This embodiment adds the electrical modal features, environmental modal features, and multi-head attention results feature by feature. This processing method allows feature information to be directly transmitted across layers, which helps to improve the accuracy of state prediction results during inference. During training, it allows the gradient to flow more smoothly when the model calculates the gradient during backpropagation, alleviating the problem of vanishing gradients of the model to a certain extent. It also helps the model learn the identity mapping, making the model easier to train.

[0092] In one embodiment, a state prediction result of a converter valve is obtained based on the residual connection feature and a state prediction unit in a converter valve state monitoring model, including:

[0093] The mean and variance of the residual connection features are calculated in the dimension of the feature vector; the residual connection features are normalized according to the mean and variance to obtain normalized features; and the state prediction results of the converter valve are obtained based on the normalized features and the state prediction unit in the converter valve state monitoring model.

[0094] For example, after obtaining the residual connection feature After that, the residual connection features can be normalized layer by layer, which can be expressed as:

[0095] .

[0096] The layer normalization process may specifically include: calculating the mean and variance of the residual connection features in the dimension of the feature vector; assuming that the residual connection features are a matrix of size a×b, all the features in the matrix are averaged according to the average value calculation formula to obtain the mean; all the features in the matrix are processed according to the variance calculation formula to obtain the variance; after obtaining the mean and variance, the features in the matrix are normalized to obtain the normalized features.

[0097] After obtaining the normalized features, the state prediction results of the converter valve can be obtained based on the normalized features and the state prediction unit in the converter valve state monitoring model.

[0098] In this embodiment, the mean and variance of the residual connection features are calculated in the dimension of the feature vector; based on the mean and variance, the residual connection features are normalized so that the features are unified to the same scale. Based on the obtained normalized features, a more accurate state prediction result of the converter valve can be obtained.

[0099] In one embodiment, a state prediction result of the converter valve is obtained based on the normalized features and a state prediction unit in the converter valve state monitoring model, including:

[0100] The normalized features are scaled and translated to obtain fused features. The fused features are input into the state prediction unit in the converter valve state monitoring model to obtain the converter valve state prediction results.

[0101] After obtaining the normalized features, we can perform scaling and translation to obtain fusion features. , the above process can be expressed as follows:

[0102] .

[0103] Next, the fusion features Input the state prediction unit in the converter valve state monitoring model to obtain the state prediction result of the converter valve.

[0104] In this embodiment, the normalized features are scaled and translated, and the results of the scaling and translation are used as fusion features, which help improve the accuracy of state prediction results during inference. During training, it helps the modality to adaptively adjust the distribution of features according to the distribution of data, which helps to accelerate the convergence of the model and improve the stability and generalization ability of the model.

[0105] In order to better understand the above method, an application example of the converter valve status monitoring method based on multimodal data of the present application is described in detail below.

[0106] Neural network-based multi-source data fusion methods for converter valves have significant potential for improving device performance and intelligence, but bottlenecks such as data quality, real-time performance, and model reliability remain to be overcome. Future efforts will require the integration of interdisciplinary technologies (such as edge computing and digital twins) and the development of industry standards to advance this approach from the laboratory to engineering applications.

[0107] In terms of current application status:

[0108] (1) Application in the field of condition monitoring and fault diagnosis: It has been applied in the insulation status of converter valves, reliability of valve control systems, and device life prediction. By integrating multi-source data such as temperature, pressure, voltage, current, and vibration, it can achieve efficient monitoring and intelligent diagnosis of equipment status.

[0109] (2) Verification of the effectiveness of neural network models: Models such as CNN (Convolutional Neural Network), LSTM (Long Short-Term Memory Network) and Transformer are applied to the multi-source data fusion task of the converter valve, verifying the advantages of the model in processing nonlinear, dynamic and high-dimensional time series data.

[0110] (3) Improved intelligence and autonomy of fusion algorithms: It has shifted from simple feature-level fusion to deep feature fusion, and gradually achieved end-to-end autonomous feature learning, improving the ability to automatically fuse multi-source data under complex working conditions.

[0111] (4) Initial advancement in practical applications: Some of these methods have been initially applied to actual converter valve systems. Through real-time monitoring and early warning, the operational reliability of the converter valves has been improved and the operation and maintenance costs have been reduced.

[0112] Disadvantages of traditional technology:

[0113] (1) Data heterogeneity and quality challenges: The converter valve involves multi-source data such as current, voltage, temperature, and vibration. Different sensors have different acquisition frequencies, accuracies, and formats, requiring complex preprocessing (such as time alignment and noise filtering). Neural networks are highly sensitive to data quality, and low-quality data can easily lead to degraded model performance.

[0114] (2) Insufficient model generalization: Traditional technology research focuses on specific scenarios (such as ultra-high voltage direct current transmission), and the model's generalization ability is limited across different voltage levels or environments (such as submarine cables and high-altitude areas). For example, simulation models based on laboratory data proposed in some solutions may fail in real complex power grids.

[0115] (3) Explanability and reliability issues: As a “black box” model, neural networks lack explainability when applied to critical power equipment. For example, fault diagnosis results are difficult to trace back to specific physical mechanisms, which may affect the trustworthiness of operation and maintenance decisions.

[0116] (4) Data privacy and security risks: Multi-source data fusion involves cross-system data interaction (such as power grid, meteorological, and equipment manufacturers), which poses risks of data leakage and cyberattacks. Traditional technology research has paid insufficient attention to security protection mechanisms.

[0117] Traditional technologies have the following problems to be solved:

[0118] (1) The heterogeneity of multi-source data: The multi-source sensors commonly used in converter valves, such as voltage, current, temperature, pressure, vibration, strain, etc., are not only of various types, but also have obvious differences in physical meaning, dimension, sampling frequency and data distribution. This data heterogeneity seriously hinders the efficient construction of data fusion models and effective data fusion. Specifically, the following issues need to be addressed: ① Data type differences: For example, the fusion strategy of continuous signals such as voltage and current and low-frequency discrete signals such as vibration and temperature. ② Data scale differences: The data amplitude gap is large (for example, the current signal may be several thousand amperes, while the temperature signal is only tens of degrees), requiring effective data normalization or standardization. ③ Data semantic differences: The device status information represented by the sensor signals is different, so how to establish a unified semantic space.

[0119] (2) Inconsistency and conflict issues in multi-source data: The operating environment of the converter valve is complex, and sensor equipment aging, installation errors, and environmental interference lead to data inconsistencies and conflicts. For example, temperature data measured at different locations may be inconsistent, and some sensors may be damaged or have decreased accuracy, resulting in erroneous data. Specific issues that need to be addressed include: ① Data conflict identification and resolution strategies: How to identify data conflicts between sensors; research on strategies for eliminating, correcting, or weighting conflicting data. ② Multi-source data quality assessment: Establish a real-time sensor data credibility assessment model to improve the reliability of fusion results.

[0120] (3) Missing and outlier data from multiple sources: During operation, converter valve sensors often experience missing data, transmission interruptions, or abnormal data fluctuations, which makes data fusion difficult and seriously affects the training effect and accuracy of the fusion model. Specific issues that need to be addressed include: ① Data missing processing: real-time data missing interpolation or prediction completion methods (such as time series prediction based on LSTM or Transformer). ② Outlier detection and processing: real-time detection methods for abnormal data; tracing the causes of data anomalies and dynamic correction mechanisms.

[0121] (4) The problem of spatiotemporal correlation of multi-source data: Different sensors have different sampling frequencies (e.g., high frequency for vibration signals, medium frequency for voltage signals, and low frequency for temperature signals) and are installed in different locations, making it difficult to accurately construct and model the spatiotemporal correlation of data. Specifically, the following issues need to be addressed: ① Data synchronization: How to effectively achieve time alignment of data with different frequencies; and collaborative expression of data from sensors at different spatial locations. ② Spatiotemporal feature fusion: Designing deep neural network models based on structures such as CNN-LSTM, CNN-Transformer, or space-time graph neural networks (ST-GNN) to learn the correlation between different data sources.

[0122] This application example involves the training phase and the inference phase.

[0123] The data involved in the training phase is called data samples, and the features are called feature samples. The training phase specifically includes several aspects:

[0124] (1) Data sample collection and cleaning:

[0125] After collecting the initial multimodal data samples from the converter valve, data cleaning can be performed, such as noise reduction, using a sliding average filter to remove high-frequency noise. Afterwards, time alignment and normalization can be performed, using dynamic time warping to align data samples collected by sensors at different frequencies.

[0126] (1.1) Noise Reduction: A sliding average filter is used to process the multimodal initial data samples to remove high-frequency noise and enhance the continuity and identifiability of the multimodal data samples. For signals with significant fluctuations in the multimodal initial data samples, a median filter or wavelet threshold denoising algorithm can be used in conjunction with the multimodal initial data samples to remove these fluctuations and improve outlier robustness. After denoising the multimodal initial data samples, the resulting data is referred to as a multimodal denoised data sample.

[0127] (1.2) Time Alignment: Considering the different sampling frequencies and activation times of the sensors in each modality, a dynamic time warping algorithm can be used to time-align the multimodal noise reduction data samples. This ensures the temporal synchronization between the data samples in each modality, laying the foundation for subsequent fusion. After time-aligning the multimodal noise reduction data samples, the result is called a multimodal aligned data sample.

[0128] (1.3) Missing and outlier handling: Due to sensor damage, communication delays, and other factors, some modal data may experience missing data points, sudden changes, or drift. This includes the following steps:

[0129] (1.3.1) Short-term missing data repair: Use interpolation (linear, nearest neighbor, or KNN) within a sliding window for fast filling. KNN stands for "K-Nearest Neighbor" in English and "K-nearest neighbor algorithm" in Chinese.

[0130] (1.3.2) Long-term missing mark processing: Use the mask matrix to participate in subsequent model training and adapt the missing perception mechanism in Transformer-type models.

[0131] (1.3.3) Anomaly Detection: This method uses a combination of statistical methods (such as IQR and Z-score) and machine learning models (such as Isolation Forest) to identify outliers that deviate from the global trend and replace or discard them. IQR stands for "Inter-Quartile Range" in English, "Z-score" in Chinese, and "Isolation Forest" in Chinese. It is a machine learning algorithm used for anomaly detection.

[0132] (1.4) Normalization: After processing missing values and outliers in the multimodal aligned data samples, the resulting results can be called multimodal processed data samples. Normalization can be performed on multimodal processed data samples to eliminate dimensional differences in physical quantities and unify the modal features to comparable scales. This facilitates efficient model training and stable gradient propagation during training, and helps improve the accuracy of prediction results during inference. Normalization can use the Z-score or Min-Max normalization methods. The normalized multimodal processed data samples are used as multimodal target data samples. Multimodal target data samples include current waveform target data samples, voltage waveform target data samples, temperature time series target data samples, and vibration time series target data samples.

[0133] (2) Extraction of multimodal features: In order to effectively mine the structural information of each modality in the time and space dimensions, a dual-branch feature extraction framework based on convolutional neural network (CNN) and long short-term memory network (LSTM) is constructed to achieve targeted modeling of electrical and environmental modal features.

[0134] (2.1) A convolutional neural network is used to extract features from electrical modal target data samples. For example, the current waveform target data samples and the voltage waveform target data samples can be segmented to extract local spatiotemporal features to obtain current feature samples and voltage feature samples. This process can be expressed as follows:

[0135] , ;

[0136] Where I represents current, V represents voltage, represents the current characteristic sample, Represents a voltage signature sample.

[0137] (2.2) Using a long short-term memory network to extract features from environmental modal target data, for example, long-term dependencies can be modeled for temperature time series target data samples and vibration time series target data samples to obtain temperature feature samples and vibration feature samples. This process can be expressed as follows:

[0138] , ;

[0139] Where T represents temperature, S represents vibration, represents the temperature feature sample, Represents a vibration feature sample.

[0140] (3) Cross-modal self-attention mechanism: In order to overcome the problem that traditional feature splicing methods are difficult to capture implicit correlations between modalities, this application example introduces a cross-modal self-attention mechanism to achieve deep semantic interaction between different signal sources. First, the current feature sample is input into the multi-head attention unit. , voltage characteristic samples , temperature characteristic samples and vibration signature samples , and then conduct cross-modal interaction; the calculation formula involved is as follows:

[0141] , ,

[0142] ,

[0143] in, ; is the dimension of the key matrix, which is used to scale the attention score to prevent the dot product result from being too large, causing the softmax function gradient to disappear.

[0144] is a learnable weight matrix used to transform the current feature samples Linear transformation to query vector . Current characteristic sample Weight matrix The query vector obtained by linear transformation.

[0145] Is a learnable weight matrix used to transform the temperature feature samples Linear transformation into key vector . Temperature characteristic sample Weight matrix The key vector obtained by linear transformation.

[0146] Temperature characteristic sample Weight matrix The value vector obtained by linear transformation participates in the attention weight calculation.

[0147] Indicates that according to the query vector and key vector The calculated attention weight is obtained by This formula is calculated to reflect the degree of correlation between different parts of the current feature samples and the temperature feature samples, thereby achieving deep semantic interaction between different signal sources.

[0148] Different heads in the multi-head attention unit independently calculate cross-modal attention weights. The specific process is as follows:

[0149] First, perform a linear transformation. After the linear transformation, the attention weights can be calculated to obtain the attention output matrix calculated by n different heads. The results calculated by n heads (such as the attention output matrix calculated by n different heads) are spliced to form a new matrix. Then, the linear transformation matrix Transform the concatenated matrix to obtain the feature representation processed by the multi-head attention mechanism (which can be called multi-head attention result) and is used for subsequent fusion feature calculation and other links.

[0150] (4) Fusion feature optimization and decision-making: In the feature fusion stage, in order to avoid information redundancy and gradient dissipation problems, the residual connection and layer normalization (LayerNorm) mechanism are introduced to normalize the multimodal data through the residual connection method. The specific calculation method is as follows:

[0151] .

[0152] Multi-head attention results To expand on the multi-head attention model, each head independently calculates cross-modal attention weights. To enhance expressiveness, a multi-head attention mechanism (MultiHead) is further constructed, which improves generalization and information interaction through parallel subspace attention learning.

[0153] A multi-task loss function is used to handle different tasks. Cross entropy loss is used for fault classification tasks, and mean square error is used for temperature prediction tasks. The overall loss function is calculated as follows:

[0154] ;

[0155] 、 、 are weight coefficients, is the L2 regularization term, is the fault classification loss function, Loss function for the temperature prediction task.

[0156] After training is completed, the converter valve state monitoring model can be obtained and applied to the reasoning stage.

[0157] The reasoning phase specifically includes the following steps:

[0158] Using a sliding average filter, high-frequency noise is removed from the multi-modal initial data of the converter valve to obtain multi-modal noise-reduced data.

[0159] The multimodal denoised data is time-aligned using a dynamic time warping algorithm to obtain multimodal aligned data.

[0160] Process missing values and outliers on the multimodal alignment data to obtain multimodal processed data;

[0161] The multimodal processed data is scaled uniformly to obtain multimodal target data;

[0162] The convolutional neural network and long short-term memory network in the pre-built converter valve state monitoring model are used to extract features from the electrical modal target data and environmental modal target data in the multimodal target data to obtain electrical modal features and environmental modal features.

[0163] The multi-head attention unit in the converter valve state monitoring model processes the electrical modal features and environmental modal features to obtain the multi-head attention results.

[0164] Add the electrical modal features, environmental modal features, and multi-head attention results feature by feature;

[0165] According to the result of adding each feature, the residual connection feature is obtained;

[0166] Calculate the mean and variance of the residual connection features in the dimension of the feature vector;

[0167] According to the mean and variance, the residual connection features are normalized to obtain normalized features;

[0168] Scaling and translation processing are performed on the normalized features to obtain fused features;

[0169] The fused features are input into the state prediction unit in the converter valve state monitoring model to obtain the state prediction result of the converter valve.

[0170] The solution provided in this application example predicts the state of the converter valve based on its multimodal data, which can improve the accuracy of state monitoring. In addition, considering that the multimodal data of the converter valve are collected by different sensors, and the collection frequency, accuracy and format of different sensors vary greatly, this application performs noise reduction and time alignment on the multimodal initial data to obtain multimodal target data, so as to avoid the difference in sensor collection affecting the accuracy of the state prediction results. In addition, this application uses the convolutional neural network and long short-term memory network in the converter valve state monitoring model to respectively process the electrical modal target data and the environmental modal target data in the multimodal target data. The feature extraction and processing of the standard data are carried out to obtain the electrical modal features and the environmental modal features, which can effectively mine the structural information of each mode in the time and space dimensions, and further improve the accuracy of the state prediction results; the multi-head attention unit in the converter valve state monitoring model is used to process the electrical modal features and the environmental modal features, which helps to capture the implicit correlation between the modes and realize the deep semantic interaction between different modal data. The residual connection features are obtained by combining the multi-head attention results, electrical modal features and environmental modal features. Combined with the state prediction unit in the converter valve state monitoring model, the state prediction results of the converter valve with high accuracy can be obtained.

[0171] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0172] Based on the same inventive concept, embodiments of the present application also provide a multimodal data-based converter valve state monitoring device for implementing the aforementioned multimodal data-based converter valve state monitoring method. The solution provided by this device is similar to the solution described in the aforementioned method. Therefore, the specific limitations of one or more embodiments of the multimodal data-based converter valve state monitoring device provided below can be found in the above-described limitations of the multimodal data-based converter valve state monitoring method and are not further elaborated here.

[0173] In one embodiment, Figure 4 As shown, a converter valve state monitoring device based on multimodal data is provided, comprising:

[0174] The data preprocessing module 401 is used to perform noise reduction and time alignment on the multimodal initial data of the converter valve to obtain multimodal target data;

[0175] A feature extraction module 402 is configured to perform feature extraction processing on the electrical modal target data and the environmental modal target data in the multimodal target data using a convolutional neural network and a long short-term memory network in a pre-built converter valve state monitoring model, respectively, to obtain electrical modal features and environmental modal features;

[0176] an attention processing module 403 for processing the electrical modal features and the environmental modal features through a multi-head attention unit in the converter valve state monitoring model to obtain a multi-head attention result;

[0177] A residual connection processing module 404 is used to integrate the electrical modal features, environmental modal features and multi-head attention results to obtain residual connection features;

[0178] The result acquisition module 405 is configured to obtain a state prediction result of the converter valve according to the residual connection feature and the state prediction unit in the converter valve state monitoring model.

[0179] In one embodiment, the data preprocessing module 401 is further configured to:

[0180] A sliding flat filter is used to remove high-frequency noise from the multimodal initial data of the converter valve to obtain multimodal noise-reduced data; a dynamic time warping algorithm is used to perform time alignment on the multimodal noise-reduced data to obtain multimodal aligned data; and missing values and outlier processing are performed on the multimodal aligned data to obtain multimodal target data.

[0181] In one embodiment, the data preprocessing module 401 is further configured to:

[0182] The multimodal alignment data is processed for missing values and outliers to obtain multimodal processed data; and the multimodal processed data is scaled to obtain multimodal target data.

[0183] In one embodiment, the residual connection processing module 404 is further configured to:

[0184] The electrical modal features, environmental modal features and multi-head attention results are added feature by feature; and residual connection features are obtained based on the results of the feature-by-feature addition.

[0185] In one embodiment, the result acquisition module 405 is further configured to:

[0186] The mean and variance of the residual connection features are calculated in the dimension of the feature vector; the residual connection features are normalized according to the mean and variance to obtain normalized features; and the state prediction result of the converter valve is obtained according to the normalized features and the state prediction unit in the converter valve state monitoring model.

[0187] In one embodiment, the result acquisition module 405 is further configured to:

[0188] The normalized features are scaled and translated to obtain fused features; the fused features are input into a state prediction unit in the converter valve state monitoring model to obtain a state prediction result of the converter valve.

[0189] Each module in the multimodal data-based converter valve condition monitoring device can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in hardware form, or stored in a computer device memory in software form, so that the processor can call and execute the corresponding operations of each module.

[0190] In an exemplary embodiment, a computer device is provided, the internal structure of which can be shown as follows: Figure 5 As shown. The computer device includes a processor, a memory, an input / output interface (I / O) and a communication interface. The processor, memory and input / output interface are connected via a system bus, and the communication interface is connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store data involved in the above method. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, a method for monitoring the state of a converter valve based on multimodal data is implemented.

[0191] Those skilled in the art will understand that Figure 5 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0192] In one embodiment, a computer device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above-mentioned method embodiments when executing the computer program.

[0193] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0194] In one embodiment, a computer program product is provided, on which a computer program is stored. The computer program is used by a processor to execute the steps in the above-mentioned various method embodiments.

[0195] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0196] Those skilled in the art will understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), quantum computing-based data processing logic devices, artificial intelligence (AI) processors, and the like.

[0197] The technical features of the above embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0198] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A method for monitoring the status of a converter valve based on multimodal data, characterized in that: The method comprises: Perform noise reduction and time alignment on the multi-modal initial data of the converter valve to obtain multi-modal target data; Using a convolutional neural network and a long short-term memory network in a pre-built converter valve state monitoring model, feature extraction processing is performed on the electrical modal target data and the environmental modal target data in the multimodal target data to obtain electrical modal features and environmental modal features; Processing the electrical modal features and the environmental modal features through a multi-head attention unit in the converter valve state monitoring model to obtain a multi-head attention result; Combining the electrical modal features, environmental modal features and multi-head attention results to obtain residual connection features; A state prediction result of the converter valve is obtained according to the residual connection feature and the state prediction unit in the converter valve state monitoring model.

2. The method according to claim 1, characterized in that The multimodal initial data of the converter valve is subjected to noise reduction and time alignment to obtain multimodal target data, including: Using a sliding average filter, high-frequency noise is removed from the multi-modal initial data of the converter valve to obtain multi-modal noise-reduced data. Performing time alignment processing on the multimodal noise reduction data using a dynamic time warping algorithm to obtain multimodal aligned data; Missing values and outlier processing are performed on the multimodal aligned data to obtain multimodal target data.

3. The method according to claim 2, characterized in that Processing missing values and outliers on the multimodal alignment data to obtain multimodal target data includes: Processing missing values and outliers on the multimodal aligned data to obtain multimodal processed data; The multimodal processed data is scaled uniformly to obtain multimodal target data.

4. The method according to claim 1, wherein Combining the electrical modal features, environmental modal features, and multi-head attention results, we obtain the residual connection features, including: Adding the electrical modal features, environmental modal features, and multi-head attention results feature by feature; According to the result of adding each feature, the residual connection feature is obtained.

5. The method according to claim 1, wherein Obtaining a state prediction result of the converter valve according to the residual connection feature and a state prediction unit in the converter valve state monitoring model includes: Calculating the mean and variance of the residual connection features in the dimension of the feature vector; Normalizing the residual connection features according to the mean and variance to obtain normalized features; A state prediction result of the converter valve is obtained according to the normalized features and the state prediction unit in the converter valve state monitoring model.

6. The method according to claim 5, characterized in that Obtaining a state prediction result of the converter valve according to the normalized feature and a state prediction unit in the converter valve state monitoring model includes: Scaling and translating the normalized features to obtain fused features; The fusion feature is input into the state prediction unit in the converter valve state monitoring model to obtain the state prediction result of the converter valve.

7. A converter valve status monitoring device based on multimodal data, characterized in that: The device comprises: A data preprocessing module is used to perform noise reduction and time alignment on the multi-modal initial data of the converter valve to obtain multi-modal target data; a feature extraction module for performing feature extraction processing on the electrical modal target data and the environmental modal target data in the multimodal target data using a convolutional neural network and a long short-term memory network in a pre-built converter valve state monitoring model, respectively, to obtain electrical modal features and environmental modal features; an attention processing module, configured to process the electrical modal features and the environmental modal features through a multi-head attention unit in the converter valve state monitoring model to obtain a multi-head attention result; A residual connection processing module, configured to synthesize the electrical modal features, the environmental modal features, and the multi-head attention results to obtain residual connection features; The result acquisition module is used to obtain the state prediction result of the converter valve according to the residual connection feature and the state prediction unit in the converter valve state monitoring model.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images