A multi-modal fusion power equipment anomaly detection method and system

Abstract

The present application relates to the technical field of power equipment anomaly detection, and provides a multi-modal fusion power equipment anomaly detection method and system, comprising: for each modal data, obtaining the modal data after cross-modal calibration through a cross-modal calibration model; for each modal data after cross-modal calibration, adopting a double-branch structure to respectively extract time sequence features and spatial topology features, and after splicing, obtaining spatio-temporal joint features; based on the spatio-temporal joint features, calculating the correlation between modes, and combining environmental parameters and the noise level of each modal data acquisition sensor, calculating the weight of each mode, weighting the spatio-temporal joint features of each mode, and obtaining fusion features; based on the fusion features, predicting the power equipment state change trend through a multi-modal large model, identifying abnormal time points, and improving the power equipment anomaly detection precision.

Description

Technical Field

[0001] This invention belongs to the field of power equipment anomaly detection technology, and particularly relates to a multimodal fusion power equipment anomaly detection method and system. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] Stable operation of power equipment is crucial to ensuring reliable power supply from the power system. As the scale of the power system continues to expand and the complexity of the equipment increases, timely and accurate detection of abnormal conditions of power equipment becomes increasingly important.

[0004] Traditional methods for detecting anomalies in power equipment mainly rely on single-modal data (such as electrical parameters, visual inspection of equipment appearance, or sound signal analysis), and use technologies such as convolutional neural networks (CNN) and transformers to identify equipment status. However, these methods have obvious limitations, namely, they are difficult to comprehensively and accurately reflect the true operating status of the equipment.

[0005] In recent years, there have also been detection schemes based on the fusion of multimodal data such as visible light and infrared. However, these schemes use simple splicing, fixed-weight weighted summation, and other multimodal fusion methods combined with target detection and other technologies for power equipment fault identification. This approach has the following limitations: (1) Limited feature quality: The reliability of multimodal data is affected by a variety of factors (such as sensor noise, environmental interference, and modal correlation). Fixed weight fusion methods (such as simple averaging) will introduce invalid information. (2) Insufficient utilization of multimodal data: Only images and text are fused, without integrating equipment operating parameters (such as current, vibration, temperature) and environmental data (such as weather, geographical location), resulting in one-sided information. Summary of the Invention

[0006] To address the technical problems mentioned above, this invention provides a multimodal fusion-based method and system for detecting power equipment anomalies. It proposes a meta-learning-driven cross-modal feature calibration, spatiotemporal joint feature extraction, and dynamic weighted fusion method. By employing dual-loss constraints to improve cross-modal consistency, and further by dynamically adjusting weights and fusing features, it can automatically filter noise, weaken interference, and focus on reliable modes, avoiding interference from invalid information on subsequent multimodal large-scale models, thus improving feature quality and consequently enhancing the accuracy of power equipment anomaly detection.

[0007] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of the present invention provides a multimodal fusion-based method for detecting anomalies in power equipment, comprising: Multimodal data of power equipment is acquired, and cross-modal calibration of each modal data is performed through a cross-modal calibration model. The training of the cross-modal calibration model adopts a weighted sum of local feature alignment loss and global context matching loss, and the model parameters and weighting adjustment factors are learned through meta-learning training. For each modal data after cross-modal calibration, a dual-branch structure is adopted to extract time series features and spatial topological features respectively, and then the features are spliced ​​together to obtain spatiotemporal joint features; Based on the spatiotemporal joint features, the correlation between modes is calculated, and combined with environmental parameters and the noise level of each mode data acquisition sensor, the modal weights are calculated. The spatiotemporal joint features of each mode are then weighted and fused to obtain the fused features. Based on fusion features, the system predicts the trend of power equipment status changes through a multimodal large model, identifies power equipment anomalies, and combines the constructed equipment fault causal graph to analyze the causal relationship of anomalies through logical reasoning.

[0008] Furthermore, the weighted sum is: ; in, The adjustment factor; local feature alignment loss , This represents the set of local features extracted from mode A. Let N represent the set of local features proposed by mode B, N represent the dimension of the local features extracted by mode A, and M represent the dimension of the local features extracted by mode B. For cosine similarity, Temperature coefficient; global context matching loss , For the global features of mode A, This represents the global features of mode B; Let K represent the global feature of mode B of the negative samples, where K is the number of negative samples.

[0009] Furthermore, the modal weights are as follows: ; in, Indicates temperature coefficient; composite score Sensor noise level , It is the noise standard deviation. It is the sensitivity coefficient; intermodal correlation score. m represents the total number of modes. The Pearson correlation coefficients of modes i and j; scene prior factors. , It is an indicator function. It is the scene gain coefficient.

[0010] Furthermore, the cross-modal calibration model uses different branch encoders to extract local and global features from different modal data, calculates the similarity matrix of local features between different modal data to obtain a local similarity matrix, calculates the similarity matrix of global features between different modal data to obtain a global similarity matrix, and based on the local and global similarity matrices, outputs the matching probability between different modal data through a classifier, and based on the matching probability, obtains the cross-modal calibrated modal data through an attention-based decoder.

[0011] Furthermore, the multimodal data includes images of the power equipment, operating noise, current data, voltage data, temperature data, humidity data, manuals, and maintenance records.

[0012] Furthermore, it also includes noise suppression, missing mode generation and compensation, and timestamp alignment of each modal data before cross-modal calibration.

[0013] A second aspect of the present invention provides a multimodal fusion power equipment anomaly detection system, comprising: The data acquisition module is configured to acquire multimodal data from power equipment. The feature extraction module is configured as follows: for each modal data, after cross-modal calibration through the cross-modal calibration model, a dual-branch structure is adopted to extract time series features and spatial topological features respectively, and then concatenate them to obtain spatiotemporal joint features; wherein, the training of the cross-modal calibration model adopts a weighted sum of local feature alignment loss and global context matching loss, and the cross-modal calibration model learns model parameters and adjustment factors used for weighting through meta-learning training; The fusion module is configured to: calculate the intermodal correlation based on spatiotemporal joint features, and calculate the modal weights by combining environmental parameters and the noise levels of the data acquisition sensors for each modality, and perform weighted fusion of the spatiotemporal joint features of each modality to obtain fused features; The anomaly detection module is configured to: predict the state change trend of power equipment based on fused features through a multimodal large model, identify power equipment anomalies, and analyze the anomaly causal relationship through logical reasoning by combining the constructed equipment fault causal graph.

[0014] Furthermore, the weighted sum is: ; in, The adjustment factor; local feature alignment loss , This represents the set of local features extracted from mode A. Let N represent the set of local features proposed by mode B, N represent the dimension of the local features extracted by mode A, and M represent the dimension of the local features extracted by mode B. For cosine similarity, Temperature coefficient; global context matching loss , For the global features of mode A, This represents the global features of mode B; Let K represent the global feature of mode B of the negative samples, where K is the number of negative samples.

[0015] Furthermore, the modal weights are as follows: ; in, Indicates temperature coefficient; composite score Sensor noise level , It is the noise standard deviation. It is the sensitivity coefficient; intermodal correlation score. m represents the total number of modes. The Pearson correlation coefficients of modes i and j; scene prior factors. , It is an indicator function. It is the scene gain coefficient.

[0016] Furthermore, the cross-modal calibration model uses different branch encoders to extract local and global features from different modal data, calculates the similarity matrix of local features between different modal data to obtain a local similarity matrix, calculates the similarity matrix of global features between different modal data to obtain a global similarity matrix, and based on the local and global similarity matrices, outputs the matching probability between different modal data through a classifier, and based on the matching probability, obtains the cross-modal calibrated modal data through an attention-based decoder.

[0017] Furthermore, the multimodal data includes images of the power equipment, operating noise, current data, voltage data, temperature data, humidity data, manuals, and maintenance records.

[0018] A third aspect of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the multimodal fusion power equipment anomaly detection method described above.

[0019] A fourth aspect of the present invention provides a computer device including a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, wherein the processor executes the program to implement the steps of the multimodal fusion power equipment anomaly detection method described above.

[0020] Compared with the prior art, the beneficial effects of the present invention are: This invention innovatively proposes a meta-learning-driven method for cross-modal feature calibration, spatiotemporal joint feature extraction, and dynamic weighted fusion. By using dual loss constraints to improve cross-modal consistency, and on this basis, by dynamically adjusting weights and fusing features, it can automatically filter noise, weaken interference, and focus on reliable modes, avoiding invalid information from interfering with subsequent multimodal large models, thus improving feature quality and thereby improving the accuracy of power equipment anomaly detection.

[0021] This invention innovatively proposes a method for detecting anomalies in power equipment based on multimodal data. By acquiring multimodal data (such as electrical quantities, temperature, images, and sounds), it can cover the multi-dimensional characteristics of equipment in terms of "electrical-mechanical-physical-environment," avoiding information loss from the source and providing a comprehensive data foundation for subsequent analysis. Attached Figure Description

[0022] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0023] Figure 1 This is a flowchart of a multimodal fusion-based power equipment anomaly detection method according to Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the structure of a computer device according to Embodiment 4 of the present invention. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings.

[0025] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0026] Example 1 This embodiment provides a multimodal fusion method for detecting anomalies in power equipment.

[0027] This embodiment provides a multimodal fusion-based power equipment anomaly detection method. By combining dynamic adaptive data preprocessing, meta-learning-driven cross-modal calibration, spatiotemporal joint modeling, dynamic multimodal weight fusion, two-layer causal reasoning, and cloud-based federated learning, it achieves accurate detection and interpretable diagnosis of power equipment anomalies. This improves detection accuracy and generalization ability in complex environments, reduces equipment operation and maintenance costs, and provides an innovative solution for intelligent operation and maintenance of power systems.

[0028] This embodiment provides a multimodal fusion method for detecting power equipment anomalies, such as... Figure 1 As shown, it includes the following steps: Step 1: Multi-source data acquisition.

[0029] Using cameras, microphone arrays, current / voltage sensors, and temperature / humidity sensors, multimodal data (including images of the power equipment, operating noise, current data, voltage data, temperature / humidity data, etc.) are simultaneously collected from power equipment (such as transformers, insulators, circuit breakers, load switches, etc.) and manuals and maintenance records of the power equipment are obtained.

[0030] Step 2: Modality-aware preprocessing chain.

[0031] Specifically, preprocessing steps are automatically matched based on the type of input modal data. For example, adaptive histogram equalization is used for images of power equipment, and word segmentation and cleaning are performed on long texts such as maintenance records.

[0032] Step 3: Noise suppression.

[0033] Specifically, for each modal data, abnormal data points are detected sequentially by combining wavelet transform and autoencoder, and interpolation or generation models are used to repair the abnormal data points.

[0034] Step 4: Generation and compensation of missing modes.

[0035] Specifically, synthetic features for missing modalities are generated based on cross-modal generative adversarial networks.

[0036] For example, maintenance records can be used to generate corresponding virtual images of electrical equipment.

[0037] Step 5: Achieve timestamp alignment using a time-series asynchronicity self-optimization algorithm.

[0038] Specifically, a bidirectional long short-term memory network (LSTM) and a multi-head attention mechanism are used to model cross-modal temporal dependencies and implicitly align the timestamps of multimodal data.

[0039] For example, when aligning an image with running noise, attention weights are used to automatically associate noise segments at the same timestamp with the corresponding images.

[0040] Step 6: Cross-modal calibration.

[0041] Example of cross-modal calibration: For “matching text-image pairs” (such as a maintenance record of transformer A and an image of transformer A), which are two modal data belonging to the same power equipment, their output features are forced to be very close in space; for “mismatched pairs” (such as a maintenance record of transformer A and an image of transformer B or a transformer), which are two modal data belonging to different power equipment, they are forced to be far apart.

[0042] Specifically, based on the timestamp-aligned modal data, cross-modal calibrated modal data is obtained through a cross-modal calibration model.

[0043] The cross-modal calibration model includes: (1) Base Model / Embedding Networks, which are responsible for extracting feature representations from each modality of data.

[0044] In this embodiment, the base model uses a dual-branch encoder.

[0045] Image branch: Using convolutional neural networks (CNNs), such as ResNet, EfficientNet, or their variants, to process images of power equipment, outputting multi-scale feature maps (preserving spatial information, used for local alignment, called local features) and globally pooled feature vectors (used for global alignment, called global features).

[0046] Text / Record Branch: Different sequence models, such as LSTM (Long Short-Term Memory Network), GRU (Gated Recurrent Unit), or Transformer (a bidirectional encoder based on Transformer representing BERT, a robust optimized BERT pre-training method RoBERTa, or a lightweight version thereof), are used to process running noise, current data, voltage data, temperature / humidity data, manuals, and maintenance records. The output can be word / sub-word level embedding sequences (for local alignment, called local features) and sentence / document level special embeddings (Classification Embedding) or pooled vectors (for global alignment, called global features).

[0047] (2) Alignment Module.

[0048] Local alignment: Through the cross-attention mechanism, the similarity matrix of local features of two different modalities is calculated to obtain the local similarity matrix; Global alignment: Calculate the cosine similarity or dot product of the global vectors of two different modalities to obtain the global similarity matrix.

[0049] (3) Prediction / Fusion Head.

[0050] For two different modalities, an MLP classifier is input based on the local and global similarity matrices, and outputs the matching probability. Based on the matching probability, an attention-based decoder (such as an LSTM or Transformer decoder) is used to align the two different modalities.

[0051] In this embodiment, a cross-modal calibration model is trained to learn model parameters and dynamically weighted adjustment factors through meta-learning, such as MAML (Model Independent Meta-Learning). Meta-learning requires only a small amount of labeled data. Through previously learned model parameters and adjustment factors, it can converge and align with the model after very few iterations. That is, it can quickly adapt to new modal combinations using a small amount of labeled data (for example, in the calibration of images and maintenance records, it can learn general mapping rules through meta-training and fine-tune them on new data). It also designs a multi-level alignment loss function, combining fine-grained local feature similarity with global context matching, and dynamically balances the contributions of both through meta-learning weights.

[0052] Meta-learning employs a dynamically weighted total loss, calculated as follows: (1) Formula for calculating the local feature alignment loss function: ; in, This represents the set of local features extracted from mode A. Let N represent the set of local features proposed by mode B, N represent the dimension of the local features extracted by mode A, and M represent the dimension of the local features extracted by mode B. For cosine similarity, For temperature coefficient, and The corresponding data is marked as belonging to the same power equipment.

[0053] (2) The formula for calculating the global context matching loss function is as follows: ; in, For the global features of mode A, This represents the global features of mode B; Let K represent the global feature of mode B of the negative samples, where K is the number of negative samples. and The corresponding data are labeled with the same power equipment, assuming and The corresponding data is labeled as power equipment 1, then The corresponding data is labeled as other electrical equipment besides electrical equipment 1.

[0054] (3) The formula for calculating the dynamic weighted total loss is as follows: ; Among them, the regulating factor It is dynamically generated by meta-learning.

[0055] Step 7: Dynamic feature extraction and feature fusion of multimodal data.

[0056] Dynamic feature extraction and feature fusion of multimodal data includes spatiotemporal joint modeling and dynamic weight allocation.

[0057] (1) Spatiotemporal joint modeling.

[0058] Specifically, for each mode, based on the modal data after cross-modal calibration, a dual-branch structure is adopted to extract time series features and spatial topological features respectively, and then splicing them together to obtain spatiotemporal joint features.

[0059] The temporal branch uses TCN-BiGRU (a deep learning model combining a temporal convolutional network and a bidirectional gated recurrent unit) to capture long-term dependencies on current, voltage, and environment, while the spatial branch introduces a graph convolutional network (GCN) to model the relationships between device nodes.

[0060] (2) Dynamic weight allocation.

[0061] Specifically, a modal importance score is designed, and the weights of each modality are dynamically adjusted based on the real-time data distribution (sensor noise level, intermodal correlation, and scene prior factors).

[0062] For example, the modal weights of infrared images are increased in high-temperature scenarios, while the weights of vibration data are enhanced in low-light conditions.

[0063] The greater the noise, the lower the modal reliability, and the weights should decay using an exponential decay function. ; in, It is the normalized noise level (standard deviation). It is the sensitivity coefficient, which controls the decay rate. The noise level is obtained by inputting data collected from different sensors, such as the camera, microphone array, current / voltage sensor, and temperature / humidity sensor, into a convolutional neural network for evaluation.

[0064] Among them, intermodal correlation reflects information redundancy, and the formula for calculating the modal correlation score is as follows: ; Where m represents the total number of modes. This is the real-time Pearson correlation coefficient between modes i and j, calculated based on joint spatiotemporal features. Modes with low correlation and low noise should be assigned higher weights.

[0065] Among them, the calculation formula for the scenario prior factor is based on dynamically adjusting the weights according to environmental parameters: ; in, It is an indicator function (1 when the condition is met, 0 otherwise). The condition is considered met when both the ambient temperature and ambient humidity are within the set range. It is the scene gain coefficient.

[0066] The above scores are multiplied to obtain a comprehensive score, and then weights are generated using Softmax (a flexible maximum activation function). ; ; in, This represents the temperature coefficient, which controls the steepness of the weight distribution.

[0067] The design modal importance score refers to the use of adaptive / gated cyclic units (GRUs) to dynamically adjust the fusion path, highlight key modal information, and suppress redundant noise.

[0068] (3) Based on modal weights, the spatiotemporal joint features of each modality are weighted and fused to obtain fused features.

[0069] Step 8: Federated learning-driven anomaly detection in multimodal large models.

[0070] The federated learning-driven multimodal large model anomaly detection model includes two-layer causal inference, lightweight hybrid fine-tuning strategy, and cloud-based federated training.

[0071] Two-layer causal reasoning refers to the introduction of a two-layer causal reasoning network to achieve anomaly localization and root cause analysis, and to achieve interpretable output and decision support.

[0072] Anomaly localization employs a multimodal large-scale model, specifically a multimodal time-series prediction model built on the Transformer architecture. This model, combined with an autocorrelation compensation mechanism, predicts the changing trends of power equipment status and identifies anomalies, specifically anomaly time points. The multimodal time-series prediction model based on the Transformer architecture predicts anomalies, and the autocorrelation compensation mechanism dynamically corrects prediction biases.

[0073] The formula for calculating the lag window in the autocorrelation compensation mechanism is defined as follows: ; in, It is the observation value at the current time step; These are the historical observations over the next k steps; ACF() is the autocorrelation function; It is the autocorrelation significance threshold, usually taken as 0.8.

[0074] The lightweight hybrid fine-tuning strategy refers to embedding a hybrid expert system (MoE) into a pre-trained multimodal large model (such as the second-generation BLIP-2 with guided language image pre-training), and only performing LoRA (low-rank adaptive) fine-tuning on the expert sub-modules. For specific power scenarios (such as substations and transmission lines), the model parameters are dynamically adjusted through small sample learning to quickly adapt to new environments.

[0075] Root cause analysis refers to constructing a causal graph of equipment failures, analyzing abnormal causal relationships through logical reasoning, and generating an interpretable fault diagnosis report. Specifically, it is achieved through forward reasoning, reverse verification, and uncertainty quantification; forward reasoning: starting from the abnormal event, searching for possible root cause paths along the causal graph; reverse verification: based on counterfactual queries (such as "if there is oxidation due to electric shock, will the temperature rise be abnormal?"); uncertainty quantification: using Bayesian networks to calculate the root cause probability.

[0076] Among them, cloud-based federated training refers to aggregating the model parameters of city-level sub-nodes through a federated averaging algorithm, updating the global model, automatically expanding the knowledge base based on new detection results, enabling continuous model learning, and ensuring that data does not leave the local area, thus protecting data privacy.

[0077] This embodiment provides a multimodal fusion-based method for detecting anomalies in power equipment, which utilizes cross-modal generative adversarial networks to complete missing modes and improve robustness.

[0078] This embodiment provides a multimodal fusion-based power equipment anomaly detection method, which solves the problem of multimodal data heterogeneity by using a time-series asynchronous self-optimizing timestamp alignment method and a cross-modal calibration method with dynamic weight balancing based on meta-learning strategy.

[0079] This embodiment provides a multimodal fusion method for detecting power equipment anomalies. It combines GCN (Graph Convolutional Network) with TCN-BiGRU (Temporal Convolutional Network-Bidirectional Gated Recurrent Unit) to achieve spatiotemporal joint modeling, optimize the extraction of power grid topology features, and dynamically adjust modal weights through real-time scoring to improve adaptability to complex scenarios.

[0080] This embodiment provides a multimodal fusion-based method for detecting power equipment anomalies. By using a two-layer causal reasoning network and an expert knowledge graph, it improves the interpretability of abnormal power equipment outputs and reduces equipment operation and maintenance costs.

[0081] This embodiment provides a multimodal fusion-based power equipment anomaly detection method. It uses federated learning to aggregate multi-node model parameters and introduces an incremental learning mechanism to adapt to changes in data distribution through online updates, thereby achieving efficient multi-node collaboration, avoiding model degradation, and protecting the data privacy of the power system.

[0082] This embodiment provides a multimodal fusion-based power equipment anomaly detection method. Through multi-level loss functions and dynamic weighting mechanisms, it achieves efficient cross-modal calibration based on meta-learning strategies with a small amount of labeled data, adapting to modal differences in different scenarios.

[0083] Example 2 This embodiment provides a multimodal fusion power equipment anomaly detection system, including: The data acquisition module is configured to acquire multimodal data from power equipment. The feature extraction module is configured as follows: for each modal data, after cross-modal calibration through the cross-modal calibration model, a dual-branch structure is adopted to extract time series features and spatial topological features respectively, and then concatenate them to obtain spatiotemporal joint features; wherein, the training of the cross-modal calibration model adopts a weighted sum of local feature alignment loss and global context matching loss, and the cross-modal calibration model learns model parameters and adjustment factors used for weighting through meta-learning training; The fusion module is configured to: calculate the intermodal correlation based on spatiotemporal joint features, and calculate the modal weights by combining environmental parameters and the noise levels of the data acquisition sensors for each modality, and perform weighted fusion of the spatiotemporal joint features of each modality to obtain fused features; The anomaly detection module is configured to: predict the state change trend of power equipment based on fused features through a multimodal large model, identify power equipment anomalies, and analyze the anomaly causal relationship through logical reasoning by combining the constructed equipment fault causal graph.

[0084] Furthermore, the weighted sum is: ; in, The adjustment factor; local feature alignment loss , This represents the set of local features extracted from mode A. Let N represent the set of local features proposed by mode B, N represent the dimension of the local features extracted by mode A, and M represent the dimension of the local features extracted by mode B. For cosine similarity, Temperature coefficient; global context matching loss , For the global features of mode A, This represents the global features of mode B; Let K represent the global feature of mode B of the negative samples, where K is the number of negative samples.

[0085] Furthermore, the modal weights are as follows: ; in, Indicates temperature coefficient; composite score Sensor noise level , It is the noise standard deviation. It is the sensitivity coefficient; intermodal correlation score. m represents the total number of modes. The Pearson correlation coefficients of modes i and j; scene prior factors. , It is an indicator function. It is the scene gain coefficient.

[0086] Furthermore, the cross-modal calibration model uses different branch encoders to extract local and global features from different modal data, calculates the similarity matrix of local features between different modal data to obtain a local similarity matrix, calculates the similarity matrix of global features between different modal data to obtain a global similarity matrix, and based on the local and global similarity matrices, outputs the matching probability between different modal data through a classifier, and based on the matching probability, obtains the cross-modal calibrated modal data through an attention-based decoder.

[0087] Furthermore, the multimodal data includes images of the power equipment, operating noise, current data, voltage data, temperature data, humidity data, manuals, and maintenance records.

[0088] It should be noted that each module in this embodiment corresponds one-to-one with each step in Embodiment 1, and their specific implementation processes are the same, so they will not be repeated here.

[0089] Example 3 This embodiment provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of a multimodal fusion power equipment anomaly detection method as described in Embodiment 1 above.

[0090] Example 4 This embodiment provides a computer device, such as... Figure 2 As shown, the system includes a computer-readable storage medium 1003, a processor 1001, a communication interface 1002, and a computer program stored on the computer-readable storage medium 1003 and executable on the processor 1001. The processor 1001, communication interface 1002, and computer-readable storage medium 1003 can be connected via a bus or other means. The communication interface 1002 is used to receive and transmit data. When the processor 1001 executes the program, it implements the steps of the multimodal fusion power equipment anomaly detection method described in Embodiment 1 above.

[0091] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A multimodal fusion method for detecting anomalies in power equipment, characterized in that, include: Multimodal data of power equipment is acquired, and cross-modal calibration of each modal data is performed using a cross-modal calibration model. The cross-modal calibration model uses different branch encoders to extract local and global features from different modal data, calculates the similarity matrix of local features between different modal data to obtain a local similarity matrix, and calculates the similarity matrix of global features between different modal data to obtain a global similarity matrix. Based on the local and global similarity matrices, a classifier outputs the matching probability between different modal data. Based on the matching probability, an attention-based decoder is used to obtain the cross-modal calibrated modal data. The training of the cross-modal calibration model uses a weighted sum of local feature alignment loss and global context matching loss, and meta-learning is used to train the cross-modal calibration model to learn model parameters and adjustment factors for weighting. The weighted sum of the local feature alignment loss and global context matching loss is: ;in, The adjustment factor; local feature alignment loss , This represents the set of local features extracted from mode A. Let N represent the set of local features proposed by mode B, N represent the dimension of the local features extracted by mode A, and M represent the dimension of the local features extracted by mode B. For cosine similarity, Temperature coefficient; global context matching loss , For the global features of mode A, This represents the global features of mode B; The global feature of mode B of the negative sample is represented by K, where K is the number of negative samples. For each modal data after cross-modal calibration, a dual-branch structure is adopted to extract time series features and spatial topological features respectively, and then the features are spliced ​​together to obtain spatiotemporal joint features; Based on spatiotemporal joint features, intermodal correlations are calculated, and modal weights are calculated by combining environmental parameters and the noise levels of the sensors acquiring data for each modality. The spatiotemporal joint features of each modality are then weighted and fused to obtain fused features; the modal weights are as follows: ;in, Indicates temperature coefficient; composite score Sensor noise level , It is the noise standard deviation. It is the sensitivity coefficient; intermodal correlation score. m represents the total number of modes. The Pearson correlation coefficients of modes i and j; scene prior factors. , It is an indicator function. It is the scene gain coefficient; Based on fusion features, the system predicts the trend of power equipment status changes through a multimodal large model, identifies power equipment anomalies, and combines the constructed equipment fault causal graph to analyze the causal relationship of anomalies through logical reasoning.

2. The multimodal fusion-based power equipment anomaly detection method as described in claim 1, characterized in that, The multimodal data includes images of the power equipment, operating noise, current data, voltage data, temperature data, humidity data, manuals, and maintenance records.

3. The multimodal fusion method for detecting power equipment anomalies as described in claim 1, characterized in that, Also includes: Before performing cross-modal calibration, noise suppression, missing mode generation and compensation, and timestamp alignment are performed on each modal data.

4. A multimodal fusion power equipment anomaly detection system, characterized in that, include: The data acquisition module is configured to acquire multimodal data from power equipment. The feature extraction module is configured as follows: For each modal data, after cross-modal calibration using a cross-modal calibration model, a dual-branch structure is adopted to extract time-series features and spatial topological features respectively, and then concatenate them to obtain spatiotemporal joint features; wherein, the cross-modal calibration model uses different branch encoders to extract local and global features from different modal data, calculates the similarity matrix of local features between different modal data to obtain a local similarity matrix, calculates the similarity matrix of global features between different modal data to obtain a global similarity matrix, and based on the local and global similarity matrices, a classifier outputs the matching probability between different modal data, and based on the matching probability, an attention-based decoder obtains the cross-modal calibrated modal data; the training of the cross-modal calibration model adopts a weighted sum of local feature alignment loss and global context matching loss, and learns model parameters and weighting adjustment factors through meta-learning training; the weighted sum of local feature alignment loss and global context matching loss is: ;in, The adjustment factor; local feature alignment loss , This represents the set of local features extracted from mode A. Let N represent the set of local features proposed by mode B, N represent the dimension of the local features extracted by mode A, and M represent the dimension of the local features extracted by mode B. For cosine similarity, Temperature coefficient; global context matching loss , For the global features of mode A, This represents the global features of mode B; The global feature of mode B of the negative sample is represented by K, where K is the number of negative samples. The fusion module is configured to: calculate the intermodal correlation based on spatiotemporal joint features, and calculate the modal weights by combining environmental parameters and the noise levels of the data acquisition sensors for each modality; and perform weighted fusion of the spatiotemporal joint features of each modality to obtain fused features; wherein the modal weights are: ;in, Indicates temperature coefficient; composite score Sensor noise level , It is the noise standard deviation. It is the sensitivity coefficient; intermodal correlation score. m represents the total number of modes. The Pearson correlation coefficients of modes i and j; scene prior factors. , It is an indicator function. It is the scene gain coefficient; The anomaly detection module is configured to: predict the state change trend of power equipment based on fused features through a multimodal large model, identify power equipment anomalies, and analyze the anomaly causal relationship through logical reasoning by combining the constructed equipment fault causal graph.

5. The multimodal fusion power equipment anomaly detection system as described in claim 4, characterized in that, The multimodal data includes images of the power equipment, operating noise, current data, voltage data, temperature data, humidity data, manuals, and maintenance records.

6. The multimodal fusion power equipment anomaly detection system as described in claim 4, characterized in that, It also includes a data preprocessing module, which is configured to perform noise suppression, missing mode generation and compensation, and timestamp alignment on each modal data before cross-modal calibration.

7. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the multimodal fusion power equipment anomaly detection method as described in any one of claims 1-3.

8. A computer device comprising a computer-readable storage medium, a processor, and a computer program stored on the computer-readable storage medium and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the multimodal fusion power equipment anomaly detection method as described in any one of claims 1-3.

Images