Electric power multi-modal data perception and digital twinborn simulation cooperative interaction method and system

Abstract

The invention provides an electric power multi-modal data perception and digital twinborn simulation cooperative interaction method and system, and relates to the field of electric power equipment management and maintenance, and the method comprises the steps: carrying out the electric power multi-modal data perception of data collected by edge equipment, and obtaining the state characteristics of single equipment; constructing a digital twinborn model based on the state features of all devices in the to-be-monitored power system; constructing a power equipment graph knowledge base by using a digital twinborn model, and extracting global state features of the equipment from the power equipment graph knowledge base through a graph neural network so as to construct a global state feature time sequence of the equipment; inputting the global state feature time sequence into the trained time sequence prediction model, and predicting the equipment state at the future moment; according to the multi-modal data, parameters of the digital twin model are adjusted in real time, environmental changes are flexibly adapted, time sequence modeling and fault trend analysis functions are combined, the running state of the equipment is predicted, and the comprehensiveness and foresight of equipment monitoring are further improved.

Description

Technical Field

[0001] The present disclosure relates to the field of power equipment management and maintenance, and specifically to a collaborative interaction method and system for power multimodal data perception and digital twin simulation. Background Art

[0002] As the application and importance of power equipment in the industrial field continue to increase, real-time monitoring, fault warning and maintenance management of equipment have become particularly critical. However, traditional equipment monitoring technology usually relies on a single data source, such as temperature or vibration data, to evaluate the operating status of the equipment. This single-mode data collection method is difficult to fully capture the operating status of the equipment in a complex environment, especially when multiple abnormalities occur in power equipment, it is difficult to quickly identify potential sources of failure. In addition, traditional monitoring methods often require manual regular inspections, or use fixed preset thresholds to judge the status of the equipment, which is time-consuming and prone to false alarms and missed reports. Therefore, existing equipment monitoring systems generally lack real-time, flexibility and the ability to accurately restore fault scenarios.

[0003] To meet these challenges, current technologies are gradually moving towards multimodal data fusion and intelligence. Through the comprehensive collection of multiple sensor data (such as images, infrared thermal imaging, voiceprints, and vibrations), combined with deep learning and digital twin technology, the operating status of the equipment can be dynamically restored and deeply analyzed in a virtual environment. However, although these technologies have achieved a certain degree of fusion application, the existing systems still have shortcomings in real-time adaptive optimization, precise predictive maintenance, and deep collaboration of multi-source data. Summary of the invention

[0004] In order to solve the above problems, the present disclosure proposes a collaborative interaction method and system for power multimodal data perception and digital twin simulation. According to the multimodal data, the digital twin model parameters are adjusted in real time to flexibly adapt to environmental changes. The time series modeling and fault trend analysis functions are combined to predict the equipment operation status, further improving the comprehensiveness and foresight of equipment monitoring.

[0005] According to some embodiments, the present disclosure adopts the following technical solutions: The collaborative interaction method of power multimodal data perception and digital twin simulation includes: Perform power multimodal data perception on the data collected by edge devices to obtain the status characteristics of a single device; Based on the status characteristics of all equipment in the power system to be monitored, a digital twin model is built to update the status characteristics of each equipment in real time; Using the digital twin model, we build a graph knowledge base for power equipment. We extract the global state features of the equipment from the graph knowledge base through a graph neural network, and then build a time series of the global state features of the equipment. The global state feature time series is input into the trained time series prediction model to predict the device state at future moments.

[0006] According to some embodiments, the present disclosure adopts the following technical solutions: The electric power multimodal data perception and digital twin simulation collaborative interaction system includes: The data perception module is configured to: perform power multimodal data perception on the data collected by the edge device to obtain the state characteristics of a single device; The model building module is configured to: build a digital twin model based on the state characteristics of all devices in the power system to be monitored, and update the state characteristics of each device in real time; The feature extraction module is configured to: construct a graph knowledge base of electric power equipment using the digital twin model, extract the global state features of the equipment from the graph knowledge base of electric power equipment through the graph neural network, and thus construct a time series of the global state features of the equipment; The state prediction module is configured to: input the global state feature time series into the trained time series prediction model to predict the device state at a future moment.

[0007] According to some embodiments, the present disclosure adopts the following technical solutions: A computer program product includes a computer program, which, when executed by a processor, implements the method for collaborative interaction of electric power multimodal data perception and digital twin simulation.

[0008] According to some embodiments, the present disclosure adopts the following technical solutions: A non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method of collaborative interaction of electric power multimodal data perception and digital twin simulation is implemented.

[0009] According to some embodiments, the present disclosure adopts the following technical solutions: An electronic device comprises: a processor, a memory and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the method for collaborative interaction of electric power multimodal data perception and digital twin simulation.

[0010] Compared with the prior art, the present invention has the following beneficial effects: The present invention utilizes the fusion of multimodal data and digital twin technology to enhance the accuracy and real-time performance of equipment status monitoring. It unifies the processing of multiple sensor data such as visual images, infrared thermal imaging, voiceprints, and vibrations, and uses an adaptive learning algorithm to dynamically adjust the parameters of the twin model to reflect the current status of the equipment in real time and accurately. This real-time monitoring capability can quickly identify potential faults, provide predictive maintenance recommendations, and help users take necessary maintenance measures in advance to avoid sudden equipment failures and operational interruptions, thereby significantly improving the equipment's operational reliability and safety.

[0011] The present invention generates personalized operation optimization strategies by deeply analyzing the historical operation data and current status of the equipment, and combines time series modeling and fault trend analysis functions to provide users with detailed equipment operation status predictions and support forward-looking maintenance decisions. This data-driven optimization solution not only reduces maintenance costs, but also improves the overall efficiency of the equipment, allowing users to manage and operate power equipment more effectively, laying a solid technical foundation for the intelligent management of industrial operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings constituting a part of the present disclosure are used to provide a further understanding of the present disclosure. The illustrative embodiments of the present disclosure and their descriptions are used to explain the present disclosure and do not constitute an improper limitation on the present disclosure.

[0013] Figure 1 The present invention is a method flow chart of an embodiment of the present invention. DETAILED DESCRIPTION The present disclosure is further described below in conjunction with the accompanying drawings and embodiments.

[0014] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present disclosure belongs.

[0015] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present disclosure. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0016] Example 1 In one embodiment of the present disclosure, a method for collaborative interaction of electric power multimodal data perception and digital twin simulation is provided. According to multimodal data, the parameters of the digital twin model are adjusted in real time to flexibly adapt to environmental changes. In addition, the equipment operation status is predicted by combining time series modeling and fault trend analysis functions, so as to further improve the comprehensiveness and foresight of equipment monitoring, including: Step S1: Perform power multimodal data perception on the data collected by the edge device to obtain the status characteristics of a single device.

[0017] Specifically, in step S1.1, multimodal data is collected through edge devices, including sensor data such as visual images, infrared thermal imaging, voiceprints and vibrations, so as to comprehensively capture the real-time operating status of power equipment; various sensors are pre-calibrated to ensure the accuracy and consistency of the collected data; at the same time, a data sampling frequency suitable for each sensor is set so that changes in temperature, sound and vibration can be accurately captured.

[0018] The visual image and infrared thermal imaging here are two different modalities. Visual images usually refer to visible light images (for example, ordinary RGB images), which capture the external shape and state of the device through visible light and are often used to monitor and detect surface problems of the device; while infrared thermal imaging is an image that captures the heat distribution of an object through an infrared sensor. It can reflect the temperature difference on the surface of the device, help find temperature anomalies, and then identify potential faults (for example, overheated electrical components). Therefore, they belong to different types of modalities. Visual images are visual modalities that capture the shape, structure and other information of the device through visible light, and infrared thermal imaging is a thermal imaging modality that captures the temperature information of the device through infrared rays and is used to detect heat distribution and temperature changes.

[0019] Furthermore, the data from different sensors are time-aligned through timestamps and synchronization algorithms to ensure the time consistency of multimodal data. Furthermore, various types of data are accurately calibrated, such as comparing infrared temperature data with standard temperature values ​​and vibration data with normal operating parameters of the equipment, so that multimodal data can accurately reflect the current status of the equipment.

[0020] Step S1.2, extracting the status characteristics of a single device through power multimodal data perception.

[0021] Furthermore, a mixture of experts (MOE) model is used for multimodal feature extraction and fusion, which can effectively improve the flexibility and performance of the model, especially when processing multimodal data. MOE is a model combination method that distributes input data to multiple experts (usually learning models), each of which is responsible for processing a subset or modality of the data; the decision mechanism of the model determines which expert should process the current input through a gating network; in this way, the MOE model can automatically select the most appropriate expert for processing according to the different characteristics of the input data, thereby improving efficiency and accuracy.

[0022] In the application of multimodal data fusion, the MOE model can select different experts to extract features for different modalities (such as infrared images, voiceprints, and vibration data), ensuring that the data of each modality can be processed by the most appropriate expert. For multimodal data, such as infrared images, voiceprints, and vibration signals, each modality is first classified through a gating network, and which expert network is used to process the data of that modality.

[0023] Assume there are three modal data, namely X IR (Infrared image), X SW (voiceprint data), and X VIB (vibration data), each mode will be processed by an expert network:

[0024] Among them, E IR 、E SW and E VIB They represent the expert networks for infrared images, voiceprints, and vibration data, respectively. For example, infrared images are processed using Vision Transformer (ViT), voiceprint data are processed using Audio Transformer, and vibration data are extracted using a temporal Transformer model. Each modality is processed through an expert network E IR 、E SW and E VIB Extract the corresponding feature representation H IR , H SW and H VIB .

[0025] In MOE, the gating network determines the expert choice for each modality through the following formula:

[0026] Among them, g iis the output of the gating network, representing the weight of each expert, σ is the activation function (such as Sigmoid), W g is the weight of the gating network, b g is the bias term, X modality is the input modal data.

[0027] Next, use the output weight g of the gating network IR , g SW and g VIB The features of each modality are weighted and fused to obtain the final multimodal fusion feature H fusion :

[0028] This weighted fusion ensures that the features of different modalities are properly adjusted according to their importance and reliability, thereby improving the quality of the fused features.

[0029] After multimodal feature fusion, the fused feature H fusion Input into the subsequent deep learning model (such as Transformer, etc.) for further processing to extract higher-level device status features H final , which comprehensively represents the current operating status of the equipment and provides the basis for subsequent predictive maintenance, fault diagnosis and operation optimization:

[0030] Through the mixed expert model (MOE), the characteristics of different modal data are integrated into a complete status view and accurately reflect the current status of the equipment.

[0031] Step S2: Based on the status characteristics of all equipment in the power system to be monitored, a digital twin model is constructed to update the status characteristics of each equipment in real time.

[0032] The digital twin model is used to simulate the state characteristics of the equipment in the power system to be monitored, and provides a global state characteristic representation of the equipment state by integrating the state characteristics of all equipment; this model can restore and reflect the current state of the power equipment in real time. The multimodal fusion state characteristics also provide reliable support for real-time monitoring and fault detection of equipment, so that the digital twin model is not limited to a single state display, but can also track the operation of the equipment in real time, facilitating the identification of potential abnormalities and fault trends at a later stage.

[0033] By simulating the power system to be monitored, a digital twin model is constructed, specifically: (1) Input data: Multimodal sensor data is extracted and fused to generate a unified device status feature H final .final As the basic input for simulation.

[0034] (2) Simulation model construction: The core of the digital twin model: Using the digital twin platform, a high-precision simulation model is established based on physical laws and equipment operation logic.

[0035] Data-driven updates: By receiving H final Update model parameters to achieve dynamic simulation of the state.

[0036] (3) Virtual scene reproduction: Real-time monitoring: Generate a dynamic visual interface of equipment operation to display the current status (such as temperature distribution, vibration frequency).

[0037] Virtual operation testing: Allows users to simulate operations in a virtual environment, such as adjusting operating parameters or performing maintenance simulations.

[0038] Fault reproduction: Restore the device fault scenario based on historical data and analyze possible fault causes.

[0039] (4) Simulation output: Current status mapping: Visualize the real-time operation status of the equipment.

[0040] Fault Analysis Report: Generates diagnostic results about current operating anomalies.

[0041] Trend forecasting support: Provide high-quality input data for subsequent time series forecasting models.

[0042] The digital twin model also includes automatic adjustment of parameters of the digital twin model through an adaptive learning algorithm, and the parameters include a calibration coefficient of a thermal imaging sensor and a sensitivity coefficient of a vibration sensor.

[0043] Specifically, the adaptive learning algorithm dynamically adjusts the key parameters of the digital twin model to ensure that the model maintains high accuracy and reliability in a real-time changing device environment. The specific implementation process is as follows: (1) Data collection and monitoring Real-time input: Collect multimodal data from sensors (such as temperature, vibration, voiceprint, etc.).

[0044] After feature extraction and fusion, the data generates the current state feature H of the device final .

[0045] Target parameters: Define the model parameters that need to be corrected, for example: calibration coefficients for thermal imaging sensors, sensitivity coefficients for vibration sensors.

[0046] (2) Parameter deviation detection Deviation formula: Use the formula to detect the deviation between the current parameter and the reference value:

[0047] Among them, Pcurrent is the real-time sensor parameter and Pbaseline is the historical baseline parameter.

[0048] If the deviation is greater than a preset threshold, a correction method is triggered.

[0049] (3) Calibration method According to different parameter types, the following correction methods are used: 1) Thermal imaging sensor calibration: Objective: Adjust the calibration factors of a thermal imaging sensor.

[0050] Calibration formula:

[0051] Among them, T measured is the current measured value, T offset It is a calibration factor, adjusted based on historical benchmarks and real-time deviations.

[0052] 2) Vibration sensor sensitivity calibration: Objective: To adjust the sensitivity factor of the vibration signal.

[0053] Correction formula:

[0054] Among them, V raw is the original data of the sensor, G factor is the sensitivity coefficient, which is adjusted based on historical benchmarks and real-time deviations.

[0055] (4) Feedback and verification Error feedback: After correction, the model error Errornew is calculated in real time.

[0056] If the error meets a preset condition (eg, is below a threshold), the correction is complete.

[0057] Verification results: Use the corrected model to simulate real-time data.

[0058] The results were compared with the true values ​​to evaluate the correction effect.

[0059] (5) Dynamic learning and optimization Continuously optimize the correction strategy through adaptive learning: Use reinforcement learning (such as PPO) to update the correction algorithm.

[0060] Dynamically adjust the correction threshold based on device historical trends.

[0061] Step S3: Use the digital twin model to build a power equipment graph knowledge base, extract the global state characteristics of the equipment from the power equipment graph knowledge base through the graph neural network, and thus construct the global state characteristic time series of the equipment.

[0062] The electric power equipment graph knowledge base is composed of a node set and an edge set; the nodes represent equipment status and fault information, and the edges represent physical connections of equipment, conduction paths of faults, and operation logic.

[0063] Furthermore, the real-time operation data of the collected power equipment is integrated with the operation rules of the equipment, historical fault cases and expert experience, and these data from different sources are introduced into the power-specific graph knowledge base to build a global graph structure containing equipment status and fault information. Specifically, each node in the knowledge base represents a device status or fault event, and the relationship between the nodes represents the physical connection of the equipment, the conduction path of the fault or the operation logic.

[0064] The nodes include the following types: (1) Equipment status node: Indicates the operating status of the device at a certain time, including normal status and abnormal status (such as overheating, abnormal vibration, etc.).

[0065] (2) Fault event node: Describe known fault types and their characteristics, such as overheating fault, abnormal vibration frequency, and current overload.

[0066] (3) Equipment component node: Represents a specific physical part of a device, e.g. generator bearings, transformer windings.

[0067] (4) Rules and experience nodes: Represents operating rules or expert experience, such as temperature threshold rules (the equipment operating temperature does not exceed 80°C) and fault conduction paths.

[0068] (5) Historical case nodes: Represents a fault instance that occurred in the past, including the complete fault context.

[0069] During the node construction process, because each component and device has multiple sensors that can collect multimodal data, the node features of the equipment and components can be obtained through multimodal fusion; for fault event nodes, which are abnormal equipment or component nodes, feature collection is similar to that of ordinary equipment or fault nodes. The rule and experience nodes are text descriptions, which are directly processed with text features, and the text is input into the large model to obtain text features or vectors. Historical case nodes are nodes that aggregate the features of surrounding nodes from past fault nodes. These nodes aggregate the features of surrounding nodes into historical case nodes through graph networks.

[0070] The relationships between nodes include the following types: (1) Physical connection relationship: Describes the physical connections between equipment components, such as the electrical connections between a generator and a transformer.

[0071] (2) Fault conduction path: Describe the logical relationship of fault propagation between devices, for example: abnormal vibration → bearing overheating → rotor damage.

[0072] (3) Time series relationship: Describes the changes of the same device status node at different time points.

[0073] (4) Causal relationship: Describe the cause-and-effect chain between the fault node and the status node, for example: excessive vibration causes the device temperature to rise.

[0074] (5) Rule association relationship: The relationship between a device state node and a rule node, for example: a state node triggers a specific rule (such as temperature exceeding a threshold).

[0075] Different types of edges have different weight values, for example, the causal edge connection weight is 2, and the fault propagation edge weight is 5.

[0076] Through this graph structure, the information in multimodal data can be aligned with the entities and relationships in the graph knowledge base to achieve unified organization of data. With the graph structure, convolution operations and calculations can be performed through graph neural networks. During the convolution and calculation process, the information of other nodes will be integrated, such as the characteristics of historical nodes and spatial nodes (nodes with adjacent relationships). In this way, the nodes have both time and space information and context information (rule experience and historical cases).

[0077] Furthermore, in order to efficiently integrate multi-source data with the graph knowledge base, a graph neural network (GNN) is used to extract and encode the feature information of each node.

[0078] After building the power equipment graph knowledge base, the graph neural network (GNN) is used to extract and encode the feature information of the nodes, fuse the multimodal data into a unified feature space, and achieve semantic alignment and unified expression of the real-time status of the equipment and historical fault information.

[0079] 1. Input The input of GNN mainly includes the following: 1.Node characteristics: The characteristics of each node represent the equipment status, component information, historical failure modes, etc.

[0080] Node feature source: From H final Extracted multimodal fusion features, such as temperature, vibration, voiceprint, etc.

[0081] Embedded representation of rules and historical cases.

[0082] Example:

[0083] x i is the feature vector of node i.

[0084] 2. Edge connection relationship: The edges (relationships) between nodes in the graph structure represent physical connections, fault transmission paths, or time series relationships.

[0085] Example:

[0086] A is the adjacency matrix, A ij =1 means there is a connection between node i and node j.

[0087] 3. Characteristics of edges: Features that represent relationships between nodes, for example: Weights (such as the magnitude of change in sensor data).

[0088] Time series trends (such as temperature changes over time).

[0089] Example:

[0090] e ij is the edge feature vector between node i and node j.

[0091] 2. Output The output of GNN is the node representation and (optional) edge representation after feature extraction and update for subsequent analysis and application.

[0092] 1. Node embedding representation: GNN generates a unified feature vector for each node, namely the global state feature, which contains multimodal data and relationship information, and is expressed as:

[0093] in, It is the updated feature vector of node i, which contains information such as device status and historical failure mode.

[0094] 2. Edge embedding representation: Encode the relationship between nodes in the graph and output the updated edge representation, which can be expressed as:

[0095] in, is the updated edge feature, which represents the connection between the equipment status and fault conduction.

[0096] 3. Global graph representation: Aggregate the entire graph to generate a global state feature vector for overall analysis, which is expressed as:

[0097] G is the embedding representation of the global graph.

[0098] GNN can represent different types of data into a unified feature vector, allowing multimodal data to interact and analyze in the same feature space. Through this processing, the graph knowledge base can not only contain information about the real-time status of the equipment, but also embed nodes and edges related to historical faults to achieve semantic alignment and unified expression of data; this structure enables various types of data to be associated in the knowledge base, forming a device status map with a global perspective, which can query and obtain comprehensive status information of the equipment in real time.

[0099] Step S4: Input the global state feature time series into the trained time series prediction model to predict the device state at the future moment.

[0100] Through in-depth analysis of multimodal data, fault trends can be identified, predictive maintenance recommendations and operation optimization strategies can be provided, and users can be supported to make data-driven decisions to improve equipment efficiency and safety.

[0101] Furthermore, on the basis of achieving real-time updates of equipment status, time series analysis and anomaly detection technology are used to deeply mine multimodal data and identify potential failure trends; by comparing current data with historical operating modes, abnormal conditions that may occur in equipment are predicted, such as component wear or deviations from key parameters.

[0102] Based on these prediction results, personalized diagnostic recommendations are generated, such as recommendations to replace specific parts, adjust operating parameters, or schedule regular inspections; users can perform proactive maintenance based on these recommendations, thereby reducing the possibility of sudden failures and improving the stability and reliability of equipment.

[0103] This embodiment uses a time series prediction model (such as a time series Transformer model) to predict future failure trends of equipment. The time series Transformer model adopts a self-attention mechanism, which can capture long-term dependencies and make full use of time series patterns and potential failure signs in the historical data of the equipment.

[0104] Among them, the temporal Transformer model consists of two parts: 1. Predict the global state characteristics at a certain time t By inputting historical global state features and timestamps, they are mapped into a series of time series feature vectors as the input of the time series transformer. Assume that the time series feature vector of the device state is X history , where X history ={x 1 ,x 2 ,…,x t}, where each x i Represents the global state characteristics of the device at time i. The time series Transformer model processes the input data through the following formula:

[0105] Among them, T = {1, 2, .., t} is the timestamp sequence corresponding to the time series feature vector, H t It is the output feature of the temporal Transformer model, that is, the predicted value of the global state feature at the current time t by integrating the historical state and time context.

[0106] 2. Predicting the device status in the next k steps The Time Series Transformer assigns weights to the input of each time step through the self-attention mechanism, thereby dynamically capturing important changes in the device state and potential failure trends; the core advantage of the Time Series Transformer model is that it can predict future failure trends based on the historical state information of the device. By analyzing historical data and current device status, it can generate predictions of the future state of the device and evaluate the possibility of failure in the next few steps. The prediction formula is as follows:

[0107] Among them, y t+k is the prediction result of the device state in the next k steps, f(⋅) is the temporal Transformer model, Ht is the global state feature at the current time t, and p is the time step of the historical data. By comparing the similarity between the historical data and the current state, the model can predict possible equipment failures or abnormal states in the future.

[0108] Based on the prediction of the equipment status at a future moment, by comparing it with the historical failure mode, the possible failure trend in the future is predicted, helping users to take maintenance measures in advance to avoid potential risks.

[0109] The failure mode here is a set of characteristic modes of the equipment in a faulty state, including abnormal characteristics and their associations that occur during equipment operation, such as: Temperature pattern: The temperature distribution characteristics of the device when it is overheated.

[0110] Vibration pattern: abnormal frequency content or excessive amplitude.

[0111] Voiceprint pattern: unusual noise or signal changes.

[0112] Comprehensive mode: multi-modal features (temperature, vibration, soundprint) simultaneously reflect the fault status.

[0113] Mode representation: Each fault mode is mapped into a feature vector through feature extraction.

[0114] Match the current state with the historical fault mode to determine whether the current state is close to a certain fault mode. The specific matching steps are: 1. Feature extraction: Extracting current device status features H from multimodal data t .

[0115] Extract the fault mode H to be matched from the historical fault database fault .

[0116] 2. Similarity calculation: The current state and the historical pattern are compared using a similarity algorithm, such as cosine similarity or Euclidean distance. A cosine similarity value close to 1 indicates a high similarity to the historical failure pattern.

[0117] 3. Classifier judgment: If the similarity exceeds a set threshold, the current state is considered to match the failure mode.

[0118] 4. Fault mode matching output: Outputs the historical fault modes that the current state may belong to (such as "overheat fault" or "mechanical loose fault").

[0119] Furthermore, based on the prediction results of the time series analysis model, corresponding early warning signals can be generated. When the similarity exceeds the set threshold, an early warning prompt will be automatically issued to remind the user to take corresponding maintenance measures in advance. For example, when a high equipment load is detected, the user may be advised to reduce the equipment load or assign operating tasks to avoid overload damage. In addition, based on the in-depth analysis results of multimodal data, a series of optimization solutions are generated to help users achieve more efficient equipment management. This collaborative optimization mechanism makes the digital twin model not only a monitoring tool, but also an important assistant to support decision-making and improve operational performance, providing users with comprehensive data-driven support.

[0120] Example 2 In one embodiment of the present disclosure, a power multimodal data perception and digital twin simulation collaborative interaction system is provided, including: The data perception module is configured to: perform power multimodal data perception on the data collected by the edge device to obtain the state characteristics of a single device; The model building module is configured to: build a digital twin model based on the state characteristics of all devices in the power system to be monitored, and update the state characteristics of each device in real time; The feature extraction module is configured to: construct a graph knowledge base of electric power equipment using the digital twin model, extract the global state features of the equipment from the graph knowledge base of electric power equipment through the graph neural network, and thus construct a time series of the global state features of the equipment; The state prediction module is configured to: input the global state feature time series into the trained time series prediction model to predict the device state at a future moment.

[0121] Example 3 In one embodiment of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the method for collaborative interaction of electric power multimodal data perception and digital twin simulation.

[0122] Example 4 In one embodiment of the present disclosure, a non-transitory computer-readable storage medium is provided, wherein the non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by a processor, the method of collaborative interaction of power multimodal data perception and digital twin simulation is implemented.

[0123] Example 5 In one embodiment of the present disclosure, an electronic device is provided, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory, so that the electronic device executes the method for collaborative interaction of power multimodal data perception and digital twin simulation.

[0124] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of the processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0125] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0126] Although the above describes the specific implementation methods of the present disclosure in conjunction with the accompanying drawings, it is not intended to limit the scope of protection of the present disclosure. Technical personnel in the relevant field should understand that on the basis of the technical solution of the present disclosure, various modifications or variations that can be made by those skilled in the art without creative work are still within the scope of protection of the present disclosure.

Claims

1. A collaborative interaction method for electric power multimodal data perception and digital twin simulation, characterized in that: include: Perform power multimodal data perception on the data collected by edge devices to obtain the status characteristics of a single device; Based on the status characteristics of all equipment in the power system to be monitored, a digital twin model is built to update the status characteristics of each equipment in real time; Using the digital twin model, we build a graph knowledge base for power equipment. We extract the global state features of the equipment from the graph knowledge base through a graph neural network, and then build a time series of the global state features of the equipment. The global state feature time series is input into the trained time series prediction model to predict the device state at future moments.

2. The method for collaborative interaction of electric power multimodal data perception and digital twin simulation according to claim 1, characterized in that: The electric power multimodal data perception is to extract features and perform weighted fusion on visual images, infrared images, voiceprint data, and vibration data through a hybrid expert model.

3. The method for collaborative interaction of electric power multimodal data perception and digital twin simulation according to claim 1, characterized in that: The digital twin model is used to simulate the state characteristics of the equipment in the power system to be monitored, and provide a global state characteristic representation of the equipment state by fusing the state characteristics of all equipment; During the construction of the digital twin model, the parameters of the digital twin model are automatically adjusted through an adaptive learning algorithm, and the parameters include the calibration coefficient of the thermal imaging sensor and the sensitivity coefficient of the vibration sensor.

4. The method for collaborative interaction of electric power multimodal data perception and digital twin simulation according to claim 1, characterized in that: The electric power equipment graph knowledge base is composed of a node set and an edge set; The nodes represent device status and fault information, and the edges represent physical connections of devices, fault conduction paths, and operation logic.

5. The method for collaborative interaction of electric power multimodal data perception and digital twin simulation according to claim 4, characterized in that: The method of extracting the global state features of the equipment from the electric equipment graph knowledge base through the graph neural network is to process the features of the nodes and edges in the electric equipment graph knowledge base, update the feature representation of the node by aggregating the features of the neighboring nodes, and obtain the global state features of the equipment.

6. The method for collaborative interaction of electric power multimodal data perception and digital twin simulation according to claim 1, characterized in that: The time series prediction model predicts the global state characteristics at future moments based on the global state characteristic time series, and predicts the equipment state through the similarity between the predicted global state characteristics and the historical failure mode.

7. The electric power multimodal data perception and digital twin simulation collaborative interactive system is characterized by: include: The data perception module is configured to: perform power multimodal data perception on the data collected by the edge device to obtain the state characteristics of a single device; The model building module is configured to: build a digital twin model based on the state characteristics of all devices in the power system to be monitored, and update the state characteristics of each device in real time; The feature extraction module is configured to: construct a graph knowledge base of electric power equipment using the digital twin model, extract the global state features of the equipment from the graph knowledge base of electric power equipment through the graph neural network, and thus construct a time series of the global state features of the equipment; The state prediction module is configured to: input the global state feature time series into the trained time series prediction model to predict the device state at a future moment.

8. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for collaborative interaction of electric power multimodal data perception and digital twin simulation as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium is used to store computer instructions. When the computer instructions are executed by the processor, the collaborative interaction method of power multimodal data perception and digital twin simulation as described in any one of claims 1 to 6 is implemented.

10. An electronic device, characterized in that: include: A processor, a memory and a computer program; wherein the processor is connected to the memory, and the computer program is stored in the memory. When the electronic device is running, the processor executes the computer program stored in the memory so that the electronic device executes the collaborative interaction method of electric power multimodal data perception and digital twin simulation as described in any one of claims 1 to 6.

Images