Electric power multi-modal data automatic labeling system and method based on deep learning

Abstract

The invention discloses an electric power multi-modal data automatic labeling system and method based on deep learning, and the system comprises a data collection module which collects multi-modal data in an electric power system; the data preprocessing module is used for preprocessing the multi-modal data; the multi-modal data fusion module is used for fusing the sensor data and the image data by adopting a multi-modal neural network to construct a unified feature vector; the fault diagnosis module is used for performing fault diagnosis according to the feature vector to obtain fault information; the labeling module is used for labeling the fault information according to a preset rule labeling library to generate a labeling text; and the visualization module is used for generating visualization content according to the annotation text and the multi-modal data, visualizing the visualization content and displaying the visualization content to a user. According to the rule base, the annotation text is automatically generated, information such as fault types, reasons and damage degrees is converted into text description easy to understand, and manual intervention is reduced.

Description

Technical Field

[0001] The present invention relates to the field of electric power multimodal data annotation, and in particular to a system and method for automatic annotation of electric power multimodal data based on deep learning. Background Art

[0002] With the continuous expansion of power system scale and increasing technical complexity, the stable operation of power system faces unprecedented challenges. Modern power networks not only need to process a large amount of data streams, but also these data often come from different sources and have multiple modalities (such as sensor readings, image information, etc.), which makes it difficult for traditional data analysis methods to effectively deal with them. In this context, how to efficiently and accurately analyze and diagnose multimodal data in power systems has become an urgent problem to be solved.

[0003] Existing power system monitoring mainly relies on a single type of data source, such as using only electrical parameters (voltage, current) to determine the system status. However, this method has obvious limitations: it cannot fully reflect the operating status of the entire power system, and has weak recognition capabilities for some complex fault modes. Summary of the invention

[0004] In view of the shortcomings of the prior art, the present invention provides a system and method for automatic labeling of electric power multimodal data based on deep learning.

[0005] The present invention provides a deep learning-based automatic labeling system for electric power multimodal data, comprising the following:

[0006] Data acquisition module: used to collect multimodal data in the power system, including data from various sensors and image data from cameras;

[0007] Data preprocessing module: used for preprocessing the multimodal data;

[0008] Multimodal data fusion module: used to fuse sensor data and image data using a multimodal neural network to construct a unified feature vector;

[0009] Fault diagnosis module: used to perform fault diagnosis according to the feature vector to obtain fault information, including fault type, fault cause and degree of damage to the equipment corresponding to the fault;

[0010] Annotation module: used to annotate the fault information according to a preset rule annotation library and generate annotated text;

[0011] Visualization module: used to generate visualization content according to the annotated text and the multimodal data, visualize the visualization content, and display it to the user.

[0012] Preferably, the data preprocessing module includes a sensor data processing submodule, an image data processing submodule and a data synchronization submodule.

[0013] The image data processing submodule comprises:

[0014] Noise processing unit: used to remove noise from sensor data using digital filtering algorithms;

[0015] Feature extraction unit: used to extract features from sensor data, including abnormal amplitudes of current and voltage, and frequency fluctuation range; LSTM is used to encode time series features to generate feature vectors of fixed length;

[0016] Anomaly detection unit: used to detect anomalies in data using time series models.

[0017] Preferably, the image processing submodule includes:

[0018] Image enhancement unit: used to improve image clarity by using image enhancement method;

[0019] Feature extraction unit: used to extract key features in the image using a convolutional neural network, including the bright spot area of ​​the spark, the edge contour of the smoke, and the deformation area of ​​the device surface;

[0020] Target image segmentation unit: used to segment the damaged area using the Mask R-CNN algorithm and generate pixel-level boundaries of device damage.

[0021] Preferably, the data synchronization submodule is used to align the time series data of the sensor data and the image frame according to the timestamp.

[0022] Preferably, the fault diagnosis module includes:

[0023] Data acquisition unit: used to acquire the feature vector in the multimodal data fusion module;

[0024] Fault identification unit: used to build a fault identification model based on characteristic phase quantities and deep neural networks, and obtain fault types including short circuit, overload, ground fault and disconnection fault;

[0025] Fault factor analysis unit: used to analyze the correlation between fault type, sensor abnormality characteristics and image damage characteristics using Bayesian network;

[0026] Fault image analysis unit: used to segment the image into damaged areas through Mask R-CNN, calculate the area ratio of the damaged area, and evaluate the degree of damage;

[0027] Damage level classification unit: used to classify the damage area into three damage levels: slight, moderate and severe.

[0028] Preferably, the marking module includes:

[0029] Diagnosis result conversion unit: used to convert the diagnosis result in the fault diagnosis module into a text description, and dynamically update the text description according to the diagnosis result;

[0030] Labeling rule library construction unit: used to construct a set of labeling rules corresponding to each fault type, fault factor, and damage degree to form a rule labeling library;

[0031] Annotation text generation unit: used to generate annotation text according to the rule annotation library and text description.

[0032] Preferably, the visualization module includes:

[0033] Sensor data curve annotation unit: used to draw dynamic waveforms of current, voltage and frequency, mark abnormal fluctuation points, and mark the abnormal waveforms with corresponding annotation texts;

[0034] Image data annotation unit: used to annotate the damaged area with different colors according to the segmentation results generated by Mask R-CNN, and annotate the corresponding text annotation of the damaged area;

[0035] Interface providing unit: used to display the annotation results of the sensor data curve annotation unit and the annotation results of the image data annotation unit to the user through the interface. The user can also zoom in or out the device image through the interface to view the details of the damaged area.

[0036] The present invention provides a method for automatic labeling of electric power multimodal data based on deep learning, characterized in that it comprises the following steps:

[0037] S1: Collect multimodal data in the power system, including data from various sensors and image data from cameras;

[0038] S2: preprocessing the multimodal data;

[0039] S3: Use a multimodal neural network to fuse sensor data and image data to construct a unified feature vector;

[0040] S4: performing fault diagnosis according to the feature vector to obtain fault information, including fault type, fault cause and degree of damage to the equipment corresponding to the fault;

[0041] S5: Annotate the fault information according to a preset rule annotation library to generate an annotation text;

[0042] S6: used to generate visualization content according to the annotated text and the multimodal data, visualize the visualization content, and display it to the user.

[0043] Preferably, step S4 is specifically as follows: obtaining the characteristic vector in the multimodal data fusion module; constructing a fault recognition model based on the characteristic phasor and using a deep neural network to obtain fault types including short circuit, overload, ground fault and disconnection fault; using a Bayesian network to analyze the correlation between the fault type, sensor abnormality characteristics and image damage characteristics; segmenting the image into damaged areas through Mask R-CNN, and calculating the area ratio of the damaged area to evaluate the degree of damage; and dividing the damage into three damage levels: slight, moderate and severe according to the damaged area.

[0044] The present invention provides a deep learning-based automatic labeling system and method for electric power multimodal data, which can combine sensor data (such as current, voltage, frequency) and image data (such as equipment damage images) to comprehensively consider the multi-dimensional characteristics of different fault types. By integrating the two, it can provide more accurate and comprehensive fault identification results than a single data source;

[0045] Through LSTM, time series analysis can effectively capture short-term abnormal fluctuations in the power system and timely discover potential fault points. By extracting key features (such as sparks, smoke, equipment deformation, etc.), image data can capture abnormal phenomena on the surface of the equipment in real time; by analyzing the correlation between fault type, sensor abnormal features and image damage features, it can reveal complex fault mechanisms; through multi-level analysis, it can accurately confirm the root cause of the fault from different angles (sensor data, image data), and combine the fault type (such as short circuit, overload, ground fault, etc.) for deep inference to reduce human judgment errors. Mask R-CNN is used for pixel-level segmentation of damaged areas, which can accurately calibrate the degree of damage in various areas on the surface of the equipment. By calculating the area ratio of the damaged area, the degree of equipment damage can be quantified, which helps to evaluate the maintenance priority and the required maintenance resources. Based on the image segmentation results, the damaged area is divided into three levels: slight, moderate, and severe, which can provide fault handling personnel with clear judgment of the degree of damage and optimize maintenance decisions. According to the rule base, the annotation text is automatically generated to convert information such as fault type, cause, and degree of damage into easy-to-understand text descriptions to reduce manual intervention. The rule base dynamically updates the text description according to different fault types and damage characteristics to ensure the real-time and accuracy of the annotation content, reducing manual annotation. Through the dynamic waveform of sensor data and the annotation display of equipment images, users can intuitively see the process and location of the fault, helping operation and maintenance personnel to quickly determine the cause and specific location of the fault, reducing the time for problem diagnosis. Users can zoom in and out of the equipment image through the interface to view the details of the damaged area. This interactive design improves the operability and visualization of the system, helping users to understand the specific situation of the fault more deeply. Through the real-time collection and analysis of sensor data and image data, the system can respond at the early stage of the fault and issue a fault alarm in time. Combining the annotation text and visualization results provided by the system, operation and maintenance personnel can respond quickly to avoid the spread of faults and reduce system downtime. Based on the accurate assessment of equipment damage and the diagnosis results of fault types, data support is provided for maintenance decisions. The system is suitable for power systems of various sizes, from small distribution networks to large power backbone networks, and can flexibly respond to the fault diagnosis and maintenance needs of equipment of different types and sizes. Automatic annotation and automated fault diagnosis greatly reduce manual participation. Operation and maintenance personnel no longer need to manually check the operating status of each device. The system can automatically identify and annotate the fault type and degree of damage. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0047] Figure 1 An architecture diagram of a deep learning-based automatic annotation system for electric power multimodal data provided by the present invention;

[0048] Figure 2 A flowchart of a method for automatic labeling of electric power multimodal data based on deep learning provided by the present invention; DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention are clearly and completely described. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0051] refer to Figure 1 The present invention provides a deep learning-based automatic labeling system for electric power multimodal data, comprising the following:

[0052] Data acquisition module 11: used to collect multimodal data in the power system, including data from various sensors and image data from cameras;

[0053] Data preprocessing module 12: used for preprocessing multimodal data;

[0054] Multimodal data fusion module 13: used to fuse sensor data and image data using a multimodal neural network to construct a unified feature vector;

[0055] Specifically, the multimodal neural network includes Transformer, which takes the feature vectors of sensor data and image data as inputs of Transformer respectively, and processes the temporal order and spatial information of different modal features through position encoding. In Transformer, the feature vector of each modality interacts with the feature vectors of other modalities to calculate the attention weight. Transformer can determine the importance of the features of each modality for the final prediction, and after multiple attention layer calculations, a unified fusion feature vector is obtained.

[0056] Fault diagnosis module 14: used to perform fault diagnosis according to the feature vector to obtain fault information, including fault type, fault cause and degree of damage to the equipment corresponding to the fault;

[0057] Annotation module 15: used to annotate fault information according to a preset rule annotation library and generate annotation text;

[0058] Visualization module 16: used to generate visualization content according to the annotated text and multimodal data, visualize the visualization content, and display it to the user.

[0059] The present invention provides a deep learning-based automatic labeling system for multimodal power data. By integrating different types of sensor data and image information, it can more comprehensively understand the equipment status, thereby improving the accuracy of fault type and cause judgment. This multi-dimensional data combination helps to discover problems that are difficult to detect in a single mode. By automatically generating label text according to preset rules, the workload of manual operations is greatly reduced, and the consistency and standardization of labeling are guaranteed. By intuitively displaying fault information and related data, it promotes communication and collaboration across departments, and also facilitates decision makers to quickly respond. It can accurately and quickly detect hidden dangers in the power system, which helps to take measures in advance to prevent accidents and ensure the safety of workers.

[0060] In a preferred embodiment, the data preprocessing module 12 includes a sensor data processing submodule, an image data processing submodule and a data synchronization submodule. The sensor data processing submodule includes:

[0061] Noise processing unit: used to remove noise from sensor data using digital filtering algorithms to enhance data stability. Digital filtering algorithms include Kalman filtering and sliding average methods, for example, to filter out abnormal frequency fluctuation signals caused by external electromagnetic interference.

[0062] Feature extraction unit: used to extract features from sensor data, including abnormal amplitudes of current and voltage, frequency fluctuation range, etc.; LSTM is used to encode time series features to generate feature vectors of fixed length;

[0063] Anomaly detection unit: used to detect abnormal points in data using time series models, including sudden increases in current, instantaneous decreases in voltage, etc.

[0064] In a preferred embodiment, the image processing submodule includes:

[0065] Image enhancement unit: used to adopt image enhancement methods such as histogram equalization and contrast enhancement to improve image clarity and ensure that smoke, sparks, etc. in the image can be clearly identified.

[0066] Feature extraction unit: used to extract key features in the image using a convolutional neural network, including the bright spot area of ​​the spark, the edge contour of the smoke, and the deformation area of ​​the device surface;

[0067] Target image segmentation unit: used to segment the damaged area using the Mask R-CNN algorithm and generate pixel-level boundaries of device damage.

[0068] In a preferred embodiment, the data synchronization submodule is used to align the time series data of the sensor data and the image frame according to the timestamp to ensure that the two correspond to the fault information within the same time window, and for data with different sampling frequencies, such as the sensor collects 100 times per second and the image collects 10 frames per second, the interpolation method is used to fill the low-frequency data.

[0069] In a preferred embodiment, the fault diagnosis module 14 includes:

[0070] Data acquisition unit: used to obtain feature vectors in the multimodal data fusion module;

[0071] Fault identification unit: used to build a fault identification model based on characteristic phase quantities and deep neural networks, and obtain fault types including short circuit, overload, ground fault and disconnection fault;

[0072] Among them, the method of training the fault recognition model is to use labeled multimodal data to train the fault recognition model, combined with data enhancement such as simulating different types of sensor anomalies and image damage to improve the generalization ability of the model.

[0073] Fault factor analysis unit: used to use Bayesian network to analyze the correlation between fault type, sensor abnormality characteristics and image damage characteristics; specifically, define sensor data nodes including current abnormality, frequency fluctuation, voltage instability, etc., define image data nodes including sparks, smoke, equipment surface deformation, etc., define fault type nodes including short circuit, overload, ground fault, etc.; define edges to represent the correlation between different nodes through directed edges, such as current abnormality → short circuit fault: current abnormality may cause short circuit fault; spark → short circuit fault: spark is one of the common manifestations of short circuit fault; smoke → fire: smoke may be related to equipment fire. Define conditional probability distribution, for each node, define its conditional probability distribution according to historical data. For example, for the "current abnormality" node, the probability of current abnormality can be estimated based on historical data; for the "spark" node, the probability of spark occurrence can be determined based on the probability of current abnormality.

[0074] Fault image analysis unit: used to segment the image into damaged areas through Mask R-CNN, calculate the area ratio of the damaged area, and evaluate the degree of damage;

[0075] Specifically, the smoke concentration and spark intensity are quantified. The smoke concentration is obtained by grayscale histogram statistics, and the spark intensity is calculated by brightness value distribution.

[0076] Damage level classification unit: used to classify the damage area into three levels: slight, moderate, and severe. Specifically, slight means the damage area is less than 10%, with no obvious functional impact. Moderate means the damage area is between 10% and 30%, with functional abnormalities. Severe means the damage area is greater than 30%, with complete failure of the device function.

[0077] In a preferred embodiment, the annotation module 15 includes:

[0078] Diagnosis result conversion unit: used to convert the diagnosis result in the fault diagnosis module 14 into a text description, and dynamically update the text description according to the diagnosis result;

[0079] Labeling rule library construction unit: used to construct a set of labeling rules for each fault type, fault factor, and damage degree to form a rule labeling library; specifically, Rule 1: When the fault type is "short circuit" and the abnormal current fluctuation is the main cause, it is marked as "the short circuit fault is caused by a sudden increase in current, which may be accompanied by sparks." Rule 2: When the image analysis detects a deformed area and there is no significant change in voltage, it is marked as "the equipment is damaged by external force, and the surface is significantly deformed", etc.

[0080] Annotation text generation unit: used to generate annotation text according to the rule annotation library and text description.

[0081] In a preferred embodiment, the visualization module 16 comprises:

[0082] Sensor data curve annotation unit: used to draw dynamic waveforms of current, voltage and frequency, mark abnormal fluctuation points, and mark the abnormal waveforms with corresponding annotation texts;

[0083] Image data annotation unit: used to annotate the damaged areas with different colors according to the segmentation results generated by Mask R-CNN, and annotate the corresponding text annotations of the damaged areas; for example, red indicates a serious area, where a short circuit fault caused by a sudden increase in current occurs, which may be accompanied by sparks.

[0084] Interface providing unit: used to display the annotation results of the sensor data curve annotation unit and the image data annotation unit to the user through the interface. The user can also zoom in or out the device image through the interface to view the details of the damaged area.

[0085] refer to Figure 2 The present invention provides a method for automatic labeling of electric power multimodal data based on deep learning, comprising the following steps:

[0086] S1: Collect multimodal data in the power system, including data from various sensors and image data from cameras;

[0087] S2: Preprocessing of multimodal data;

[0088] S3: Use a multimodal neural network to fuse sensor data and image data to construct a unified feature vector;

[0089] S4: Perform fault diagnosis based on the feature vector to obtain fault information, including fault type, fault cause and degree of damage to the equipment corresponding to the fault;

[0090] S5: Annotate the fault information according to the preset rule annotation library to generate annotation text;

[0091] S6: used to generate visualization content based on the annotated text and multimodal data, visualize the visualization content, and display it to the user.

[0092] In a preferred embodiment, step S4 is specifically as follows: obtaining a feature vector in a multimodal data fusion module; constructing a fault recognition model based on the feature phasor and using a deep neural network to obtain fault types including short circuit, overload, ground fault and disconnection fault; using a Bayesian network to analyze the causal relationship between the fault type, sensor abnormality features and image damage features; using Mask R-CNN to segment the image into damaged areas, and calculating the area ratio of the damaged area to evaluate the degree of damage; and dividing the damage into three levels: slight, moderate and severe according to the damaged area.

[0093] The present invention provides a deep learning-based automatic labeling system and method for electric power multimodal data, which can combine sensor data (such as current, voltage, frequency) and image data (such as equipment damage images) to comprehensively consider the multi-dimensional characteristics of different fault types. By integrating the two, it can provide more accurate and comprehensive fault identification results than a single data source;

[0094] Through LSTM, time series analysis can effectively capture short-term abnormal fluctuations in the power system and timely discover potential fault points. By extracting key features (such as sparks, smoke, equipment deformation, etc.), image data can capture abnormal phenomena on the surface of the equipment in real time; by analyzing the correlation between fault type, sensor abnormal features and image damage features, it can reveal complex fault mechanisms; through multi-level analysis, it can accurately confirm the root cause of the fault from different angles (sensor data, image data), and combine the fault type (such as short circuit, overload, ground fault, etc.) for deep inference to reduce human judgment errors. Mask R-CNN is used for pixel-level segmentation of damaged areas, which can accurately calibrate the degree of damage in various areas on the surface of the equipment. By calculating the area ratio of the damaged area, the degree of equipment damage can be quantified, which helps to evaluate the maintenance priority and the required maintenance resources. Based on the image segmentation results, the damaged area is divided into three levels: slight, moderate, and severe, which can provide fault handling personnel with clear judgment of the degree of damage and optimize maintenance decisions. According to the rule base, the annotation text is automatically generated to convert information such as fault type, cause, and degree of damage into easy-to-understand text descriptions to reduce manual intervention. The rule base dynamically updates the text description according to different fault types and damage characteristics to ensure the real-time and accuracy of the annotation content, reducing manual annotation. Through the dynamic waveform of sensor data and the annotation display of equipment images, users can intuitively see the process and location of the fault, helping operation and maintenance personnel to quickly determine the cause and specific location of the fault, reducing the time for problem diagnosis. Users can zoom in and out of the equipment image through the interface to view the details of the damaged area. This interactive design improves the operability and visualization of the system, helping users to understand the specific situation of the fault more deeply. Through the real-time collection and analysis of sensor data and image data, the system can respond at the early stage of the fault and issue a fault alarm in time. Combining the annotation text and visualization results provided by the system, operation and maintenance personnel can respond quickly to avoid the spread of faults and reduce system downtime. Based on the accurate assessment of equipment damage and the diagnosis results of fault types, data support is provided for maintenance decisions. The system is suitable for power systems of various sizes, from small distribution networks to large power backbone networks, and can flexibly respond to the fault diagnosis and maintenance needs of equipment of different types and sizes. Automatic annotation and automated fault diagnosis greatly reduce manual participation. Operation and maintenance personnel no longer need to manually check the operating status of each device. The system can automatically identify and annotate the fault type and degree of damage.

Claims

1. A deep learning-based automatic annotation system for electric power multimodal data, characterized in that: These include: Data acquisition module: used to collect multimodal data in the power system, including data from various sensors and image data from cameras; Data preprocessing module: used for preprocessing the multimodal data; Multimodal data fusion module: used to fuse sensor data and image data using a multimodal neural network to construct a unified feature vector; Fault diagnosis module: used to perform fault diagnosis according to the feature vector to obtain fault information, including fault type, fault cause and degree of damage to the equipment corresponding to the fault; Annotation module: used to annotate the fault information according to a preset rule annotation library and generate annotated text; Visualization module: used to generate visualization content according to the annotated text and the multimodal data, visualize the visualization content, and display it to the user.

2. According to the deep learning-based automatic annotation system for electric power multimodal data according to claim 1, it is characterized in that: The data preprocessing module includes a sensor data processing submodule, an image data processing submodule and a data synchronization submodule. The sensor data processing submodule includes: Noise processing unit: used to remove noise from sensor data using digital filtering algorithms; Feature extraction unit: used to extract features from sensor data, including abnormal amplitudes of current and voltage, and frequency fluctuation range; LSTM is used to encode time series features to generate feature vectors of fixed length; Anomaly detection unit: used to detect anomalies in data using time series models.

3. According to the deep learning-based automatic annotation system for electric power multimodal data according to claim 2, it is characterized in that: The image processing submodule comprises: Image enhancement unit: used to improve image clarity by using image enhancement method; Feature extraction unit: used to extract key features in the image using a convolutional neural network, including the bright spot area of ​​the spark, the edge contour of the smoke, and the deformation area of ​​the device surface; Target image segmentation unit: used to segment the damaged area using the Mask R-CNN algorithm and generate pixel-level boundaries of device damage.

4. According to the deep learning-based automatic annotation system for electric power multimodal data according to claim 2, it is characterized in that: The data synchronization submodule is used to align the time series data of the sensor data and the image frame according to the timestamp.

5. The deep learning-based automatic annotation system for electric power multimodal data according to claim 1 is characterized in that: The fault diagnosis module comprises: Data acquisition unit: used to acquire the feature vector in the multimodal data fusion module; Fault identification unit: used to build a fault identification model based on characteristic phase quantities and deep neural networks, and obtain fault types including short circuit, overload, ground fault and disconnection fault; Fault factor analysis unit: used to analyze the correlation between fault type, sensor abnormality characteristics and image damage characteristics using Bayesian network; Fault image analysis unit: used to segment the image into damaged areas through Mask R-CNN, calculate the area ratio of the damaged area, and evaluate the degree of damage; Damage level classification unit: used to classify the damage area into three damage levels: slight, moderate and severe.

6. The deep learning-based automatic annotation system for electric power multimodal data according to claim 1 is characterized in that: The marking module comprises: Diagnosis result conversion unit: used to convert the diagnosis result in the fault diagnosis module into a text description, and dynamically update the text description according to the diagnosis result; Labeling rule library construction unit: used to construct a set of labeling rules corresponding to each fault type, fault factor, and damage degree to form a rule labeling library; Annotation text generation unit: used to generate annotation text according to the rule annotation library and text description.

7. The deep learning-based automatic annotation system for electric power multimodal data according to claim 1 is characterized in that: The visualization module comprises: Sensor data curve annotation unit: used to draw dynamic waveforms of current, voltage and frequency, mark abnormal fluctuation points, and mark the abnormal waveforms with corresponding annotation texts; Image data annotation unit: used to annotate the damaged area with different colors according to the segmentation results generated by Mask R-CNN, and annotate the corresponding text annotation of the damaged area; Interface providing unit: used to display the annotation results of the sensor data curve annotation unit and the annotation results of the image data annotation unit to the user through the interface. The user can also zoom in or out the device image through the interface to view the details of the damaged area.

8. A method for automatic labeling of electric power multimodal data based on deep learning, characterized in that: The steps include: S1: Collect multimodal data in the power system, including data from various sensors and image data from cameras; S2: preprocessing the multimodal data; S3: Use a multimodal neural network to fuse sensor data and image data to construct a unified feature vector; S4: performing fault diagnosis according to the feature vector to obtain fault information, including fault type, fault cause and degree of damage to the equipment corresponding to the fault; S5: Annotate the fault information according to a preset rule annotation library to generate an annotation text; S6: used to generate visualization content according to the annotated text and the multimodal data, visualize the visualization content, and display it to the user.

9. The method for automatic labeling of electric power multimodal data based on deep learning according to claim 9 is characterized in that: The step S4 is specifically as follows: obtaining the feature vector in the multimodal data fusion module; constructing a fault recognition model based on the feature phasor and using a deep neural network to obtain fault types including short circuit, overload, ground fault and disconnection fault; using a Bayesian network to analyze the correlation between the fault type, sensor abnormality characteristics and image damage characteristics; segmenting the image into damaged areas through Mask R-CNN, and calculating the area ratio of the damaged area to evaluate the degree of damage; and dividing the damage into three levels of slight, moderate and severe according to the damaged area.

Images