Deep fusion system and method for multi-modal data of power distribution network

Abstract

The invention discloses a deep fusion system and method for multi-modal data of a power distribution network. The deep fusion system comprises an acquisition and uploading module, a classification and marking module, a feature extraction module and an integration and summarization module, the acquisition and uploading module is used for performing data monitoring and acquisition by using different types of sensors, acquiring the working state through a monitoring probe, and performing acquisition according to an uploaded work diary; the classification marking module is used for stamping timestamps on the information acquired by the acquisition and uploading module and pre-classifying the acquired information; the feature extraction module is used for performing feature extraction on the multi-modal data classified by the classification marking module; and the integrating and summarizing module is used for integrating the extracted features to obtain a multi-modal fusion data block, so that deep fusion of the data is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of power systems, and in particular to a system and method for deep fusion of multimodal data of a distribution network. Background Art

[0002] Multimodal data is data composed of two or more modes, which can be different types of data, such as pictures, texts, audio, video, etc. After searching, it is found that the current technology involves the field of data fusion technology. The system includes a data receiving module, a data classification module and a data fusion module. The data receiving module, the data classification module and the data fusion module are connected in series in turn; the data receiving module is used to receive data from different data sources of each power equipment, and the data includes equipment monitoring data, power load data and meteorological data; the data classification module is used to perform feature classification on the equipment monitoring data, power load data and meteorological data of each power equipment, respectively, to obtain feature classification data of each power equipment, and the feature classification data includes equipment feature data, load feature data and meteorological feature data; the data fusion module is used to perform weighted fusion on the feature classification data of each power equipment to obtain fused data. Its main feature is that it can improve the readability and availability of data and improve the accuracy of data fusion.

[0003] In summary, most of the existing distribution network data fusion is single-modal fusion, that is, the data monitored by multiple types of sensors are classified and summarized, which requires manual sorting and fusion of the received summary data with the staff's work log, which is not convenient to operate. In view of the above problems, the existing equipment needs to be improved. Summary of the invention

[0004] In view of the above existing problems, the present invention is proposed.

[0005] Therefore, the present invention provides a deep fusion system and method for multimodal data of a distribution network to solve the problem that most of the existing distribution network data fusion is single-modal fusion, that is, the data monitored by multiple types of sensors are classified and summarized, which requires manual sorting and fusion of the received summarized data with the work logs of the staff, which is not convenient to operate.

[0006] In order to solve the above technical problems, the present invention provides the following technical solutions:

[0007] In a first aspect, the present invention provides a deep fusion system for multimodal data of a distribution network, including: a collection and uploading module, a classification and marking module, a feature extraction module, and an integration and summary module;

[0008] The collection and uploading module is used to monitor and collect data using different types of sensors, collect the working status through monitoring probes, and collect data based on the uploaded work diary;

[0009] The classification marking module is used to add a timestamp to the information collected by the collection and upload module, and to pre-classify the collected information;

[0010] The feature extraction module is used to extract features from the multimodal data after classification by the classification and marking module;

[0011] The integration and summary module is used to integrate the extracted features to obtain a multimodal fusion data block.

[0012] As a preferred solution of the deep fusion system of multimodal data of distribution network described in the present invention, wherein: the classification marking module classifies data through a classification algorithm, and determines cluster data through an evaluation method.

[0013] As a preferred solution of the deep fusion system of multimodal data of distribution network described in the present invention, wherein: the feature extraction module includes linear discriminant analysis, wavelet transform and principal component analysis;

[0014] Linear discriminant analysis is used to find the direction that maximizes the difference between categories and minimizes the difference within categories, and is used for feature extraction of text information;

[0015] The wavelet transform is used for feature extraction of image information;

[0016] The principal component analysis is used for feature extraction of data information.

[0017] In a second aspect, the present invention provides a method for deep fusion of multimodal data of a distribution network, comprising:

[0018] Acquire a multimodal data set, classify the multimodal data set using a classification algorithm, and add a timestamp to the classified multimodal data set to obtain a first multimodal data set;

[0019] Extracting features from the first multimodal dataset using different feature extraction techniques;

[0020] The extracted features are aggregated into blocks according to the corresponding attributes and stored encrypted.

[0021] As a preferred solution of the method for deep fusion of multimodal data of distribution network described in the present invention, wherein: classifying the multimodal data by using a classification algorithm includes:

[0022] Randomly select K data points as the initial cluster centers;

[0023] For each data point in the multimodal dataset, the distance between the data point and each cluster center is calculated, and the data point is assigned to the cluster with the closest cluster center.

[0024] Calculate the mean of all data points in each cluster and use it as the center of the cluster;

[0025] Recalculate the distance from each data point in the cluster to the cluster center and calculate the total square error;

[0026] The data points are assigned repeatedly until a stopping criterion is met.

[0027] As a preferred solution of the method for deep fusion of multimodal data of distribution network described in the present invention, wherein: using different feature extraction techniques to extract features from the first multimodal data set includes:

[0028] For the first multimodal data set of different categories, the mean of the samples as well as the intra-class scatter matrix and the inter-class scatter matrix are calculated;

[0029] Based on the mean of the samples and the intra-class scatter matrix and the inter-class scatter matrix, the generalized eigenvalue is solved;

[0030] Select the eigenvector corresponding to the maximum eigenvalue to form a new feature space;

[0031] Project the original data into the new feature space to obtain the reduced-dimensional data, namely the first eigenvalue.

[0032] As a preferred solution of the method for deep fusion of multimodal data of distribution network described in the present invention, it also includes:

[0033] Decomposing the first multimodal data set into different frequency sub-bands using a wavelet basis function, and extracting a second eigenvalue therefrom;

[0034] Standardizing the first multimodal data set, calculating a covariance matrix of the standardized data, and calculating eigenvalues ​​and eigenvectors of the covariance matrix;

[0035] The principal component is selected according to the size of the eigenvalue, and the original data is projected onto the principal component to obtain the reduced-dimensional data, namely the third eigenvalue.

[0036] As a preferred solution of the method for deep fusion of multimodal data of distribution network described in the present invention, the encrypted storage includes:

[0037] When the encrypted stored data is attacked or the number of key inputs reaches the set upper limit, a warning message is selectively sent to warn.

[0038] In a third aspect, the present invention provides a computing device, comprising:

[0039] Memory and processor;

[0040] The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the deep fusion method of distribution network multimodal data are implemented.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the steps of the method for deep fusion of multimodal data of the distribution network.

[0042] Compared with the prior art, the present invention has the following beneficial effects: the present invention uses electrical, magnetic, temperature and humidity, and mechanical sensors, and the system can monitor the electrical parameters, environmental conditions, and physical status of the distribution network, and provide real-time operation data; monitoring probes (such as infrared cameras) provide visual monitoring, capture visual information such as equipment anomalies and environmental changes, and enhance the intuitive understanding of the status of the distribution network; the work logs uploaded by staff provide operation records and maintenance information, and provide additional contextual information for data analysis; automated data collection and processing reduce the need for manual operation, reduce human errors, and increase the speed of data processing; the integration and summary module fuses data from different sources and types to form a unified data view, and improves the readability of the data; combined with sensor data, visual information, and work logs, the system can provide more comprehensive data analysis results, improve the accuracy of data fusion, detect data anomalies and potential security threats, respond in a timely manner, and improve the security of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0044] Figure 1 A system flow chart of a system and method for deep fusion of multimodal data of a distribution network according to an embodiment of the present invention;

[0045] Figure 2 The present invention is a schematic diagram of the overall process logic of a system and method for deep fusion of multimodal data of a distribution network according to an embodiment of the present invention. DETAILED DESCRIPTION

[0046] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0047] Example 1

[0048] Reference Figure 1-Figure 2 , is an embodiment of the present invention, and provides a deep fusion system for multimodal data of a distribution network, comprising:

[0049] Figure 1 The system structure diagram of a deep fusion system and method for multimodal data of a distribution network is shown, including: an acquisition and uploading module 100, a classification and marking module 200, a feature extraction module 300, and an integration and summary module 400;

[0050] The collection and uploading module 100 is used to monitor and collect data using different types of sensors 101, collect the working status through the monitoring probe 102, and collect data based on the uploaded work diary;

[0051] The classification marking module 200 is used to add a timestamp to the information collected by the collection and uploading module 100 and pre-classify the collected information;

[0052] The feature extraction module 300 is used to extract features from the multimodal data classified by the classification and labeling module 200;

[0053] The integration and summary module 400 is used to integrate the extracted features to obtain a multimodal fusion data block.

[0054] It should be noted that the collection and uploading module 100 can use various types of sensors 101 to monitor and collect data, including electrical, magnetic, temperature and humidity, and mechanical sensors, as well as image information captured by the monitoring probe 102 and work logs uploaded by staff. The multi-source data collection method improves the comprehensiveness and accuracy of the data; the classification and marking module 200 adds a timestamp to the collected data and pre-classifies it, which helps with time synchronization and preliminary organization of the data, and provides convenience for subsequent processing.

[0055] The feature extraction module 300 performs feature extraction on the pre-classified multimodal data, and can identify and extract key information in the data; the integration and summary module 400 integrates the extracted features to form a multimodal fusion data block, which can provide a more comprehensive data view and enhance the depth and breadth of data analysis.

[0056] Specifically, the sensor 101 used by the acquisition and uploading module 100 includes three aspects: electrical sensor, magnetic sensor, temperature and humidity sensor, and mechanical sensor, and the electrical sensor includes a pressure transmitter or pressure sensor, a capacitive sensor, and a resistive sensor;

[0057] Magnetic sensors include electromagnetic flow sensors, thermal flow sensors, inductive voltage sensors, magnetic induction current sensors, magnetoelectric sensors and electromagnetic sensors;

[0058] The temperature and humidity sensors include temperature sensors and humidity sensors, and the temperature sensors include thermocouple type, thermistor type and infrared thermometer type, while the humidity sensors include capacitive sensors and resistive sensors;

[0059] The monitoring probe 102 is an infrared camera, and there are two types of infrared cameras, namely, priority transmission and wireless transmission.

[0060] In the embodiment of the present application, the classification labeling module 200 classifies data through a classification algorithm and determines clustered data through an evaluation method.

[0061] Specifically, the classification and labeling module 200 classifies data using a K-means clustering algorithm, and its principles include initial selection, sample allocation, centroid update, and repeated iterations. When selecting the K value, the number of clusters is determined by the elbow rule or the silhouette coefficient method;

[0062] In the embodiment of the present application, the feature extraction module 300 includes linear discriminant analysis, wavelet transform and principal component analysis;

[0063] Linear discriminant analysis is used to find the direction that maximizes the difference between categories and minimizes the difference within categories, and is used for feature extraction of text information;

[0064] Wavelet transform is used to extract features from image information;

[0065] Principal component analysis is used to extract features from data information.

[0066] Specifically, wavelet transform is used for feature extraction of image information, and wavelet transform completes feature extraction by decomposing an image into components of different scales and frequencies; principal component analysis is used for feature extraction of data information, and completes feature extraction by transforming the original random vector into a new random vector with unrelated components through orthogonal transformation.

[0067] It should be noted that the system can comprehensively collect the operating data of the distribution network, including electrical parameters, environmental parameters and visual information, by using various types of sensors 101 (electrical, magnetic, temperature and humidity, mechanical) and infrared cameras, thereby improving the comprehensiveness and diversity of the data. Through the real-time monitoring function of the monitoring probe 102 (infrared camera), the system can timely capture the operating status of the distribution network and respond quickly to abnormal situations. It uses the K-means clustering algorithm for data classification, and determines the optimal number of clusters through the elbow rule or silhouette coefficient method, thereby improving the efficiency and accuracy of data processing, extracting key features for different types of data, and enhancing the expression ability of features.

[0068] like Figure 2 As shown, this embodiment also provides a method for deep fusion of multimodal data of a distribution network, including:

[0069] S100: Acquire a multimodal data set, classify the multimodal data set using a classification algorithm, and add a timestamp to the classified multimodal data set to obtain a first multimodal data set;

[0070] S200: extracting features from the first multimodal data set using different feature extraction techniques;

[0071] S300: The extracted features are aggregated into blocks according to corresponding attributes and encrypted and stored.

[0072] It should be noted that the coordinated use of the collection and upload module, the classification and marking module, the feature extraction module and the integration and summary module can effectively solve the problem that most of the existing distribution network data fusion is single-modal fusion, that is, the data monitored by multiple types of sensors are classified and summarized, which requires manual sorting and fusion of the received summary data with the staff's work log, and the operation is not convenient. The collection and upload module collects multi-modal data from sensors, monitoring probes and work logs uploaded by staff in real time, and then the classification and marking module and the feature extraction module classify and extract features from the collected data in turn. Finally, the corresponding extracted text, images and data are integrated through the integration and summary module, thereby realizing deep data fusion.

[0073] In the embodiment of the present application, the above step S100 includes the following sub-steps A1-A5;

[0074] In A1: K data points are randomly selected as initial cluster centers;

[0075] In A2: For each data point in the multimodal dataset, the distance between the data point and each cluster center is calculated, and the data point is assigned to the cluster with the closest cluster center;

[0076] In A3: Calculate the mean of all data points in each cluster and use it as the center of the cluster;

[0077] In A4: recalculate the distance from each data point in the cluster to the cluster center and calculate the total square error;

[0078] In A5: Data points are assigned repeatedly until a stopping condition is met.

[0079] Specifically, the mean of all data points in each cluster is calculated. The mean may include the arithmetic mean, geometric mean, median, etc.;

[0080] The stopping condition can be that the change of cluster center is less than a certain threshold, or the preset maximum number of iterations is reached, or the decrease of error function is less than a certain value;

[0081] It should be noted that the K-means clustering algorithm has the characteristics of simple principle, relatively easy implementation, strong interpretability, fast convergence speed and optimized iteration function; that is, high computational efficiency, that is, the time complexity is approximately linear, and results can be obtained relatively quickly for large-scale data sets; strong interpretability, the cluster center has practical interpretability; fast convergence speed, in most cases, the K-means algorithm can converge to the local optimal solution relatively quickly; optimized iteration function, it can iteratively correct based on the already obtained clustering to improve the accuracy of clustering.

[0082] In the embodiment of the present application, the above step S200 includes the following sub-steps B1-B5;

[0083] In B1: for the first multimodal data set of different categories, the mean of the samples as well as the intra-class scatter matrix and the inter-class scatter matrix are calculated;

[0084] In B2: Based on the mean of the samples and the intra-class scatter matrix and the inter-class scatter matrix, the generalized eigenvalue is solved;

[0085] In B3: select the eigenvector corresponding to the maximum eigenvalue to form a new eigenspace;

[0086] In B4: the original data is projected into the new feature space to obtain the data after dimension reduction, i.e., the first eigenvalue;

[0087] In B5: using a wavelet basis function, decomposing the first multimodal data set into different frequency sub-bands, and extracting a second eigenvalue therefrom;

[0088] In B6: normalizing the first multimodal data set, calculating a covariance matrix of the normalized data, and calculating eigenvalues ​​and eigenvectors of the covariance matrix;

[0089] In B7: The principal component is selected according to the size of the eigenvalue, and the original data is projected onto the principal component to obtain the data after dimensionality reduction, i.e., the third eigenvalue.

[0090] It should be noted that each feature extraction method has its own unique advantages and applicable scenarios. Linear discriminant analysis is good at handling classification problems, wavelet transform is suitable for processing signals with time-frequency characteristics, and principal component analysis is suitable for dimensionality reduction and extracting main features. Combining these methods can complement the shortcomings of these methods and improve the comprehensiveness of feature extraction. By combining the features extracted by different methods, the system can capture the essential characteristics of the data from multiple angles and improve the accuracy of classification, prediction and fault diagnosis.

[0091] In the embodiment of the present application, the above step S300 includes the following sub-step C1;

[0092] In C1: When the encrypted stored data is attacked or the number of key inputs reaches the set upper limit, a warning message is selectively sent to warn.

[0093] Specifically, the upper limit value is set according to the system security policy and the requirements of the actual application scenario, and the system selectively sends warning information to the mailbox, mobile terminal or terminal device of the relevant management personnel for warning.

[0094] It should be noted that by monitoring abnormal behavior, the system can promptly detect potential security threats and take warning measures, thereby enhancing the security of the system. The timely sending of warning information enables managers to respond quickly to security incidents and reduce potential damage. By limiting the number of key attempts and setting up a warning mechanism, the system reduces the risk of data leakage caused by password cracking.

[0095] The above is a schematic scheme of a deep fusion system of multimodal data of a distribution network in this embodiment. It should be noted that the technical scheme of the deep fusion system of multimodal data of the distribution network and the technical scheme of the deep fusion system of multimodal data of the distribution network mentioned above belong to the same concept. The details of the technical scheme of the deep fusion system of multimodal data of the distribution network in this embodiment that are not described in detail can be found in the description of the technical scheme of the deep fusion system of multimodal data of the distribution network mentioned above.

[0096] The system of the method for deep fusion of multimodal data of distribution network in this embodiment includes:

[0097] A classification module, used to obtain a multimodal data set, classify the multimodal data set using a classification algorithm, and add a timestamp to the classified multimodal data set to obtain a first multimodal data set;

[0098] A feature extraction module, used to extract features from the first multimodal data set using different feature extraction techniques;

[0099] The storage module is used to aggregate the extracted features into blocks according to the corresponding attributes and store them in encrypted form.

[0100] This embodiment further provides a computing device, which is applicable to the deep fusion of multimodal data of a distribution network, and includes:

[0101] Memory and processor; the memory is used to store computer executable instructions, and the processor is used to execute computer executable instructions to implement a deep fusion system for distribution network multimodal data as proposed in the above embodiment.

[0102] This embodiment also provides a storage medium on which a computer program is stored. When the program is executed by a processor, a deep fusion system for multimodal data of a distribution network as proposed in the above embodiment is implemented.

[0103] The storage medium proposed in this embodiment and the system for realizing deep fusion of multimodal data of distribution network proposed in the above embodiment belong to the same inventive concept. The technical details not fully described in this embodiment can be referred to the above embodiment, and this embodiment has the same beneficial effects as the above embodiment.

[0104] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course can also be implemented by hardware. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ReadOnly, Memory, ROM), random access memory (Random Access Memory, RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform the methods of various embodiments of the present invention.

Claims

1. A deep fusion system for multimodal data of distribution network, characterized in that: include: A collection and uploading module (100), a classification and marking module (200), a feature extraction module (300), and an integration and summary module (400); The collection and uploading module (100) is used to use different types of sensors (101) to monitor and collect data, collect the working status through the monitoring probe (102), and collect data according to the uploaded work diary; The classification marking module (200) is used to add a timestamp to the information collected by the collection and uploading module (100) and to pre-classify the collected information; The feature extraction module (300) is used to extract features from the multimodal data classified by the classification and labeling module (200); The integration and summary module (400) is used to integrate the extracted features to obtain a multi-modal fusion data block.

2. The deep fusion system of multimodal data of distribution network according to claim 1, characterized in that: include: The classification labeling module (200) performs data classification through a classification algorithm and determines clustered data through an evaluation method.

3. The deep fusion system of multimodal data of distribution network according to claim 2, characterized in that: include: The feature extraction module (300) includes linear discriminant analysis, wavelet transform and principal component analysis; Linear discriminant analysis is used to find the direction that maximizes the difference between categories and minimizes the difference within categories, and is used for feature extraction of text information; The wavelet transform is used for feature extraction of image information; The principal component analysis is used for feature extraction of data information.

4. A method for deep fusion of multimodal data of a distribution network using any one of claims 1 to 3, characterized in that: include: Acquire a multimodal data set, classify the multimodal data set using a classification algorithm, and add a timestamp to the classified multimodal data set to obtain a first multimodal data set; Extracting features from the first multimodal dataset using different feature extraction techniques; The extracted features are aggregated into blocks according to the corresponding attributes and stored encrypted.

5. The method for deep fusion of multimodal data of distribution network according to claim 4, characterized in that: Classifying the multimodal data using a classification algorithm includes: Randomly select K data points as the initial cluster centers; For each data point in the multimodal dataset, the distance between the data point and each cluster center is calculated, and the data point is assigned to the cluster with the closest cluster center. Calculate the mean of all data points in each cluster and use it as the center of the cluster; Recalculate the distance from each data point in the cluster to the cluster center and calculate the total square error; The data points are assigned repeatedly until a stopping criterion is met.

6. The method for deep fusion of multimodal data of distribution network according to claim 4, characterized in that: The first multimodal dataset is subjected to feature extraction using different feature extraction techniques including: For the first multimodal data set of different categories, the mean of the samples as well as the intra-class scatter matrix and the inter-class scatter matrix are calculated; Based on the mean of the samples and the intra-class scatter matrix and the inter-class scatter matrix, the generalized eigenvalue is solved; Select the eigenvector corresponding to the maximum eigenvalue to form a new feature space; Project the original data into the new feature space to obtain the reduced-dimensional data, namely the first eigenvalue.

7. The method for deep fusion of multimodal data of distribution network according to claim 5 or 6, characterized in that: Also includes: Decomposing the first multimodal data set into different frequency sub-bands using a wavelet basis function, and extracting a second eigenvalue therefrom; Standardizing the first multimodal data set, calculating a covariance matrix of the standardized data, and calculating eigenvalues ​​and eigenvectors of the covariance matrix; The principal component is selected according to the size of the eigenvalue, and the original data is projected onto the principal component to obtain the reduced-dimensional data, namely the third eigenvalue.

8. The method for deep fusion of multimodal data of distribution network according to claim 6, characterized in that: Encrypted storage includes: When the encrypted stored data is attacked or the number of key inputs reaches the set upper limit, a warning message is selectively sent to warn.

9. An electronic device, comprising: Memory and processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the method for deep fusion of multimodal data of the distribution network described in any one of claims 4 to 8 are implemented.

10. A computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, can implement the steps of the method for deep fusion of multimodal data of a distribution network as described in any one of claims 4 to 8.

Images