Substation operation regulation and control method and device based on multiple modes

Abstract

The invention discloses a multi-mode-based substation operation regulation and control method and device, and relates to the field of substation operation regulation and control, and the method comprises the steps: obtaining historical monitoring features of a to-be-regulated and controlled substation; based on a self-attention mechanism, respectively calculating attention weights of the voiceprint feature, the visible light feature and the infrared feature, and according to the attention weights, fusing the voiceprint feature, the visible light feature and the infrared feature to obtain a multi-modal fusion feature; based on a graph neural network, enhancing the multi-modal fusion feature to obtain a fusion graph feature; according to the fusion image features and preset multitask loss, training a plurality of preset parallel models to obtain a joint regulation and control model; and obtaining real-time monitoring data of a substation to be regulated and controlled, inputting the real-time monitoring data into the joint regulation and control model to obtain an operation decision, and regulating and controlling the substation to be regulated and controlled according to the operation decision.

Description

Technical Field

[0001] The present application relates to the field of substation operation control, and in particular to a substation operation control method and device based on multi-modality. Background Art

[0002] As an important node in the power system, the safe and stable operation of the substation is crucial to improving the overall reliability of the power grid. To this end, it is necessary to improve the standards for the timeliness and accuracy of the operation and regulation of substations. In the prior art, the operation and regulation of substations are mainly based on various types of monitoring equipment in the station, such as monitoring equipment, sensors, and voiceprint detection equipment. However, each monitoring device often operates independently and lacks coordination, resulting in low efficiency in the comprehensive analysis of the monitoring data of each monitoring device and low accuracy in substation regulation. Therefore, how to improve the accuracy of substation operation and regulation is still a technical problem that needs to be solved urgently in the prior art. Summary of the invention

[0003] The present application provides a substation operation control method and device based on multi-modality to solve the technical problem of insufficient control accuracy of existing substations.

[0004] According to a first aspect of an embodiment of the present application, a multi-modal substation operation control method is provided, comprising:

[0005] Acquire historical monitoring features of the substation to be regulated; wherein the historical monitoring features include voiceprint features, visible light features, and infrared features;

[0006] Based on the self-attention mechanism, the attention weights of the voiceprint feature, the visible light feature and the infrared feature are calculated respectively, and according to the attention weights, the voiceprint feature, the visible light feature and the infrared feature are fused to obtain a multimodal fusion feature;

[0007] Based on a graph neural network, the multimodal fusion features are enhanced to obtain fusion graph features;

[0008] According to the fusion graph features and the preset multi-task loss, multiple preset parallel models are trained to obtain a joint control model; wherein the multiple parallel models are based on multiple preset tasks; the multiple tasks include device anomaly detection, device classification and device status prediction;

[0009] Real-time monitoring data of the substation to be regulated is obtained, the real-time monitoring data is input into the joint regulation model to obtain an operation decision, and the substation to be regulated is regulated according to the operation decision.

[0010] The present application first obtains the historical monitoring features of the substation to be regulated, calculates the attention weight of each feature based on the self-attention mechanism, and then fuses the features to obtain multimodal fusion features. It can integrate the monitoring features of different aspects and modes of the substation to improve the multimodal adaptability of the substation operation and control, and then enhance the multimodal fusion features based on the graph neural network to obtain the fusion graph features. It can continue to use the graph structure to capture and enhance the multimodal features, and then train multiple parallel models according to the fusion graph features and preset multi-task losses to obtain a joint control model. By first dividing the substation operation and control into different situations and using different models for training, it can meet the different task requirements of the substation operation and control, and then improve the accuracy of the substation operation and control when making operation decisions based on real-time monitoring data and regulating the substation.

[0011] In certain embodiments of the present application, the obtaining of historical monitoring characteristics of the substation to be regulated specifically includes:

[0012] Acquire historical monitoring data of the substation to be regulated; wherein the historical monitoring data includes voiceprint data, visible light images and infrared images;

[0013] Based on a bidirectional recurrent neural network, extracting time series features from the voiceprint data to obtain voiceprint features;

[0014] Based on a self-attention image encoder, performing image feature extraction on the visible light image to obtain visible light features;

[0015] Based on a convolutional neural network, image features are extracted from the infrared image to obtain infrared features.

[0016] This application first obtains historical monitoring data, and then extracts features of voiceprint data, visible light images and infrared images based on different types of neural networks to obtain voiceprint features, visible light features and infrared features. By selecting different feature extraction networks based on the category of the data, the corresponding features can be extracted more accurately, thereby improving the accuracy of the acquired features.

[0017] In certain embodiments of the present application, the voiceprint feature, the visible light feature and the infrared feature are fused according to the attention weight to obtain a multimodal fusion feature, specifically including:

[0018] Based on a convolutional neural network, the dimensions of the voiceprint feature and the infrared feature are unified into the dimensions of the visible light feature to obtain a first voiceprint feature and a first infrared feature;

[0019] According to the attention weight, based on a multi-layer self-attention encoder, the first voiceprint feature, the visible light feature and the first infrared feature are spliced ​​and fused to obtain a multimodal fusion feature.

[0020] This application first unifies the dimensions of voiceprint features and infrared features into the dimensions of visible light features based on a convolutional neural network, which can ensure data consistency and prevent data dimensional conflicts, and then concatenates and fuses the first voiceprint features, visible light features and the first infrared features according to the attention weights and multi-layer self-attention encoders to obtain multimodal fusion features, which can integrate monitoring data of different types and modes of substations and improve the multimodal adaptability of substation operation and control.

[0021] In certain embodiments of the present application, the multimodal fusion feature is enhanced based on the graph neural network to obtain the fusion graph feature, specifically including:

[0022] Using the time step of the multimodal fusion feature as a graph node, generating a first multimodal fusion graph corresponding to the multimodal fusion feature;

[0023] Based on a preset edge connection threshold, edges between graph nodes of the first multimodal fusion graph are removed to obtain a second multimodal fusion graph;

[0024] Based on the self-attention mechanism, the edge weights between the graph nodes in the second multimodal fusion graph are calculated, and according to the edge weights, based on a multi-layer graph convolutional network, the second multimodal fusion graph is enhanced to obtain fusion graph features.

[0025] This application first generates a first multimodal fusion graph using the time step of the multimodal fusion feature as a graph node, which can capture multimodal features using the graph structure, and then removes some edges based on a preset edge connection threshold to obtain a second multimodal fusion graph, which can increase the weights of edges that meet the conditions, and then calculate the edge weights based on the self-attention mechanism, and enhance the second multimodal fusion graph based on the edge weights and a multi-layer graph convolutional network. Through the graph convolutional network, the relationship between different modalities and different time steps can be captured, thereby improving the matching degree between the output fusion graph features and the actual ones.

[0026] In certain embodiments of the present application, the preset multiple parallel models are trained according to the fusion graph features and the preset multi-task loss to obtain a joint regulation model, specifically including:

[0027] According to the task categories of the multiple tasks, respectively set the output types of the multiple parallel models;

[0028] Taking the fusion graph features as model input, under the premise of setting the output type, and based on the preset multi-task loss, the multiple parallel models are trained to obtain a joint control model;

[0029] The preset multi-task loss is specifically:

[0030] L=λ1*L anomaly +λ2*L device +λ3*L state ;

[0031] Among them, L is the multi-task loss, L anomaly , L device , L state are equipment abnormality loss, equipment type loss and equipment status loss respectively. λ1, λ2 and λ3 are equipment abnormality loss weight, equipment type loss weight and equipment status loss weight respectively.

[0032] This application first sets the output types of multiple parallel models according to the task categories of multiple tasks. By differentiating the models in output type, a single model can be specialized in a single task, which is beneficial to improving the accuracy of the model. Then, the fusion graph features are used as the model input, and multiple parallel models are trained based on the preset multi-task loss to obtain a joint control model, which can meet the different task requirements of substation operation and control, and then when the substation operation and control are performed according to the joint control model, the accuracy of the substation operation and control is improved.

[0033] According to a second aspect of the implementation of the present application, a multi-modal-based substation operation control device is provided, comprising a feature acquisition module, a feature fusion module, a feature enhancement module, a model training module and an operation control module;

[0034] The feature acquisition module is used to acquire the historical monitoring features of the substation to be regulated; wherein the historical monitoring features include voiceprint features, visible light features and infrared features;

[0035] The feature fusion module is used to calculate the attention weights of the voiceprint feature, the visible light feature and the infrared feature respectively based on the self-attention mechanism, and fuse the voiceprint feature, the visible light feature and the infrared feature according to the attention weights to obtain a multimodal fusion feature;

[0036] The feature enhancement module is used to enhance the multimodal fusion feature based on a graph neural network to obtain a fusion graph feature;

[0037] The model training module is used to train a plurality of preset parallel models according to the fusion graph features and the preset multi-task loss to obtain a joint control model; wherein the plurality of parallel models are based on a plurality of preset tasks; the plurality of tasks include device anomaly detection, device classification and device status prediction;

[0038] The operation control module is used to obtain real-time monitoring data of the substation to be controlled, input the real-time monitoring data into the joint control model, obtain operation decisions, and control the substation to be controlled according to the operation decisions.

[0039] In certain embodiments of the present application, the feature acquisition module includes a historical data acquisition unit, a voiceprint feature extraction unit, a visible light feature extraction unit, and an infrared feature extraction unit;

[0040] The historical data acquisition unit is used to acquire historical monitoring data of the substation to be regulated; wherein the historical monitoring data includes voiceprint data, visible light images and infrared images;

[0041] The voiceprint feature extraction unit is used to extract time series features from the voiceprint data based on a bidirectional recurrent neural network to obtain voiceprint features;

[0042] The visible light feature extraction unit is used to extract image features from the visible light image based on a self-attention image encoder to obtain visible light features;

[0043] The infrared feature extraction unit is used to extract image features from the infrared image based on a convolutional neural network to obtain infrared features.

[0044] In certain embodiments of the present application, the feature fusion module includes a dimension unification unit and a feature fusion unit;

[0045] The dimension unification unit is used to unify the dimension of the voiceprint feature and the dimension of the infrared feature into the dimension of the visible light feature based on a convolutional neural network, so as to obtain a first voiceprint feature and a first infrared feature;

[0046] The feature fusion unit is used to splice and fuse the first voiceprint feature, the visible light feature and the first infrared feature according to the attention weight and based on a multi-layer self-attention encoder to obtain a multimodal fusion feature.

[0047] In certain embodiments of the present application, the feature enhancement module includes a fusion graph generation unit, a fusion edge removal unit, and a fusion graph enhancement unit;

[0048] The fusion graph generating unit is used to generate a first multimodal fusion graph corresponding to the multimodal fusion feature by taking the time step of the multimodal fusion feature as a graph node;

[0049] The fusion edge removal unit is used to remove the edges between the graph nodes of the first multimodal fusion graph based on a preset edge connection threshold to obtain a second multimodal fusion graph;

[0050] The fusion graph enhancement unit is used to calculate the edge weights between the graph nodes in the second multimodal fusion graph based on the self-attention mechanism, and enhance the second multimodal fusion graph based on the edge weights and a multi-layer graph convolutional network to obtain fusion graph features.

[0051] In certain embodiments of the present application, the model training module includes a type setting unit and a model training unit;

[0052] The type setting unit is used to set the output types of the multiple parallel models respectively according to the task categories of the multiple tasks;

[0053] The model training unit is used to use the fusion graph feature as a model input, and under the premise of the setting of the output type, train the multiple parallel models based on the preset multi-task loss to obtain a joint control model;

[0054] The preset multi-task loss is specifically:

[0055] L=λ1*L anomaly +λ2*L device +λ3*L state

[0056] Among them, L is the multi-task loss, L anomaly , L device , L state are equipment abnormality loss, equipment type loss and equipment status loss respectively. λ1, λ2 and λ3 are equipment abnormality loss weight, equipment type loss weight and equipment status loss weight respectively.

[0057] The present application first obtains the historical monitoring features of the substation to be regulated, calculates the attention weight of each feature based on the self-attention mechanism, and then fuses the features to obtain multimodal fusion features. It can integrate the monitoring features of different aspects and modes of the substation to improve the multimodal adaptability of the substation operation and control, and then enhance the multimodal fusion features based on the graph neural network to obtain the fusion graph features. It can continue to use the graph structure to capture and enhance the multimodal features, and then train multiple parallel models according to the fusion graph features and preset multi-task losses to obtain a joint control model. By first dividing the substation operation and control into different situations and using different models for training, it can meet the different task requirements of the substation operation and control, and then improve the accuracy of the substation operation and control when making operation decisions based on real-time monitoring data and regulating the substation. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 : A flowchart of a substation operation control method based on multi-mode shown in certain embodiments of the present application;

[0059] Figure 2 : A module structure diagram of a multi-modal-based substation operation and control device shown in certain embodiments of the present application. DETAILED DESCRIPTION

[0060] The embodiments of the present application are described in detail below, and examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions from beginning to end. The embodiments described below in conjunction with the accompanying drawings are exemplary and are only used to explain some embodiments of the present application, and should not be construed as limitations on the embodiments of the present application. Based on the embodiments shown in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0061] In the description of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, unless otherwise clearly and specifically defined, "multiple" and "several" mean two or more.

[0062] The operation and control of substations are mainly based on various types of monitoring equipment in the station, such as monitoring equipment, sensors and voiceprint detection equipment. However, each monitoring equipment often operates independently and lacks coordination, resulting in low efficiency in the comprehensive analysis of each monitoring data. At the same time, the existing operation and control methods for substations usually only focus on monitoring data under a single mode, lacking the integration and analysis of multi-modal monitoring data, resulting in low accuracy in the operation and control of substations. Therefore, how to improve the accuracy of substation operation and control is still a technical problem that needs to be solved urgently in the existing technology.

[0063] Based on the above technical background, please refer to Figure 1 The embodiment of the present application provides a substation operation control method based on multi-mode, including steps S101 to S105, each step is as follows:

[0064] Step S101: Acquire historical monitoring features of the substation to be regulated; wherein the historical monitoring features include voiceprint features, visible light features and infrared features.

[0065] In certain embodiments of the present application, the obtaining of historical monitoring characteristics of the substation to be regulated specifically includes:

[0066] Acquire historical monitoring data of the substation to be regulated; wherein the historical monitoring data includes voiceprint data, visible light images and infrared images;

[0067] Based on a bidirectional recurrent neural network, extracting time series features from the voiceprint data to obtain voiceprint features;

[0068] Based on a self-attention image encoder, performing image feature extraction on the visible light image to obtain visible light features;

[0069] Based on a convolutional neural network, image features are extracted from the infrared image to obtain infrared features.

[0070] Specifically, the voiceprint data in the historical monitoring data is represented as X[b, 1, T], the visible light image is represented as V[b, 3, h, w], and the infrared image is represented as I[b, 1, h, w], wherein b is the amount of data processed at a single time, T is the audio duration, h and ω are the height and width of the image, respectively, and the second digit in each data representation is the original dimension corresponding to each data.

[0071] In certain embodiments of the present application, the voiceprint data X[b, 1, T] is subjected to temporal feature extraction based on a bidirectional recurrent neural network to obtain voiceprint features, specifically:

[0072] Performing adaptive time-frequency analysis on the voiceprint data X[b, 1, T] to obtain a first voiceprint time-frequency feature;

[0073] Based on the mask, data enhancement is performed on the first voiceprint time-frequency feature in frequency and time to obtain a second voiceprint time-frequency feature, and local time-frequency features of the second voiceprint time-frequency feature are extracted based on two-dimensional convolution to obtain a third voiceprint time-frequency feature;

[0074] Based on a bidirectional recurrent neural network, extracting time series features from the third voiceprint time-frequency features to obtain initial voiceprint features;

[0075] The initial voiceprint feature is averaged and pooled to obtain the voiceprint feature F x [b,D x ]=H avg [b, 2*D].

[0076] Among them, D x is the voiceprint feature dimension, and D is the hidden state dimension of the bidirectional recurrent neural network.

[0077] In certain embodiments of the present application, the bidirectional recurrent neural network includes but is not limited to a bidirectional RNN and a bidirectional LSTM, and a preferred embodiment is a bidirectional LSTM.

[0078] In certain embodiments of the present application, the self-attention image encoder is used to extract image features from the visible light image to obtain visible light features, specifically:

[0079] Based on the self-attention image encoder, the image feature extraction is performed on the visible light image V[b, 3, h, w] to obtain the visible light feature F v [b, N, D v].

[0080] Where N is the number of tiles, which is related to the height and width h, w of the image and the preset patch size of the self-attention image encoder. v is the feature dimension of the visible light image.

[0081] In certain embodiments of the present application, a preferred implementation of the self-attention image encoder is VisionTransformer (ViT).

[0082] In certain embodiments of the present application, the image feature extraction is performed on the infrared image based on a convolutional neural network to obtain infrared features, specifically:

[0083] Based on the convolutional neural network, the infrared image I[b, 1, h, w] is subjected to image feature extraction to obtain the infrared feature F i [b,D i ].

[0084] Among them, D i is the characteristic dimension of infrared image.

[0085] In certain embodiments of the present application, the preferred embodiment of the convolutional neural network is CNN.

[0086] This application first obtains historical monitoring data, and then extracts features of voiceprint data, visible light images and infrared images based on different types of neural networks to obtain voiceprint features, visible light features and infrared features. By selecting different feature extraction networks based on the category of the data, the corresponding features can be extracted more accurately, thereby improving the accuracy of the acquired features.

[0087] Step S102: Based on the self-attention mechanism, the attention weights of the voiceprint feature, the visible light feature and the infrared feature are calculated respectively, and according to the attention weights, the voiceprint feature, the visible light feature and the infrared feature are fused to obtain a multimodal fusion feature.

[0088] In certain embodiments of the present application, the voiceprint feature, the visible light feature and the infrared feature are fused according to the attention weight to obtain a multimodal fusion feature, specifically including:

[0089] Based on a convolutional neural network, the dimensions of the voiceprint feature and the infrared feature are unified into the dimensions of the visible light feature to obtain a first voiceprint feature and a first infrared feature;

[0090] According to the attention weight, based on a multi-layer self-attention encoder, the first voiceprint feature, the visible light feature and the first infrared feature are spliced ​​and fused to obtain a multimodal fusion feature.

[0091] In certain embodiments of the present application, based on the convolutional neural network, the dimension of the voiceprint feature and the dimension of the infrared feature are unified into the dimension of the visible light feature to obtain the first voiceprint feature and the first infrared feature, specifically:

[0092]

[0093] Among them, F x-seq [b, N, D v ],F i-seq [b, N, D i ] are the first voiceprint feature and the first infrared feature respectively; F x [b, N, D x / N],F i [b, N, D i / N] represent the intermediate voiceprint features and intermediate infrared features respectively; resize represents the sampling operation; CNN represents the dimensionality change operation based on the convolutional neural network.

[0094] In certain embodiments of the present application, according to the attention weight, based on a multi-layer self-attention encoder, the first voiceprint feature, the visible light feature and the first infrared feature are spliced ​​and fused to obtain a multimodal fusion feature, specifically:

[0095] The first voiceprint feature F x-seq [b, N, D v ], the visible light feature F v [b, N, D v ] and the first infrared feature F i-seq [b, N, D i ] and concatenate at dim = 1 to obtain the multimodal concatenation feature F cat [b,3*N,D v ], and based on the multi-layer self-attention encoder, the multimodal encoding feature F is obtained trans [b,3*N,D t ]; among them, D t is the hidden state dimension of the multi-layer self-attention encoder;

[0096] The attention weight {W x [b, 1], W v [b, 1], W i [b, 1]} and the multimodal encoding feature F trans [b,3*N,Dt ] correspond to multiplication, and obtain the multimodal fusion feature F fused [b, N, D t ].

[0097] This application first unifies the dimensions of voiceprint features and infrared features into the dimensions of visible light features based on a convolutional neural network, which can ensure data consistency and prevent data dimensional conflicts, and then concatenates and fuses the first voiceprint features, visible light features and the first infrared features according to the attention weights and multi-layer self-attention encoders to obtain multimodal fusion features, which can integrate monitoring data of different types and modes of substations and improve the multimodal adaptability of substation operation and control.

[0098] Step S103: Based on the graph neural network, the multimodal fusion features are enhanced to obtain fusion graph features.

[0099] In certain embodiments of the present application, the multimodal fusion feature is enhanced based on the graph neural network to obtain the fusion graph feature, specifically including:

[0100] Using the time step of the multimodal fusion feature as a graph node, generating a first multimodal fusion graph corresponding to the multimodal fusion feature;

[0101] Based on a preset edge connection threshold, edges between graph nodes of the first multimodal fusion graph are removed to obtain a second multimodal fusion graph;

[0102] Based on the self-attention mechanism, the edge weights between the graph nodes in the second multimodal fusion graph are calculated, and according to the edge weights, based on a multi-layer graph convolutional network, the second multimodal fusion graph is enhanced to obtain fusion graph features.

[0103] In certain embodiments of the present application, the preferred value of the preset edge connection threshold k is N / 4.

[0104] In certain embodiments of the present application, the edges between the graph nodes of the first multimodal fusion graph are removed based on a preset edge connection threshold to obtain the second multimodal fusion graph, specifically:

[0105] Based on a preset edge connection threshold k, only the top-k connection edges of each graph node in the first multimodal fusion graph are retained to obtain a second multimodal fusion graph.

[0106] It should be noted that the connection edge described in this application is used to describe the degree of association between each graph node, which can reflect the correlation and importance between different modal features. Specifically, the connection edge obtains the connection edge strength based on the similarity of multimodal features at the semantic level.

[0107] In certain embodiments of the present application, the connection edge strength is the size of the matrix product of the feature vector of each graph node and the feature vectors of other graph nodes for which the connection edge strength needs to be calculated.

[0108] Specifically, after obtaining the connection edge strengths of all connection edges of each graph node according to the first multimodal fusion graph, all connection edges of each graph node are arranged in descending order according to the connection edge strength, and the top-k connection edges of each graph node are selected to construct the second multimodal fusion graph.

[0109] In certain embodiments of the present application, the self-attention mechanism is used to calculate the edge weights between the graph nodes in the second multimodal fusion graph, and the second multimodal fusion graph is enhanced based on the edge weights and a multi-layer graph convolutional network to obtain fusion graph features, specifically:

[0110] Based on the self-attention mechanism, the edge weights E[b, N, k, D] between the nodes in the second multimodal fusion graph are calculated. edge ]; among them, D edge is the edge feature dimension;

[0111] According to the edge weights, based on a multi-layer graph convolutional network, the second multimodal fusion graph is enhanced to obtain a fusion graph feature F graph [b, N, D graph ]; among them, D graph is the convolutional graph feature dimension.

[0112] In certain embodiments of the present application, a preferred implementation of the multi-layer graph convolutional network is a graph convolutional network GCN.

[0113] This application first generates a first multimodal fusion graph using the time step of the multimodal fusion feature as a graph node, which can capture multimodal features using the graph structure, and then removes some edges based on a preset edge connection threshold to obtain a second multimodal fusion graph, which can increase the weights of edges that meet the conditions, and then calculate the edge weights based on the self-attention mechanism, and enhance the second multimodal fusion graph based on the edge weights and a multi-layer graph convolutional network. Through the graph convolutional network, the relationship between different modalities and different time steps can be captured, thereby improving the matching degree between the output fusion graph features and the actual ones.

[0114] Step S104: According to the fusion graph features and the preset multi-task loss, the preset multiple parallel models are trained to obtain a joint control model; wherein the multiple parallel models are based on the preset multiple tasks; the multiple tasks include equipment anomaly detection, equipment classification and equipment status prediction.

[0115] In certain embodiments of the present application, the preset multiple parallel models are trained according to the fusion graph features and the preset multi-task loss to obtain a joint regulation model, specifically including:

[0116] According to the task categories of the multiple tasks, respectively set the output types of the multiple parallel models;

[0117] Taking the fusion graph features as model input, under the premise of setting the output type, and based on the preset multi-task loss, the multiple parallel models are trained to obtain a joint control model;

[0118] The preset multi-task loss is specifically:

[0119] L=λ1*L anomaly +λ2*L device +λ3*L state ;

[0120] Among them, L is the multi-task loss, L anomaly , L device , L state are equipment abnormality loss, equipment type loss and equipment status loss respectively. λ1, λ2 and λ3 are equipment abnormality loss weight, equipment type loss weight and equipment status loss weight respectively.

[0121] In certain embodiments of the present application, the multiple tasks include equipment anomaly detection, equipment classification and equipment status prediction, and the multiple tasks are classification tasks; specifically, equipment anomaly monitoring is a binary classification task, and equipment classification and equipment status prediction are multi-classification tasks.

[0122] In certain embodiments of the present application, the output types of the multiple parallel models are respectively set according to the task categories of the multiple tasks, specifically:

[0123] Set the output type of the parallel model corresponding to the task category of device anomaly monitoring to score anomaly [b, 1];

[0124] Set the output type of the parallel model corresponding to the task category of device classification to score device [b,num device ]; where num device is the total number of device types;

[0125] Set the output type of the parallel model corresponding to the task category of device status prediction to score state [b,num state ]; where num state The total number of device states.

[0126] In certain embodiments of the present application, the model structures of the multiple parallel models are all the fusion graph features connected to a fully connected layer after pooling.

[0127] This application first sets the output types of multiple parallel models according to the task categories of multiple tasks. By differentiating the models in output type, a single model can be specialized in a single task, which is beneficial to improving the accuracy of the model. Then, the fusion graph features are used as the model input, and multiple parallel models are trained based on the preset multi-task loss to obtain a joint control model, which can meet the different task requirements of substation operation and control, and then when the substation operation and control are performed according to the joint control model, the accuracy of the substation operation and control is improved.

[0128] Step S105: acquiring real-time monitoring data of the substation to be regulated, inputting the real-time monitoring data into the joint regulation model, obtaining an operation decision, and regulating the substation to be regulated according to the operation decision.

[0129] The present application first obtains the historical monitoring features of the substation to be regulated, calculates the attention weight of each feature based on the self-attention mechanism, and then fuses the features to obtain multimodal fusion features. It can integrate the monitoring features of different aspects and modes of the substation to improve the multimodal adaptability of the substation operation and control, and then enhance the multimodal fusion features based on the graph neural network to obtain the fusion graph features. It can continue to use the graph structure to capture and enhance the multimodal features, and then train multiple parallel models according to the fusion graph features and preset multi-task losses to obtain a joint control model. By first dividing the substation operation and control into different situations and using different models for training, it can meet the different task requirements of the substation operation and control, and then improve the accuracy of the substation operation and control when making operation decisions based on real-time monitoring data and regulating the substation.

[0130] Corresponding to the previous method, see Figure 2 , the embodiment of the present application provides a substation operation control device based on multi-modality, including a feature acquisition module 210, a feature fusion module 220, a feature enhancement module 230, a model training module 240 and an operation control module 250;

[0131] The feature acquisition module 210 is used to acquire the historical monitoring features of the substation to be regulated; wherein the historical monitoring features include voiceprint features, visible light features and infrared features;

[0132] The feature fusion module 220 is used to calculate the attention weights of the voiceprint feature, the visible light feature and the infrared feature respectively based on the self-attention mechanism, and fuse the voiceprint feature, the visible light feature and the infrared feature according to the attention weights to obtain a multimodal fusion feature;

[0133] The feature enhancement module 230 is used to enhance the multimodal fusion feature based on a graph neural network to obtain a fusion graph feature;

[0134] The model training module 240 is used to train a plurality of preset parallel models according to the fusion graph features and the preset multi-task loss to obtain a joint control model; wherein the plurality of parallel models are based on a plurality of preset tasks; the plurality of tasks include device anomaly detection, device classification and device status prediction;

[0135] The operation control module 250 is used to obtain real-time monitoring data of the substation to be controlled, input the real-time monitoring data into the joint control model, obtain operation decisions, and control the substation to be controlled according to the operation decisions.

[0136] In certain embodiments of the present application, the feature acquisition module 210 includes a historical data acquisition unit, a voiceprint feature extraction unit, a visible light feature extraction unit, and an infrared feature extraction unit;

[0137] The historical data acquisition unit is used to acquire historical monitoring data of the substation to be regulated; wherein the historical monitoring data includes voiceprint data, visible light images and infrared images;

[0138] The voiceprint feature extraction unit is used to extract time series features from the voiceprint data based on a bidirectional recurrent neural network to obtain voiceprint features;

[0139] The visible light feature extraction unit is used to extract image features from the visible light image based on a self-attention image encoder to obtain visible light features;

[0140] The infrared feature extraction unit is used to extract image features from the infrared image based on a convolutional neural network to obtain infrared features.

[0141] In certain embodiments of the present application, the feature fusion module 220 includes a dimension unification unit and a feature fusion unit;

[0142] The dimension unification unit is used to unify the dimension of the voiceprint feature and the dimension of the infrared feature into the dimension of the visible light feature based on a convolutional neural network, so as to obtain a first voiceprint feature and a first infrared feature;

[0143] The feature fusion unit is used to splice and fuse the first voiceprint feature, the visible light feature and the first infrared feature according to the attention weight and based on a multi-layer self-attention encoder to obtain a multimodal fusion feature.

[0144] In certain embodiments of the present application, the feature enhancement module 230 includes a fusion graph generation unit, a fusion edge removal unit, and a fusion graph enhancement unit;

[0145] The fusion graph generating unit is used to generate a first multimodal fusion graph corresponding to the multimodal fusion feature by taking the time step of the multimodal fusion feature as a graph node;

[0146] The fusion edge removal unit is used to remove the edges between the graph nodes of the first multimodal fusion graph based on a preset edge connection threshold to obtain a second multimodal fusion graph;

[0147] The fusion graph enhancement unit is used to calculate the edge weights between the graph nodes in the second multimodal fusion graph based on the self-attention mechanism, and enhance the second multimodal fusion graph based on the edge weights and a multi-layer graph convolutional network to obtain fusion graph features.

[0148] In certain embodiments of the present application, the model training module 240 includes a type setting unit and a model training unit;

[0149] The type setting unit is used to set the output types of the multiple parallel models respectively according to the task categories of the multiple tasks;

[0150] The model training unit is used to use the fusion graph feature as a model input, and under the premise of the setting of the output type, train the multiple parallel models based on the preset multi-task loss to obtain a joint control model;

[0151] The preset multi-task loss is specifically:

[0152] L=λ1*L anomaly +λ2*L device +λ3*L state ;

[0153] Among them, l is the multi-task loss, L anomaly , L device , L state are equipment abnormality loss, equipment type loss and equipment status loss respectively. λ1, λ2 and λ3 are equipment abnormality loss weight, equipment type loss weight and equipment status loss weight respectively.

[0154] The present application first obtains the historical monitoring features of the substation to be regulated, calculates the attention weight of each feature based on the self-attention mechanism, and then fuses the features to obtain multimodal fusion features. It can integrate the monitoring features of different aspects and modes of the substation to improve the multimodal adaptability of the substation operation and control, and then enhance the multimodal fusion features based on the graph neural network to obtain the fusion graph features. It can continue to use the graph structure to capture and enhance the multimodal features, and then train multiple parallel models according to the fusion graph features and preset multi-task losses to obtain a joint control model. By first dividing the substation operation and control into different situations and using different models for training, it can meet the different task requirements of the substation operation and control, and then improve the accuracy of the substation operation and control when making operation decisions based on real-time monitoring data and regulating the substation.

[0155] It should be understood that the device provided in the embodiment of the present application corresponds to the aforementioned method, and a substation operation and control device based on multimodality provided in the embodiment of the present application can implement a substation operation and control method based on multimodality provided in any embodiment of the present application.

[0156] Adaptively, the embodiments of the present application also provide a computer device and a computer-readable storage medium.

[0157] The computer device comprises: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor;

[0158] Among them, when the processor executes the computer program, a multi-modal-based substation operation control method of the present application is implemented.

[0159] The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute a multi-modal-based substation operation control method of the present application.

[0160] The above are some implementation methods of the present application, and the purpose, technical scheme and beneficial effects of the present application are further described in detail. It should be clear that the above-mentioned partial implementation methods of the present application cannot be understood as limiting the present application. It is particularly pointed out that for those skilled in the art, any changes, modifications, equivalent substitutions and variations made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A multi-modal substation operation control method, characterized in that: include: Acquire historical monitoring features of the substation to be regulated; wherein the historical monitoring features include voiceprint features, visible light features, and infrared features; Based on the self-attention mechanism, the attention weights of the voiceprint feature, the visible light feature and the infrared feature are calculated respectively, and according to the attention weights, the voiceprint feature, the visible light feature and the infrared feature are fused to obtain a multimodal fusion feature; Based on a graph neural network, the multimodal fusion features are enhanced to obtain fusion graph features; According to the fusion graph features and the preset multi-task loss, multiple preset parallel models are trained to obtain a joint control model; wherein the multiple parallel models are based on multiple preset tasks; the multiple tasks include device anomaly detection, device classification and device status prediction; Real-time monitoring data of the substation to be regulated is obtained, the real-time monitoring data is input into the joint regulation model to obtain an operation decision, and the substation to be regulated is regulated according to the operation decision.

2. A multi-modal substation operation control method according to claim 1, characterized in that: The obtaining of historical monitoring characteristics of the substation to be regulated specifically includes: Acquire historical monitoring data of the substation to be regulated; wherein the historical monitoring data includes voiceprint data, visible light images and infrared images; Based on a bidirectional recurrent neural network, extracting time series features from the voiceprint data to obtain voiceprint features; Based on a self-attention image encoder, performing image feature extraction on the visible light image to obtain visible light features; Based on a convolutional neural network, image features are extracted from the infrared image to obtain infrared features.

3. A multi-modal substation operation control method according to claim 1, characterized in that: The method of fusing the voiceprint feature, the visible light feature and the infrared feature according to the attention weight to obtain a multimodal fusion feature specifically includes: Based on a convolutional neural network, the dimensions of the voiceprint feature and the infrared feature are unified into the dimensions of the visible light feature to obtain a first voiceprint feature and a first infrared feature; According to the attention weight, based on a multi-layer self-attention encoder, the first voiceprint feature, the visible light feature and the first infrared feature are spliced ​​and fused to obtain a multimodal fusion feature.

4. A multi-modal substation operation control method according to claim 1, characterized in that: The multimodal fusion feature is enhanced based on the graph neural network to obtain the fusion graph feature, which specifically includes: Using the time step of the multimodal fusion feature as a graph node, generating a first multimodal fusion graph corresponding to the multimodal fusion feature; Based on a preset edge connection threshold, edges between graph nodes of the first multimodal fusion graph are removed to obtain a second multimodal fusion graph; Based on the self-attention mechanism, the edge weights between the graph nodes in the second multimodal fusion graph are calculated, and according to the edge weights, based on a multi-layer graph convolutional network, the second multimodal fusion graph is enhanced to obtain fusion graph features.

5. A multi-modal substation operation control method according to claim 1, characterized in that: The method of training the preset multiple parallel models according to the fusion graph features and the preset multi-task loss to obtain a joint control model specifically includes: According to the task categories of the multiple tasks, respectively set the output types of the multiple parallel models; Taking the fusion graph features as model input, under the premise of setting the output type, and based on the preset multi-task loss, the multiple parallel models are trained to obtain a joint control model; The preset multi-task loss is specifically: L=λ1*L anomaly +λ2*L device +λ3*L state ; Among them, L is the multi-task loss, L anomay , L device , L state are equipment abnormality loss, equipment type loss and equipment status loss respectively, λ1, λ2 and λ3 are equipment abnormality loss weight, equipment type loss weight and equipment status loss weight respectively.

6. A substation operation control device based on multi-mode, characterized in that: It includes feature acquisition module, feature fusion module, feature enhancement module, model training module and operation control module; The feature acquisition module is used to acquire the historical monitoring features of the substation to be regulated; wherein the historical monitoring features include voiceprint features, visible light features and infrared features; The feature fusion module is used to calculate the attention weights of the voiceprint feature, the visible light feature and the infrared feature respectively based on the self-attention mechanism, and fuse the voiceprint feature, the visible light feature and the infrared feature according to the attention weights to obtain a multimodal fusion feature; The feature enhancement module is used to enhance the multimodal fusion feature based on a graph neural network to obtain a fusion graph feature; The model training module is used to train a plurality of preset parallel models according to the fusion graph features and the preset multi-task loss to obtain a joint control model; wherein the plurality of parallel models are based on a plurality of preset tasks; the plurality of tasks include device anomaly detection, device classification and device status prediction; The operation control module is used to obtain real-time monitoring data of the substation to be controlled, input the real-time monitoring data into the joint control model, obtain operation decisions, and control the substation to be controlled according to the operation decisions.

7. A multi-modal substation operation control device according to claim 6, characterized in that: The feature acquisition module includes a historical data acquisition unit, a voiceprint feature extraction unit, a visible light feature extraction unit and an infrared feature extraction unit; The historical data acquisition unit is used to acquire historical monitoring data of the substation to be regulated; wherein the historical monitoring data includes voiceprint data, visible light images and infrared images; The voiceprint feature extraction unit is used to extract time series features from the voiceprint data based on a bidirectional recurrent neural network to obtain voiceprint features; The visible light feature extraction unit is used to extract image features from the visible light image based on a self-attention image encoder to obtain visible light features; The infrared feature extraction unit is used to extract image features from the infrared image based on a convolutional neural network to obtain infrared features.

8. The multi-modal substation operation control device according to claim 6, characterized in that: The feature fusion module includes a dimension unification unit and a feature fusion unit; The dimension unification unit is used to unify the dimension of the voiceprint feature and the dimension of the infrared feature into the dimension of the visible light feature based on a convolutional neural network, so as to obtain a first voiceprint feature and a first infrared feature; The feature fusion unit is used to splice and fuse the first voiceprint feature, the visible light feature and the first infrared feature according to the attention weight and based on a multi-layer self-attention encoder to obtain a multimodal fusion feature.

9. The multi-modal substation operation control device according to claim 6, characterized in that: The feature enhancement module includes a fusion graph generation unit, a fusion edge removal unit and a fusion graph enhancement unit; The fusion graph generating unit is used to generate a first multimodal fusion graph corresponding to the multimodal fusion feature by taking the time step of the multimodal fusion feature as a graph node; The fusion edge removal unit is used to remove the edges between the graph nodes of the first multimodal fusion graph based on a preset edge connection threshold to obtain a second multimodal fusion graph; The fusion graph enhancement unit is used to calculate the edge weights between the graph nodes in the second multimodal fusion graph based on the self-attention mechanism, and enhance the second multimodal fusion graph based on the edge weights and a multi-layer graph convolutional network to obtain fusion graph features.

10. A multi-modal substation operation control device according to claim 6, characterized in that: The model training module includes a type setting unit and a model training unit; The type setting unit is used to set the output types of the multiple parallel models respectively according to the task categories of the multiple tasks; The model training unit is used to use the fusion graph feature as a model input, and under the premise of the setting of the output type, train the multiple parallel models based on the preset multi-task loss to obtain a joint control model; The preset multi-task loss is specifically: L=λ1*L anomaly +λ2*L device +λ3*L state ; Among them, L is the multi-task loss, L anomaly , L device , L state are equipment abnormality loss, equipment type loss and equipment status loss respectively. λ1, λ2 and λ3 are equipment abnormality loss weight, equipment type loss weight and equipment status loss weight respectively.

Images