Multi-modal fusion power grid dispatching knowledge graph generation method and device

Abstract

The invention discloses a multi-modal fusion power grid dispatching knowledge graph generation method and device, and the method comprises the steps: obtaining historical multi-modal data, including text data, image data and audio data, in power grid dispatching, inputting the historical multi-modal data into a pre-trained feature extraction fusion model, and obtaining the fusion features of power grid equipment, a power grid dispatching knowledge graph is constructed based on fusion features of power grid equipment, each node in the power grid dispatching knowledge graph represents a power grid equipment entity and a state feature thereof, and edges among the nodes represent a topology or dispatching operation relationship among the power grid equipment, so that the power grid dispatching knowledge graph is constructed through multi-modal data; through the power grid dispatching knowledge graph, reasoning basis and decision support are provided for power grid dispatching, the problems that the subjectivity is high and a correct and effective regulation and control decision cannot be made by using multi-modal data when a dispatching transaction is manually processed are solved, and a dispatcher can objectively, accurately and quickly make a regulation and control decision by referring to the power grid dispatching knowledge graph.

Description

Technical Field

[0001] The present invention relates to the field of power grid dispatching technology, and in particular to a method and device for generating a multi-modal fusion power grid dispatching knowledge graph. Background Art

[0002] The operating characteristics and control modes of the existing power system are highly complex and continue to evolve during the development process. The operating mode is flexible and changeable. As the power grid models, various power grid equipment and the structure between power grids have increased exponentially, a large amount of multimodal data such as audio, video, images, text, etc. has been generated.

[0003] In power grid dispatching, dispatching affairs are currently mainly handled manually, which has a strong human-subjective and objective nature. It is impossible to use multimodal data to make effective and correct control tasks, which poses potential risks to the safe operation of the power grid. Summary of the invention

[0004] The present invention provides a method and device for generating a multimodal fusion knowledge graph for power grid dispatching, so as to solve the problem that manual dispatching decisions in power grid dispatching are highly subjective and it is impossible to use multimodal data to make effective and correct control task decisions.

[0005] In a first aspect, the present invention provides a method for generating a multi-modal fusion knowledge graph for power grid dispatching, comprising:

[0006] Acquire historical multimodal data of power grid equipment in power grid dispatching, wherein the multimodal data includes at least text data, image data, and audio data;

[0007] Inputting the historical multimodal data into a pre-trained feature extraction fusion model to obtain fusion features of the power grid equipment;

[0008] A power grid dispatching knowledge graph is constructed based on the fusion characteristics of the power grid equipment. Each node in the power grid dispatching knowledge graph represents a power grid equipment entity and its state characteristics, and the edges between the nodes represent the topology or dispatching operation relationship between the power grid equipment.

[0009] Optionally, before inputting the historical multimodal data into a pre-trained feature extraction and fusion model, the method further includes:

[0010] The historical multimodal data is preprocessed to obtain preprocessed historical multimodal data.

[0011] Optionally, the historical multimodal data is input into a pre-trained feature extraction fusion model to obtain fusion features of the power grid device, including:

[0012] Inputting text data, image data and audio data in the multimodal data into the feature extraction fusion model;

[0013] Extracting text features, image features and audio features in the feature extraction fusion model;

[0014] The text features, image features and audio features are fused to obtain fused features of the power grid device.

[0015] Optionally, the feature extraction and fusion model includes a text feature extraction network, an image feature extraction network, an audio feature extraction network, a feature dimension adjustment network and a feature fusion network, wherein the text feature extraction network, the image feature extraction network and the audio feature extraction network are all connected to the feature dimension adjustment network, and the feature dimension adjustment network is connected to the feature fusion network;

[0016] Inputting text data, image data, and audio data in the multimodal data into the feature extraction fusion model includes:

[0017] Inputting text data into a text feature extraction network, inputting image data into an image feature extraction network, and inputting audio data into an audio feature extraction network;

[0018] The feature extraction fusion model extracts text features, image features and audio features, including:

[0019] Extract features from text data in the text feature extraction network to obtain initial text features;

[0020] Extract features from image data in the image feature extraction network to obtain initial image features;

[0021] Extracting features from audio data in an audio feature extraction network to obtain initial audio features;

[0022] The initial text features, the initial image features and the initial audio features are input into a feature dimension adjustment network to obtain target text features, target image features and target audio features with equal feature dimensions.

[0023] Optionally, the feature fusion network includes an autoencoder based on an attention mechanism, which fuses the text features, image features, and audio features to obtain fusion features of the power grid device, including:

[0024] Inputting the target text features, target image features and target audio features into the autoencoder, and calculating weights of the target text features, target image features and target audio features in the autoencoder based on an attention mechanism;

[0025] The target text feature, the target image feature and the target audio feature are weightedly fused based on the weights to obtain the fused feature of the power grid device.

[0026] Optionally, constructing a power grid dispatching knowledge graph based on the fusion features of the power grid equipment includes:

[0027] The fusion characteristics of power grid equipment are used as nodes to represent the characteristics of different power grid equipment;

[0028] Based on the topological relationship of the power grid equipment in the distribution network, set the edges of each node and add edge attributes;

[0029] Calculate the similarity of the fusion features of any two nodes, add an edge to the two nodes whose similarity exceeds the speech threshold, and set the attributes of the edge.

[0030] Optionally, after constructing the power grid dispatching knowledge graph based on the fusion features of the power grid equipment, the method further includes:

[0031] Acquire detected multimodal data;

[0032] The multimodal data and the power grid dispatching knowledge graph are input into a pre-trained multimodal reasoning model to obtain a target dispatching strategy.

[0033] In a second aspect, the present invention provides a multi-modal fusion power grid dispatching knowledge graph generation device, comprising:

[0034] A historical multimodal data acquisition module, used to acquire historical multimodal data of power grid equipment in power grid dispatching, wherein the multimodal data includes at least text data, image data and audio data;

[0035] A feature extraction and fusion module, used for inputting the historical multimodal data into a pre-trained feature extraction and fusion model to obtain fusion features of the power grid equipment;

[0036] A knowledge graph construction module is used to construct a power grid dispatching knowledge graph based on the fusion characteristics of the power grid equipment. Each node in the power grid dispatching knowledge graph represents a power grid equipment entity and its state characteristics, and the edges between nodes represent the topology or dispatching operation relationship between the power grid equipment.

[0037] In a third aspect, the present invention provides an electronic device, the electronic device comprising:

[0038] at least one processor; and

[0039] a memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multi-modal fusion power grid dispatching knowledge graph generation method described in any one of the first aspects of the present invention.

[0041] In a fourth aspect, the present invention provides a computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable a processor to implement the method for generating a multi-modal fusion power grid dispatching knowledge graph as described in any one of the first aspect of the present invention when executed.

[0042] The embodiment of the present invention obtains historical multimodal data such as text data, image data and audio data of power grid equipment in power grid dispatching, and inputs the historical multimodal data into a feature extraction and fusion model to obtain fusion features of the power grid equipment, and constructs a power grid dispatching knowledge graph based on the fusion features. In the constructed power grid dispatching knowledge graph, each node represents a power grid equipment entity and its state features, and the edges between the nodes represent the topology or dispatching operation relationship between the power grid equipment, so that the multimodal data such as text data, image data and audio data of the power grid equipment generated in the power grid dispatching are extracted and fused to form a power grid dispatching knowledge graph. The power grid dispatching knowledge graph provides reasoning basis and decision support for power grid dispatching, which solves the problem of strong subjectivity when manually handling dispatching affairs and the inability to use multimodal data to make correct and effective control decisions, so that dispatchers can refer to the power grid dispatching knowledge graph to make control decisions objectively, accurately and quickly.

[0043] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 It is a flowchart of a method for generating a multi-modal fusion knowledge graph for power grid dispatching provided in the first embodiment of the present invention;

[0046] Figure 2 It is a flowchart of a method for generating a multi-modal fusion knowledge graph for power grid dispatching provided in the second embodiment of the present invention;

[0047] Figure 3 This is a schematic diagram of the network structure of the feature extraction fusion model;

[0048] Figure 4 This is a partial schematic diagram of the knowledge graph for power grid dispatching;

[0049] Figure 5 It is a structural schematic diagram of a multi-modal fusion power grid dispatching knowledge graph generation device provided in Embodiment 3 of the present invention;

[0050] Figure 6 It is a schematic diagram of the structure of an electronic device provided in Embodiment 4 of the present invention. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0052] Embodiment 1

[0053] Figure 1 This is a flowchart of a method for generating a multimodal fusion knowledge graph for power grid dispatching provided in the first embodiment of the present invention. This embodiment can be applied to generating a knowledge graph for power grid dispatching to provide a reference for power grid dispatching decisions. This method can be executed by a multimodal fusion knowledge graph generation device for power grid dispatching. The multimodal fusion knowledge graph generation device for power grid dispatching can be implemented in the form of hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the multi-modal fusion power grid dispatching knowledge graph generation method includes:

[0054] S101. Acquire historical multimodal data of power grid equipment in power grid dispatching, where the multimodal data includes at least text data, image data, and audio data.

[0055] The historical multimodal data in this embodiment may be data of power grid equipment generated by power grid dispatching operations, for example, may be text data, image data, and audio data, wherein the text data may be text describing power grid dispatching control events. Taking the transformer as a power grid device as an example, the text data may be text information describing the voltage, current, load rate, transformer temperature, transformer insulating oil temperature, operation ticket, work ticket, dispatching log, and even operation manual of the transformer before, during, and after the dispatching control of the transformer. The image data may be an infrared image of the transformer, an infrared image of the transformer insulating oil, an indicator light image, or an image extracted from a monitoring video of the power grid equipment, etc. The audio data may be audio such as sounds and alarm sounds generated by mechanical vibrations during the operation of the transformer collected by a microphone or other pickup.

[0056] In one embodiment, a collection of text data, image data, and audio data of power grid equipment involved in a scheduling event can be obtained as a historical multimodal data. For example, if a transformer, primary-side switch, and secondary-side switch are involved in a scheduling event, the text data, image data, and audio data of the transformer, primary-side switch, and secondary-side switch involved in this scheduling event can be obtained.

[0057] S102: Input the historical multimodal data into a pre-trained feature extraction fusion model to obtain fusion features of the power grid equipment.

[0058] This embodiment can pre-train a feature extraction fusion model, which is used to extract text features, image features and audio features from the text data, image data and audio data of each power grid device, respectively, and fuse the text features, image features and audio features to obtain the fusion features of the power grid device.

[0059] S103. Construct a power grid dispatching knowledge graph based on the fusion characteristics of the power grid equipment. Each node in the power grid dispatching knowledge graph represents a power grid equipment entity and its state characteristics, and the edges between the nodes represent the topology or dispatching operation relationship between the power grid equipment.

[0060] Specifically, the fusion features of each power grid device can be used as a node, the topological relationship of the power grid devices in the distribution network can be used to establish edges, and the attributes of the edges can be set according to the topological relationship. The similarity can be calculated using the fusion features represented by any two nodes, and the edges and edge attributes can be added to obtain a power grid dispatching knowledge graph.

[0061] In the constructed power grid dispatch knowledge graph, each node represents the power grid equipment entity and its state characteristics, and the edges between nodes represent the topology or dispatch operation relationship between power grid equipment. During the operation of the power grid, the multimodal data of each power grid equipment is detected in real time, and the features extracted from the detected multimodal data are compared with the features of the nodes in the power grid dispatch knowledge graph to determine the target node. The relationship between the characteristics and edges of the target node and other nodes in the power grid dispatch knowledge graph is used to recommend power grid dispatch decision references to dispatchers.

[0062] The embodiment of the present invention obtains historical multimodal data such as text data, image data and audio data of power grid equipment in power grid dispatching, and inputs the historical multimodal data into a feature extraction and fusion model to obtain fusion features of the power grid equipment, and constructs a power grid dispatching knowledge graph based on the fusion features. In the constructed power grid dispatching knowledge graph, each node represents a power grid equipment entity and its state features, and the edges between the nodes represent the topology or dispatching operation relationship between the power grid equipment, so that the multimodal data such as text data, image data and audio data of the power grid equipment generated in the power grid dispatching are extracted and fused to form a power grid dispatching knowledge graph. The power grid dispatching knowledge graph provides reasoning basis and decision support for power grid dispatching, which solves the problem that the subjectivity is strong when manually handling dispatching affairs and the multimodal data cannot be used to make correct and effective control decisions, so that dispatchers can refer to the power grid dispatching knowledge graph to make control decisions objectively, accurately and quickly.

[0063] Embodiment 2

[0064] Figure 2 A flowchart of a method for generating a multi-modal fusion knowledge graph for power grid dispatching provided in the second embodiment of the present invention is provided. The embodiment of the present invention is optimized on the basis of the above-mentioned first embodiment. Figure 2 As shown, the multi-modal fusion power grid dispatching knowledge graph generation method includes:

[0065] S201. Acquire historical multimodal data of power grid equipment in power grid dispatching, where the multimodal data includes at least text data, image data, and audio data.

[0066] In one embodiment, the dispatching and control events in the power grid can be used as a guide to obtain text data, image data and audio data of one or more power grid devices involved in each dispatching and control event from the database, wherein the text data can be text describing a power grid dispatching and control event. Taking the transformer as a power grid device as an example, the text data can be text information describing the voltage, current, load rate, transformer temperature, transformer insulating oil temperature, operation ticket, work ticket, dispatch log and even operation manual of the transformer before, during and after the dispatching and control of the transformer. The image data can be an infrared image of the transformer, an infrared image of the transformer insulating oil, an indicator light image or an image extracted from a monitoring video of the power grid equipment, etc. The audio data can be audio such as sound, alarm sound, etc. generated by mechanical vibration during the operation of the transformer collected by a microphone or other pickup.

[0067] S202: preprocess the historical multimodal data to obtain preprocessed historical multimodal data.

[0068] Specifically, for text data preprocessing, it can include operations such as removing punctuation marks, removing auxiliary words (such as "de", "zhi", "shi", etc.), stemming, and lemmatization, converting the text into a standard format. Exemplarily, for the text description of transformer overload "At 5:30 pm on December 5, 2023, the output power is too high, reaching 90% of the rated output power", after preprocessing, the text "202312051730, Power≥98%Power_max" is obtained.

[0069] For image data preprocessing, it can include operations such as cropping to a unified scale size, enhancement (processing of clarity, brightness, etc.), and removing occluded images. For audio data, it can include operations such as unifying the audio duration, noise reduction, and filtering.

[0070] In this embodiment, preprocessing text data, image data, and audio data can improve the quality of multimodal data and the accuracy of the subsequent constructed power grid dispatching knowledge graph.

[0071] S203. Input the text data, image data, and audio data in the multimodal data into the feature extraction and fusion model.

[0072] As Figure 3 shown is a schematic diagram of the network structure of the feature extraction and fusion model of the embodiment of the present invention. The feature extraction and fusion model of this embodiment includes a text feature extraction network, an image feature extraction network, an audio feature extraction network, a feature dimension adjustment network, and a feature fusion network. The text feature extraction network, the image feature extraction network, and the audio feature extraction network are all connected to the feature dimension adjustment network, and the feature dimension adjustment network is connected to the feature fusion network. Among them, the text feature extraction network is used to extract text features from text data, the image feature extraction network is used to extract image features from image data, the audio feature extraction network is used to extract audio features from audio data, the feature dimension adjustment network is used to uniformly adjust the dimensions of the extracted text features, image features, and audio features to obtain text features, image features, and audio features with equal scales, and the feature fusion network is used to fuse the text features, image features, and audio features with equal scales to obtain fused features.

[0073] As Figure 3 shown, the preprocessed text data text in the historical multimodal data can be input into the text feature extraction network, the image data image can be input into the image feature extraction network, and the audio data audio can be input into the audio feature extraction network.

[0074] S204. Extract text features, image features, and audio features in the feature extraction and fusion model.

[0075] For text data, features can be extracted from the text data in a text feature extraction network to obtain initial text features. Exemplarily, the text feature extraction network can be a trained BERT (Bidirectional Encoder Representations from Transformers), and the text data is converted into a text feature vector through BERT. When the text data is a power grid dispatch log, the text features extracted from the text data include the control instruction information in the power grid dispatch log, and also include the dispatcher's judgment on the dispatch operation and the description of its operation rules. When the text data is an operation ticket, the text features include the information of each operation step in the operation ticket.

[0076] For image data, features can be extracted from the image data in an image feature extraction network to obtain initial image features. Exemplarily, the image feature extraction network can be a convolutional neural network (CNN), such as an existing ResNet obtained after training with image samples. For example, the image data can be a monitoring image of a power grid device. The monitoring image of the power grid device is input into the image feature extraction network to extract image features. The image features are used to represent the device status and environmental information of the power grid device, such as the position of the switch in the switch cabinet, the status of the circuit breaker, whether there is smoke, water vapor condensation and other environmental information.

[0077] For audio data, features can be extracted from the audio data in an audio feature extraction network to obtain initial audio features. Exemplarily, the audio feature extraction network can be CNN-LSTM, which combines CNN (convolutional neural network) and LSTM (long short-term memory network) to form a network structure suitable for processing spatiotemporal data. The audio data of the power grid equipment (such as the audio generated by the mechanical vibration during the operation of the transformer) is input into the CNN-LSTM, and the CNN-LSTM extracts time series and spectrum features to obtain audio features. The extracted audio features are used to characterize abnormal sounds when the power grid equipment is abnormal (such as high-frequency frequency changes, abnormal vibrations, etc.).

[0078] Since text, image, and audio are data of different modalities, the extracted features need to be represented in a unified manner. Specifically, the extracted initial text features, initial image features, and initial audio features can be input into a feature dimension adjustment network to obtain target text features, target image features, and target audio features with equal feature dimensions. Specifically, in the feature dimension adjustment network, the longest-scale feature can be first determined from the initial text features, initial image features, and initial audio features, and then the scales of the other two features can be expanded to the length of the longest-scale feature to obtain text features, image features, and audio features with the same scale, and then the text features, image features, and audio features are normalized to obtain target text features, target image features, and target audio features with consistent feature vector ranges.

[0079] S205: Fusing text features, image features, and audio features to obtain fused features of the power grid equipment.

[0080] In one embodiment, the target text features, target image features and target audio features can be input into an autoencoder, and the weights of the target text features, target image features and target audio features are calculated in the autoencoder based on an attention mechanism. The target text features, target image features and target audio features are weightedly fused based on the weights to obtain the fused features of the power grid equipment.

[0081] The autoencoder expression is as follows:

[0082] Z k =f enc (X) = σ(W enc X+b enc );

[0083] X={X text ,X image ,X audio};

[0084] X text represents text features, X image represents the image features, X audio represents the audio feature, σ is the coefficient, W enc represents the weight of the encoder, b enc represents the bias term, f enc (X) represents the encoder transfer function, the above text feature X text , image feature X image , audio feature X audio After the autoencoder, the corresponding low-dimensional feature Z is obtained k , k=text, image, audio.

[0085] In the autoencoder based on the attention mechanism, the feedforward neural network is trained on the input text feature X. text , image feature X image , audio feature X audio Importance scoring is performed and attention weights are calculated as follows:

[0086]

[0087] Among them, e k represents the importance score of the k-th feature.

[0088] The final weighted fusion feature is expressed as:

[0089] Z fused =∑ k α k Z k .

[0090] The fused features can not only integrate the key information of each modality (text, image, audio), but also give higher weights to important features in the power grid dispatching process. For example, greater weights are given to features in text features that represent status, operation, control, etc., and greater weights are given to features in image features that represent the status of power grid equipment. Greater weights are given to features in audio features that represent abnormalities or actions of power grid equipment. This makes the fused features of power grid equipment include comprehensive status information (equipment status) and operating behavior patterns (dispatching operation relationships) of power grid equipment during dispatching, providing entities and relationships for the construction of a knowledge graph for power grid dispatching.

[0091] S206. Use the fusion characteristics of the power grid equipment as nodes to represent the characteristics of different power grid equipment.

[0092] Specifically, in the distribution network, the fusion characteristics of the power grid equipment can be used as entity nodes, such as substations, transformers, circuit breakers, power grid lines or monitoring devices, as nodes in the knowledge graph, and the nodes are associated with fusion characteristics, that is, one node represents a power grid equipment and its fusion characteristics.

[0093] S207. Based on the topological relationship of the power grid devices in the distribution network, set the edge of each node and add the attributes of the edge.

[0094] Specifically, the topological circuit diagram of the power grid equipment in the distribution network can be obtained, and edges can be added to the nodes of the corresponding power grid equipment in the knowledge graph based on the topological relationship of the power grid equipment in the topological circuit diagram. The attributes of the edges can be set to electrical connection, control, monitoring, etc. to describe the interactive process of power grid dispatching. The "operation sequence" can also be established through time series. The attributes of nodes and edges can be specific information represented in the fusion features, such as alarm status, equipment failure probability, dispatching operation priority, etc.

[0095] S208. Calculate the similarity of the fusion features of any two nodes, add an edge to the two nodes whose similarity exceeds the speech threshold, and set the attributes of the edge.

[0096] In order to strengthen the association between different modes, the embodiment of the present invention can adopt a similarity calculation method to connect nodes with similar related features through edges to achieve multi-modal association of power grid dispatching knowledge. In an example, the cosine similarity can be used to calculate the similarity of the fusion features of the nodes:

[0097]

[0098] Where Z i and Z j It represents the fusion features of two power grid devices. When the similarity exceeds the threshold, an edge is added to the two nodes in the knowledge graph to achieve cross-modal association of information.

[0099] like Figure 4 A schematic diagram of the power grid dispatching knowledge graph is shown, which includes a primary side switch, a camera, a transformer, a secondary side switch, a current sensor, a voltage sensor, a sound sensor and an automation terminal. The edge relationship between the camera, the current sensor, the voltage sensor and the transformer is monitoring, the edge relationship between the primary side switch, the transformer, the secondary side switch and the automation terminal is control, and the edge relationship between the transformer and the primary side switch and the secondary side switch is connection. The features in the transformer include multiple fused features to describe different states of the transformer. Similarly, other nodes also include fused features. Figure 4 This is just an example of a knowledge graph for power grid dispatching, which includes the states represented by different fusion features of transformer nodes, as well as the operating states of primary and secondary switches under different states.

[0100] S209: Acquire detected multimodal data.

[0101] Specifically, during the operation of the power grid, multimodal data of various devices in the power grid are collected through cameras, current and voltage sensors, sound sensors, etc. Figure 4 As an example, images, voltage and current (in text form), and audio data of the transformer are collected, and multi-modal data of the primary side switch and the secondary side switch are collected.

[0102] S210: Input the multimodal data and the power grid dispatching knowledge graph into a pre-trained multimodal reasoning model to obtain a target dispatching strategy.

[0103] Specifically, the multimodal reasoning model includes the aforementioned feature extraction and fusion model, and also includes a feature matching model. After the multimodal data is input into the feature extraction and fusion model, the current fusion features of the transformer, the current fusion features of the primary side switch, and the current fusion features of the secondary side switch are obtained. The current fusion features of the transformer, the primary side switch, and the secondary side switch and the power grid dispatching knowledge graph are input into the feature matching model to obtain the target features and states of each node (transformer, primary side switch, secondary side switch). The edges between the nodes are determined in the power grid dispatching knowledge graph through the target features and states, thereby obtaining the corresponding target dispatching strategy. For example, when it is matched that the transformer currently belongs to feature 3, the primary side switch belongs to feature 1, and the secondary side switch belongs to feature 1, the dispatching strategy can be to first control the secondary side switch to open, and then control the primary side switch to open. Of course, the above are just examples. In more complex power grid dispatching, a more complex knowledge graph can be constructed based on the historical multimodal data of power grid dispatching. For example, if a power outage due to an interval fault requires dispatching adjacent intervals to supply power to the faulty interval, the corresponding knowledge graph can be established by extracting features from historical multimodal data. In the application, when the multimodal data of the monitored interval is consistent with the fault, the target decision is obtained through reasoning through the knowledge graph to restore power to the faulty interval.

[0104] In an optional embodiment, the knowledge graph involved in the target decision can also be displayed on a display screen so that the dispatcher can refer to the node status, scheduling operation relationship between nodes, priority, etc. on the knowledge graph to review whether the generated scheduling strategy is reasonable, thereby realizing the visualization of the knowledge graph to support the regulation business.

[0105] The embodiment of the present invention obtains historical multimodal data such as text data, image data and audio data of power grid equipment in power grid dispatching, extracts text features, image features and audio features in a feature extraction and fusion model, fuses the text features, image features and audio features to obtain fusion features of the power grid equipment, and constructs a power grid dispatching knowledge graph based on the fusion features. In the constructed power grid dispatching knowledge graph, each node represents a power grid equipment entity and its state features, and the edges between the nodes represent the topology or dispatching operation relationship between the power grid equipment, so that multimodal data such as text data, image data and audio data of power grid equipment generated in power grid dispatching are extracted and fused to form a power grid dispatching knowledge graph. The power grid dispatching knowledge graph provides reasoning basis and decision support for power grid dispatching, which solves the problem that the subjectivity is strong when manually handling dispatching affairs and it is impossible to use multimodal data to make correct and effective control decisions, so that dispatchers can refer to the power grid dispatching knowledge graph to make control decisions objectively, accurately and quickly.

[0106] Furthermore, based on the attention mechanism, the autoencoder fuses text features, image features and audio features, which can add higher weight to the information related to the power grid dispatching business in the text features, image features and audio features, so that the fused features can characterize the joint information and operations of the power grid equipment in the dispatching business, and provide support for the construction of the power grid dispatching knowledge graph.

[0107] Embodiment 3

[0108] Figure 5 This is a schematic diagram of the structure of a multi-modal fusion power grid dispatching knowledge graph generation device provided in the third embodiment of the present invention. Figure 5 As shown, the multi-modal fusion power grid dispatching knowledge graph generation device includes:

[0109] A historical multimodal data acquisition module 501 is used to acquire historical multimodal data of power grid equipment in power grid dispatching, wherein the multimodal data includes at least text data, image data and audio data;

[0110] A feature extraction and fusion module 502 is used to input the historical multimodal data into a pre-trained feature extraction and fusion model to obtain fusion features of the power grid equipment;

[0111] The knowledge graph construction module 503 is used to construct a power grid dispatching knowledge graph based on the fusion characteristics of the power grid equipment. Each node in the power grid dispatching knowledge graph represents a power grid equipment entity and its state characteristics, and the edges between nodes represent the topology or dispatching operation relationship between the power grid equipment.

[0112] Optionally, also include:

[0113] The preprocessing module is used to preprocess the historical multimodal data to obtain preprocessed historical multimodal data.

[0114] Optionally, the feature extraction and fusion module 502 includes:

[0115] A data input unit, used for inputting text data, image data and audio data in the multimodal data into the feature extraction fusion model;

[0116] A feature extraction unit, used to extract text features, image features and audio features in the feature extraction fusion model;

[0117] The feature fusion unit is used to fuse the text features, image features and audio features to obtain the fusion features of the power grid equipment.

[0118] Optionally, the feature extraction and fusion model includes a text feature extraction network, an image feature extraction network, an audio feature extraction network, a feature dimension adjustment network and a feature fusion network, wherein the text feature extraction network, the image feature extraction network and the audio feature extraction network are all connected to the feature dimension adjustment network, and the feature dimension adjustment network is connected to the feature fusion network;

[0119] A data input unit, used to input text data into a text feature extraction network, input image data into an image feature extraction network, and input audio data into an audio feature extraction network;

[0120] The feature extraction unit includes:

[0121] A question feature extraction subunit, used for extracting features from text data in a text feature extraction network to obtain initial text features;

[0122] An image feature extraction subunit, used for extracting features from image data in an image feature extraction network to obtain initial image features;

[0123] An audio feature extraction subunit, used to extract features from audio data in an audio feature extraction network to obtain initial audio features;

[0124] The feature dimension adjustment subunit is used to input the initial text features, the initial image features and the initial audio features into the feature dimension adjustment network to obtain target text features, target image features and target audio features with equal feature dimensions.

[0125] Optionally, the feature fusion network includes an autoencoder based on an attention mechanism, and the feature fusion unit includes:

[0126] An attention weight calculation subunit, used for inputting the target text feature, the target image feature and the target audio feature into the autoencoder, and calculating weights of the target text feature, the target image feature and the target audio feature in the autoencoder based on an attention mechanism;

[0127] The fusion subunit is used to perform weighted fusion on the target text feature, the target image feature and the target audio feature based on the weight to obtain the fusion feature of the power grid device.

[0128] Optionally, the knowledge graph construction module 503 includes:

[0129] A node construction unit, used to use the fusion characteristics of the power grid equipment as nodes to represent the characteristics of different power grid equipment;

[0130] A first edge construction unit, used to set the edge of each node and add edge attributes based on the topological relationship of the power grid equipment in the distribution network;

[0131] The second edge construction unit is used to calculate the similarity of the fusion features of any two nodes, add an edge to the two nodes whose similarity reaches the speech threshold, and set the attributes of the edge.

[0132] Optionally, also include:

[0133] A multimodal data detection module, used to obtain detected multimodal data;

[0134] The reasoning module is used to input the multimodal data and the power grid dispatching knowledge graph into a pre-trained multimodal reasoning model to obtain a target dispatching strategy.

[0135] The multimodal fusion power grid dispatching knowledge graph generation device provided in an embodiment of the present invention can execute the multimodal fusion power grid dispatching knowledge graph generation method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0136] Embodiment 4

[0137] Figure 6 A schematic diagram of the structure of an electronic device 40 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0138] like Figure 6 As shown, the electronic device 40 includes at least one processor 41, and a memory connected to the at least one processor 41, such as a read-only memory (ROM) 42, a random access memory (RAM) 43, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 41 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 42 or the computer program loaded from the storage unit 48 to the random access memory (RAM) 43. In the RAM 43, various programs and data required for the operation of the electronic device 40 can also be stored. The processor 41, the ROM 42, and the RAM 43 are connected to each other through a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0139] A number of components in the electronic device 40 are connected to the I / O interface 45, including: an input unit 46, such as a keyboard, a mouse, etc.; an output unit 47, such as various types of displays, speakers, etc.; a storage unit 48, such as a disk, an optical disk, etc.; and a communication unit 49, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 49 allows the electronic device 40 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0140] The processor 41 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The processor 41 executes the various methods and processes described above, such as a method for generating a knowledge graph for power grid dispatching with multimodal fusion.

[0141] In some embodiments, the method for generating a knowledge graph for power grid dispatching with multimodal fusion may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 40 via the ROM 42 and / or the communication unit 49. When the computer program is loaded into the RAM 43 and executed by the processor 41, one or more steps of the method for generating a knowledge graph for power grid dispatching with multimodal fusion described above may be performed. Alternatively, in other embodiments, the processor 41 may be configured to perform the method for generating a knowledge graph for power grid dispatching with multimodal fusion by any other appropriate means (e.g., by means of firmware).

[0142] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0143] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0144] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in combination with an instruction execution system, device or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0145] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0146] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0147] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0148] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0149] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for generating a multi-modal fusion knowledge graph for power grid dispatching, characterized in that: include: Acquire historical multimodal data of power grid equipment in power grid dispatching, wherein the multimodal data includes at least text data, image data, and audio data; Inputting the historical multimodal data into a pre-trained feature extraction fusion model to obtain fusion features of the power grid equipment; A power grid dispatching knowledge graph is constructed based on the fusion characteristics of the power grid equipment. Each node in the power grid dispatching knowledge graph represents a power grid equipment entity and its state characteristics, and the edges between the nodes represent the topology or dispatching operation relationship between the power grid equipment.

2. The method for generating a multi-modal fusion knowledge graph for power grid dispatching according to claim 1 is characterized in that: Before inputting the historical multimodal data into the pre-trained feature extraction fusion model, the method further includes: The historical multimodal data is preprocessed to obtain preprocessed historical multimodal data.

3. The method for generating a multi-modal fusion knowledge graph for power grid dispatching according to claim 1 is characterized in that: Inputting the historical multimodal data into a pre-trained feature extraction fusion model to obtain fusion features of the power grid equipment includes: Inputting text data, image data and audio data in the multimodal data into the feature extraction fusion model; Extracting text features, image features and audio features in the feature extraction fusion model; The text features, image features and audio features are fused to obtain fused features of the power grid device.

4. The method for generating a multi-modal fusion knowledge graph for power grid dispatching according to claim 3 is characterized in that: The feature extraction and fusion model includes a text feature extraction network, an image feature extraction network, an audio feature extraction network, a feature dimension adjustment network and a feature fusion network, wherein the text feature extraction network, the image feature extraction network and the audio feature extraction network are all connected to the feature dimension adjustment network, and the feature dimension adjustment network is connected to the feature fusion network; Inputting text data, image data, and audio data in the multimodal data into the feature extraction fusion model includes: Inputting text data into a text feature extraction network, inputting image data into an image feature extraction network, and inputting audio data into an audio feature extraction network; The feature extraction fusion model extracts text features, image features and audio features, including: Extract features from text data in the text feature extraction network to obtain initial text features; Extract features from image data in the image feature extraction network to obtain initial image features; Extracting features from audio data in an audio feature extraction network to obtain initial audio features; The initial text features, the initial image features and the initial audio features are input into a feature dimension adjustment network to obtain target text features, target image features and target audio features with equal feature dimensions.

5. The method for generating a multi-modal fusion knowledge graph for power grid dispatching according to claim 4 is characterized in that: The feature fusion network includes an autoencoder based on an attention mechanism, which fuses the text features, image features, and audio features to obtain fusion features of the power grid device, including: Inputting the target text features, target image features and target audio features into the autoencoder, and calculating weights of the target text features, target image features and target audio features in the autoencoder based on an attention mechanism; The target text feature, the target image feature and the target audio feature are weightedly fused based on the weights to obtain the fused feature of the power grid device.

6. The method for generating a multi-modal fusion knowledge graph for power grid dispatching according to any one of claims 1 to 5, characterized in that: A power grid dispatching knowledge graph is constructed based on the fusion features of the power grid equipment, including: The fusion characteristics of power grid equipment are used as nodes to represent the characteristics of different power grid equipment; Based on the topological relationship of the power grid equipment in the distribution network, set the edges of each node and add edge attributes; Calculate the similarity of the fusion features of any two nodes, add an edge to the two nodes whose similarity exceeds the speech threshold, and set the attributes of the edge.

7. The method for generating a multi-modal fusion knowledge graph for power grid dispatching according to any one of claims 1 to 5, characterized in that: After constructing a power grid dispatching knowledge graph based on the fusion features of the power grid equipment, it also includes: Acquire detected multimodal data; The multimodal data and the power grid dispatching knowledge graph are input into a pre-trained multimodal reasoning model to obtain a target dispatching strategy.

8. A multi-modal fusion power grid dispatching knowledge graph generation device, characterized in that: include: A historical multimodal data acquisition module, used to acquire historical multimodal data of power grid equipment in power grid dispatching, wherein the multimodal data includes at least text data, image data and audio data; A feature extraction and fusion module, used for inputting the historical multimodal data into a pre-trained feature extraction and fusion model to obtain fusion features of the power grid equipment; A knowledge graph construction module is used to construct a power grid dispatching knowledge graph based on the fusion characteristics of the power grid equipment. Each node in the power grid dispatching knowledge graph represents a power grid equipment entity and its state characteristics, and the edges between nodes represent the topology or dispatching operation relationship between the power grid equipment.

9. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the multi-modal fusion power grid dispatching knowledge graph generation method described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the multi-modal fusion power grid dispatching knowledge graph generation method described in any one of claims 1-7 when executed.

Images