A multi-modal knowledge graph method based on power grid scheduling

Abstract

The present application relates to a kind of multi-modal knowledge graph methods based on power grid scheduling, the multi-modal data of text, image, audio and video in power grid scheduling are structured representation, by feature extraction to multi-modal data set, obtain corresponding feature set, introduce adaptive feature selection mechanism, the feature representation of different modal is fused to obtain multi-modal feature representation;The relationship and interaction between different modal data are considered, and the graph structure information is constructed;By learning the feature representation of different modal data, various information in power grid scheduling can be uniformly encoded and form a computable knowledge graph representation, which facilitates the understanding, reasoning and application of power grid scheduling information;Through graph neural network technology, complex cross-modal reasoning can be carried out on the knowledge graph, helping power grid dispatchers quickly understand the current state, predict future trends, and make appropriate decisions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of power grid dispatching, and particularly relates to a multi-modal knowledge graph method based on power grid dispatching. BACKGROUND

[0002] Multi-modal knowledge graphs have a wide range of applications in power grid dispatching. For example, by fusing and analyzing multi-source data such as power equipment information, monitoring data, and weather data, accurate prediction of power load can be achieved, thereby optimizing power supply. In addition, multi-modal knowledge graphs can also be used for safety control and fault diagnosis of power systems, improving the reliability and security of the power grid.

[0003] CN115270881A "Line fault identification method based on multi-modal information fusion knowledge graph" discloses "including the following steps, a, multi-modal information preprocessing, b, multi-modal information vectorization, multi-modal information is encoded based on deep neural network, and its distributed representation vector is obtained; c, multi-modal information fusion knowledge graph construction, multi-modal information fusion knowledge graph construction takes line equipment as entity node, and constructs line fault identification knowledge graph in the form of "line equipment-fault condition-fault type" triplets; d, line fault diagnosis, by identifying power grid monitoring signals, meteorological information and protection oscillogram images, the corresponding fault type is inferred online based on the triplets knowledge organization form of the knowledge graph, the method solves the problem that the line fault cause recognition is not accurate due to the reliance on single signal or artificial experience analysis, and improves the line fault cause diagnosis accuracy", but the features of different modalities usually have different dimensions, and the features of different modalities may differ in representation, which cannot be directly fused, and the features of different modalities may also have different distributions and data characteristics, which may lead to the dominance of the features of some modalities in the fusion process, while the features of other modalities are masked or ignored, which may result in incomplete or biased feature expression after fusion. SUMMARY

[0004] Based on the above technical problems, the present application provides a multi-modal knowledge graph method based on power grid dispatching, and the specific steps include:

[0005] Obtaining a power grid dispatching historical multi-modal data set, including a text data set, an image data set, an audio data set, and a video data set;

[0006] Standardizing the power grid dispatching historical multi-modal data set to obtain a multi-modal sample data set;

[0007] Feature extraction is performed on the multi-modal sample data set to obtain a corresponding feature set, and a self-adaptive feature selection mechanism is introduced to perform feature fusion on the feature representations of different modalities to obtain multi-modal feature representations;

[0008] considering the relationship and interaction between different modal data, constructing graph structure information;

[0009] constructing a graph neural network model, learning the representation vector of the node, using the representation vector of the node to perform cross-modal relationship reasoning, and optimizing the associated relationship in the knowledge graph.

[0010] Preferably, the text data set includes power grid historical scheduling instructions and power grid historical operation and maintenance records; the image data set includes thermal images and fault scene photos of power grid equipment; the audio data set includes power grid equipment operation audio; and the video data set includes power grid equipment and line fault conditions when a fault occurs.

[0011] Preferably, the standardization processing of the power grid scheduling historical multi-modal data set is specifically converting the power grid scheduling historical multi-modal data set of different modalities into a unified format, wherein:

[0012] The standardization processing of the text data set includes removing special characters, punctuation marks and stop words, and unifying the text letter case;

[0013] The standardization processing of the image data set includes pixel value normalization and size adjustment;

[0014] The standardization processing of the audio data set includes audio amplitude normalization and sampling rate unification;

[0015] The standardization processing of the video data set includes video frame rate unification and video size adjustment.

[0016] Preferably, the feature extraction of the multi-modal sample data set obtains the corresponding feature set, which is specifically:

[0017] The text sample data set is trained using a word embedding model Word2Vec to learn the distributed representation of words; each word in the text data is replaced by a corresponding word vector, and the word vector is weighted and averaged to obtain the feature representation of the text;

[0018] The image sample data set is subjected to feature extraction using a multi-scale spatial perception image feature extraction algorithm to obtain the feature representation of the image;

[0019] The audio sample data set is subjected to feature extraction using a multi-resolution time-frequency deep learning feature extraction algorithm to obtain the feature representation of the audio;

[0020] The video sample data set is subjected to feature extraction using a spatio-temporal attention convolutional network algorithm to obtain the feature representation of the video.

[0021] Preferably, the feature extraction of the image sample data set is performed using a multi-scale spatial perception image feature extraction algorithm to obtain the feature representation of the image, specifically:

[0022] A plurality of scales of the image sample data are constructed through a pyramid structure, each scale containing different image detail information, which is expressed by a formula as follows:

[0023]

[0024] In the formula, is a pixel value at position (i, j) in the image sample data at scale s, is a Gaussian filter at scale s, and σ s is a standard deviation of the Gaussian filter;

[0025] At each scale, a convolution operation is performed on the image sample data, which is expressed by a formula as follows:

[0026]

[0027] In the formula, is a convolution result at position (i, j) in the image sample data at scale s, is a Gaussian filter at scale s;

[0028] The convolution results at each scale are fused to obtain the features of the image sample data at different scales, which is expressed by a formula as follows:

[0029]

[0030] In the formula, F i,j is a feature value at position (i, j) after fusing the features of the image sample data at all scales;

[0031] The features of the image sample data at different scales are nonlinearly mapped to obtain the feature representation of the image, which is expressed by a formula as follows:

[0032] F i,j = g(F i,j );

[0033] In the formula, F i,j is a feature value processed by the nonlinear mapping function g, i.e., the feature representation of the image.

[0034] Preferably, a multi-resolution time-frequency deep learning feature extraction algorithm is used to perform feature extraction on the audio sample data set to obtain the feature representation of the audio, specifically:

[0035] The audio signal in the audio sample data is converted to the time-frequency domain, which is expressed by a formula as follows:

[0036] X(n,k) = ∑ t x(t)w(t-n)e -j2πkt / N ;

[0037] where X(n,k) is a complex value in the time-frequency domain at time frame n and frequency band k, x(t) is the original audio signal, w(t-n) is a window function, N is the number of points of FFT, and k is the index of the frequency band;

[0038] Local and global features of the audio signal are extracted at different resolutions in the time-frequency domain, and the local and global features of the audio signal are input into a deep neural network for feature extraction, and a deep learning feature of the audio is obtained through multiple layers of convolution, pooling and full connection layer, and a feature representation of the audio is obtained through nonlinear mapping of the deep learning feature.

[0039] Preferably, a spatio-temporal attention convolutional network algorithm is used to extract features of the video sample data set to obtain a feature representation of the video, specifically:

[0040] The spatio-temporal relationship between video frames in the video sample data is extracted through 3D convolution, and an attention mechanism is introduced to model important spatio-temporal regions in the video sample data to obtain spatio-temporal attention features, which are expressed by the formula:

[0041]

[0042] where i and j are the rows and columns of the image in the two-dimensional space, A(i,j,t) is the attention weight at (i,j) and time step t in the two-dimensional space, and f(i,j,t) is a function for calculating the attention weight;

[0043] The spatio-temporal attention features are fused with the original features, and a nonlinear mapping function is applied to obtain a feature representation of the video, which is expressed by the formula:

[0044] F(i,j,t) = g(aX(i,j,t) + βA(i,j,t));

[0045] where X(i,j,t) is the original feature at position (i,j) and time step t, g() is a nonlinear mapping function, and a and β are the fusion weights of the original feature and the spatio-temporal attention feature, respectively.

[0046] Preferably, an adaptive feature selection mechanism is introduced to fuse the feature representations of different modalities to obtain a multi-modal feature representation, specifically:

[0047] The mutual information between different modalities is calculated, which is expressed by the formula:

[0048]

[0049] where I(A, B) is mutual information between modality A and modality B, p(a, b) is the probability of modality A and modality B occurring simultaneously, p(a) is the probability of modality A occurring alone, and p(b) is the probability of modality B occurring alone;

[0050] The mutual information of the feature representations of different modalities is calculated, which is expressed in a formula as follows:

[0051]

[0052] where I(X|Y) is the mutual information between the feature representation X of modality A and the feature representation Y of modality B, p(x, y) is the probability of the feature representation A and the feature representation B occurring simultaneously, p(x) is the probability of the feature representation X occurring alone, and p(x|y) is the probability of the feature representation X occurring given the feature representation Y;

[0053] The importance score I(X) of the feature representation X in modality A is calculated, which is expressed in a formula as follows:

[0054] I(X)=∑ Y≠A I(X|Y);

[0055] where Y≠A represents other modalities except modality A;

[0056] The importance scores of the feature representations of different modalities are obtained according to the formula for calculating the importance score I(X) of the feature representation X in modality A;

[0057] The corresponding feature representation weights are assigned according to the importance scores of the feature representations of different modalities, and a multi-modal feature representation is obtained using linear weighting.

[0058] Preferably, the relationship and interaction between different modal data are considered to construct graph structure information, specifically as follows:

[0059] Nodes are constructed according to four types of data, i.e., text, image, audio, and video;

[0060] Edges are constructed according to the mutual information between different modalities and the mutual information of the feature representations of different modalities;

[0061] The edges are weighted according to the importance scores of the feature representations of different modalities;

[0062] The graph structure information is constructed according to the connection relationship of the nodes and edges.

[0063] Preferably, a graph neural network model is constructed, the representation vector of a node is learned, cross-modal relationship reasoning is performed using the representation vector of the node, and the associated relationship in the knowledge graph is optimized, specifically as follows:

[0064] The graph neural network is a GCN model, the GCN model utilizes the feature representation of nodes and the graph structure information to perform information transmission and feature updating between nodes, learns the representation vector of the node, and performs cross-modal relationship reasoning by using the node representation learned by the GCN model, and the formula expression is:

[0065]

[0066] In the formula, H (I) is the node representation matrix of the Ith layer, is the normalized adjacency matrix, is the diagonal matrix, W (I) is the weight matrix of the Ith layer, and sigma is an activation function.

[0067] By information transmission and aggregation between nodes, the correlation and semantic relationship between different modalities are learned, and the formula expression is:

[0068]

[0069] In the formula, is the representation vector of the Ith layer node i, is the attention weight between nodes i and j, W (I) is the weight matrix of the Ith layer, and sigma is an activation function.

[0070] Compared with the prior art, the beneficial effects of the present application are:

[0071] 1. The present application provides a multi-modal knowledge graph method based on power grid dispatching, which structures the multi-modal data of text, image, audio and video in power grid dispatching, and through feature representation learning of different modal data, various information in power grid dispatching can be uniformly encoded and form a computable knowledge graph representation, which is convenient for understanding, reasoning and application of power grid dispatching information.

[0072] 2. The present application provides a multi-modal knowledge graph method based on power grid dispatching, which can perform complex cross-modal reasoning on the knowledge graph through graph neural network technology, help power grid dispatchers quickly understand the current state, predict future development trend, and make corresponding decisions. BRIEF DESCRIPTION OF DRAWINGS

[0073] Figure 1 is a method flowchart of the embodiment of the present application. DETAILED DESCRIPTION

[0074] With reference to the accompanying drawings, the technical solutions in the embodiments of the present application will be clearly and completely described below, obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work belong to the protection scope of the present application.

[0075] Embodiment one

[0076] Embodiment one of the present application discloses a multi-modal knowledge graph method based on power grid dispatching, in the embodiment, the method obtains a power grid dispatching history multi-modal data set through multiple ways, including a text data set, an image data set, an audio data set and a video data set, the text data set includes power grid historical dispatching instructions and power grid historical operation and maintenance records in various documents, records, reports; the image data set includes thermal images and fault scene photos of power grid equipment obtained by various sensors and monitoring devices; the audio data set includes power grid equipment operation audio obtained by various sensors and monitoring devices; the video data set includes power grid equipment and line fault conditions when a fault occurs, collected by monitoring cameras and infrared cameras, specifically, the method of the embodiment includes:

[0077] S1, standardizing the power grid dispatching history multi-modal data set to obtain a multi-modal sample data set, so that data of different modalities can be processed and analyzed under the same framework, aiming to improve data quality, reduce noise and redundancy, and provide a unified input representation for subsequent multi-modal tasks;

[0078] Preferably, the standardization processing of the power grid dispatching history multi-modal data set specifically converts the power grid dispatching history multi-modal data set of different modalities into a unified format, wherein:

[0079] S11, the standardization processing of the text data set includes removing special characters, punctuation marks and stop words, unifying the case of text letters, specifically, removing special characters and punctuation marks through using a regular expression or a string operation method to delete special characters and punctuation marks in the text, removing stop words is to delete common meaningless words from the text according to a predefined stop word list, such as prepositions, conjunctions, etc., unifying the case of text letters is to convert the letters in the text to lowercase or uppercase to eliminate the difference between uppercase and lowercase;

[0080] S12, the standardization processing of the image data set includes pixel value normalization and size adjustment, specifically, the pixel value normalization is to normalize the pixel values of the image, so that the range is between 0 and 1 or between -1 and 1, the size adjustment is to scale or crop the image, so that it has the same size and aspect ratio;

[0081] S13, the standardization processing of the audio dataset includes audio amplitude normalization and sampling rate unification. Specifically, the audio amplitude normalization is to scale or normalize the amplitude range of the audio signal to ensure that different audios have similar amplitude ranges. The sampling rate unification is to unify the sampling rate of the audio signal to the same value for subsequent processing and analysis.

[0082] S14, the standardization processing of the video dataset includes video frame rate unification and video size adjustment. Specifically, the video frame rate unification is to unify the frame rate of the video to the same value to ensure that the video has the same time interval. The video size adjustment is to scale or crop the video frame to have the same size and aspect ratio.

[0083] It is worth noting that the multi-modal dataset described in the embodiment can also include current or voltage data. The standardization processing of the current or voltage data includes:

[0084] Using the statistical method Z-score to calculate the difference between the standard deviation of the current or voltage data points and the mean value, and marking the data points exceeding the set threshold as outliers to detect outliers, which can be expressed in the formula as:

[0085] Z=(x-μ) / σ;

[0086] In the formula, x is the current or voltage data value, μ is the mean value, and σ is the standard deviation.

[0087] When the sample size is large and the proportion of missing values is very small, the missing values will not have a significant impact on the analysis results, so the samples containing missing values are directly deleted, otherwise interpolation is used to fill in the missing values, which can be expressed in the formula as:

[0088] Interpolation=y1+(y2-y1)*(x-x1) / (x2-x1);

[0089] In the formula, (x1, y1) and (x2, y2) are known data points, and x is the missing value corresponding to the horizontal coordinate.

[0090] S2, feature extraction is performed on the multi-modal sample dataset to obtain the corresponding feature set;

[0091] S21, the text sample dataset is trained using the word embedding model Word2Vec to learn the distributed representation of words; the words in each text data are replaced by the corresponding word vectors, and the word vectors are weighted and averaged to obtain the feature representation of the text;

[0092] S22, the multi-scale spatial perception image feature extraction algorithm is used to extract the features of the image sample dataset to obtain the feature representation of the image;

[0093] The multiple scales of the image sample data are constructed by the pyramid structure, each scale contains different image detail information, which is expressed by the formula as follows:

[0094]

[0095] In the formula, is the pixel value of the image sample data at position (i, j) in scale s, that is, the gray value of the image at a certain scale, is the Gaussian filter in scale s, used for the construction operation of the scale space, and σ s is the standard deviation of the Gaussian filter, used to control the scale size of the filter;

[0096] At each scale, the image sample data is subjected to a convolution operation, but the size and shape of the convolution kernel are adaptively adjusted according to the scale, which is expressed by the formula as follows:

[0097]

[0098] In the formula, is the convolution result of the image sample data at position (i, j) in scale s, that is, the feature value after the spatial perception convolution operation, is the Gaussian filter in scale s, used for the spatial perception convolution operation;

[0099] The convolution results at each scale are fused to obtain the features of the image sample data at different scales, which is expressed by the formula as follows:

[0100]

[0101] In the formula, F i,j is the feature value at position (i, j) after the fusion of the features of the image sample data at all scales, that is, the image feature after the feature fusion;

[0102] The features of the image sample data at different scales are subjected to nonlinear mapping to obtain the feature representation of the image, so as to enhance the expression ability of the features, which is expressed by the formula as follows:

[0103] F i,j =g(F i,j );

[0104] In the formula, F i,j is the feature value after the nonlinear mapping function g processing, that is, the feature representation of the image;

[0105] S23, using a multi-resolution time-frequency deep learning feature extraction algorithm to perform feature extraction on the audio sample data set to obtain a feature representation of the audio;

[0106] The audio signal in the audio sample data is converted to the time-frequency domain, which is expressed in a formula as follows:

[0107] X(n, k) = å t x(t)w(t-n)e -j2πkt / N ;

[0108] In the formula, X(n, k) is a complex value at time frame n and frequency band k in the time-frequency domain, that is, the audio signal after time-frequency transformation, x(t) is the original audio signal, where t represents time, w(t-n) is a window function for windowing operation in the time domain, where n represents the offset of the window, N is the number of points of FFT, used to control the resolution of time-frequency transformation, usually taking a power of 2, k is the index of the frequency band, usually taking a value range of 0 to N-1;

[0109] Local and global features of the audio signal are extracted at different resolutions in the time-frequency domain to capture the audio content at different frequency and time scales, the local and global features of the audio signal are input into a deep neural network for feature extraction, and a deep learning feature of the audio is obtained through multiple layers of convolution, pooling and full connection layer, and a feature representation of the audio is obtained through nonlinear mapping of the deep learning feature;

[0110] S24, feature extraction of the video sample data set is performed using a spatio-temporal attention convolution network algorithm to obtain a feature representation of the video;

[0111] The spatio-temporal relationship between video frames in the video sample data is extracted through 3D convolution, and an attention mechanism is introduced to model important spatio-temporal regions in the video sample data to obtain spatio-temporal attention features, so as to improve the attention to key information, which is expressed in a formula as follows:

[0112]

[0113] In the formula, (i, j) is a position coordinate, i and j are respectively a row and a column of an image in a two-dimensional space, A(i, j, t) is an attention weight at (i, j) and a time step t in the two-dimensional space, used to indicate important spatio-temporal regions in the video, and f(i, j, t) is a function for calculating the attention weight, which can be a feature representation based on position and time, such as an output of a convolutional neural network;

[0114] The spatio-temporal attention features are fused with original features, and a nonlinear mapping function is applied to obtain a feature representation of the video, which is expressed in a formula as follows:

[0115] F(i, j, t) = g(aX(i, j, t) + bA(i, j, t));

[0116] In the formula, X(i, j, t) is the original feature at position (i, j) and time step t, g() is a nonlinear mapping function for enhancing the expression ability of the feature, and alpha and beta are the fusion weights of the original feature and the spatiotemporal attention feature, respectively, for balancing the importance of the original feature and the attention feature;

[0117] S3, introducing an adaptive feature selection mechanism to fuse the feature representations of different modalities to obtain a multi-modal feature representation;

[0118] The mutual information between different modalities is calculated, and the mutual information can measure the correlation and dependence between two random variables, which is expressed in the formula as:

[0119]

[0120] In the formula, I(A, B) is the mutual information between modality A and modality B, p(a, b) is the probability of modality A and modality B occurring simultaneously, p(a) is the probability of modality A occurring alone, and p(b) is the probability of modality B occurring alone;

[0121] The mutual information of the feature representations of different modalities is calculated, and the conditional mutual information can measure the importance of a feature to the task given other modality features, which is expressed in the formula as:

[0122]

[0123] In the formula, I(X|Y) is the mutual information between the feature representation X of modality A and the feature representation Y of modality B, p(x, y) is the probability of the feature representation A and the feature representation B occurring simultaneously, p(x) is the probability of the feature representation X occurring alone, and p(x|y) is the probability of the feature representation X occurring given the feature representation Y;

[0124] For each modality feature, the conditional mutual information of the feature with other modality features is summed to calculate the importance score I(X) of the feature representation X in modality A, which is expressed in the formula as:

[0125] I(X) = ∑ Y≠A I(X|Y);

[0126] In the formula, Y≠A represents other modalities except modality A;

[0127] The importance scores of the feature representations of different modalities are calculated according to the formula for calculating the importance score I(X) of the feature representation X in modality A;

[0128] According to the importance scores of the feature representations of the different modalities, corresponding feature representation weights are assigned, and a multi-modal feature representation is obtained using linear weighting;

[0129] It is worth noting that in addition to the multi-modal feature representation obtained by the steps described in the present example, the multi-modal feature representation can also be obtained by model training learning. Specifically:

[0130] The selected different modal feature representations are fused to obtain a multi-modal feature representation, which fully utilizes the complementarity of multi-modal data and improves the expression ability of the comprehensive feature. The formula is expressed as:

[0131] z=F(x1, x2, …, x N );

[0132] In the formula, z is a multi-modal feature representation, and F is a feature fusion function that maps the input modal feature representation to a multi-modal feature space.

[0133] The fused multi-modal feature is input into a convolutional neural network (CNN) to learn a more expressive multi-modal representation through training. First, define a convolutional neural network model including convolutional layers, pooling layers, fully connected layers, etc. The goal of the model is to learn the representation of the input features, which can be represented by the following formula:

[0134] s=h(z)

[0135] In the formula, s is the learned feature representation, and h() is the CNN model learning function.

[0136] S4, considering the relationship and interaction between different modal data, constructing a graph structure information;

[0137] According to the entities in the multi-modal data, a node is created for each entity. In the present embodiment, nodes corresponding to text, image, audio and video data are constructed according to the four types of data;

[0138] According to the relationship information in the multi-modal data, an edge is created between the nodes. The edge can represent the connection between entities, semantic association, similarity, etc. In the present embodiment, the edge is constructed according to the mutual information between different modalities and the mutual information of different modal feature representations;

[0139] The edge in the graph is assigned a weight to represent the importance or relevance of different edges. The weight can be determined according to the strength of the relationship, similarity or other specific criteria. In the present embodiment, the edge is assigned a weight according to the importance score of the feature representation of different modalities;

[0140] According to the connection relationship between nodes and edges, the structure of the graph is constructed. The graph can be a directed graph or an undirected graph, in which the nodes represent entities and the edges represent the relationship between entities.

[0141] S5, construct a graph neural network model, learn the representation vector of the node, use the representation vector of the node to perform cross-modal relationship reasoning, and optimize the associated relationship in the knowledge graph;

[0142] The graph neural network is a GCN model, the GCN model uses the feature representation of the node and the graph structure information to perform information transmission and feature updating between nodes, learns the representation vector of the node, and uses the node representation learned by the GCN model to perform cross-modal relationship reasoning, which is expressed by a formula as follows:

[0143]

[0144] In the formula, H (I) is the node representation matrix of the Ith layer, is the normalized adjacency matrix, is a diagonal matrix, W (I) is the weight matrix of the Ith layer, and sigma is an activation function.

[0145] Through information transmission and aggregation between nodes, the correlation and semantic relationship between different modalities are learned, which is expressed by a formula as follows:

[0146]

[0147] In the formula, is the representation vector of the Ith layer node i, is the attention weight between nodes i and j, W (I) is the weight matrix of the Ith layer, and sigma is an activation function.

[0148] The above description is only an embodiment of the present application, and does not limit the patent range of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection range of the present application.

Claims

1. A method for grid scheduling based multi-modal knowledge graph, characterized in that, The specific steps include: acquiring a power grid dispatching history multi-modal data set, including a text data set, an image data set, an audio data set and a video data set; standardizing the power grid dispatching history multi-modal data set to obtain a multi-modal sample data set; extracting features from the multi-modal sample data set to obtain a corresponding feature set, introducing an adaptive feature selection mechanism, fusing the feature representations of different modalities to obtain multi-modal feature representations, specifically: calculating the mutual information between different modalities, expressed in the formula as: ； wherein is the mutual information between modalities A and B, is the probability of modalities A and B occurring simultaneously, is the probability of modality A occurring alone, is the probability of modality B occurring alone; calculating the mutual information of the feature representations of different modalities, expressed in the formula as: ； wherein is the mutual information between the feature representation X of modality A and the feature representation Y of modality B, is the probability of the feature representation A and the feature representation B occurring simultaneously, is the probability of the feature representation X occurring alone, is the probability of the feature representation X occurring given the feature representation Y. computing an importance score for the feature representation X in modality A In formula expression is: ； In the formulae, denotes the other modalities than modal A; According to the formula, the importance score of the feature representation X in the computing mode A is obtained The importance score of the feature representation of different modes is obtained according to the formula assigning corresponding feature representation weights according to the importance scores of the feature representations of different modalities, and obtaining multi-modal feature representations using linear weighting; considering the relationship and interaction between different modal data, constructing graph structure information; constructing a graph neural network model to learn the representation vector of the node, and using the representation vector of the node to perform cross-modal relationship reasoning to optimize the associated relationship in the knowledge graph. 2.The method of claim 1, wherein, The text data set includes power grid historical dispatching instructions and power grid historical operation and maintenance records; the image data set includes thermal images and fault scene photos of power grid equipment; the audio data set includes power grid equipment operation audio; and the video data set includes power grid equipment and line fault conditions when a fault occurs. 3.The method of claim 2, wherein, The standardization processing of the power grid dispatching history multi-modal data set specifically converts the power grid dispatching history multi-modal data set of different modalities into a unified format, wherein: the standardization processing of the text data set includes removing special characters, punctuation marks and stop words, and unifying the text letter case; the standardization processing of the image data set includes pixel value normalization and size adjustment; the standardization processing of the audio data set includes audio amplitude normalization and sampling rate unification; the standardization processing of the video data set includes video frame rate unification and video size adjustment. 4.The method of claim 3, wherein, The feature extraction of the multi-modal sample data set to obtain the corresponding feature set specifically includes: training the text sample data set using the word embedding model Word2Vec to learn the distributed representation of words; replacing the words in each text data with the corresponding word vector, and performing weighted average operation on the word vector to obtain the feature representation of the text; using a multi-scale spatial perception image feature extraction algorithm to extract features from the image sample data set to obtain the feature representation of the image; using a multi-resolution time-frequency deep learning feature extraction algorithm to extract features from the audio sample data set to obtain the feature representation of the audio; using a spatio-temporal attention convolutional network algorithm to extract features from the video sample data set to obtain the feature representation of the video. 5.The grid scheduling based multi-modal knowledge graph method of claim 4, wherein, The feature extraction of the image sample data set using the multi-scale spatial perception image feature extraction algorithm to obtain the feature representation of the image specifically includes: constructing multiple scales of the image sample data through a pyramid structure, each scale containing different image detail information, expressed in the formula as: ； wherein is the pixel value in the image sample data at position on scale s, is a Gaussian filter on scale s, is the standard deviation of the Gaussian filter; performing convolution operation on the image sample data at each scale, expressed in the formula as: ； wherein is the convolution result of the image sample data at scale s located at position in the image sample data at scale s, is a Gaussian filter at scale s; fusing the convolution results at each scale to obtain the features of the image sample data at different scales, expressed in the formula as: ； In the formula, is the feature value of the image sample data at the position of all scales, and then Nonlinear mapping is performed on the features of image sample data of different scales to obtain the feature representation of the image, which is expressed by the formula as follows: ； In the formula, is a non-linear mapping function processed feature values, i.e., a feature representation of the image. 6.The method of claim 5, wherein, The multi-resolution time-frequency deep learning feature extraction algorithm is used to extract the features of the audio sample data set to obtain the feature representation of the audio, which is specifically as follows: The audio signal in the audio sample data is converted to the time-frequency domain, which is expressed by the formula as follows: ； wherein is a complex value in the time-frequency domain located in a time frame and a frequency band , is an original audio signal, is a window function, is the number of points of the FFT, is an index of the frequency band; Local and global features of the audio signal are extracted at different resolutions in the time-frequency domain, and the local and global features of the audio signal are input into a deep neural network for feature extraction. The deep learning features of the audio are obtained through multiple layers of convolution, pooling and full connection layer. Nonlinear mapping is performed on the deep learning features to obtain the feature representation of the audio.

7. The method of claim 6, wherein, The spatio-temporal attention convolution network algorithm is used to extract the features of the video sample data set to obtain the feature representation of the video, which is specifically as follows: The spatio-temporal relationship between video frames in the video sample data is extracted through 3D convolution, and the attention mechanism is introduced to model important spatio-temporal regions in the video sample data to obtain spatio-temporal attention features, which are expressed by the formula as follows: ； wherein and are the rows and columns of the image in two-dimensional space, respectively, is the image in two-dimensional space and the attention weight at time step , is a function that computes the attention weight; The spatio-temporal attention features and the original features are fused, and a nonlinear mapping function is applied to obtain the feature representation of the video, which is expressed by the formula as follows: ； In the formula, is the original feature at position and time step , is a nonlinear mapping function, and are the fusion weights of the original feature and the spatiotemporal attention feature, respectively. 8.The method of claim 7, wherein, The relationship and interaction between different modal data are considered, and the graph structure information is constructed, which is specifically as follows: According to the four types of data of text, image, audio and video, the corresponding nodes are constructed; According to the mutual information between different modalities and the mutual information of the feature representations of different modalities, the edges are constructed; According to the importance score of the feature representation of different modalities, the importance weight distribution is performed on the edges; According to the connection relationship of nodes and edges, the graph structure information is constructed. 9.The grid scheduling based multi-modal knowledge graph method of claim 8, wherein, The graph neural network model is constructed, the representation vector of the node is learned, and the representation vector of the node is used for cross-modal relationship reasoning to optimize the associated relationship in the knowledge graph, which is specifically as follows: The graph neural network is a GCN model, which uses the feature representation of the node and the graph structure information to perform information transmission and feature updating between nodes, learns the representation vector of the node, and uses the node representation learned by the GCN model to perform cross-modal relationship reasoning, which is expressed by the formula as follows: ； wherein, is the first layer of nodes represents a matrix, is the normalized adjacency matrix, is the diagonal matrix, is the first layer of weight matrices, is the activation function; Through information transmission and aggregation between nodes, the correlation and semantic relationship between different modalities are learned, which is expressed by the formula as follows: ； wherein is the layer node representation vector, is the importance weight of the edge between nodes and , is the weight matrix of the layer, is the activation function.

Images