An intelligent query method and system for electric power scientific research knowledge based on knowledge graph

Abstract

The present invention discloses a method and system for intelligent query of electric power scientific research knowledge based on a knowledge graph. The method includes the following steps: collecting electric power scientific research literature information, crawling a keyword dictionary in the electric power scientific research field, storing the collected electric power scientific research literature information and electric power scientific research field keywords in a graph database Neo4j; collecting sentences about keywords and keyword interpretations in electric power articles, and manually annotating the data; constructing a knowledge graph of keywords and words as an entity, and obtaining a neighborhood matrix of each word that integrates neighbor information based on the graph; constructing an intelligent query model for electric power scientific research knowledge that integrates the neighborhood information of the knowledge graph, and the model is trained to output the identification of electric power keywords in sentences and the analysis of sentence intent. The present invention improves the commonly used character embedding model and incorporates the aggregated information of the knowledge graph between words and electric power keywords based on neighbors, which can better match electric power keywords.

Description

Technical Field

[0001] The present invention relates to the technical field of electric power scientific research knowledge extraction, and in particular to a method and system for intelligent query of electric power scientific research knowledge based on a knowledge graph and a neural network model. Background Art

[0002] As one of the fundamental infrastructures of modern society, the power industry plays a vital role worldwide. A wealth of literature provides valuable information for power science research. Effectively exploring and organizing this information can support scientific research in the power sector, promote problem-solving and innovation, and provide a scientific basis for policymaking.

[0003] The current extraction of power scientific research knowledge mainly has the following difficulties:

[0004] 1) The lack of publicly available, annotated Chinese electricity text datasets limits researchers from conducting systematic experiments and evaluations;

[0005] 2) Named entity recognition methods in general fields, such as name and place name recognition, and common text classification methods, such as sentiment classification and news classification, have ideal recognition results on general datasets. However, directly transferring these general field technologies to the power sector does not produce ideal results.

[0006] 3) Few models can simultaneously perform named entity recognition and intent recognition of keywords in the power field. Summary of the Invention

[0007] Purpose of the invention: In order to overcome the shortcomings of the existing technology, the present invention provides a method for extracting scientific research knowledge from the power industry based on a neural network model, which solves the problems of the above-mentioned existing technology. The present invention also provides a system for extracting scientific research knowledge from the power industry based on a neural network model.

[0008] Technical solution: According to a first aspect of the present invention, a method for intelligent query of electric power scientific research knowledge based on a knowledge graph is provided, the method comprising the following steps:

[0009] S1 collects power research literature information, crawls the power research field keyword dictionary, and stores the collected power research literature information and power research field keywords in the graph database Neo4j;

[0010] S2 collects sentences with keywords and keyword definitions from articles about electricity and manually annotates the data;

[0011] S3 constructs a knowledge graph of keywords and words by treating each word in the keyword in step S2 as an entity, and obtains a neighborhood matrix of each word that integrates neighborhood information based on the graph;

[0012] S4 builds an intelligent query model for electric power research knowledge that integrates knowledge graph neighborhood information, denoted as the ESQM model. After training, the model outputs the power keyword recognition and sentence intent analysis in sentences.

[0013] The ESQM model includes an input feature layer and a feature extraction and prediction output layer. The input feature layer is a character embedding model combined with a knowledge graph neighborhood matrix. The feature extraction and prediction output layer includes: a BILSTM layer, a named entity layer, and a text classification layer. After the input character vector passes through the BILSTM layer, it passes through the named entity layer and the text classification layer respectively to obtain the named entity keywords and the required classification of the query sentence.

[0014] S5 uses the corresponding database query language to obtain the correct classification in the graph database based on the obtained sentence intention analysis and power keywords, and completes the intelligent query of power scientific research knowledge.

[0015] Further, including:

[0016] The manual annotation in step S2 includes: constructing electricity query sentences with different intentions, and annotating the electricity keywords therein and the query intention of the sentence.

[0017] Further, including:

[0018] In step S3, each character in the keyword is taken as an entity to construct a knowledge graph of keywords and characters, specifically, each character in the keyword is separated, and a graph connection structure is formed with all keywords to construct a neighborhood matrix between each character.

[0019] Further, including:

[0020] In step S4, the character embedding model combined with the knowledge graph neighborhood matrix includes two inputs: one input is converted into a 256-dimensional vector representation through the embedding layer, and the other input vector incorporates the neighborhood information of the word and the entity. The word and the keyword are similar to the interaction relationship between the user and the selected item;

[0021] For a pair of candidate users u and items i, K represents the entity-relationship-entity set in the knowledge graph, and N v ={(i,v)|(i,r,v)∈K} represents the set of entities directly connected to i in the knowledge graph, that is, the first-order connectivity information of item i on the knowledge graph. Through this first-order connectivity information, the neighborhood information of the item entity can be integrated and embedded with the user to realize the entity neighborhood representation of user u, as shown in the following formula:

[0022]

[0023] Where: e(1)u,r i,vis the first-order entity neighborhood embedding representation of the user, e(0)u,r i,v is the embedding representation of the ID corresponding to item i, e(0)u,r i,v’ is the embedding representation of the entity v' corresponding ID, er i,v’ is the embedding representation of the relation ID corresponding to item i and entity v', r i,v is the relationship between item i and adjacent entities in the knowledge graph, e(0)u is the embedding representation of the user's corresponding ID, · represents the multiplication of elements at the same position one by one, score(u,r i,v ) is the user u to relationship r i,v The normalized user relationship score. At the same time, since the value of |Nv| varies for different entities in the actual knowledge graph, in order to keep the calculation mode of each batch fixed and more efficient, a set of fixed-size neighbor entities is uniformly sampled for each entity, that is, a fixed receptive field is used, and the receptive field is set to 10. Therefore, a 256-dimensional output vector is finally obtained.

[0024] Therefore, the embedding vector after the input feature layer will become 256+256=512 dimensions.

[0025] The neighborhood information vector is a keyword vector that integrates the neighborhood information of the knowledge graph. Each word in the keyword is an entity, and each keyword is also an entity in the knowledge graph. A knowledge graph of keywords and words is constructed. For example, "power system" and "electricity", "force", "system" and "system" are defined as a containment relationship. Based on this graph, the neighborhood matrix of each word that integrates the neighborhood information is obtained, that is, "electricity" and "force" are neighbor nodes to each other, so the high-dimensional vector represented by "electricity" will contain the information contained in "force".

[0026] Further, including:

[0027] After the input character vector passes through the BILSTM layer, it passes through the named entity layer and the text classification layer respectively to obtain the named entity keywords and the required classification of the query sentence, specifically including:

[0028] The memory network layer BILSTM includes: input gate, forget gate and output gate. If the current time step is t, the character of the current time step is embedded in the vector x t , the hidden state h at the previous moment t-1 (forward) and memory cell state c t-1 (forward) As input, the hidden state h t (forward) The calculation formula is as follows:

[0029]

[0030] in,

[0031] are all weight parameters, σ represents the sigmoid function, and ⊙ represents element-level multiplication.

[0032] Similar to the forward calculation process above, the reverse process is calculated to obtain its hidden state h t (backward) ; The output of the memory network layer is these two vectors.

[0033] Assume that each training batch is batch_size and the maximum length of a sentence is length. Then the two output dimensions of BILSTM are batch_size*length*hidden_dim, where hidden_dim is the dimension of the hidden state sequence of the word embedding vector.

[0034] Further, including:

[0035] The named entity layer includes: a biaffine attention layer and a CRF layer. The vector output by the BILSTM layer passes through the Biaffine Attention layer to obtain batch_size*class_num*length, where class_num is the classification of the named entity recognition. In the power system (BIII), B represents the starting part of the power keyword, and I represents the beginning part of the power keyword. The CRF layer is used to calculate the loss.

[0036] The biaffine attention layer is an attention mechanism for capturing the relationship between elements in an input sequence. It is commonly used in tasks such as dependency parsing. It uses two MLPs to map the two input tensors to the same dimension, and then uses a bilinear layer to calculate the biaffine attention score matrix.

[0037] Further, including:

[0038] Assume that the forward time series vector X1 and the backward time series vector X2 output by BILSTM have dimensions [batch_size, length, hidden_dim]. The formula and input dimension changes are as follows:

[0039] (1) Suppose X1 and X2 pass through a multi-layer perceptron (MLP) consisting of two linear layers and a ReLU activation function. The output of the MLP is:

[0040] X1=Linear2(ReLU(Linear1(X1)))

[0041] X2=Linear2(ReLU(Linear1(X2)))

[0042] Here, Linear1 represents the first linear layer, ReLU represents the activation function, and Linear2 represents the second linear layer. The function of this MLP is to linearly map the two input tensors, perform nonlinear transformations using the ReLU activation function, and then pass the resulting X1 and X2 into the Bilinear layer.

[0043] (2) Define the weight matrices W, U, and V, where the dimensions of W are [class_num, hidden_dim, hidden_dim], the dimensions of U are [class_num, hidden_dim], and the dimensions of V are [class_num], where hidden_dim is the dimension of the hidden state sequence of the word embedding vector and class_num is the classification of named entity recognition;

[0044] (3) Perform linear transformation on the input tensors X1 and X2 respectively:

[0045] X′1=X1·W; where X′1 dimension is [batch_size, length, class_num, hidden_dim]

[0046] X′2=X2·W, where the dimension of X2′ is [batch_size, length, class_num, hidden_dim]

[0047] (4) Transpose X′2:

[0048] in, The dimension is [batch_size, length, hidden_dim, class_num];

[0049] (5) Calculate the score matrix S of Biaffine Attention:

[0050]

[0051] Among them, the dimension of S is [batch_size, length, class_num, class_num];

[0052] (6) Perform softmax operation on the score matrix S to obtain the final attention distribution matrix A:

[0053] A = softmax(S);

[0054] The dimensions of A are [batch_size, length, class_num, class_num] and the first three dimensions are [batch_size, length, class_num];

[0055] Therefore, the Biaffine Attention layer can simultaneously consider the relationship between two input tensors by introducing linear mapping and bilinear operations. It is widely used in natural language processing tasks. The vector passing through BiaffineAttention is passed to the CRF layer for prediction enhancement.

[0056] The CRF layer considers the dependencies between labels in the sequence labeling task and jointly models and infers the entire label sequence. In the CRF layer, each label is regarded as a state of the model, and each time step in the input sequence is regarded as an observation value of the model. The CRF layer models the overall structure of the label sequence by learning transition probabilities and emission probabilities.

[0057] Transition probability: represents the probability of transferring from one label to another, emission probability: represents the probability of observing a specific input given a given label;

[0058] For a given input sequence X and corresponding label sequence Y in the training set, the goal of the model is to maximize the conditional probability P(Y|X);

[0059]

[0060] Among them, Score(X, Y) is the score of the model given the input sequence X and the label sequence Y, which is calculated by the emission probability and transition probability, and Y' represents the possible label sequence.

[0061] Further, including:

[0062] The text classification layer includes: CNN feature extraction prediction output layer, cross entropy loss calculation layer, vector h output by the BILSTM layer t (forward) and h t (backward) Concatenation, as input to the text classification layer After CNN feature extraction and prediction output layer, we get batch_size*n_class, where n_class is the classification of the input text.

[0063] Further, including:

[0064] The loss function of the ESQM model is a superposition of the named entity recognition layer and the text classification layer, and the calculation formula is:

[0065] loss = αloss 命名实体层 +βloss 文本分类层 , where α and β are weight parameters. Text classification is more important in intelligent query, so the loss is increased. 文本分类层 Importance, that is, increasing the proportion of β.

[0066] On the other hand, the present invention also provides an intelligent query system for electric power scientific research knowledge based on knowledge graph, which includes:

[0067] The first acquisition module is used to collect power research literature information, crawl the power research field keyword dictionary, and store the collected power research literature information and power research field keywords in the graph database Neo4j;

[0068] The second acquisition module is used to collect sentences with keywords and keyword definitions from articles about electricity and manually annotate the data;

[0069] A knowledge graph construction module is used to construct a knowledge graph of keywords and words by treating each word in the keywords in the second acquisition module as an entity, and obtain a neighborhood matrix of each word that integrates neighborhood information based on the graph;

[0070] The model building module is used to build an intelligent query model for electric power research knowledge that integrates knowledge graph neighborhood information, denoted as the ESQM model. After training, the model outputs the power keyword recognition and sentence intent analysis in sentences;

[0071] The ESQM model includes an input feature layer and a feature extraction and prediction output layer. The input feature layer is a character embedding model combined with a knowledge graph neighborhood matrix. The feature extraction and prediction output layer includes: a BILSTM layer, a named entity layer, and a text classification layer. After the input character vector passes through the BILSTM layer, it passes through the named entity layer and the text classification layer respectively to obtain the named entity keywords and the required classification of the query sentence.

[0072] The query module is used to parse the sentence intent and power keywords, use the corresponding database query language to obtain the correct answer in the graph database, and complete the intelligent query of power scientific research knowledge. The correct answer here refers to the role of classification, such as intelligent search, what are the related keywords of ***, or **** related articles.

[0073] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0074] (1) The present invention improves the commonly used character embedding model and incorporates the knowledge graph between characters and power keywords based on the aggregated information of neighbors, which can better match power keywords;

[0075] (2) The present invention proposes an electric power scientific research knowledge intelligent query model (ESQM), which can simultaneously perform named entity recognition and text classification in the electric power field in one model, and can be applied to electric power scientific research knowledge intelligent query. At the same time, the accuracy of the model is improved compared with the common named entity recognition and text classification, reaching the industry-leading level. The training time and response time of the model are also lower than using two models to perform named entity recognition and text classification separately. BRIEF DESCRIPTION OF THE DRAWINGS

[0076] Figure 1 This is a flow chart of the extraction method according to an embodiment of the present invention;

[0077] Figure 2 This is the power data neo4j storage diagram described in the embodiment of the present invention;

[0078] Figure 3 This is a schematic diagram of the structure of the electric power scientific research knowledge intelligent query model (ESQM) according to an embodiment of the present invention;

[0079] Figure 4 Schematic diagram of the BILSTM structure according to an embodiment of the present invention. DETAILED DESCRIPTION

[0080] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention and not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0081] First, if Figure 1 As shown, the present invention provides an intelligent query method for electric power scientific research knowledge, which includes:

[0082] S1 collects power research literature information, crawls the power research field keyword dictionary, and stores the collected power research literature information and power research field keywords in the graph database Neo4j;

[0083] The collection of literature information described in step 1 is to use legal crawler methods to crawl the power research literature knowledge in CNKI, including the literature title, literature abstract, and literature keywords;

[0084] The crawling of the keyword dictionary in the field of power research refers to using the scrapy crawler framework to search Baidu Encyclopedia for the keywords to obtain the definitions of the keywords and related keywords;

[0085] The power data in the graph database Neo4j is as follows Figure 2 As shown, it contains entity power keywords, power articles, power keywords include attributes, keyword interpretations, power articles include attributes document title, document abstract, document keywords; the relationships in the graph database include the relationship between articles and keywords and the relationship between keywords.

[0086] S2 collected sentences about electricity articles, keywords and keyword definitions, and manually annotated the data;

[0087] The collection of sentences about electricity articles, keywords and keyword definitions refers to manually constructing a data set for training the model, manually constructing 2,000 query sentences containing electricity keywords, and annotating them according to the keywords and related definitions in step 1, annotating the sentence types, such as article type, keyword type, keyword definition type, and using BIO annotation for each character in each sentence to perform named entity recognition of the sentence, and obtain a data set for model training.

[0088] S3 takes each word in the keyword as an entity to construct a knowledge graph of keywords and words, and obtains the neighborhood matrix of each word based on the graph, which integrates the neighborhood information.

[0089] In order to improve the representation ability of characters corresponding to keywords, this paper compares characters to users and keywords to interactive users who have positive feedback with users.

[0090] This paper incorporates the entity neighborhood information of user items. For a pair of candidate users u and items i, K represents the “entity-relationship-entity” set in the knowledge graph, and N v ={(i,v)|(i,r,v)∈K} represents the set of entities directly connected to item i in the knowledge graph, i.e., the first-order connectivity information of item i on the knowledge graph. This first-order connectivity information can be used to integrate the neighborhood information of the item entity and embed it with the user to represent the entity neighborhood of user u.

[0091]

[0092] Where: e(1)u,r i,v is the first-order entity neighborhood embedding representation of the user, e(0)u,r i,v is the embedding representation of the ID corresponding to item i, e(0)u,r i,v’ is the embedding representation of the entity v' corresponding ID, er i,v’is the embedding representation of the relation ID corresponding to item i and entity v', r i,v is the relationship between item i and adjacent entities in the knowledge graph, e(0)u is the embedding representation of the user's corresponding ID, · represents the multiplication of elements at the same position one by one, score(u,r i,v ) is the user u to relationship r i,v The normalized user relationship score.

[0093] At the same time, since in the actual knowledge graph, |N v The value of | varies for different entities. To maintain a consistent computational model for each batch and improve efficiency, a fixed-sized set of neighboring entities is uniformly sampled for each entity, i.e., a fixed receptive field is used. In this paper, the receptive field is set to 10.

[0094] Therefore, the vector passing through the input feature layer will become 256+256=512 dimensions.

[0095] Specifically:

[0096] The neighborhood information vector is a keyword vector that integrates the neighborhood information of the knowledge graph. Each word in the keyword is an entity, and each keyword is also an entity in the knowledge graph. A knowledge graph of keywords and words is constructed. For example, "power system" and "electricity", "force", "system" and "system" are defined as a containment relationship. Based on this graph, the neighborhood matrix of each word that integrates the neighborhood information is obtained, that is, "electricity" and "force" are neighbor nodes to each other, so the high-dimensional vector represented by "electricity" will contain the information contained in "force".

[0097] A knowledge graph of electric power keywords is constructed, where each keyword and each character in the word is an entity. The correlation between the characters is obtained. Based on this graph, an adjacency list is generated for each character. The length of the adjacency list is the set receptive field. Since electric power research keywords are generally four characters long, this is set to 4 in this article. Finally, the adjacency matrix of the character entities is formed. Based on this adjacency matrix, the information aggregation vector of the character embedding layer is constructed. The details are as follows.

[0098] This paper compares characters to users and keywords to interactive users who have positive feedback with users. This paper incorporates the entity neighborhood information of user items. For a pair of candidate users u and items i, K represents the "entity-relationship-entity" set in the knowledge graph, which is "character entity-relationship-word entity" in this paper. N v ={(i,v)|(i,r,v)∈K} represents the set of entities directly connected to i in the knowledge graph, that is, the first-order connectivity information of the word entity on the knowledge graph. Through this first-order connectivity information, the neighborhood information of the word entity can be integrated to finally obtain the entity neighborhood embedding representation of the word entity u.

[0099] At the same time, since in the actual knowledge graph, |Nv The value of | varies for different entities. To maintain a consistent computational model for each batch and improve efficiency, a fixed-sized set of neighboring entities is uniformly sampled for each entity, i.e., a fixed receptive field is used. In this paper, the receptive field is set to 4.

[0100] Each word input is vectorized and then the word embedding is obtained. It represents the embedding representation of the ID corresponding to the word entity, and finally obtains the first-order entity neighborhood embedding representation of the input word entity through weighted aggregation.

[0101]

[0102]

[0103] in, is the first-order entity neighborhood embedding representation of the word entity, is the embedding representation of the ID corresponding to the keyword entity i, r i,v is the relationship between keyword entity i and adjacent entities in the knowledge graph, · represents the multiplication of elements in the same position one by one, score(u,r i,v ) is the word entity u to relationship r i,v The normalized user relationship score is calculated as shown in formula (4).

[0104] Through the above model, each input word can be converted into a keyword vector containing knowledge graph neighborhood information.

[0105] S4 builds an electric power scientific research knowledge intelligent query model (ESQM). The network can use the data set to simultaneously identify electric power keywords in sentences and analyze sentence intent, such as Figure 3 As shown;

[0106] The electric power scientific research knowledge intelligent query model (ESQM) includes an input feature layer and a feature extraction and prediction output layer. The input feature layer is a character embedding model combined with a knowledge graph neighborhood matrix. The feature extraction and prediction output layer includes a BILSTM layer, a named entity layer, and a text classification layer. After the input character vector passes through the BILSTM layer, it passes through the named entity part and the text classification part respectively to obtain the named entity keywords and the required classification of the query sentence.

[0107] The character embedding model combined with the knowledge graph neighborhood matrix means that each character in the input text is used as two inputs, one input is converted into a 264-dimensional vector representation through the embedding layer, and the other input vector is pre-trained with the embedding layer with dictionary information;

[0108] The other input vector is pre-trained with a dictionary information embedding layer, which means that for the input character t, all words containing t are matched in the power dictionary constructed above, and all matched power keywords are encoded into four sets of B(t), M(t), E(t), and S(t), respectively, where B(t) indicates that the number of power keywords is greater than 1 and starts with t, M(t) indicates that t in the power keyword is not at the beginning or end, E(t) indicates that the number of characters in the power keyword is greater than 1 and ends with the character t, and S(t) indicates that the power keyword consists only of t; after obtaining the B, M, E, and S word sets corresponding to each character, the power keyword set needs to be compressed, the purpose of which is to compress the word embedding of each set into a feature vector;

[0109] x s =[v(B),v(M),v(E),v(S)]

[0110]

[0111] Among them, v(S) represents the feature vector of each set, which is a 64-dimensional vector obtained by the keyword through the pre-trained embedding vector model (word2vec), x s The concatenation of the 64-dimensional vectors generated by the four word sets is a 4*64=256-dimensional feature vector;

[0112] S42 Therefore, the vector passing through the input feature layer will become 256+256=512 dimensions.

[0113] The BILSTM layer is a bidirectional time memory network model, including input gate, forget gate and output gate. If the current time step is t, the character of the current time step is embedded in the vector x t , the hidden state h at the previous moment t-1 (forward) and memory cell state c t-1 (forward) As input, the hidden state h t (forward) The calculation formula is as follows:

[0114]

[0115] in,

[0116] are all weight parameters, σ represents the sigmoid function, and ⊙ represents element-level multiplication.

[0117] Similar to the forward calculation process above, the reverse process is calculated to obtain its hidden state h t (backward); The output of the memory network layer is these two vectors.

[0118] Assume that each training batch is batch_size and the maximum length of a sentence is length. Then the two output dimensions of BILSTM are batch_size*length*hidden_dim, where batch_size is the sentence batch, length is the set length of the sentence (the number of characters in the sentence), and hidden_dim is the dimension of the hidden state sequence of the word embedding vector.

[0119] The named entity recognition layer includes a biaffine attention layer and a CRF layer. The vector output by the BILSTM layer passes through the biaffine attention layer to obtain batch_size*class_num*length. Class_num is the classification of the named entity recognition. For example, the output of "power system" is "BIII", where B represents the starting part of the power keyword and I represents the beginning part of the power keyword. The CRF layer is used to calculate the loss.

[0120] The biaffine attention layer is an attention mechanism used to capture the relationship between elements in an input sequence. It is often used in tasks such as dependency parsing. This paper uses two MLPs to map the two input tensors to the same dimension and then uses a bilinear layer to calculate the biaffine attention score matrix.

[0121] Assume that the forward time series vector X1 and the backward time series vector X2 output by BILSTM have dimensions [batch_size, length, hidden_dim]. The formula and input dimension changes are as follows:

[0122] Suppose X1 and X2 pass through a multi-layer perceptron (MLP) consisting of two linear layers and a ReLU activation function. The output of the MLP is

[0123] X1=Linear2(ReLU(Linear1(X1)))

[0124] X2=Linear2(ReLU(Linear1(X2)))

[0125] Here, Linear1 represents the first linear layer, ReLU represents the activation function, and Linear2 represents the second linear layer. The function of this MLP is to linearly map the two input tensors, perform nonlinear transformations using the ReLU activation function, and then pass the resulting X1 and X2 into the Bilinear layer.

[0126] Define weight matrices W, U, and V, where the dimensions of W are [class_num, hidden_dim, hidden_dim], the dimensions of U are [class_num, hidden_dim], and the dimensions of V are [class_num]. Hidden_dim is the dimension of the hidden state sequence of the word embedding vector, and class_num is the classification of named entity recognition.

[0127] Perform linear transformation on the input tensors X1 and X2 respectively:

[0128] X′1=X1·W

[0129] Among them, the dimension of X′1 is [batch_size, length, class_num, hidden_dim]:

[0130] X′2=X2·W

[0131] Among them, the dimension of X2' is [batch_size, length, class_num, hidden_dim]:

[0132] Transpose X′2:

[0133] in, The dimension is [batch_size, length, hidden_dim, class_num];

[0134] Calculate the score matrix S of Biaffine Attention:

[0135]

[0136] Among them, the dimension of S is [batch_size, length, class_num, class_num];

[0137] Perform a softmax operation on the score matrix S to obtain the final attention distribution matrix A:

[0138] A=softmax(S)

[0139] The dimensions of A are [batch_size, length, class_num, class_num] and the first three dimensions are [batch_size, length, class_num];

[0140] Biaffine Attention introduces linear mapping and bilinear operations, which can simultaneously consider the relationship between two input tensors and is widely used in natural language processing tasks.

[0141] The [batch_size, length, class_num] vector of Biaffine Attention passes through the CRF layer. The CRF feature extraction layer is used to calculate the dependency between labels to ensure that the generated label sequence is reasonable. When using CRF, the label sequence output by the model is the most likely label sequence in the entire sequence, which helps solve the local consistency problem in sequence labeling and ensures that the output label sequence is consistent across the entire sequence.

[0142] In the CRF layer, each label is regarded as a state of the model, and each time step in the input sequence is regarded as an observation value of the model. The CRF layer models the overall structure of the label sequence by learning transition probabilities and emission probabilities.

[0143] Transition probability: represents the probability of transferring from one label to another, emission probability: represents the probability of observing a specific input given a given label;

[0144] For a given input sequence X and corresponding label sequence Y in the training set, the goal of the model is to maximize the conditional probability P(Y|X);

[0145]

[0146] Among them, Score(X, Y) is the score of the model given the input sequence X and the label sequence Y, which is calculated by the emission probability and transition probability, and Y' represents the possible label sequence.

[0147] The text classification layer refers to the CNN feature extraction prediction output layer, the cross entropy loss calculation layer, and the vector output by the BILSTM layer. and Concatenation, as input to the text classification layer After CNN feature extraction and prediction output layer, we get batch_size*n_class, where n_class is the classification of the input text. For example, if the input is: (Please give me some articles related to power electronics and transistors), its text classification belongs to the retrieval article part.

[0148] The loss function of this model is superimposed by the named entity recognition layer and the text classification layer, and the calculation formula is: loss = αloss 命名实体层 +βloss 文本分类层, where α and β are weight parameters. Text classification is more important in this intelligent query, so the loss can be increased. 文本分类层 Importance, that is, increasing the proportion of β.

[0149] Based on the obtained sentence intent and power keywords, S5 uses the corresponding cypher statements to obtain the correct answer in the graph database and complete the intelligent query of power scientific research knowledge.

[0150] The corresponding cypher statement is a language for querying the graph database neo4j. For example, if the power keywords obtained from the S3 model are ['PID controller', 'matlab'], and the sentence is intended to associate keywords, the following cypher statement can be used:

[0151] "MATCH(p:Paper)-[:HAS_KEYWORD]->(k:Keyword)"

[0152] "WHERE k.name IN $keywords"

[0153] "WITH p,count(k)as keywordCount"

[0154] "RETURN p, keywordCount"

[0155] "ORDER BY keywordCount DESC"

[0156] Get the remaining keywords associated with ['PID controller', 'matlab'];

[0157] For example, if the power keywords obtained from the S3 model are ['deep learning', 'recommendation system'] and the sentence intent is the article type, the following cypher statement can be used:

[0158] MATCH(p:Paper)-[:HAS_KEYWORD]->(k:Keyword)

[0159] WHERE k.name IN $keywords

[0160] WITH p,count(k)as KeywordCount

[0161] RETURN p.title AS Title, KeywordCount, p AS Paper

[0162] ORDER BY KeywordCount DESC

[0163] Get power articles containing the keyword ['deep learning', 'recommendation system'] and sort them by the number of matches.

[0164] On the other hand, the present invention also provides an intelligent query system for electric power scientific research knowledge based on knowledge graph, which includes:

[0165] The first acquisition module is used to collect power research literature information, crawl the power research field keyword dictionary, and store the collected power research literature information and power research field keywords in the graph database Neo4j;

[0166] The second acquisition module is used to collect sentences with keywords and keyword definitions from articles about electricity and manually annotate the data;

[0167] A knowledge graph construction module is used to construct a knowledge graph of keywords and words by treating each word in the keywords obtained by the second acquisition module as an entity, and to obtain a neighborhood matrix of each word that integrates neighborhood information based on the graph;

[0168] The model building module is used to build an intelligent query model for electric power research knowledge that integrates knowledge graph neighborhood information, denoted as the ESQM model. After training, the model outputs the power keyword recognition and sentence intent analysis in sentences;

[0169] The ESQM model includes an input feature layer and a feature extraction and prediction output layer. The input feature layer is a character embedding model combined with a knowledge graph neighborhood matrix. The feature extraction and prediction output layer includes: a BILSTM layer, a named entity layer, and a text classification layer. After the input character vector passes through the BILSTM layer, it passes through the named entity layer and the text classification layer respectively to obtain the named entity keywords and the required classification of the query sentence.

[0170] The query module is used to parse the sentence intent and power keywords based on the database in the first acquisition module, use the corresponding database query language to obtain the correct answer in the graph database, and complete the intelligent query of power scientific research knowledge.

[0171] Other technical features of this system are similar to the methods described in this application and will not be repeated here.

[0172] Finally, the present invention also provides a computer-readable storage medium, which stores computer instructions, and the computer instructions are used to enable a processor to implement the above-mentioned method for intelligent extraction of scientific research knowledge in the power industry when executed.

[0173] It will be understood by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Thus, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0174] Although the preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.

[0175] Obviously, those skilled in the art may make various changes and modifications to the embodiments of the present invention without departing from the spirit and scope of the embodiments of the present invention. Thus, if such changes and modifications of the embodiments of the present invention fall within the scope of the claims and their equivalents, the present invention is intended to include such changes and modifications.

Claims

1. An intelligent query method for electric power scientific research knowledge based on knowledge graph, characterized in that: The method comprises the following steps: S1 collects power research literature information, crawls the power research field keyword dictionary, and stores the collected power research literature information and power research field keywords in the graph database Neo4j; S2 collects sentences containing keywords and their meanings from articles about electricity and manually annotates the data; S3 constructs a knowledge graph of keywords and words by treating each word in the keywords and related meanings obtained in step S2 as an entity, and obtains a neighborhood matrix of each word that integrates neighborhood information based on the knowledge graph; Among them, in constructing the knowledge graph of keywords and characters, since each keyword and each character in the word are entities, the correlation between the characters is obtained, each character in the keyword is separated, and a graph connection structure is formed with all keywords. Based on this graph connection structure, an adjacency list of each character is obtained, and the length of the adjacency list is the set receptive field. Finally, a neighborhood matrix of the character entity is combined, and the information aggregation vector of the character embedding layer is constructed based on this neighborhood matrix; the neighborhood matrix of the character entity is specifically composed of: S31 compares characters to users and keywords to interactive users who have positive feedback with users, thereby integrating the entity neighborhood information of user items. For a pair of candidate users u and items i, K represents the "entity-relationship-entity" set in the knowledge graph, that is, corresponding to "character entity-relationship-word entity", using N v ={(i,v)|(i,r,v)∈K} represents the set of entities directly connected to item i in the knowledge graph, that is, the first-order connectivity information of the word entity on the knowledge graph; S32 integrates the first-order connectivity information into the neighborhood information of the character entity, and finally obtains the first-order entity neighborhood embedding of the input character entity u through weighted aggregation, which is specifically expressed as: in, is the first-order entity neighborhood embedding representation of the user, is the embedding representation of the ID corresponding to item i, is the embedding representation of the entity v' corresponding ID, is the embedding representation of the relation ID corresponding to item i and entity v', r i,v is the relationship between item i and adjacent entities in the knowledge graph, is the embedding representation of the user's corresponding ID, · represents the multiplication of elements at the same position one by one, score(u,r i,v' ) is the user u to relationship r i,v' The normalized user relationship score of ,Finally, each input word obtains a keyword vector containing the knowledge graph neighborhood information; S4 builds an intelligent query model for electric power scientific research knowledge that integrates neighborhood information, denoted as the ESQM model. After training, the model outputs the recognition of electric power keywords in sentences and the corresponding sentence intent analysis; The ESQM model includes an input feature layer and a feature extraction and prediction output layer. The input feature layer is a character embedding model combined with a knowledge graph neighborhood matrix. The feature extraction and prediction output layer includes: a BILSTM layer, a named entity layer, and a text classification layer. After the input character vector passes through the BILSTM layer, it passes through the named entity layer and the text classification layer respectively to obtain the named entity keywords and intent analysis of the query sentence. S5 uses the sentence intention analysis and power keywords obtained in step S1, and uses the corresponding database query language to obtain the correct classification in the graph database based on the power research literature information and power research field keywords obtained in step S1, and completes the intelligent query of power research knowledge.

2. The intelligent query method for electric power scientific research knowledge based on knowledge graph according to claim 1 is characterized in that: The manual annotation in step S2 includes: constructing electricity query sentences with different intentions, and annotating the electricity keywords therein and the intention analysis of the sentence.

3. The intelligent query method for electric power scientific research knowledge based on knowledge graph according to claim 2 is characterized in that: In step S4, the input of the character embedding model combined with the knowledge graph neighborhood matrix includes two: one input is converted into a 256-dimensional vector representation through the embedding layer, and the other input vector incorporates the neighborhood information of the word and entity; Since in the actual knowledge graph, |N v The value of | varies for different entities. To keep the computational mode of each batch fixed and more efficient, a fixed-size set of neighboring entities is uniformly sampled for each entity. That is, a fixed receptive field is used, which is set to 10. Therefore, the final output vector is 256-dimensional. Therefore, the embedding vector after passing through the input feature layer becomes 256 + 256 = 512 dimensions.

4. The method for intelligent query of electric power scientific research knowledge based on knowledge graph according to claim 3 is characterized in that: After the input character vector passes through the BILSTM layer, it passes through the named entity layer and the text classification layer respectively to obtain the named entity keywords and the required classification of the query sentence, specifically including: The memory network layer BILSTM includes: input gate, forget gate and output gate. If the current time step is t, the character of the current time step is embedded in the vector x t , the hidden state h at the previous moment t-1 (forward) and memory cell state c t-1 (forward) As input, the hidden state h t (forward) The calculation formula is as follows: in, are all weight parameters, σ represents the sigmoid function, and ⊙ represents element-level multiplication; Similar to the forward calculation process, the reverse process is calculated to obtain its hidden state h t (backward) ; The output of the memory network layer is these two vectors; Assume that each training batch is batch_size and the maximum length of a sentence is length. Then the two output dimensions of BILSTM are batch_size*length*hidden_dim, where hidden_dim is the dimension of the hidden state sequence of the word embedding vector.

5. The method for intelligent query of electric power scientific research knowledge based on knowledge graph according to claim 4 is characterized in that: The named entity layer includes: a biaffine attention layer and a CRF layer. The vector output by the BILSTM layer passes through the Biaffine Attention layer to obtain a vector of batch_size*class_num*length dimensions, where class_num is the classification of the named entity recognition. The CRF layer is used to calculate the loss. The biaffine attention layer is an attention mechanism for capturing the relationship between elements in an input sequence. It uses two MLPs to map the two input tensors to the same dimension, and then uses a Bilinear layer to calculate the score matrix of the biaffine attention.

6. The intelligent query method for electric power scientific research knowledge based on knowledge graph according to claim 5 is characterized in that The forward time series vector X1 and the backward time series vector X2 output by BILSTM both have dimensions [batch_size, length, hidden_dim]. The formulas and input dimension changes are as follows: (1) Suppose X1 and X2 pass through a multi-layer perceptron (MLP) consisting of two linear layers and a ReLU activation function. The output of the MLP is: X1=Linear2(ReLU(Linear1(X1))); X2=Linear2(ReLU(Linear1(X2))); Among them, Linear1 represents the first linear layer, ReLU represents the activation function, and Linear2 represents the second linear layer. The function of this MLP is to linearly map the two input tensors, perform nonlinear transformation through the ReLU activation function, and then pass the obtained X1 and X2 into the Bilinear layer; (2) Define the weight matrices W, U, and V, where the dimensions of W are [class_num, hidden_dim, hidden_dim], the dimensions of U are [class_num, hidden_dim], and the dimensions of V are [class_num], where hidden_dim is the dimension of the hidden state sequence of the word embedding vector and class_num is the classification of named entity recognition; (3) Perform linear transformation on the input tensors X1 and X2 respectively: X1′=X1·W; where X1′ dimension is [batch_size, length, class_num, hidden_dim] X2′=X2·W; where the dimension of X2′ is [batch_size, length, class_num, hidden_dim] (4) Transpose X2′: in, The dimension is [batch_size, length, hidden_dim, class_num]; (5) Calculate the score matrix S of Biaffine Attention: Among them, the dimension of S is [batch_size, length, class_num, class_num]; (6) Perform softmax operation on the score matrix S to obtain the final attention distribution matrix A: A=softmax(S); The dimensions of A are [batch_size, length, class_num, class_num] and the first three dimensions are [batch_size, length, class_num]; Therefore, the Biaffine Attention layer can simultaneously consider the relationship between two input tensors by introducing linear mapping and bilinear operations. It is widely used in natural language processing tasks. The vector passed through the Biaffine Attention is passed to the CRF layer for prediction enhancement. The CRF layer considers the dependencies between labels in the sequence labeling task and jointly models and infers the entire label sequence. In the CRF layer, each label is regarded as a state of the model, and each time step in the input sequence is regarded as an observation value of the model. The CRF layer models the overall structure of the label sequence by learning transition probabilities and emission probabilities. Transition probability: represents the probability of transferring from one label to another, emission probability: represents the probability of observing a specific input given a given label; For a given input sequence X and corresponding label sequence Y in the training set, the goal of the model is to maximize the conditional probability P(Y|X); Among them, Score(X, Y) is the score of the model given the input sequence X and the label sequence Y, which is calculated by the emission probability and transition probability, and Y' represents the possible label sequence.

7. The method for intelligent query of electric power scientific research knowledge based on knowledge graph according to claim 6 is characterized in that: The text classification layer includes: CNN feature extraction prediction output layer, cross entropy loss calculation layer, vector h output by the BILSTM layer t (forward) and h t (backward) Splicing, as the input h of the text classification layer t =[h t (forward) ,h t (backward) ], after CNN feature extraction and prediction output layer, we get batch_size*n_class, where n_class is the classification of the input text.

8. The method for intelligent query of electric power scientific research knowledge based on knowledge graph according to claim 7 is characterized in that: The loss function of the ESQM model is a superposition of the named entity recognition layer and the text classification layer, and the calculation formula is: loss = αloss 命名实体层 +βloss 文本分类层 , where α and β are weight parameters. Text classification is more important in intelligent query, so increasing loss 文本分类层 Importance, that is, increasing the proportion of β.

9. An intelligent query system for electric power scientific research knowledge based on knowledge graph, characterized in that: The system includes: The first acquisition module is used to collect power research literature information, crawl the power research field keyword dictionary, and store the collected power research literature information and power research field keywords in the graph database Neo4j; The second acquisition module is used to collect sentences with keywords and keyword definitions from articles about electricity and manually annotate the data; this data is used as a training dataset for subsequent model training. A knowledge graph construction module, configured to construct a knowledge graph of keywords and characters by treating each character in the keywords in the second acquisition module as an entity, and to obtain a neighborhood matrix of each character that incorporates neighborhood information based on the graph; Among them, in constructing the knowledge graph of keywords and characters, since each keyword and each character in the word are entities, the correlation between the characters is obtained, each character in the keyword is separated, and a graph connection structure is formed with all keywords. Based on this graph connection structure, an adjacency list of each character is obtained, and the length of the adjacency list is the set receptive field. Finally, a neighborhood matrix of the character entity is combined, and the information aggregation vector of the character embedding layer is constructed based on this neighborhood matrix; the neighborhood matrix of the character entity is specifically composed of: The characters are compared to users, and the keywords are compared to the interactive users who have positive feedback with the users, so as to integrate the entity neighborhood information of the user items. For a pair of candidate users u and items i, K represents the "entity-relationship-entity" set in the knowledge graph, that is, it corresponds to "character entity-relationship-word entity", and N is used. v ={(i,v)|(i,r,v)∈K} represents the set of entities directly connected to item i in the knowledge graph, that is, the first-order connectivity information of the word entity on the knowledge graph; The first-order connected information is integrated into the neighborhood information of the character entity, and finally the first-order entity neighborhood embedding of the input character entity u is obtained through weighted aggregation, which is specifically expressed as: in, is the first-order entity neighborhood embedding representation of the user, is the embedding representation of the ID corresponding to item i, is the embedding representation of the entity v' corresponding ID, is the embedding representation of the relation ID corresponding to item i and entity v', r i,v is the relationship between item i and adjacent entities in the knowledge graph, is the embedding representation of the user's corresponding ID, · represents the multiplication of elements at the same position one by one, score(u,r i,v' ) is the user u to relationship r i,v' The normalized user relationship score of ,Finally, each input word obtains a keyword vector containing the knowledge graph neighborhood information; The model building module is used to build an intelligent query model for electric power research knowledge that integrates knowledge graph neighborhood information, denoted as the ESQM model. After training, the model outputs the power keyword recognition and sentence intent analysis in sentences; The ESQM model includes an input feature layer and a feature extraction and prediction output layer. The input feature layer is a character embedding model combined with a knowledge graph neighborhood matrix. The feature extraction and prediction output layer includes: a BILSTM layer, a named entity layer, and a text classification layer. After the input character vector passes through the BILSTM layer, it passes through the named entity layer and the text classification layer respectively to obtain the named entity keywords and the required classification of the query sentence. The query module is used to parse the sentence intent and power keywords based on the content of the graph database Neo4j in the first acquisition module, use the corresponding database query language to obtain the correct answer in the graph database, and complete the intelligent query of power scientific research knowledge.

Images