A graph topology learning method and system based on data statistics and knowledge guidance

Abstract

The present invention discloses a graph topology learning method and system based on data statistics and knowledge guidance, which can learn graph topology from text data under supervised downstream tasks. The method includes: inputting text data, obtaining entities in the text data as nodes of the graph topology, calculating the conditional probability between nodes according to historical text data, and constructing an adjacency matrix of the initial graph topology. Secondly, a knowledge attention network for each text data is constructed based on the domain knowledge graph. Finally, the adjacency matrix of the initial graph topology is input into the DK_Transformer (Data and Knowledge drivenTransformer) model, and the knowledge attention network is used as expert knowledge, and the attention mechanism is combined to learn the graph topology adapted to downstream tasks. This aspect can effectively solve the problems that the existing graph topology construction method based on text data only considers data statistical information, does not fully utilize knowledge, and cannot adapt to downstream tasks.

Description

Technical Field

[0001] The present invention relates to the fields of natural language processing and graph deep learning, and in particular to a graph topology learning method and system based on data statistics and knowledge guidance. Background Art

[0002] Graph models have become a hot topic in recent years due to their powerful data fusion, analysis, and reasoning capabilities. Obtaining graph data that conforms to the real world from actual scene data is the premise of graph model research. Graph topology can convey important insights about actual scene data, and its vector representation can improve the performance of various downstream tasks. However, through a comprehensive analysis of the research on graph topology construction based on text data, it is found that the common defects of existing methods are: first, existing methods only construct graph topology based on data statistical information or dependency syntax analysis, and do not make full use of the rich knowledge accumulated in this field; second, the constructed graph topology is generally a predefined static graph that cannot adapt to downstream tasks. Therefore, how to learn graph topology for text data in actual scenes and obtain graph topology that is suitable for downstream tasks is the focus of research. Summary of the invention

[0003] The main purpose of the present invention is to overcome the above-mentioned defects in the prior art and propose a graph topology learning method and system based on data statistics and knowledge guidance, so as to realize learning of graph topology that meets the requirements from text data under supervised downstream tasks.

[0004] The present invention adopts the following technical solution:

[0005] On the one hand, a graph topology learning method based on data statistics and knowledge guidance is characterized by comprising the following steps:

[0006] Step 1: Receive input text data, obtain entities in the text data as nodes of the graph topology, calculate the conditional probabilities between nodes based on historical text data, and construct an adjacency matrix of the initial graph topology;

[0007] Step 2: Build a knowledge attention network for each text data based on the domain knowledge graph;

[0008] Step 3: Input the adjacency matrix of the initial graph topology into the DK_Transformer model, and use the knowledge attention network as expert knowledge to jointly learn the graph topology that adapts to downstream tasks with the attention mechanism.

[0009] Preferably, the step 1 specifically includes:

[0010] Step 1.1: Receive text data D = {d1, d2, ..., d N}, get the entity set E = {e1, e2, …, e num}, and take it as the node set V = {v1,v2,…,v |V|}, where num is the number of entity sets, and |V| is the number of nodes in each graph topology; if num=|V|, all entities are taken as nodes of the graph topology; if num>|V|, entities with node indexes greater than |V| are deleted; if num<|V|, empty nodes with node indexes greater than num but less than or equal to |V| are filled with special codes;

[0011] Step 1.2: Based on the node set V of the graph topology determined in step 1.1, calculate any two nodes v based on all text data D. i and v j The conditional probability of (i,j=1,2,..,|V|) is used as the edge weight a ij , that is, v j In v i The probability of occurrence under the condition of occurrence p(v j |v i ), and its calculation formula is:

[0012]

[0013] Obtain the adjacency matrix P based on conditional probability, expressed as:

[0014]

[0015] Step 1.3: Normalize the adjacency matrix P obtained in step 1.2 to obtain the normalized adjacency matrix

[0016] Preferably, the step 2 specifically includes:

[0017] According to the node set V of the graph topology determined in step 1, the connection relationship of these nodes in the knowledge graph is obtained from the domain knowledge graph, so as to build a knowledge attention network M for each text data d. The calculation method is as follows:

[0018]

[0019] Where u is v i and v j The number of common ancestors, l is v i to v j The number of paths.

[0020] Preferably, the step 3 specifically includes:

[0021] Step 3.1: embed the node set V of the graph topology determined in step 1 into vectors, where the vectors of empty nodes are set to 0, and obtain the initial feature matrix X of the nodes in the graph. (0) ;

[0022] Step 3.2: The results of step 1 and step 3.1, that is, the normalized adjacency matrix of the graph topology and the initial feature matrix X of the nodes in the graph (0) Input into the first block module of the DK_Transformer model for learning;

[0023] Step 3.3: Feed the output of step 3.2 into the n block modules of the DK_Transformer model, use the knowledge attention network M obtained in step 2 as expert knowledge, and combine the attention matrix S of the DK_Transformer model to guide graph topology learning, and finally obtain the normalized adjacency matrix of the learned graph topology.

[0024] When n=1,

[0025] When n>1,

[0026] Among them, softmax is a normalized exponential function, is the attention matrix S of the nth block module of the DK_Transformer model; and They represent the query vector and key vector of the nth block module of DK_Transformer respectively; and They represent the learnable matrix parameters in the nth block module of DK_Transformer, X (n-1) is the feature matrix of the nodes in the n-1th block module; d is The dimension of the column;

[0027] Step 3.4: Calculate the loss L of the DK_Transformer model in graph topology learning reg :

[0028] When n=1,

[0029] When n>1,

[0030] Among them, D KL is the KL divergence, The matrix is ​​the normalized adjacency matrix output by the nth block module of DK_Transformer The fine-tuning matrix of . That is, the position index of the non-zero value in the label knowledge attention network M, and in the normalized adjacency matrix These non-zero values ​​are retained in the corresponding positions of to form a fine-tuning matrix

[0031] Step 3.5: Calculate the loss L of the downstream task of the DK_Transformer model main :

[0032]

[0033] Among them, the downstream task can be a binary classification task, the loss function is the binary cross entropy, N is the total number of samples of text data, y i is the label of the i-th text data, is the prediction result of the i-th text data;

[0034] Step 3.6. Based on the results of steps 3.4 and 3.5, comprehensively calculate the total loss function L of the DK_Transformer model:

[0035]

[0036] Among them, the regularization coefficient λ is used to balance the loss L of the downstream task main and the loss L for graph topology learning reg ;

[0037] Step 3.7, minimize the total loss function L, update the network parameters until the maximum number of iterations T is reached, and obtain the trained DK_Transformer model;

[0038] Step 3.8: Input a test text data testd into the trained DK_Transformer model and predict the adjacency matrix of the output graph topology. and the results of downstream tasks

[0039] On the other hand, a graph topology learning system based on data statistics and knowledge guidance includes:

[0040] The module for constructing the adjacency matrix of the initial graph topology is used to receive the input text data, obtain the entities in the text data as nodes of the graph topology, calculate the conditional probability between the nodes according to the historical text data, and construct the adjacency matrix of the initial graph topology;

[0041] Construct a knowledge attention network module to construct a knowledge attention network for each text data based on the domain knowledge graph;

[0042] The graph topology learning module is used to input the adjacency matrix of the initial graph topology into the DK_Transformer model, and use the knowledge attention network as expert knowledge and the joint attention mechanism to learn the graph topology that adapts to downstream tasks.

[0043] It can be seen from the above description of the present invention that, compared with the prior art, the present invention has the following beneficial effects:

[0044] The present invention uses data statistical information and domain knowledge guidance to learn a corresponding graph topology from a text data of an actual scenario. On the one hand, the graph topology can convey important insights about the actual scenario data and clearly reflect the structured correlation information of the data; on the other hand, using graph topology for machine learning can significantly improve the prediction performance of downstream tasks. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 A flowchart of a graph topology learning method based on data statistics and knowledge guidance of the present invention;

[0046] Figure 2 This is a research framework diagram of graph topology learning based on data statistics and knowledge guidance of the present invention;

[0047] Figure 3 It is a pseudo code diagram of the DK_Transformer algorithm of the present invention;

[0048] Figure 4 It is a DK_Transformer model framework diagram of the present invention;

[0049] Figure 5 It is a visualization diagram of graph topology learning of the present invention;

[0050] Figure 6 It is a structural block diagram of the graph topology learning system based on data statistics and knowledge guidance of the present invention. DETAILED DESCRIPTION

[0051] The present invention will be further described below in conjunction with specific embodiments. It should be understood that these embodiments are only used to illustrate the present invention and are not intended to limit the scope of the present invention. In addition, it should be understood that after reading the content taught by the present invention, those skilled in the art can make various changes or modifications to the present invention, and these equivalent forms fall within the scope limited by the appended claims of the application equally.

[0052] See also Figure 1 and Figure 2 The graph topology learning method based on data statistics and knowledge guidance of the present invention comprises the following steps:

[0053] Step 1: Receive input text data, obtain entities in the text data as nodes of the graph topology, calculate the conditional probabilities between nodes based on historical text data, and construct an adjacency matrix of the initial graph topology;

[0054] Step 2: Build a knowledge attention network for each text data based on the domain knowledge graph;

[0055] Step 3: Input the adjacency matrix of the initial graph topology into the DK_Transformer model, use the knowledge attention network as expert knowledge, and combine the attention mechanism to learn a graph topology that adapts to downstream tasks.

[0056] For details, see the pseudo code diagram of DK_Transformer algorithm and the framework diagram of DK_Transformer model. Figure 3 and Figure 4 shown.

[0057] Taking the disease prediction task of pediatric Chinese electronic medical records as an example, the goal is to learn a graph topology that meets the requirements from each electronic medical record data, and predict whether the patient is diagnosed with acute upper respiratory tract infection based on the graph topology. This is a binary classification task, and the sample size N = 144,170 electronic medical record data sets were collected. The Jieba word segmentation tool was used to segment the electronic medical record text, and stop words and other words without medical meaning were removed from the segmented results and used as nodes of the graph topology.

[0058] The step 1 specifically includes:

[0059] Step 1.1: Input the electronic medical record text D = {d1, d2, …, d 144170}, get the entity set E = {e1, e2, …, e num}, and take it as the node set V = {v1,v2,…,v |V|}, where num is the number of entity sets, |V| is the number of nodes in each graph topology, where |V| = 150. If num = |V|, all entities are used as nodes of the graph topology; if num>|V|, entities with node indexes greater than |V| are deleted; if num<|V|, empty nodes with node indexes greater than num but less than or equal to |V| are filled with special codes;

[0060] Step 1.2: Based on the node set V of the graph topology determined in step 1.1, calculate any two nodes v based on all electronic medical record data D. i and v j The conditional probability of (i,j=1,2,..,|V|) is used as the edge weight a ij , that is, v j In vi The probability of occurrence under the condition of occurrence p(v j |v i ), and its calculation formula is:

[0061]

[0062] Obtain the adjacency matrix P based on conditional probability, expressed as:

[0063]

[0064] Step 1.3: Normalize the adjacency matrix P obtained in step 1.2 to obtain the normalized adjacency matrix

[0065] The step 2 specifically includes:

[0066] Step 2: According to the node set V of the graph topology determined in step 1, the connection relationship of these nodes in the medical knowledge graph CMeKG is obtained, so as to construct a knowledge attention network M for each electronic medical record d. The calculation method is as follows:

[0067]

[0068] Where u is v i and v j The number of common ancestors, l is v i to v j The number of paths.

[0069] The step 3 specifically includes:

[0070] Step 3.1: embed the node set V of the graph topology determined in step 1 into vectors, where the vectors of empty nodes are set to 0, and obtain the node initial feature matrix X with a vector dimension of 128. (0) ;

[0071] Step 3.2: The results of step 1 and step 3.1, that is, the normalized adjacency matrix of the graph topology and the initial feature matrix X of the node (0) Input into the first block module of the DK_Transformer model (without using the attention matrix) for learning;

[0072] Step 3.3: Send the output of step 3.2 to the n block modules of the DK_Transformer model (except the first block module), set n=4; at the same time, use the knowledge attention network M obtained in step 2 as expert knowledge, and combine the attention matrix S of the DK_Transformer model to guide the graph topology learning, and finally obtain the normalized adjacency matrix of the learned graph topology.

[0073] When n=1,

[0074] When n>1,

[0075] Among them, softmax is a normalized exponential function, is the attention matrix S of the nth block module of the DK_Transformer model; and They represent the query vector and key vector of the nth block module of DK_Transformer respectively; and They represent the learnable matrix parameters in the nth block module of DK_Transformer, X (n-1) is the feature matrix of the nodes in the n-1th block module; d is The dimension of the column;

[0076] Step 3.4: Calculate the loss L of the DK_Transformer model in graph topology learning reg :

[0077] When n=1,

[0078] When n>1,

[0079] Among them, D KL is the KL divergence, The matrix is ​​the normalized adjacency matrix output by the nth block module of DK_Transformer The fine-tuning matrix of . That is, the position index of the non-zero value in the label knowledge attention network M, and in the normalized adjacency matrix These non-zero values ​​are retained in the corresponding positions of to form a fine-tuning matrix

[0080] Step 3.5: Calculate the loss L of the downstream task of the DK_Transformer model main :

[0081]

[0082] Among them, the downstream task is a binary classification risk prediction task, the loss function is the binary classification cross entropy, N is the total number of samples of electronic medical records, y i is the category to which the i-th electronic medical record sample belongs, is the predicted value of the i-th electronic medical record sample.

[0083] Step 3.6. Based on the results of steps 3.4 and 3.5, comprehensively calculate the total loss function L of the DK_Transformer model:

[0084]

[0085] Among them, the regularization coefficient λ is used to balance the loss L of the downstream task main and the loss L for graph topology learning reg , where λ = 0.1.

[0086] Step 3.7, minimize the total loss function L (Formula 7), update the network parameters until the maximum number of iterations T = 1000 is reached, and obtain the trained DK_Transformer model. In the experiment, the Chinese electronic medical record dataset was randomly divided into training set, validation set and test set in a ratio of 8:1:1. The model was trained on the training set using the Adam optimization method, with the learning rate set to 0.0001, dropout set to 0.5, and minibatch set to 64;

[0087] Step 3.8: Input a test electronic medical record data testd into the trained DK_Transformer model to predict the adjacency matrix of the output graph topology. and the results of downstream tasks

[0088] In order to verify the effectiveness of the DK_Transformer model, three types of baseline models were used: the first type is the CNN (Convolutional Neural Network) and LSTM (Long Short-Term Memory) non-graph algorithms based on text data, the second type is the existing graph algorithms GCT (Graph Convolutional Transformer) and GCN (Graph Convolutional Network) based on graph data, and the third type is the models involved in the DK_Transformer ablation experiment, including Transformer, Transformer (P,) 、Transformer (,M) The detailed description of all baseline models is shown in Table 1. The experimental comparison results of the DK_Transformer model and the baseline model are shown in Table 2. AUROC (Area Under the Receiver Operating Characteristic Curve) is used as the evaluation indicator to evaluate the performance of the model.

[0089] Table 1 Baseline model settings

[0090]

[0091] Table 2 Performance of the model in the disease prediction task of the Chinese electronic medical record dataset

[0092]

[0093]

[0094] In addition, the echar technology is used to visualize the graph topology. Figure 5 The graph topology of a Chinese electronic medical record data (preliminarily diagnosed as acute upper respiratory tract infection) after graph topology learning by the DK_Transformer model is shown.

[0095] At this point, the graph topology learning method based on data statistics and knowledge guidance has been completed. It is not difficult to find that the present invention can convert text data in actual scenarios into structured graph topology, and use graph topology for machine learning to effectively improve the performance of downstream tasks.

[0096] See also Figure 6 As shown, the present invention provides a graph topology learning system based on data statistics and knowledge guidance, including:

[0097] The module 601 for constructing an adjacency matrix of an initial graph topology is used to receive input text data, obtain entities in the text data as nodes of the graph topology, calculate conditional probabilities between nodes according to historical text data, and construct an adjacency matrix of the initial graph topology;

[0098] A knowledge attention network construction module 602 is used to construct a knowledge attention network for each text data based on the domain knowledge graph;

[0099] The graph topology learning module 603 is used to input the adjacency matrix of the initial graph topology into the DK_Transformer model, and use the knowledge attention network as expert knowledge and the joint attention mechanism to learn the graph topology that is suitable for downstream tasks.

[0100] A specific implementation of a graph topology learning system based on data statistics and knowledge guidance. The same graph topology learning method based on data statistics and knowledge guidance will not be repeated in this embodiment.

[0101] The above is only a specific implementation of the present invention, but the design concept of the present invention is not limited thereto. Any non-substantial changes to the present invention using this concept shall be deemed as an infringement of the protection scope of the present invention.

Claims

1. A graph topology learning method based on data statistics and knowledge guidance, characterized in that: The steps include: Step 1: Receive input text data, obtain entities in the text data as nodes of the graph topology, calculate the conditional probabilities between nodes based on historical text data, and construct an adjacency matrix of the initial graph topology; Step 2: Build a knowledge attention network for each text data based on the domain knowledge graph; Step 3: Input the adjacency matrix of the initial graph topology into the DK_Transformer model, and use the knowledge attention network as expert knowledge to learn the graph topology that adapts to the downstream tasks through the joint attention mechanism; The step 1 specifically includes: Step 1.1: Receive text data D = {d1, d2, ..., d N }, get the entity set E = {e1, e2, ..., e num }, and take it as the node set V = {v1, v2, ..., v |V| }, where num is the number of entity sets, and |V| is the number of nodes in each graph topology; if num=|V|, all entities are taken as nodes of the graph topology; if num>|V|, entities with node indexes greater than |V| are deleted; if num<|V|, empty nodes with node indexes greater than num but less than or equal to |V| are filled with special codes; Step 1.2: Based on the node set V of the graph topology determined in step 1.1, calculate any two nodes v based on all text data D. i and v j The conditional probability of (i, j = 1, 2, .., |V|) is used as the edge weight a ij , that is, v j In v i The probability of occurrence under the condition of occurrence p(v j |v i ), and its calculation formula is: Obtain the adjacency matrix P based on conditional probability, expressed as: Step 1.3: Normalize the adjacency matrix P obtained in step 1.2 to obtain the normalized adjacency matrix The step 2 specifically includes: According to the node set V of the graph topology determined in step 1, the connection relationship of these nodes in the knowledge graph is obtained from the domain knowledge graph, so as to build a knowledge attention network M for each text data d. The calculation method is as follows: Where u is v i and v j The number of common ancestors, l is v i to v j The number of paths.

2. The graph topology learning method based on data statistics and knowledge guidance according to claim 1 is characterized in that: The step 3 specifically includes: Step 3.1: embed the node set V of the graph topology determined in step 1 into vectors, where the vectors of empty nodes are set to 0, and obtain the initial feature matrix X of the nodes in the graph. (0) ; Step 3.2: The results of step 1 and step 3.1, that is, the normalized adjacency matrix of the graph topology and the initial feature matrix X of the nodes in the graph (0) Input into the first block module of the DK_Transformer model for learning; Step 3.3: Feed the output of step 3.2 into the n block modules of the DK_Transformer model, use the knowledge attention network M obtained in step 2 as expert knowledge, and combine the attention matrix S of the DK_Transformer model to guide graph topology learning, and finally obtain the normalized adjacency matrix of the learned graph topology. When n=1, When n>1, Among them, softmax is a normalized exponential function, is the attention matrix S of the nth block module of the DK_Transformer model; and They represent the query vector and key vector of the nth block module of DK_Transformer respectively; and They represent the learnable matrix parameters in the nth block module of DK_Transformer, X (n-1) is the feature matrix of the nodes in the n-1th block module; d is The dimension of the column; Step 3.4: Calculate the loss L of the DK_Transformer model in graph topology learning reg : When n=1, When n>1, Among them, D KL is the KL divergence, The matrix is ​​the normalized adjacency matrix output by the nth block module of DK_Transformer The fine-tuning matrix of M is the position index of the non-zero value in the knowledge attention network M, and the normalized adjacency matrix These non-zero values ​​are retained in the corresponding positions of to form a fine-tuning matrix Step 3.5: Calculate the loss L of the downstream task of the DK_Transformer model main : Among them, the downstream task can be a binary classification task, the loss function is the binary cross entropy, N is the total number of samples of text data, y i is the label of the i-th text data, is the prediction result of the i-th text data; Step 3.

6. Based on the results of steps 3.4 and 3.5, comprehensively calculate the total loss function L of the DK_Transformer model: Among them, the regularization coefficient λ is used to balance the loss L of the downstream task main and the loss L for graph topology learning reg ; Step 3.7, minimize the total loss function L, update the network parameters until the maximum number of iterations T is reached, and obtain the trained DK_Transformer model; Step 3.8: Input a test text data testd into the trained DK_Transformer model and predict the adjacency matrix of the output graph topology. and the results of downstream tasks 3. A graph topology learning system based on data statistics and knowledge guidance, characterized in that: include: The module for constructing the adjacency matrix of the initial graph topology is used to receive the input text data, obtain the entities in the text data as nodes of the graph topology, calculate the conditional probability between the nodes according to the historical text data, and construct the adjacency matrix of the initial graph topology; Construct a knowledge attention network module to construct a knowledge attention network for each text data based on the domain knowledge graph; The graph topology learning module is used to input the adjacency matrix of the initial graph topology into the DK_Transformer model, and use the knowledge attention network as expert knowledge and the joint attention mechanism to learn the graph topology that adapts to the downstream tasks; The adjacency matrix module for constructing the initial graph topology is specifically implemented as follows: Step 1.1: Receive text data D = {d1, d2, ..., d N }, get the entity set E = {e1, e2, …, e num }, and take it as the node set V = {v1,v2,…,v |V| }, where num is the number of entity sets, and |V| is the number of nodes in each graph topology; if num=|V|, all entities are taken as nodes of the graph topology; if num>|V|, entities with node indexes greater than |V| are deleted; if num<|V|, empty nodes with node indexes greater than num but less than or equal to |V| are filled with special codes; Step 1.2: Based on the node set V of the graph topology determined in step 1.1, calculate any two nodes v based on all text data D. i and v j The conditional probability of (i,j=1,2,..,|V|) is used as the edge weight a ij , that is, v j In v i The probability of occurrence under the condition of occurrence p(v j |v i ), and its calculation formula is: Obtain the adjacency matrix P based on conditional probability, expressed as: Step 1.3: Normalize the adjacency matrix P obtained in step 1.2 to obtain the normalized adjacency matrix The knowledge attention network module is specifically implemented as follows: According to the node set V of the graph topology determined by the adjacency matrix module of the initial graph topology, the connection relationship of these nodes in the knowledge graph is obtained from the domain knowledge graph, so as to construct a knowledge attention network M for each text data d. The calculation method is as follows: Where u is v i and v j The number of common ancestors, l is v i to v j The number of paths.

Images