A joint information extraction method based on spatially aware dependency graph and lossless decoding

Abstract

The present invention provides a joint information extraction method based on space-aware dependency graph and lossless decoding, comprising: step 1, encoding an input text sequence using a pre-trained language model; step 2, identifying entities and event triggers; step 3, adding location information to the entity and event trigger representation; step 4, constructing the information extraction task as a graph prediction model, with entities and event triggers as nodes of the graph, and relationships and event arguments as edges of the graph; step 5, calculating the joint probability distribution of nodes and edges; step 6, obtaining labels of nodes and edges; step 7, inputting text, and outputting the entity, relationship, event trigger, and event argument categories contained in the text. The present invention improves the existing local decoding strategy, adopts stochastic gradient variational inference to avoid the loss of dependency between labels of the same task during decoding, so as to achieve better global optimal allocation of instance labels. Effectively improve the accuracy and robustness of joint information extraction.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a joint information extraction method based on space-aware dependency graph and lossless decoding. Background Art

[0002] Traditional information extraction methods usually process named entity recognition, relation extraction and other subtasks separately. This pipeline processing method has the problems of error accumulation and information fragmentation. For example, in the relation extraction task, if the entity recognition is wrong, it will directly affect the result of relation extraction.

[0003] In order to solve these problems, joint information extraction methods have emerged in recent years. Different from the traditional pipeline method, joint information extraction integrates multiple subtasks into a unified framework and performs joint modeling and optimization. This method can make full use of the correlation information between different subtasks, thereby effectively improving the overall performance of information extraction. Previous information extraction did not take into account the temporal information of the text and adopted a decoding method that lost task dependencies. Summary of the invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide a joint information extraction method based on spatial perception dependency graph and lossless decoding in view of the shortcomings of the prior art, comprising the following steps:

[0005] Step 1: Encode the input text sequence using a pre-trained language model;

[0006] Step 2: Use conditional random fields to annotate the start, internal, and external BIOs of the encoded sequence to identify entities and event triggers;

[0007] Step 3, using a continuous dynamic location information encoding method, adding location information to the entity and event trigger representations identified in step 2;

[0008] Step 4: construct the information extraction task as a graph prediction model, with entities and event triggers as nodes of the graph, and relationships and event arguments as edges of the graph;

[0009] Step 5, using the scoring factor, calculate the joint probability distribution of nodes and edges;

[0010] Step 6: Use stochastic gradient variational inference on the joint probability distribution in step 5 to obtain the labels of nodes and edges.

[0011] Step 7: Use the trained model to input text and output the entities, relations, event triggers, and event argument categories contained in the text.

[0012] Step 1 includes: converting a text sequence of length n into , which is input into the pre-trained BERT model for encoding to obtain the sequence The operation is recorded as :

[0013] ,

[0014] Get the sequence containing the context meaning ,in represents the nth intermediate parameter, Indicates the nth word. Next, we can get the entity and event trigger through the sequence H.

[0015] Since different BERT models are trained using different corpora, such as BERT-Base-Chinese for Chinese and BERT-Base-Uncased for English. Choosing different pre-trained language models will have an impact on the accuracy of solution recognition. In addition, some BERT models are fine-tuned for specific tasks, such as BERT-for-Sequence-Classification for sentence classification and BERT-for-Question-Answering for question-answering systems. Choosing a fine-tuned model that matches the target task can achieve better performance. Pre-trained language models have a limit on the length of the input text, usually 512 tokens. Text that exceeds the length limit needs to be truncated or segmented.

[0016] Furthermore, in step 2, in order to obtain the dependency relationship between task instances, the present invention uses a method of enumerating all possible spans to form graph nodes of the dependency graph. However, this method will not only retain many non-entity and event trigger word text fragments, generating noise that affects the performance of the model, but also cause unacceptable computational complexity. For this reason, the present invention uses entity conditional random fields and event trigger conditional random fields to represent the input text. Specifically, the sequence encoded by the pre-trained language model is input into two feed-forward neural networks to obtain the scores of the sequence belonging to entities and event triggers, and then input into the entity conditional random field layer and the event trigger conditional random field layer to calculate the two distributions of event trigger words and entity labels, and then the Viterbi algorithm is used to determine The optimal tag sequence for events and entity mentions in the text. If a word is divided into multiple word fragments, the vector representations of all fragments are averaged as its word representation. In order to fully utilize the dependencies between tasks, the recognition module only provides spans marked as B and I as graph nodes, but does not provide the predicted entity labels or trigger types of these nodes. Since the recognized instances will directly affect the correctness of subsequent classification work, the present invention trains the recognition module separately to avoid the propagation of errors. Step 2 specifically includes: using two linear chain conditional random fields to perform BIO annotation for each word in the text sequence, firstly, the sequence The two feedforward neural networks FFN have the same structure. One feedforward neural network FFN is used to extract entities, and the other feedforward neural network FFN is used to extract event triggers. The representation of the task of extracting entities and event triggers is obtained:

[0017] Represents the hidden layer representation in the entity extraction task:

[0018] ,

[0019] Represents the hidden layer representation in the task of extracting event triggers:

[0020] ,

[0021] Will and Input into the conditional random field for BIO sequence labeling, first the transfer matrix of the conditional random field Initialize, constrain the transfer matrix during initialization, set For label arrive The transfer fraction of , the constraint is expressed as:

[0022] It is not possible to transfer from any label to the terminal label end, which is represented as:

[0023] ,

[0024] Indicates the transfer score from any label to the terminal label end;

[0025] The start tag start cannot be transferred to any non-start tag, which is represented as:

[0026] ,

[0027] Indicates the transfer score from the starting label start to any label;

[0028] For invalid transfers, e.g. Tags cannot be directly connected to The label, assigned a low score, is expressed as:

[0029] ,

[0030] Using forward variables Represents the sequence at time step The status on the label The path score at the time of All possible paths in time steps, this variable is derived from the time step by the recursive formula Calculate step by step to the time step , let the hidden layer in the entity extraction task and the hidden layers in the task of extracting event triggers is the current time step The linear layer output on represents the score of each label at the current time step, in order to update the forward variable , we need to consider the scores of all labels transferred from the previous time step to the labels of the current time step. The formula for calculating the path score is:

[0031] ,

[0032] ,

[0033] in Indicates that the state at the t+1th time step is the label The path score of the entity, Indicates that the state at the t+1th time step is the label The path score of the event trigger, exp function represents the exponential function with the natural constant e as the base; Used to prevent overflow and efficiently calculate the sum under logarithm;

[0034] After traversing all time steps, we need to consider the transfer score from the last label to the end label end and convert the forward variable The score of the last time step in is added to the transition score of the terminal label end:

[0035] ,

[0036] in Indicates the path score of the state at the tth time step with the termination label end, Representation Tags The fraction of transitions to the termination label end.

[0037] In order to calculate the subsequent log-likelihood and training loss, the score and the normalization constant are calculated for all possible label sequences. :

[0038] ,

[0039] In order to predict the optimal label sequence, the Viterbi algorithm is used. The goal of Viterbi decoding is to find the path with the highest score, that is, to maximize:

[0040] ,

[0041] ,

[0042] in Representative sentences The optimal prediction score of the entity, Representative sentences The optimal prediction score of the event trigger, Expressing sentences The tth word of Expressing sentences The hidden layer representation of the t-th word in the entity extraction task, Expressing sentences The hidden layer representation of the t-th word in the task of extracting event triggers, argmax means maximization;

[0043] The maximum score path is calculated through dynamic programming, the optimal previous state is selected at each step, and the optimal label at each time step is gradually backtracked to obtain the entities and event triggers in the text sequence.

[0044] Step 3 includes: Although the conditional random field can use the feature function to implicitly capture some position information in the sequence when identifying the node, this information will not be explicitly used in the entire process, especially when the node is constructed as a graph. This is due to the characteristics of the undirected graph itself. In order for the model to utilize the order and context information of the sequence, some information about the relative or absolute position of the word in the sequence must be injected. Through the position information of the word in the sequence, the global context information is captured and the flexibility of retaining the sequence order is retained. The continuous dynamic model encodes the position information by treating it as part of a continuous dynamic system, which means that the position information is not static, but a continuous dynamic system is used to generate the position encoding. ,in Represents the position encoding of the nth word, the evolution function of the continuous dynamic system It is the key function that describes how the hidden state changes with the sequence position i. It defines the rate of change of the continuous-time dynamic system at any sequence position and determines how the hidden state changes from the initial state to the Evolving to subsequent positions The state of the continuous dynamic system evolution function The formula is:

[0045] ,

[0046] in represents the subsequent position point, is the parameter of the evolution function, W is a learnable weight matrix, Represents the sigmoid function;

[0047] The position encoding of the continuous dynamic model is based on differential equations to model the position information. The formula of the differential equation is:

[0048] ,

[0049] in represents the differential symbol;

[0050] Through the continuous dynamic model, the position encoding can produce continuous changes according to the characteristics of the sequence, rather than relying solely on a fixed formula. The learned differential equation affects the evolution of the position encoding through the following formula:

[0051] ,

[0052] in is the change in sequence position, Representative Positional encoding of a sequence;

[0053] According to the evolution formula, the code of each position is gradually updated to obtain the final position code sequence. .

[0054] This dynamic and learnable encoding method can not only capture the position information in the sequence, but also learn more complex sequence dependencies, such as long-range dependencies.

[0055] Step 4 includes: given a sentence, representing the information extraction task of the sentence as a dependency factor graph, where nodes correspond to entities and event trigger words, and edges represent relations and event arguments;

[0056] Based on the dependency factor graph, the following three information extraction tasks are implemented:

[0057] Named Entity Recognition: Identify specific types of entities (such as names of people, places, organizations, etc.) from text; entities can be any object with real-world meaning;

[0058] Relation extraction: Capture the relationship between entity pairs in the text and mark the relationship type;

[0059] Event extraction: Identify and extract events and related information from text; an event usually consists of a trigger word and its related arguments;

[0060] Perform named entity recognition, relation extraction, and event extraction in a joint information extraction model: Given an input sentence, formalized as a graph Prediction tasks, where represents a node set, Represents a directed edge set, each node Indicates an entity mention or event trigger, and Respectively represent the start and end positions of the span, is the tag of the span, for entity tags , for event trigger tags , Represents a set of entity tags, Represents a set of event trigger labels; each edge represents the relationship label from the i-th graph node to the j-th graph node, .

[0061] Step 5 includes: In order to solve the problem that the traditional information extraction algorithm based on graph prediction cannot perceive the sequence position information, a graph prediction information extraction algorithm based on space-aware dependency graph is proposed, and the position information of the text sequence is added to the dependency graph to better integrate the correlation across instances and tasks, and the score of each node label and edge label is calculated accordingly. By constructing a factor graph and designing three different scoring factors to simulate the local distribution of each label, the correlation between instances of the same task, and the correlation between instances of different tasks, the three different scoring factors specifically include unary factors, binary factors and ternary factors:

[0062] Unary factor: average the representations of all words in the prediction span and add them to the positional encoding to obtain the representation z of each node, and input the representation of each graph node into the feedforward neural network FFN of different tasks at the same time to obtain an independent label score for each graph node:

[0063] ,

[0064] ,

[0065] in Representative The first graph node word fragments, avg means taking the average value, Represents different graph node tasks, Feedforward neural network FFN representing different graph node tasks, Represents the unary score of the i-th graph node under different graph node tasks;

[0066] For edge , using two different feedforward neural networks FFN to simulate the influence of nodes as subjects and objects on the relationship, and using the biaffine function to model the interaction of different feedforward neural networks FFN:

[0067] ,

[0068] in Represents different side tasks, represents element-wise product, is a randomly initialized learnable parameter matrix, Representative side Unary scores under different side tasks, The feedforward neural network FFN represents different node tasks when the graph node is used as the head node. Feedforward neural network FFN representing different node tasks when graph nodes are used as tail nodes;

[0069] Binary factors: Binary factors are used to capture the dependencies of edges that share the same node, using different feed-forward neural networks (FFNs) to calculate the representation of each node as a shared joint node and an independent sole node:

[0070] ,

[0071] ,

[0072] in Represents the representation of the i-th graph node as an independent sole node, Represents the i-th graph node as a representation of a shared joint node;

[0073] For two edges that share the i-th graph node, the binary factor of the edge is represented as:

[0074] ,

[0075] in represents the label of the first edge, is the first edge label Embedded, represents the label of the second edge, is the second edge label For symmetry, let , which means that for the edge that shares the i-th graph node and edge ,by Binary factor score With Binary factor score same;

[0076] Ternary Factor: Ternary factor is used to model the mutual dependence between the two endpoints of an edge, using different feed-forward neural networks FFN to generate representations for the head node and the tail node:

[0077] ,

[0078] ,

[0079] in Represents the representation of a graph node as a head node. Represents the representation of a graph node as a tail node;

[0080] The representation of the ternary factor is:

[0081] ,

[0082] in represents the ternary score of the ith graph node and the jth graph node and the edge connecting the ith graph node and the jth graph node, represents the embedding of the edge label between the i-th graph node and the j-th graph node belonging to category m, Represents the head node The label belongs to category p, Represents the head node The label belongs to category q, is the label of the head node of the ternary factor Embedded, is the tail node label of the ternary factor Embedding

[0083] Aggregate the three scoring factors to obtain the joint probability distribution of all variables. The joint probability distribution of each graph node is calculated as:

[0084] ,

[0085] ,

[0086] in represents the probability that the i-th graph node label belongs to category p when the model parameter is w, represents the probability that the jth graph node label belongs to category q when the model parameter is w, Represents a univariate scoring factor No. elements, Represents a univariate scoring factor No. elements, Represents a univariate scoring factor No. elements, It is the weight parameter used to adjust the contribution of each factor;

[0087] For each edge, the joint probability distribution is calculated as:

[0088] ,

[0089] in When the model parameter is w, the label of the edge between the i-th graph node and the j-th graph node is The probability of It is the weight parameter used to adjust the contribution of each factor.

[0090] Step 6 includes: for different types of factors, simulating the correlation between the same task instances and the correlation between different task instances to obtain the joint probability distribution of all variables. It is usually not feasible to directly and accurately infer the labels of the variables because all visible variables need to be integrated, which is computationally expensive in high-dimensional cases. Variational inference can simplify the calculation by introducing a controllable approximate distribution. Although mean field variational inference is a simple and commonly used variational inference method, it has some limitations when dealing with complex problems. It is different from the previous use of mean field theory to simplify the approximate distribution by making assumptions about the independence of variable labels. , the present invention uses stochastic gradient variational inference to continue to maintain the complex dependencies between latent variables and iteratively infer each variable Approximate posterior distribution , let the true posterior distribution be , by reducing distance to optimize the approximate posterior distribution :

[0091] ,

[0092] in represents the approximate posterior distribution after iteration, KL represents KL divergence, and the evidence lower bound The calculation formula is:

[0093] ,

[0094] Then by calculating The gradient of , and use gradient descent to continuously update the posterior distribution The parameters of , the model loss is calculated by forward propagation, and the final joint information extraction model is obtained after a predetermined number of back propagation iterations. It is worth noting that directly calculating The gradient of will cause high variance problem. In order to reduce the variance and make SGVI effective, the present invention uses the reparameterization technique. In short, the present invention obtains The parameters of , transfer the randomness to this simple distribution to avoid direct calculation The present invention adopts Dirichlet distribution as the approximate posterior distribution of the model. , the parameters of the Dirichlet distribution are initialized to the exponential form of the normalized univariate fraction and optimized during backpropagation. The performance of stochastic gradient variational inference is sensitive to the choice of hyperparameters such as the learning rate and needs to be carefully tuned.

[0095] In step 7, the joint information extraction model obtained in step 6 is used to input data and output entities, event triggers, relations, and event arguments;

[0096] After the information extraction model is trained, the text sequence is input into the model, and the model will extract the information contained in it according to the training content to achieve the desired effect.

[0097] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores program code, and when the program code is executed by the processor, the processor executes the steps of the described method.

[0098] The present invention also provides a storage medium storing a computer program or instruction. When the computer program or instruction is run on a computer, the steps of the method described are executed.

[0099] This paper proposes a classification model based on graph prediction. The core idea is to use a factor graph that combines sequence position information to capture cross-task and cross-instance dependency information, and use stochastic gradient variational inference to obtain the classification results, thereby realizing joint information extraction.

[0100] The present invention has the following beneficial effects: 1. The present invention designs a dependency graph structure that can perceive sequence position information, and integrates the sequence position information into the representation learning of the joint information extraction task to enhance the model's ability to capture cross-instance and cross-task dependencies.

[0101] 2. This paper improves the existing local decoding strategy and adopts stochastic gradient variational inference to avoid the loss of dependency between labels of the same task during the decoding process, so as to achieve a better global optimal allocation of instance labels and effectively improve the accuracy and robustness of joint information extraction. BRIEF DESCRIPTION OF THE DRAWINGS

[0102] Figure 1 It is a flow chart of the overall working steps of the present invention. DETAILED DESCRIPTION

[0103] The present invention will be further described in detail below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more clear.

[0104] like Figure 1 As shown, an embodiment of the present invention discloses a joint information extraction method based on a spatially-aware dependency graph and lossless decoding, comprising the following steps:

[0105] Step 1: Encode the input text sequence using a pre-trained language model.

[0106] Step 2: Use conditional random fields to perform start, internal, and external BIO annotations on the encoded sequence to identify entities and event triggers; BIO annotation is a commonly used natural language processing (NLP) annotation method, mainly used for tasks such as named entity recognition. The principle of BIO annotation is to annotate each character of a word (including the starting character, middle character, and end character of the word) as one of the three types: B, I, and O. Among them, B means that the character is the starting character of the word, I means that the character is the middle character of the word, and O means that the character is not the starting or middle character of any word;

[0107] Step 3: Use the continuous dynamic location information encoding method to add location information to the entity and event trigger representations identified in step 2.

[0108] Step 4: Construct the information extraction task as a graph prediction model, with entities and event triggers as nodes, and relationships and event arguments as edges.

[0109] Step 5: Using the three scoring factors, calculate the joint probability distribution of nodes and edges.

[0110] Step 6: Use stochastic gradient variational inference on the joint probability distribution in step 5 to obtain node and edge labels.

[0111] Step 7: Use the trained model to input text and output the entities, relations, event triggers, and event argument categories contained in the text.

[0112] Step 1 includes: Since different BERT models are trained using different corpora, such as BERT-Base-Chinese for Chinese and BERT-Base-Uncased for English. Choosing different pre-trained language models will have an impact on the accuracy of solution recognition. In addition, some BERT models are fine-tuned for specific tasks, such as BERT-for-Sequence-Classification for sentence classification and BERT-for-Question-Answering for question-answering systems. Choosing a fine-tuned model that matches the target task can achieve better performance. The pre-trained language model has a limit on the length of the input text, usually 512 tokens. Text that exceeds the length limit needs to be truncated or segmented.

[0113] Step 2 includes: In order to obtain the dependency relationship between task instances, the present invention uses a method of enumerating all possible spans to form graph nodes of the dependency graph. However, this method will not only retain many non-entity and event trigger word text fragments, generating noise that affects the performance of the model, but also cause unacceptable computational complexity. For this reason, the present invention uses entity conditional random fields and event trigger conditional random fields to represent the input text. Specifically, the sequence encoded by the pre-trained language model is input into two feed-forward neural networks to obtain the scores of the sequence belonging to entities and event triggers, and then input into the entity conditional random field layer and the event trigger conditional random field layer to calculate the two distributions of event trigger words and entity labels, and then the Viterbi algorithm is used to determine The optimal tag sequence for events and entity mentions in . If a word is divided into multiple word fragments, the vector representations of all fragments are averaged as its word representation. In order to fully utilize the dependencies between tasks, the recognition module only provides spans labeled B and I as graph nodes, without providing the predicted entity labels or trigger types of these nodes. Since the recognized instances will directly affect the correctness of subsequent classification work, we train the recognition module separately to avoid error propagation.

[0114] Step 3 includes: Although the conditional random field can use the feature function to implicitly capture some position information in the sequence when identifying nodes, this information will not be explicitly used in the entire process, especially when the nodes are constructed as a graph. This is due to the characteristics of the undirected graph itself. In order for the model to utilize the order and context information of the sequence, we must inject some information about the relative or absolute position of the word in the sequence. Due to the uncertainty of the input sequence length and the number of instances it contains, we adopt the position information encoding method of the continuous dynamic model, which can generate more flexible dynamic position encoding by simulating the time evolution process. Position encoding is a vector sequence that can be added to the input representation. Unlike traditional sine / cosine encoding that generates a fixed code for each position pos, we model the position information based on differential equations. By considering the position information as the time evolution process of a dynamic system, a series of continuous vectors that change over time are generated to represent the dynamic characteristics of each position in the input sequence. The core of the position encoding of the continuous dynamic model is to model the position information based on differential equations. The formula of the differential equation is as follows:

[0115] ,

[0116] Through this dynamic model, the position encoding can produce continuous changes according to the characteristics of the sequence, rather than relying solely on a fixed formula. The learned differential equation affects the evolution of the position encoding through the following formula:

[0117] ,

[0118] This dynamic and learnable encoding method can not only capture the position information in the sequence, but also learn more complex sequence dependencies, such as long-range dependencies.

[0119] Step 4 includes: Given an input text sequence, the present invention aims to represent the information extraction task of the sentence as a dependency factor graph, where nodes correspond to entity mentions and event triggers, and edges represent relations and event arguments. Given an input sentence, formalized as a graph Prediction tasks, where represents a node set, Represents a directed edge set, each node Indicates an entity mention or event trigger, and Respectively represent the start and end positions of the span, is the tag of the span, for entity tags , for event trigger tags , Represents a set of entity tags, Represents a set of event trigger labels; each edge represents the relationship label from the i-th graph node to the j-th graph node, .

[0120] Step 5 includes: constructing a factor graph and designing three different scoring factors to simulate the local distribution of each label, the correlation between the same task instances, and the correlation between different task instances. The three different scoring factors specifically include a unary factor, a binary factor, and a ternary factor:

[0121] Unary factor: used to simulate the local distribution of each label, average the representation of all words in the prediction span, add it to the position code to obtain the representation z of each node, and input it into different task-specific FFNs to obtain independent label scores for each node:

[0122] ,

[0123] ,

[0124] For the unary factors of edges, the present invention also uses two different FFNs to simulate the influence of nodes as subjects and objects on the relationship, and adopts the biaffine function to model their interaction:

[0125] ,

[0126] Binary factors: Binary factors are used to capture the dependencies of edges that share the same node, using different feed-forward neural networks (FFNs) to calculate the representation of each node as a shared joint node and an independent sole node:

[0127] ,

[0128] ,

[0129] For two edges that share the i-th graph node, the binary factor of the edge is represented as:

[0130] ,

[0131] Ternary Factor: Ternary factor is used to model the mutual dependence between the two endpoints of an edge, using different feed-forward neural networks FFN to generate representations for the head node and the tail node:

[0132] ,

[0133] ,

[0134] The representation of the ternary factor is:

[0135] ,

[0136] The local distribution of each label, the correlation between instances of the same task, and the correlation between instances of different tasks are calculated through three scoring factors. These factors are aggregated to obtain the joint probability distribution of all variables. The joint probability distribution of each node is calculated as follows:

[0137] ,

[0138] ,

[0139] For each edge, the joint probability distribution is calculated as follows:

[0140] ,

[0141] Step 6 includes: for different types of factors, simulating the correlation between the same task instances and the correlation between different task instances to obtain the joint probability distribution of all variables. It is usually not feasible to directly and accurately infer the labels of the variables because all visible variables need to be integrated, which is computationally expensive in high-dimensional cases. Variational inference can simplify the calculation by introducing a controllable approximate distribution. Although mean field variational inference is a simple and commonly used variational inference method, it has some limitations when dealing with complex problems. It is different from the previous use of mean field theory to simplify the approximate distribution by making assumptions about the independence of variable labels. The present invention uses stochastic gradient variational inference to continue to maintain the complex dependencies between latent variables and iteratively infer each variable The approximate posterior distribution of , let the true posterior distribution be , by reducing distance to optimize the approximate posterior distribution :

[0142] ,

[0143] The evidence lower bound The calculation formula is as follows:

[0144] ,

[0145] Then by calculating The gradient of the posterior distribution is continuously updated through back propagation It is worth noting that the direct calculation of The gradient of will cause high variance problem. In order to reduce the variance and make stochastic gradient variational inference effective, the present invention uses the reparameterization technique. In short, the present invention obtains The parameters of , transfer the randomness to this standard normal distribution to avoid direct calculation The present invention adopts Dirichlet distribution as the approximate posterior distribution of the joint information extraction model. , the parameters of the Dirichlet distribution are initialized to the normalized exponential form of the univariate fraction and optimized during the back-propagation process.

[0146] Step 7 includes: after the model training is completed, the text sequence is input into the model, and the model will extract the corresponding entities, event triggers, relations and event arguments according to the training content to achieve the expected effect. In practical applications, the present invention shows significant improvements in multiple tasks on the benchmark data set ACE05 compared to the prior art. Specifically, the F1 score of the entity naming recognition task is improved by 0.8%, the classification accuracy of the relationship argument is improved by 1.3%, and the recognition rate of the event trigger is improved by 3.8%. In addition, the stochastic gradient variational inference decoding module proposed in the present invention has strong versatility and can be directly extended to other complex natural language processing tasks, such as multi-task learning, cross-domain transfer learning, etc., which further enhances the adaptability and extensibility of the model.

[0147] The present invention provides a joint information extraction method based on spatial perception dependency graph and lossless decoding. There are many methods and ways to implement the technical solution. The above is only a preferred implementation of the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principle of the present invention, several improvements and modifications can be made, and these improvements and modifications should also be regarded as the protection scope of the present invention. All components not specified in this embodiment can be implemented by existing technologies.

Claims

1. A joint information extraction method based on spatially aware dependency graph and lossless decoding, characterized in that: The following steps are involved: Step 1: Encode the input text sequence using a pre-trained language model; Step 2: Use conditional random fields to annotate the start, internal, and external BIOs of the encoded sequence to identify entities and event triggers; Step 3, using a continuous dynamic location information encoding method, adding location information to the entity and event trigger representations identified in step 2; Step 3 includes: using a continuous dynamic system to generate a position code PE = {p1, p2, p3, ..., p n }, where p n Representing the position encoding of the nth word, the evolution function f(p(i),i|θ) of the continuous dynamic system is: f(p(i),i|θ)=σ(Wp(i)+t x ), Where p(i) represents the subsequent position point, θ is the parameter of the evolution function, W is a learnable weight matrix, and σ(·) represents the sigmoid function; The position encoding of the continuous dynamic model is based on differential equations to model the position information. The formula of the differential equation is: Where d represents the differential symbol; The learned differential equation affects the evolution of the positional encoding via the following formula: Where Δi is the change in sequence position, and p(i+Δi) represents the position code of the i+Δith sequence; According to the evolution formula, the code of each position is gradually updated to obtain the final position code sequence PE = {p1, p2, p3, ..., p n }; Step 4: construct the information extraction task as a graph prediction model, with entities and event triggers as nodes of the graph, and relationships and event arguments as edges of the graph; Step 5, using the scoring factor, calculate the joint probability distribution of nodes and edges; Step 6: Use stochastic gradient variational inference on the joint probability distribution in step 5 to obtain the labels of nodes and edges. Step 7: Use the trained model to input text and output the entities, relations, event triggers, and event argument categories contained in the text.

2. The method according to claim 1, characterized in that: Step 1 includes: converting a text sequence W of length n into {w1,w2,w 3,…, w n }, and the operation of encoding the sequence H into the pre-trained BERT model is recorded as BERT(w1,w2,w3,...,w n ): H=BERT(w1,w2,w3,...,w n )={h1,h2,h3,...,h n ), Get the sequence H containing the context meaning = {h1,h2,h3,…,h n }, where h n Indicates the nth intermediate parameter, w n Represents the nth word.

3. The method according to claim 2, characterized in that Step 2 includes: using two linear chain conditional random fields to perform BIO annotation for each word in the text sequence, firstly, the sequence H = {h1,h2,h3,…,h n } are respectively input into two feedforward neural networks FFN, the two feedforward neural networks FFN have the same structure, one feedforward neural network FFN is used to extract entities, and the other feedforward neural network FFN is used to extract event triggers, and the representation of the task of extracting entities and event triggers is obtained: H entiiy Represents the hidden layer representation in the entity extraction task: H entity =FFN(H), H ttigger Represents the hidden layer representation in the task of extracting event triggers: H trigger =FFN(H), H entity and H trigger Input into the conditional random field for BIO sequence labeling, first initialize the transfer matrix T of the conditional random field, constrain the transfer matrix during initialization, let T[u,v] be the transfer score from label u to v, and the constraint is expressed as: It is not possible to transfer from any label to the terminal label end, which is represented as: T(:,end)=-100, T(:,end) represents the transition score from any label to the terminal label end; The start tag start cannot be transferred to any non-start tag, which is represented as: T(start,:)=-100, T(start,:) represents the transfer score from the starting label start to any label; For invalid transfers, low scores are given, expressed as: T(u,v)=-100, Use the forward variable α t (v) represents the path score when the state of the sequence at time step t is label v, let the hidden layer in the entity extraction task and the hidden layers in the task of extracting event triggers is the linear layer output at the current time step t, in order to update the forward variable α t (v) It is necessary to consider the scores of all labels transferred from the previous time step to the labels of the current time step. The formula for calculating the path score is: in represents the path score of the entity with label v at the t+1th time step, represents the path score of the event trigger whose state is label v at the t+1th time step, and the exp function represents an exponential function with the natural constant e as the base; After traversing all time steps, it is necessary to consider the transfer score from the last label to the terminal label end, and add the score of the last time step in the forward variable α to the transfer score of the terminal label end: α t (end)=a n (m)+T(u,end), where α t (end) represents the path score of the state at the t-th time step being the termination label end, and T(u,end) represents the transition score from label m to the termination label end; Calculate the score and normalization constant Z(x) of all possible label sequences: In order to predict the optimal label sequence, the Viterbi algorithm is used. The goal of Viterbi decoding is to find the path with the highest score, that is, to maximize: where y entity* Represents the optimal prediction score of the entity in sentence y, y trigger* represents the optimal prediction score of the event trigger of sentence y, y t represents the tth token of sentence y, represents the hidden layer representation of the t-th token of sentence y in the entity extraction task, represents the hidden layer representation of the t-th token of sentence y in the task of extracting event triggers, and argmax represents maximization; The maximum score path is calculated through dynamic programming, the optimal previous state is selected at each step, and the optimal label at each time step is gradually backtracked to obtain the entities and event triggers in the text sequence.

4. The method according to claim 3, characterized in that: Step 4 includes: given a sentence, representing the information extraction task of the sentence as a dependency factor graph, where nodes correspond to entities and event trigger words, and edges represent relations and event arguments; Based on the dependency factor graph, the following three information extraction tasks are implemented: Named Entity Recognition: Identifying entities from text; Relation extraction: Capture the relationship between entity pairs in the text and mark the relationship type; Event extraction: identifying and extracting events and related information from text; Perform named entity recognition, relation extraction, and event extraction in a joint information extraction model: Given an input sentence, it is formalized as a graph G = (V, E) prediction task, where V represents a set of nodes, E represents a set of directed edges, each node v = (a, b, l) ∈ V represents an entity or event trigger, a and b represent the start and end positions of the span respectively, l is the label of the span, and for entity labels l ∈ L enriry , for an event trigger label l∈L event , L entity represents the entity label set, L event Represents a set of event trigger labels; each edge e ij represents the relationship label from the i-th graph node to the j-th graph node, e ij =(i,j,r)∈E.

5. The method according to claim 4, characterized in that Step 5 includes: constructing a factor graph and designing three different scoring factors to simulate the local distribution of each label, the correlation between the same task instances, and the correlation between different task instances. The three different scoring factors specifically include a unary factor, a binary factor, and a ternary factor: Unary factor: average the representations of all words in the prediction span and add them to the positional encoding to obtain the representation z of each node, and input the representation of each graph node into the feedforward neural network FFN of different tasks at the same time to obtain an independent label score for each graph node: in represents the nth word fragment of the i-th graph node, avg represents the average value, ntask represents different graph node tasks, FFN nrask Feedforward neural network FFN representing different graph node tasks, Represents the unary score of the i-th graph node under different graph node tasks; For edge e ij , using two different feedforward neural networks FFN to simulate the influence of nodes as subjects and objects on the relationship, and using the biaffine function to model the interaction of different feedforward neural networks FFN: Among them, etask represents different side tasks. represents the element-wise product, G etask is a randomly initialized learnable parameter matrix, Represents edge e ij Unary scores under different side tasks, FFN etask-s Feedforward neural network FFN, FFN representing different node tasks when the graph node is used as the head node etask-e Feedforward neural network FFN representing different node tasks when graph nodes are used as tail nodes; Binary factors: Binary factors are used to capture the dependencies of edges that share the same node, using different feed-forward neural networks (FFNs) to calculate the representation of each node as a shared joint node and an independent sole node: in Represents the representation of the i-th graph node as an independent sole node, Represents the i-th graph node as a representation of a shared joint node; For two edges that share the i-th graph node, the binary factor of the edge is represented as: where r m represents the label of the first edge, is the first edge label r m The embedding of n represents the label of the second edge, is the second edge label r n For symmetry, let For the edge e that shares the i-th graph node ij and edge ik , with e ij Binary factor score With e ik Binary factor score same; Ternary Factor: Ternary factor is used to model the mutual dependence between the two endpoints of an edge, using different feed-forward neural networks FFN to generate representations for the head node and the tail node: in Represents the representation of a graph node as a head node. Represents the representation of a graph node as a tail node; The representation of the ternary factor is: in represents the ternary score of the ith graph node and the jth graph node and the edge connecting the ith graph node and the jth graph node, represents the embedding of the edge label between the i-th graph node and the j-th graph node belonging to category m, l p ∈L s Represents the head node v i The label belongs to category p, l q ∈L e Represents the head node v i The label belongs to category q, is the label of the head node of the ternary factor l p Embedded, is the tag l of the tail node of the ternary factor q Embedding Aggregate the three scoring factors to obtain the joint probability distribution of all variables. The joint probability distribution of each graph node is calculated as: Where P(x i = l p ,w) represents the probability that the i-th graph node label belongs to category p when the model parameter is w, P(x j = l q ,w) represents the probability that the jth graph node label belongs to category q when the model parameter is w, Represents a univariate scoring factor The pth element of Represents a univariate scoring factor The qth element of Represents a univariate scoring factor The mth element of , α1, α2∈[0,1] is the weight parameter used to adjust the contribution of each factor; For each edge, the joint probability distribution is calculated as: Where P(x ij =r m ,w) means that when the model parameter is w, the label of the edge between the i-th graph node and the j-th graph node is r m , α3, α4∈[0,1] are weight parameters used to adjust the contribution of each factor.

6. The method according to claim 5, characterized in that Step 6 includes: using stochastic gradient variational inference to continue to maintain the complex dependencies between hidden variables, and iteratively inferring the approximate posterior distribution Q(X) of each variable X, letting the true posterior distribution be P(X), and optimizing the approximate posterior distribution Q(X) by reducing the distance from P(X): in represents the approximate posterior distribution after iteration, KL represents the KL divergence, and the calculation formula of the evidence lower bound L(Q(X)) is: L(Q(X))=log(P(X)-log(Q(X))), Then, by calculating the gradient of L(Q(X)) and continuously updating the parameters of the posterior distribution Q(X) using gradient descent, the model loss is calculated using forward propagation based on the posterior distribution Q(X), and the final joint information extraction model is obtained after a predetermined number of back propagation iterations.

7. The method according to claim 6, characterized in that In step 7, the joint information extraction model obtained in step 6 is used to input data and output entities, event triggers, relations, and event arguments; After the information extraction model is trained, the text sequence is input into the model, and the model will extract the information contained in it based on the training content.

8. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores program codes, and when the program codes are executed by the processor, the processor executes the steps of the method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: A computer program or instruction is stored, and when the computer program or instruction is run on a computer, the steps of the method according to any one of claims 1 to 7 are executed.

Images