A fine-grained entity classification method based on label inference network
By adopting a label inference network-based method in fine-grained entity classification and using label dependency relationships and attribute networks for inference, the problem of difficulty in generalization and long-tail label prediction in the existing technology is solved, and a more efficient fine-grained entity classification is achieved.
Patent Information
- Application Number
- CN202210320490.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Priority Date
- 2021-04-28
- Filing Date
- 2022-03-29
- Publication Date
- 2025-04-11
- Estimated Expiration
- 2042-03-29
AI Technical Summary
Existing fine-grained entity classification methods are difficult to effectively utilize the dependencies between labels and are difficult to generalize to new data sets, especially in long-tail label prediction.
Using a method based on a tag inference network, a tag is generated through pre-training language models and decoders, combining the context attention mechanism and prerequisite attention mechanism, a entity mentions tags are predicted in sequence, and an inductive reasoning is performed through the attribute network to generate new tags.
It realizes that without using predefined tag dependencies, effectively utilizes tag dependencies, can generalize to new data sets, and alleviates the problem of long-tail tag prediction, and improves the accuracy of fine-grained entity classification.
Smart Images
Figure CN114706982B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a fine-grained entity classification method, in particular to a fine-grained entity classification method based on a label inference network, and belongs to the technical field of natural language processing. Background Art
[0002] Fine-grained entity classification is an important task in information extraction. Given a sentence and the entity mentions in it as input, the purpose of fine-grained entity classification is to classify the entity mentions in fine grains in combination with the sentence information. For example, given the input "They walked with the staff of the City Calligraphy Association", where the entity mention is "City Calligraphy Association", it is necessary to classify "City Calligraphy Association" as: {"organization", "social institution", "institution", "staff", "employee"}.
[0003] The major challenges of fine-grained entity recognition tasks come from their interdependent, large-scale, and fine-grained label sets. First, entity labels usually have complex dependency structures: hierarchical dependency structures and associative dependency structures. Figure 1 As shown in the figure, the labels "person", "musician", and "composer" form a three-layer taxonomy structure; while "commander", "theorist", and "scientist" form an association structure because they share the attributes of "expert" and "scholar". Second, since the label set is large-scale and fine-grained, there are a large number of long-tail labels in the label set. For example, in the Ultra-Fine dataset, more than 80% of the labels have less than 5 training instances, and even 25% of the labels have no training instances.
[0004] In order to utilize the dependency relationships in the label set, existing methods usually use the dependency structure between labels as a label prediction constraint. However, these methods require pre-defined label dependency knowledge and then encode the dependency knowledge using specific methods, which means that these methods can only utilize limited dependency relationships and are difficult to generalize to new datasets. Summary of the invention
[0005] The purpose of the present invention is to propose a fine-grained entity classification method based on a label reasoning network, which can utilize the dependency relationship between labels without using a predefined label dependency structure, can be generalized to new data sets, and can alleviate the problem of long-tail label prediction.
[0006] To achieve the above purpose, the technical solution adopted by the present invention is as follows:
[0007] A fine-grained entity classification method based on a label inference network includes the following steps:
[0008] Given a sentence consisting of entity words, context, entity tags, and sentence tags, input it into the pre-trained language model, obtain the source-side hidden state representation of the sentence, and use the hidden state representation of the sentence tag as the context-related entity mention representation;
[0009] The source hidden state and the entity mention representation are input into a decoder, and the decoder obtains the target hidden state according to the entity mention representation, and then uses the context attention mechanism to capture the context information of different time steps according to the source hidden state, and uses the premise attention mechanism to capture the generated label information according to the target hidden state, and sequentially predicts all entity mention labels to obtain a label set; according to the entity mention representation, context information and the generated label information, the decoder calculates the prediction probability of different labels on the label set, and selects the label with the highest prediction probability in each time step as the label generated based on deductive reasoning;
[0010] The label as the prediction result is split into multiple attributes, and an attribute network is constructed according to the relationship between the attributes and the labels, wherein the attribute network includes an attribute graph, and the attribute graph includes attribute nodes and label nodes, and edges representing the association relationship between the attribute nodes and the label nodes; at each time step of the decoder, the attribute nodes in the attribute graph are activated by the hidden state of the decoder; based on the activation state of the attribute node and the association between the attribute and the label, the activation score of the label is calculated: if the number of activation copies of the label is greater than a threshold, the label is generated and used as the label generated based on inductive reasoning;
[0011] The labels generated based on deductive reasoning and the labels generated based on inductive reasoning are used as the label set of the prediction results, and compared with the label set of the standard answer. The parameters of the above decoder and attribute network are optimized by combining the set prediction loss function and the attribute network activation loss function.
[0012] The pre-trained language model, optimized decoder and attribute network are used to formally process the input sentences and classify the entities in the sentences.
[0013] Furthermore, after the sentence is input into the pre-trained language model, each token and word in the sentence has a hidden state representation.
[0014] Furthermore, the BERT model is selected as the pre-trained language model.
[0015] Furthermore, the LSTM model is selected as the decoder based on deductive reasoning.
[0016] Furthermore, the decoder initializes the first state of the target hidden state using the entity mention representation.
[0017] Furthermore, the method of using the contextual attention mechanism to capture contextual information at different time steps according to the source hidden state is as follows: at each time step, the contextual attention mechanism calculates the correlation between the hidden state of the decoder at each time step and each source hidden state through an activation function; the weight is calculated using the correlation; and a weight is assigned to each source hidden state to obtain the contextual information of the current time step.
[0018] Furthermore, the method of using the premise attention mechanism to capture the generated label information according to the hidden state of the target end is as follows: at each time step, the premise attention mechanism calculates the correlation between the hidden state of the decoder at each time step and the hidden state of each target end through an activation function; the weight is calculated using the correlation; weights are assigned to each target end hidden state to obtain the generated label information.
[0019] Furthermore, the method for calculating the prediction probability on the label set is as follows: first, the hidden state of the current time step is obtained through the decoder using the hidden state of the decoder at the previous time step and the prediction probability of the previous time step; then the entity mention representation, context information, the generated label information and the hidden state of the current time step are used as input to calculate an intermediate quantity; then, based on the intermediate quantity and an introduced mask vector to avoid generating duplicate labels, all prediction probabilities of the current time step are calculated; finally, the label with the highest prediction probability of the current time step is selected.
[0020] Furthermore, the method of activating the attribute nodes in the attribute graph through the hidden state of the decoder is: using cosine similarity to evaluate the similarity between the hidden state of the decoder and the attribute node, and then using the activation function RELU to activate the attribute.
[0021] Furthermore, the cross entropy loss function is used as the set prediction loss function, and the cross entropy loss function is used to calculate the loss between the label set of the prediction result and the optimal sequence of answers obtained based on the Hungarian algorithm.
[0022] Furthermore, the attribute network activation loss function calculates the attribute graph supervision loss.
[0023] The method of the present invention is based on a context-related entity mention encoder of a pre-trained language model; a label deductive reasoning mechanism based on a sequence-to-set generation framework, combining the generated labels and context information to generate new labels; a label inductive reasoning mechanism based on an attribute network, combining the attributes activated by the generated labels to generate new labels; corresponding to the two reasoning mechanisms described above, the network parameters are optimized using the set prediction loss and the attribute graph activation loss. The label reasoning network composed of the pre-trained language model, the decoder and the attribute network can model, learn and utilize label dependencies end-to-end, and can solve the long-tail label prediction problem in fine-grained entity classification. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 These are two example graphs of label dependencies.
[0025] Figure 2 Schematic diagram of label inference in an embodiment of the present invention.
[0026] Figure 3 It is a schematic diagram of fine-grained entity classification based on a label inference network in an embodiment of the present invention.
[0027] Figure 4 It is an attribute instance in the embodiment of the present invention. DETAILED DESCRIPTION
[0028] In order to make the above features and advantages of the present invention more obvious and easy to understand, embodiments are given below and described in detail with reference to the accompanying drawings.
[0029] by Figure 1 For example, in order to simultaneously utilize the hierarchical dependency and associative dependency of tags, this embodiment designs two tag inference mechanisms to decode tags. For the hierarchical dependency, this embodiment proposes a tag deductive reasoning mechanism to use context information and the generated tags to predict tags. Figure 2 As shown in the figure, in the deductive reasoning process, the model can infer that the entity mention "they" is a "theorist" through the generated label "person" and the context information "they theorize", which implies the hierarchical dependency relationship between the label "theorist" and "person". In view of the associative dependency relationship, this embodiment splits the label into a set of attributes and proposes a label inductive reasoning mechanism to use the attributes activated by the generated labels to further activate new labels. Figure 2 As shown, the generated labels "theorist" and "commander" activate the attributes "scholar" and "expert" respectively, and these two attributes further activate the label "scientist", which implies the association relationship between "theorist", "commander" and "scientist". Combining the above two reasoning mechanisms, this embodiment constructs a label reasoning network, which can sequentially generate fine-grained labels using a sequence-to-set generation framework. Therefore, the network is well modeled in an end-to-end form, and learns and utilizes these two types of label dependencies. In addition, since the labels are split into multiple attributes, and the attributes associated with the long-tail labels are not necessarily long-tail, the long-tail labels are easier to predict, which can solve the long-tail label prediction problem in fine-grained entity classification.
[0030] In order to utilize the dependency relationship between tags and not use a predefined tag dependency structure, while alleviating the difficulty of long-tail tag prediction, this embodiment provides a fine-grained tag classification method based on a tag inference network. Figure 3 The network architecture is shown, and the technical solution is described as follows:
[0031] 1. Context-dependent entity mention encoder based on pre-trained language model: Take entity mentions and their context as input to the pre-trained language model to obtain context-dependent entity mention representation.
[0032] Given a sentence X = [CLS], x1,…, [E1], m1,…, m k ,[E2],…,x n , where [E1], [E2] are entity tags, [CLS] is a sentence tag, m is an entity word, x is a context, and k and n represent quantities. In this embodiment, sentence X is fed into the pre-trained language model BERT. Each tag and word in sentence X will have a hidden state representation, and finally the hidden state representation H = {h1, ..., h n}, and finally adopt the hidden state representation of [CLS] as the context-dependent entity mention representation g.
[0033] Second, the label deductive reasoning mechanism based on the sequence-to-set generation framework generates new labels at each prediction time step (i.e., each prediction) by combining the context information and the labels that have been generated by the deductive reasoning mechanism at the previous time step. Specifically, the above-obtained representation is input, and the dynamic attention mechanism is used to obtain the context information and the label information that has been generated by the deductive reasoning mechanism, and then the entity mention labels are sequentially generated to obtain the label set L. This step describes the label deductive reasoning process, which is a process of continuously generating a label sequence, such as Figure 3 As shown in Figure 1, the label sequence generated by label deductive reasoning corresponds to person→theorist→commandar→… Each time a label is generated, a new label is generated by combining the already generated labels and context information.
[0034] For the target hidden state and entity mention representation g, the decoder based on deductive reasoning generates labels using context information and the generated label information sequence. Specifically, this embodiment uses the LSTM model as a decoder to obtain the target hidden state S = {s0, ..., s k}, where k is the number of predicted labels. In this embodiment, g is first used to initialize s0. Then, at each time step, the two attention mechanisms capture the context information and the generated label information respectively, and then predict the next label.
[0035] (1) Contextual attention mechanism: The contextual attention mechanism is used to capture the context related to the current state s t The contextual attention mechanism uses the context information related to the source hidden state h at each time step t to help predict the label. For example, “they theorize” provides rich information for predicting the label “theorist”. i Assign different weights α ti To capture contextual information:
[0036]
[0037]
[0038] Where tanh is the activation function, W c , U c , v c is the network parameter, s t is the hidden state of the decoder at time step t, e ti is the correlation between the hidden state at the tth time step and the i-th hidden state of the sentence; T represents the transposition calculation, which swaps the dimensions of the vector. Yes c The transpose of . In this way, the representation c of the context information can be obtained. t :
[0039]
[0040] (2) Precondition attention mechanism: The premise attention mechanism is used to capture and utilize the dependencies between labels to help predict labels. For example, if the label "person" has been generated, based on the context of "they theorize", the model is likely to predict the hyponym "theorist" of "person". At each time step t, the premise attention mechanism will be used to give different target hidden states s j Assign different weights α tj To capture information about generated tags:
[0041]
[0042]
[0043] Where W p , U p , v p is the network parameter, e tj refers to the current state S t The correlation with the hidden state of the jth target terminal at the tth time step. T represents the transposition calculation, which swaps the dimensions of the vector. Yes p This way, we can get the representation u of the generated label information. t :
[0044]
[0045] (3) Label prediction: Based on the context information mixed with the premise information, this embodiment will m t =[c t +g;u t +s t ] as the input of the next time step of LSTM, and calculate the predicted probability on the label set:
[0046] s t =LSTM(s {t-1} , W b y {t-1} )
[0047] o t =W o m t
[0048] y t =softmax(o t +I t )
[0049] Where W o is the network parameter, W b is the label embedding parameter, y t is the predicted probability at time step t, o t is the intermediate quantity, s t is the hidden state of the decoder at time step t. In order to avoid the decoder model generating repeated labels, this embodiment uses a mask vector
[0050]
[0051] in is the label predicted by the model before time step t, l i is the i-th label in the label set L. Finally, this embodiment selects y t The label with the highest prediction probability is used as the prediction result of the current time step. If the model generates the [EOS] label, the prediction is stopped.
[0052] 3. The label inductive reasoning mechanism based on the attribute network generates new labels by combining the attributes activated by the already generated labels; the labels activated in the above deductive reasoning process will gradually activate the corresponding attributes through the linear neural network model, and the activated attributes will further activate the labels through the attribute network.
[0053] This embodiment splits the label into multiple attributes and designs an attribute network to represent the relationship between the attribute and the label. Based on the constructed attribute network, the hidden state of the decoder will activate the attribute, and the attribute will infer a new label.
[0054] (1) Attribute graph: The attribute graph G = {V, E} is a bipartite graph used to represent the relationship between attributes and labels. V contains the attribute nodes V a and label node V1, where edge E only exists between attribute nodes and label nodes, and is used to indicate the association between attributes and labels.
[0055] (2) Inductive reasoning based on attribute graphs: At each time step of LSTM decoding, the hidden state s of the decoder at time step t t Will activate the attribute nodes in the attribute graph:
[0056]
[0057] Where W s is the attribute network model parameter, W a is the attribute embedding parameter. The present invention uses cosine distance to evaluate the similarity between the hidden state and the attribute (i.e., sim in the formula), and uses RELU as the activation function to activate the attribute. Based on the activation state of the attribute node, the model calculates the activation score of the label:
[0058]
[0059] where n a is the number of attributes, is the jth th Tags, E ij is the correlation between the i-th attribute and the j-th label. If the activation score of the label is greater than the threshold θ s Finally, this embodiment uses the labels generated by deductive reasoning and inductive reasoning as the final result of fine-grained entity classification.
[0060] (3) Update of attribute graph: Since labels are associated with many attributes, in order to make the calculation more efficient, this embodiment dynamically constructs an attribute graph for each training instance. This embodiment uses two methods to obtain attributes: using the mask language model to predict the words at the location of the entity mention, and directly splitting the entity mention into words. The above two parts of the vocabulary are preprocessed to obtain the attributes related to each instance, such as Figure 4 shown.
[0061] 4. Model optimization based on set prediction loss and attribute graph loss; The label set predicted by the two reasoning mechanisms in steps two and three is used as the final prediction result, and compared with the label set of the standard answer. The network parameters in steps two and three are optimized through the set prediction loss function and the attribute network activation loss function.
[0062] Since there is no order relationship between labels in fine-grained recognition and classification tasks, the set prediction loss should be used as the loss of label prediction. Specifically, this embodiment uses the bipartite matching loss (cross entropy loss) to measure the gap between the above prediction results and the answer:
[0063]
[0064] Where CE is the cross entropy loss, Y * is the model prediction result, is the optimal sequence of answers obtained using the Hungarian algorithm (i.e., a label set serving as a standard answer in this embodiment).
[0065] At the same time, the attribute network activation loss function of this embodiment uses attribute graph supervision loss to enable the model to correctly activate attributes:
[0066]
[0067]
[0068] Among them, y j Indicates whether the j-th category label is the correct answer or the wrong answer, the value is 1 for correct, and the value is -1 for wrong; Y is the standard answer label set; vj is the j-th label node.
[0069] The final loss is the sum of the two:
[0070]
[0071] The following is a specific application example:
[0072] Take the fine-grained classification of the entity mention “they” in “On such a potent force, they theorize, could collapse some of the war ship.” as an example.
[0073] Implementation:
[0074] The input sentence is constructed as “[CLS]On such a potent force,[E1]they[E2]theorize,could collapse some of the war ship.” and input into the pre-trained language model (BERT is used in this work) to obtain the context-related entity mention representation g.
[0075] Based on the label deductive reasoning mechanism of the sequence-to-set generation framework, g is input into LSTM to sequentially generate entity labels until the [EOS] label is generated. In the above example, the model will sequentially generate labels of "person", "theorist", "commander", and "[EOS]".
[0076] Based on the label inductive reasoning mechanism of the attribute graph, the labels generated during the sequence generation process will activate the attributes in the attribute graph, and the attributes will further activate the labels. If the activation score of the label is greater than a certain threshold, the label will be generated. In the above example, the "person" label will activate the "human" attribute, the "theorist" label will activate the "scholar" attribute, and the "commander" label will activate the "expert" attribute. These three activated attributes will jointly activate the "scientific" label.
[0077] Although the present invention has been disclosed as above by way of embodiments, it is not intended to limit the present invention. Appropriate modifications or equivalent substitutions of the technical solutions of the present invention made by ordinary technicians in the field should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on what is defined in the claims.
Claims
1. A fine-grained entity classification method based on label inference network, characterized in that: The following steps are involved: Given a sentence consisting of entity words, context, entity tags, and sentence tags, input it into the pre-trained language model, obtain the source-side hidden state representation of the sentence, and use the hidden state representation of the sentence tag as the context-related entity mention representation; The source hidden state and the entity mention representation are input into a decoder, and the decoder obtains the target hidden state according to the entity mention representation, and then uses the context attention mechanism to capture the context information of different time steps according to the source hidden state, and uses the premise attention mechanism to capture the generated label information according to the target hidden state, and sequentially predicts all entity mention labels to obtain a label set; according to the entity mention representation, context information and the generated label information, the decoder calculates the prediction probability of different labels on the label set, and selects the label with the highest prediction probability in each time step as the label generated based on deductive reasoning; Splitting the label as the prediction result into multiple attributes, constructing an attribute network according to the relationship between the attributes and the labels, the attribute network includes an attribute graph, the attribute graph includes attribute nodes and label nodes, and edges representing the association relationship between the attribute nodes and the label nodes; at each time step of the decoder, activating the attribute nodes in the attribute graph through the hidden state of the decoder; Based on the activation status of the attribute node and the correlation between the attribute and the label, the activation score of the label is calculated: if the activation number of the label is greater than a threshold, the label is generated and used as the label generated based on inductive reasoning; The labels generated based on deductive reasoning and the labels generated based on inductive reasoning are used as the label set of the prediction results, and compared with the label set of the standard answer. The parameters of the above decoder and attribute network are optimized by combining the set prediction loss function and the attribute network activation loss function. The pre-trained language model, optimized decoder and attribute network are used to formally process the input sentences and classify the entities in the sentences.
2. The method according to claim 1, characterized in that When a sentence is fed into a pre-trained language model, each token and word in the sentence has a hidden state representation.
3. The method according to claim 1, characterized in that The BERT model is used as the pre-trained language model; the LSTM model is used as the decoder based on deductive reasoning.
4. The method according to claim 1, characterized in that The decoder initializes the first state of the target hidden state with the entity mention representation.
5. The method according to claim 1, characterized in that The method of using the contextual attention mechanism to capture contextual information at different time steps according to the hidden state of the source end is as follows: at each time step, the contextual attention mechanism calculates the correlation between the hidden state of the decoder at each time step and the hidden state of each source end through an activation function; uses the correlation to calculate the weight; assigns weights to each source end hidden state to obtain the contextual information of the current time step.
6. The method according to claim 1, characterized in that The method of using the premise attention mechanism to capture the generated label information according to the hidden state of the target end is as follows: at each time step, the premise attention mechanism calculates the correlation between the hidden state of the decoder at each time step and the hidden state of each target end through an activation function; uses the correlation to calculate the weight; assigns weights to each hidden state of the target end to obtain the generated label information.
7. The method according to claim 1, characterized in that The method for calculating the prediction probability on the label set is as follows: first, the decoder uses the hidden state of the decoder at the previous time step and the prediction probability of the previous time step to obtain the hidden state of the current time step; then the entity mention representation, context information, the generated label information and the hidden state of the current time step are used as input to calculate an intermediate quantity; then, based on the intermediate quantity and an introduced mask vector to avoid generating duplicate labels, all prediction probabilities of the current time step are calculated; finally, the label with the highest prediction probability at the current time step is selected.
8. The method according to claim 1, characterized in that The method of activating the attribute nodes in the attribute graph through the hidden state of the decoder is: using cosine similarity to evaluate the similarity between the hidden state of the decoder and the attribute node, and then using the activation function RELU to activate the attribute.
9. The method according to claim 1, characterized in that The cross entropy loss function is used as the set prediction loss function. The cross entropy loss function is used to calculate the loss between the label set of the prediction result and the optimal sequence of answers obtained based on the Hungarian algorithm.
10. The method according to claim 1, characterized in that The attribute network activation loss function calculates the attribute graph supervision loss.