Entity co-reference digestion method without interval representation vector
Through the method of uninterval representation vectors, the Bert fine-tuning model generates word embedding representation vectors, and calculates interval scores and precursor scores, which solves the problem of large memory usage and difficulty in dealing with complete documents in the prior art, and achieves a more efficient and simple co-referential digestion task.
Patent Information
- Application Number
- CN202411901285.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-23
- Publication Date
- 2025-04-18
AI Technical Summary
The prior art relies on dynamically constructed interval vectors and interval pair representation vectors in the common reference digestion task, resulting in large memory usage and difficulty in handling complete documents and multiple instances training in a single batch.
A method of entity co-referential digestion with no interval representation vector is proposed. The word embedding representation vector of each word is generated through the Bert fine-tuning model, and the interval score and precursor score are calculated by the start token and end token to eliminate the dependence on the interval representation vector.
This method is able to process multiple documents in a batch without truncating or sliding windows, takes up less memory, and is simpler and more efficient, and is competitive compared to the baseline model.
Smart Images
Figure CN119849492A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of natural language processing, and in particular to an entity coreference resolution method without interval representation vectors. Background Art
[0002] Coreference Resolution is a common task in NLP (natural language processing), which can identify parts of a text that refer to the same object. Entity coreference resolution is formally the process of dividing different references representing the same entity into an equivalent set (reference chain). It is usually modeled by identifying entity mentions (continuous text intervals) and predicting the antecedent mentions of each interval that refers to the previously mentioned entity, otherwise predicting an empty interval query. Coreference resolution can effectively solve the problem of ambiguous references in texts. It is a basic research in the field of NLP and plays an important role in tasks such as machine reading comprehension, information extraction, and multi-round dialogue.
[0003] Before 2017, coreference resolution usually required grammatical parsing and manual feature engineering, but the results were not ideal. Later, Lee et al. published a paper titled End-to-end Neural Coreference Resolution, which used neural networks for end-to-end coreference resolution. This method surpassed all previous coreference resolution methods, and many subsequent technologies used the ideas of this paper.
[0004] Before the introduction of the Transformer architecture, the most widely used sequence transformation models were built on top of high-level CNN or RNN models that included both encoding and decoding processes. To improve performance, the best architecture versions included attention mechanisms to connect the encoder and decoder together. Vaswani et al. (2017) proposed the Transformer, a new basic network designed based on the attention mechanism without recurrent or convolutional structures. The Transformer is much faster to train than recurrent or convolutional encoders in seq2seq tasks such as language translation. BERT (Bidirectional Encoder Representations from Transformers) (Devlin et al. 2019) is a transformer-based pre-trained language model. Recent BERT-based models have reported huge gains on multiple semantic benchmarks, including question answering, natural language inference, and named entity recognition. BERT aims to pre-train representations from unlabeled text by conditioning all layers on the entire sentence. As a result, BERT can be fine-tuned with an additional layer to provide cutting-edge models for a wide variety of tasks, such as question answering and language inference, without requiring significant architectural modifications.
[0005] Currently, the introduction of pre-trained language models has reduced many complex task-specific NLP models to simple lightweight layers. An exception to this trend is coreference resolution, where a complex task-specific model is attached to a pre-trained Transformer encoder. While very effective, this model has a very large memory footprint, mainly due to the dynamically constructed interval vectors and interval pair representation vectors, which hinders the processing of full documents and the ability to train on multiple instances in a single batch.
[0006] Currently, no effective solution has been proposed for the problems in the related technologies. Summary of the invention
[0007] In response to the problems in the related art, the present invention proposes an entity co-reference resolution method without interval representation vectors to overcome the task of clustering multiple mentions of the same entity in a given text to solve the technical problem of ambiguous reference in the text.
[0008] The technical solution of the present invention is achieved in this way:
[0009] A method for entity coreference resolution without interval representation vectors comprises the following steps:
[0010] The information inside and outside the encoding interval is encoded in advance through the Bert fine-tuning model to generate a word embedding representation vector containing context information for each word, and the word embedding representation vector of each word is used as input;
[0011] The score of whether the interval is an entity is calculated by the start token and end token of the interval, which is expressed as:
[0012]
[0013] Among them, qs is the start mark parameter, qe is the end mark parameter, and the vector v s 、v e and matrix B m is the scoring function f m The trainable parameters of
[0014] is the antecedent scoring function f a Extract the start token and end token, which are represented as:
[0015]
[0016] And the sum of four bilinear functions is expressed as:
[0017]
[0018] Calculate the score that interval c and interval q refer to the same entity object.
[0019] Furthermore, the word embedding representation vector includes: a fixed pre-trained word vector and a vector obtained by performing a one-dimensional convolution on letters.
[0020] Furthermore, the step of generating a word embedding representation vector containing context information for each word includes the following steps:
[0021] Extract the start token and end token from the context token of each token in the sequence, expressed as:
[0022]
[0023] Among them, GeLU is the activation function, and are learnable parameters.
[0024] Furthermore, the step of calculating the scores that interval c and interval q refer to the same entity object includes the following steps:
[0025] Adding the probability score of the antecedent and the entity interval score gives the score that interval c and interval q refer to the same object, expressed as:
[0026]
[0027] Furthermore, the method further comprises the following steps:
[0028] For all the coreference scores obtained for a word, after adding a softmax activation function, a series of probabilities are obtained in the output layer. The largest probability is selected, and the two intervals c and interval q pointed to represent that interval c is most likely to refer to the same object as interval q.
[0029] Furthermore, the method further comprises the following steps:
[0030] Calibrate the maximum log-likelihood function, which includes: maximizing the conditional probability, expressed as:
[0031]
[0032] Beneficial effects of the present invention:
[0033] The invention eliminates the reliance on interval representation vectors, hand-crafted features, and grammatical parsing, making it possible to process multiple documents in a batch without truncation or sliding windows, and without using prior knowledge and pruning algorithms. The model is competitive with the baseline model, takes up less memory, and is simpler and more efficient. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0035] Figure 1 is a flow chart of an entity coreference resolution method without interval representation vectors according to an embodiment of the present invention;
[0036] Figure 2 is a schematic diagram of SpanRepresentations of an entity coreference resolution method without interval representation vectors according to an embodiment of the present invention;
[0037] Figure 3 Schematic diagram of ScoringArchitecture of an entity coreference resolution method without interval representation vectors according to an embodiment of the present invention. DETAILED DESCRIPTION
[0038] The following will be combined with the accompanying drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of the present invention.
[0039] According to an embodiment of the present invention, a method for entity coreference resolution without interval representation vectors is provided.
[0040] This technical solution takes all spans in the text into account and calculates the probability that each span refers to the same object. For example, a simple sentence "Cat eat fish" contains 6 spans, namely "Cat", "eat", "fish", "Cat eat", "eat fish", "Cat eat fish". Assuming the length of the sentence is T, the total number of spans is expressed as: N = (T+1)T / 2.
[0041] For interval i, its starting position is represented by START(i) and its ending position is represented by END(i). Intervals are sorted from small to large by START. If START is the same, they are sorted from small to large by END. The set of all possible antecedents of interval i is represented by Y(i), which is expressed as:
[0042] Y(i)={ε,1,2,…,i-1}
[0043] Here, ε means that interval i has no antecedent.
[0044] In addition, key terms are defined, specifically, as follows:
[0045] Mention, mention, can be understood as those words in the text that refer to entities;
[0046] Antecedent, antecedent, in a pair of referential relationships, the phrase corresponding to the starting position of the referential relationship is called the anaphora (the word that refers to the language unit above), and the ending position is called the antecedent (the word pointed to by the anaphora). Example: Recently, a certain fruit company sued a certain Tong company, accusing it of not cooperating in accordance with the relevant contract, and Tong has not yet responded. Tong company is a real entity, and {Tong company, its, Tong} correspond to the full name, alias, and abbreviation of the entity respectively, and all three are references. There are two pairs of referential relationships: (Tong company, its), (Tong company, Tong). Since the referential relationship is transitive, the three together constitute an equivalent set, namely the referential chain {Tong company, its, Tong}. A text can contain multiple entities, so there can be multiple referential chains.
[0047] like Figure 1-Figure 3 As shown, a method for entity coreference resolution without interval representation vectors includes the following steps:
[0048] Step S1, pre-encoding the information inside and outside the interval through the Bert fine-tuning model, generating a word embedding representation vector containing context information for each word, and taking the word embedding representation vector of each word as input, wherein the word embedding representation vector includes: a fixed pre-trained word vector and a vector obtained by performing a one-dimensional convolution on the letters;
[0049] The technical solution, the Bert fine-tuning model, includes: Span Representations, which is responsible for calculating the representation vector of each span and Scoring Architecture, which is responsible for calculating the probability that the span is an entity and the probability that two spans refer to the same object.
[0050] The step of generating a word embedding representation vector containing context information for each word includes the following steps:
[0051] First, extract the start token and end token from the context token of each token in the sequence, expressed as:
[0052]
[0053] Among them, GeLU is the activation function, and are learnable parameters.
[0054] Step S2, using the start token and end token of the interval, calculates the score of whether the interval is an entity, expressed as:
[0055]
[0056] Among them, qs is the start mark parameter, qe is the end mark parameter, and the vector v s 、v e and matrix B m is the scoring function f m The trainable parameters of .
[0057] In this technical solution, the first two factors measure the likelihood that the start / end markers qs / qe of the span are the start / end of an entity mention. The third measures whether these markers are boundary points of the same entity mention. By calculating the mention scores of all possible spans, spans exceeding a certain length are shielded. Only the ones with the highest scores are retained. n mention candidates to avoid O(n) when computing the antecedent 4 ) complexity.
[0058] Step S3, the antecedent scoring function f a Extract the start token and end token, which are represented as:
[0059]
[0060]
[0061] And the sum of four bilinear functions is expressed as:
[0062]
[0063] Each factor measures compatibility by the interaction of the interval boundary tokens of interval c and interval q. The first parameter compares the start tokens of interval c and interval q, the fourth parameter compares the end tokens, and the second and third parameters further enhance the accuracy of the results by cross-comparing the start token / end token of the span of c with the end token / start token of the interval q, and finally the probability score of c being the antecedent of q can be obtained.
[0064] Step S4, calculating the score that interval c and interval q refer to the same entity object;
[0065] Adding the probability score of the antecedent and the entity interval score gives the score that interval c and interval q refer to the same object, expressed as:
[0066]
[0067] Step S5, calibrating the maximum log-likelihood function;
[0068] In this technical solution, the model needs to maximize the following conditional probability, expressed as:
[0069]
[0070] In step S6, after adding a softmax activation function to all the coreference scores of a word, a series of probabilities are finally obtained in the output layer. The two intervals c and q pointed to by the largest probability are selected, which means that interval c is most likely to refer to the same object as interval q.
[0071] In summary, with the help of the above technical solutions of the present invention, the present invention eliminates the dependence on interval representation vectors, manually constructed features and grammatical parsing, so that multiple documents can be processed in one batch without truncation or sliding windows, and without using prior knowledge and pruning algorithms. The model is competitive with the baseline model, occupies less memory, and is simpler and more efficient.
[0072] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. After considering the disclosure of the specification and the examples, those skilled in the art will easily think of other embodiments of the present disclosure. This application is intended to cover any modification, use or adaptation of the present disclosure, which follows the general principles of the present disclosure and includes common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The specification and the examples are only regarded as exemplary, and the true scope and spirit of the present disclosure are indicated by the claims.
[0073] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. A method for entity coreference resolution without interval representation vectors, characterized in that: The following steps are involved: The information inside and outside the encoding interval is encoded in advance through the Bert fine-tuning model to generate a word embedding representation vector containing context information for each word, and the word embedding representation vector of each word is used as input; The score of whether the interval is an entity is calculated by the start token and end token of the interval, which is expressed as: Among them, qs is the start mark parameter, qe is the end mark parameter, and the vector v s 、v e and matrix B m is the scoring function f m The trainable parameters of is the antecedent scoring function f a Extract the start token and end token, which are represented as: And the sum of four bilinear functions is expressed as: Calculate the score that interval c and interval q refer to the same entity object; The word embedding representation vector includes: a fixed pre-trained word vector and a vector obtained by performing one-dimensional convolution on letters.
2. The entity coreference resolution method without interval representation vector according to claim 1 is characterized in that: The steps described above generate a word embedding representation vector containing context information for each word, including the following steps: Extract the start token and end token from the context token of each token in the sequence, expressed as: Among them, GeLU is the activation function, and are learnable parameters.
3. The entity coreference resolution method without interval representation vector according to claim 2 is characterized in that: The step of calculating the scores of interval c and interval q referring to the same entity object includes the following steps: Adding the probability score of the antecedent and the entity interval score gives the score that interval c and interval q refer to the same object, expressed as:
4. The entity coreference resolution method without interval representation vector according to claim 3 is characterized in that: The following steps are also included: For all the coreference scores obtained for a word, after adding a softmax activation function, a series of probabilities are obtained in the output layer. The largest probability is selected, and the two intervals c and interval q pointed to represent that interval c is most likely to refer to the same object as interval q.
5. The entity coreference resolution method without interval representation vector according to claim 4 is characterized in that: The following steps are also included: Calibrate the maximum log-likelihood function, which includes: Maximize the conditional probability, expressed as: