Coreference resolution method and apparatus based on reference to external knowledge

Abstract

Disclosed in the present invention are a coreference resolution method and apparatus based on reference to external knowledge. The method comprises: generating training data, and establishing and training a mention recognition model and a relation classification model; firstly, inputting a sentence to train the mention recognition model, wherein the model marks mentions in the sentence; and inputting the sentence in which two or three mentions are designated to be spliced with knowledge corresponding to the mentions, and training the relation classification model to determine whether a coreference relation exists among the designated mentions, and marking the mentions having a coreference relation. After model training is completed, the models are used for coreference resolution. According to the method of the present invention, in the process of referring to external knowledge for coreference resolution, semantic information of the whole sentence is considered. In the training process of the relation classification model, the model is trained to determine whether a coreference relation exists among mentions, the model is trained to mark the mentions having a coreference relation, and the input comprises designated three mentions. Thus, the training method enables the model to have deeper understanding for the mentions and the coreference relation, so that the model has stronger coreference resolution capability.

Description

A method and device for reference resolution with reference to external knowledge Technical Field

[0001] The present invention belongs to the technical field of natural language processing and reference resolution technology, and in particular relates to a reference resolution method and device with reference to external knowledge. Background Art

[0002] In linguistics, using an abbreviation or pronoun to replace a previously used word is called cross-reference. Cross-reference can avoid bloated sentences and redundancy caused by repeated use of the same word, but it can also lead to unclear references. Cross-reference resolution is the process of grouping different references to the same object into an equivalent set. Cross-reference resolution is a fundamental research area in the field of natural language processing (NLP), playing a vital role in tasks such as reading comprehension, information extraction, and multi-turn conversations.

[0003] Co-reference resolution has evolved primarily from rule-based models to neural network models. Hobbs et al. proposed a co-reference resolution algorithm that uses a breadth-first algorithm to facilitate the generation of syntactic parse trees from text parsing and effectively matches coreference relationships between entity phrases and antecedents based on handcrafted rules. Rule-based co-reference resolution methods rely on strict rule constraints to achieve the goal of matching pronouns and antecedents, but lack scalability. With the expansion of data volumes, statistical co-reference algorithms have emerged. The co-occurrence frequency between antecedents and pronouns has become a metric for studying relationships. Dagan et al. proposed a method that automatically collects statistical information on co-occurrence patterns in large corpora, effectively eliminating anaphoric references and syntactic ambiguity. With the rise of machine learning, machine learning-based co-reference resolution methods have emerged. Machine learning methods can be considered a combination of rule-based and statistical methods. McCarthy et al. proposed a reference pair model, which transforms the co-reference resolution task into a classification task of determining whether the antecedent and pronoun match. Luo et al. represented all referential phrases in the text using a Bell tree structure and used the feature representation of "ANY" to obtain the characteristics of the reference chain. Yang et al. used an inductive logic programming algorithm to construct an entity-reference model, providing a relational method for organizing different entity knowledge and references. Iida et al. proposed a "tournament" model based on the central theory, which compares two candidate antecedents and selects the one with a higher probability to establish a co-reference relationship. In recent years, the development of deep learning has enabled more and more neural network models to be used for reference resolution. "Improving Coreference Resolution by Learning Entity-Level Distributed Representations" proposed a method for generating Cluster-Pair vectors and used clustering to perform reference resolution.

[0004] However, coreference resolution is a challenging task, requiring a deep understanding of the text and common sense. For example, in the sentences "I have a big, round apple in my pocket, and it's about to burst." and "I have a big, round apple in my pocket, and it must be delicious," the object "it" refers to can be either "pocket" or "apple." To achieve coreference resolution, one must have some understanding of the sentence semantics and the knowledge mentioned. "Knowledge-aware Pronoun Coreference Resolution" proposes a coreference resolution method that can reference external knowledge. However, the model structure used in this method has weak feature extraction capabilities, and the overall sentence semantics are not taken into account when referencing external knowledge. Subsequent coreference resolution methods based on large models and text generation are unable to reference external knowledge for coreference resolution.

[0005] Summary of the Invention

[0006] In view of the shortcomings of the existing technology, the present invention proposes a reference resolution method with reference to external knowledge.

[0007] To achieve the above object, the technical solution of the present invention is:

[0008] The first aspect of the present invention is a method for reference resolution with reference to external knowledge, the method comprising the following steps:

[0009] (1) Generating training data: Obtaining a target text and marking the mentions in the target text sentences as training data for the mention recognition model; the training data includes mention recognition training data and relation classification training data;

[0010] (2) Build a mention recognition model and a relationship classification model;

[0011] (3) Using the data obtained in step (1) to train the mention recognition model and the relationship classification model;

[0012] (4) The trained model is used for coreference resolution; that is, the mention recognition model is first used to identify the mentions in the sentence, and any two mentions in the sentence are specified. The knowledge of the specified mentions is retrieved from the knowledge base, and the knowledge is spliced ​​into the sentence and then input into the relation classification model. The relation classification model predicts whether the input mentions have a coreference relationship.

[0013] Specifically, the mentions in the target text sentence are marked in step (1) as training data for the mention recognition model, specifically, two or three mentions in the specified text sentence are retrieved, and the knowledge corresponding to the specified mentions is retrieved, and whether there is a coreference relationship between the specified mentions and the mentions with a coreference relationship are marked as training data for the relationship classification model.

[0014] Specifically, the mention recognition training data in step (1) is used to train the mention recognition model, that is, for the input sentence, 0 is used to represent non-mention, 1 is used to represent the starting position of the mention, and 2 is used to represent the non-start position of the mention to label the sentence to obtain a mention recognition label sequence.

[0015] Specifically, when training the mention recognition model, a sentence is input and the training model predicts the label of each character; the loss function of the training mention recognition model is:

[0016] Where Loss_tag is the loss of mention recognition, i is the i-th position, and n is the length of the input sentence. is the predicted type value of the i-th position model, y i is the type label at the i-th position, and s is the integer index vector of the input sentence.

[0017] Specifically, the relation classification training data in step (1) is used to train the relation classification model, and the relation classification model starts with "[CLS]", followed by the input sentence, followed by "[SEP]", and then by the knowledge corresponding to the specified mention, keeping the length no more than 512; the specified mention code is used to represent the specified mention information, and the specified mention information is represented in the form of a 012 sequence; 0 represents non-specified content, 1 represents the starting position of the specified mention, and 2 represents the non-starting position of the specified mention.

[0018] Furthermore, the relationship classification model in step (4) predicts whether the input mention has a coreference relationship. During training, the relationship classification model not only determines whether the specified mention has a coreference relationship, but also marks the mentions with a coreference relationship. Specifically:

[0019] (6.1) When two mentions are specified in the input of the relational classification model, if the two mentions refer to the same object, the relational classification model should determine that there is a coreference relationship and mark both mentions; if the two mentions refer to different objects, the relational classification model should determine that there is no coreference relationship and not mark either mention;

[0020] (6.2) When three mentions are specified in the input of the relational classification model, if these three mentions refer to the same object, the relational classification model should determine that there is a co-reference relationship and mark all three mentions; if only two of them refer to the same object, the relational classification model should determine that there is a co-reference relationship and mark these two mentions; if the three mentions refer to different objects, the relational classification model should determine that there is no co-reference relationship and not mark any mention.

[0021] Specifically, when training the relationship classification model, it is necessary not only to determine whether there is a coreference relationship in the specified mentions, but also to mark the mentions with the coreference relationship. That is, when training the relationship classification model, two or three mentions are specified, and the training model determines whether there is a coreference relationship in the specified mentions, and marks the mentions with the coreference relationship. The loss function of the training relationship classification model is:

[0022] Where Loss_ref is the loss of the relation classification model, r^ is the model prediction value of the relation classification, r is the label of the relation classification, sk is the integer index vector corresponding to the concatenated text of the input sentence and knowledge, mids is the input specified mention code, i is the i-th position of the sentence, and n is the length of the sentence. is the predicted value of the model at position i of the sentence, t i is the label of the i-th position in the sentence.

[0023] Furthermore, when the training relation classification model predicts, if only two mentions need to be specified, the relation classification model determines whether the two referenced mentions have a co-reference relationship.

[0024] Specifically, the recognition model mentioned in step (2) adopts the SpanBERT model pre-trained on Chinese corpus; the relation classification model adopts the RoBERTa model pre-trained on Chinese corpus.

[0025] A second aspect of the present invention is a device for deconstructing references to external knowledge, the device comprising the following modules:

[0026] Training data generation module: obtains target text and marks mentions in sentences of the target text as training data for the mention recognition model; the training data includes mention recognition training data and relationship classification training data;

[0027] Building modules: Building a mention recognition model and a relationship classification model;

[0028] Model training module: Use the data obtained in step (1) to train the mention recognition model and the relationship classification model;

[0029] Coreference resolution module: The trained model is used for coreference resolution. That is, the mention recognition model is first used to identify mentions in a sentence. Given any two mentions in a sentence, the knowledge of the specified mentions is retrieved from the knowledge base, concatenated into the sentence, and then input into the relation classification model. The relation classification model predicts whether the input mentions have a coreference relationship.

[0030] The beneficial effects of the present invention are as follows:

[0031] First, a coreference resolution method that can reference external knowledge is proposed. This process considers the overall semantic information of the sentence, enabling the model to better understand and filter external knowledge based on the sentence's semantics. Second, the proposed method incorporates three mentions into the training of the relation classification model. In this case, the model is not only trained to determine whether there is a coreference relationship between mentions, but also to identify mentions that do. This training method provides a deeper understanding of mentions and coreference relationships, empowering the model with stronger coreference resolution capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] FIG1 is a flow chart of the method of the present invention;

[0033] Figure 2 is a schematic diagram of the input and output of the mention recognition model;

[0034] Figure 3 is a schematic diagram of the input and output of the relational classification model;

[0035] FIG4 is a flow chart of the device of the present invention. DETAILED DESCRIPTION

[0036] The following describes in detail a method for resolving references to external knowledge proposed by the present invention in conjunction with the accompanying drawings. The features of the following embodiments and implementations may be combined with each other unless they conflict.

[0037] As shown in FIG1 , the present invention proposes a method for reference resolution using external knowledge, which includes the following steps:

[0038] (1) Generate training data

[0039] Article paragraphs are crawled from the Internet and annotated. First, the mentions in the sentences are marked, and then the corresponding knowledge is searched from the knowledge base for each mention to generate training data; the training data includes mention recognition training data and relationship classification training data.

[0040] Mention recognition training data is used to train the mention recognition model. As shown in Figure 2, sentences are annotated with 0 representing a non-mention, 1 representing the start of a mention, and 2 representing the non-start of a mention to obtain a mention recognition label sequence. For example, the sentence "I have a big, round apple in my pocket. It's about to burst." contains four mentions: "I," "pocket," "apple," and "it." The corresponding mention recognition label sequence is: "101200000000001201000000."

[0041] Relation classification training data is used to train the relationship classification model. This data includes two-mention relationship classification data and three-mention relationship classification data. Two-mention relationship classification data specifies two mentions in a sentence, and the model is trained to determine whether the two mentions have a coreference relationship. Three-mention relationship classification data specifies three mentions in a sentence, and the model is trained to determine whether the three mentions have a coreference relationship and mark the mentions that have a coreference relationship. If all three mentions have a coreference relationship, all three mentions are marked.

[0042] As shown in Figure 3, taking the sentence “I have a big, round apple in my pocket, and it’s about to burst,” as an example, the following is a sample of the classification data of the two-mention relationship:

[0043] Input text: "[CLS] I have a big, round apple in my pocket, and it's about to burst. [SEP] Pocket: A pocket refers to a bag-shaped part sewn into clothes for holding things, a pocket; or a container, usually made of cloth or leather; a container made of cloth, leather, etc. Apple: Also known as nai or lingo, it is the fruit of the apple tree, usually red, but it depends on the variety. It is rich in minerals and vitamins and is one of the most commonly consumed fruits. Sometimes apple also refers to an American technology company headquartered in Cupertino, California, USA."

[0044] Enter the specified mention code: "000120000000000120000000000...";

[0045] Relationship classification label: 0;

[0046] There is a coreference relationship mentioning the tag: 000000000000000000000000;

[0047] Input text: "[CLS] I have a big, round apple in my pocket, and it's about to burst. [SEP] Pocket: A pocket refers to a bag-shaped part sewn on clothes for holding things, such as a pocket. Or a container for holding things, usually made of cloth or leather; a container made of cloth, leather, etc. It: a pronoun, referring to non-human things."

[0048] Enter the specified mention code: "000120000000000000100000000...";

[0049] Relationship classification label: 1;

[0050] There is a coreference relationship mentioning the tag: 001200000000000001000000;

[0051] The classification data of the three-mentioned relationship is as follows:

[0052] Input text: "[CLS] I have a big, round apple in my pocket, and it's about to burst. [SEP] I: first-person pronoun. Pocket: A pocket refers to a bag-shaped part sewn on clothes for holding things, a pocket; or a container, usually made of cloth or leather; a container made of cloth, leather, etc. Apple: Also known as nai or lingo, it is the fruit of the apple tree, usually red, but it depends on the variety. It is rich in minerals and vitamins and is one of the most commonly consumed fruits. Sometimes apple also refers to an American technology company headquartered in Cupertino, California, USA."

[0053] Enter the specified mention code: "010120000000000120000000000...";

[0054] Relationship classification label: 0;

[0055] There is a coreference relationship mentioning the tag: 000000000000000000000000;

[0056] Input text: "[CLS] I have a big, round apple in my pocket, and it's about to burst. [SEP] Pocket: A pocket refers to a bag-shaped part sewn into clothes for holding things, such as a pocket. Or a container, usually made of cloth or leather; a container made of cloth, leather, etc. Apple: Also known as nai or lingo, it is the fruit of the apple tree, usually red, but it depends on the variety. It is rich in minerals and vitamins and is one of the most commonly consumed fruits. Sometimes apple also refers to an American technology company headquartered in Cupertino, California. It: A pronoun, referring to non-human things."

[0057] Enter the specified mention code: 000120000000000120100000000……;

[0058] Relationship classification label: 1;

[0059] There is a coreference relationship mentioning the tag: 001200000000000001000000;

[0060] The input text begins with [CLS] before the sentence requiring coreference resolution, followed by [SEP]. After [SEP], the designated mention is added. The designated mention code is entered to represent the designated mention information in a 012 sequence, where 0 represents non-designated content, 1 represents the start position of the designated mention, and 2 represents a non-start position. In the above example, the ellipsis after the coreference code represents the omitted 0. The relationship classification label indicates whether the designated mention has a coreference relationship: 0 indicates no coreference relationship, and 1 indicates a coreference relationship. Since the three-mention classification data input specifies three mentions, their relationships can have the following possibilities: none of the three mentions have a coreference relationship, only two of the mentions have a coreference relationship, or all three have a coreference relationship. When none of the three mentions have a coreference relationship, the relationship classification label is 0; otherwise, it is 1. When the relationship classification label is 1, the coreference relationship mentions must be marked to distinguish between only two mentions or all three. To maintain a consistent data format for the relationship classification task, a coreference mention label is also added to the two-mention relationship classification task. The coreference mention label expresses the coreference mention in a 012 sequence; 1 indicates the start position of the coreference mention, 2 indicates the non-start position of the coreference mention, and 0 indicates the others.

[0061] (2) Build the model

[0062] Model building includes the construction of mention recognition model and relationship classification model.

[0063] The mention recognition model is used to annotate mentions in sentences. It uses the SpanBERT model, which was pre-trained on Chinese corpus. SpanBERT has the same structure as the BERT model, but has improvements in training tasks. It uses Span Masking to mask continuous random spans, rather than generating training data using random tags. A new training objective, Span Boundary Objective, is added to train span boundary representations to predict the entire content of the masked span, rather than relying on individual tag representations. These improvements in training tasks make SpanBERT well-suited for tasks such as entity recognition and mention recognition.

[0064] The relation classification model determines whether coreference relationships exist in input mentions and marks those that do. The input to the relation classification model is a sentence specifying the mention and related knowledge. The RoBERTa model, pre-trained on large-scale predictions, has excellent feature extraction capabilities. We used the RoBERTa Chinese pre-trained model as the relation classification model.

[0065] (3) Training model

[0066] The input sentences of the mention recognition training data are converted into integer index vectors and input into SpanBERT to obtain the output vector corresponding to each position. The output vector corresponding to each position is then input into the linear layer for classification to obtain the label of that position. The loss is obtained by comparing the model's predicted label sequence with the annotation. The model is trained by minimizing the loss. The training process of the mention recognition model is expressed as follows: s =SpanBERT_Encode(s);

[0067] Among them, M s is the encoding matrix output by the SpanBERT model, s is the integer index vector of the input sentence, SpanBERT_Encode is the encoding operation of SpanBERT, MLP is the feedforward neural network, Loss_tag is the loss of mention recognition, i is the i-th position, n is the length of the input sentence, is the predicted type value of the i-th position model, y i is the type label of the i-th position, and s is the input sentence.

[0068] The input sentences of the relation classification training data are converted into integer index vectors and input into the model together with the specified mention code. The model is trained to determine whether there is a coreference relationship and mark the mentions that have a coreference relationship. The training loss function is the sum of the relation classification loss and the mention marking loss. The training process of the relation classification model is expressed as: sk =RoBERTa_Encode(sk,mids); r^=MLP1(M sk [0]);

[0069] Among them, M sk is the encoding matrix output by the RoBERTa model, RoBERTa_Encode is the encoding operation of RoBERTa, MLP1 and MLP2 are feedforward neural network operations, Loss_ref is the loss of the relation classification model, r^ is the model prediction value of the relation classification, r is the label of the relation classification, sk is the integer index vector corresponding to the concatenated text of the input sentence and knowledge, mids is the input specified mention code, i is the i-th position of the sentence, and n is the length of the sentence. is the predicted value of the model at position i of the sentence, t i is the label of the i-th position in the sentence.

[0070] (4) Model prediction

[0071] The trained mention recognition model and relation classification model are used for coreference resolution. The mention recognition model first identifies mentions in a sentence and then retrieves the knowledge corresponding to each mention. All combinations of two mentions are identified, and the corresponding knowledge is concatenated and input into the relation classification model. The relation classification model then predicts whether the specified mentions have a coreference relationship.

[0072] When the mention recognition model makes a prediction, the sentence is converted into a list of integer indices and input into the model. The model adds the embedding vector corresponding to the integer index and the position encoding information to obtain a matrix, which is expressed as: L 0 =Embedding_Lookup(s)+Position_Encoding(s);

[0073] Among them L 0 is the result matrix, s is the integer index list of the input sentence, Embedding_Lookup is the operation to obtain the word embedding matrix, and Position_Encoding is the operation to obtain the position encoding matrix.

[0074] The position encoding matrix operation is obtained by constructing trigonometric functions to represent the position information. Specifically:

[0075] where p k,2i is the 2ith component of the encoding vector at position k, p k,2i+1 is the 2i+1th component of the encoding vector at position k, where k is the position and i is the dimension component.

[0076] Then the obtained matrix is ​​input into the multi-layer encoder to obtain the encoded output. The calculation process can be expressed as: i =encoder(L i-1 );0 <i≤6; M s =L 6 ;

[0077] Among them L i is the output of the encoder at layer i, which is obtained by the encoder-side encoding operation of Transformer. s The encoding matrix output by the model.

[0078] The encoder operation input is processed by multi-head attention, residual connection, layer normalization, linear layer, residual connection and layer normalization. The formula is as follows:

[0079] Among them L i-1 is the input of the i-th layer encoder, is the intermediate result matrix, L i is the output of the i-th layer encoder, LayerNormalization is layer normalization. MultiHeadAttention is multi-head attention calculation, the specific calculation formula is: MultiHeadAttention(Q,K,V)=Concat(head1,…,head h )W;

[0080] Where Q, K, and V are input matrices, Concat is a vector concatenation operation, h is the number of heads, and W i Q 、W i K 、W i V is the weight of the i-th head.

[0081] The above process of passing the integer index vector of the sentence through the model to obtain the encoding matrix is ​​the encoding process of the Transformer. This operation process of the model can be summarized as follows: s = Transformer_Encode(s);

[0082] After obtaining the encoding matrix, the encoding of each position is input into the feedforward network layer for classification to obtain the label of each position. The formula is as follows:

[0083] Where i is the i-th position, is the predicted value of the i-th position, and MLP is a feedforward neural network.

[0084] After the mention recognition model identifies mentions in a sentence, it pairs the identified mentions and concatenates the corresponding knowledge into the relational classification model. The relational classification model predicts whether the input mentions have a coreference relationship. It's important to note that during the prediction phase, the input doesn't need to specify three mentions; only two are sufficient. For example, consider the sentence "I have a big, round apple in my pocket; it must be delicious." Assume the mention recognition model identifies the mentions: "I," "pocket," "apple," and "it." The knowledge corresponding to each mention is retrieved from the knowledge base:

[0085] I: First-person pronoun.

[0086] Pocket: A pocket refers to a bag-shaped part sewn on clothes for holding things, or a container for holding things, usually made of cloth or leather; a container made of cloth, leather, etc.

[0087] Apple: Also known as nai or lingo, it is the fruit of the apple tree. It is generally red, but varies depending on the variety. Rich in minerals and vitamins, it is one of the most commonly consumed fruits. Apple is also sometimes used to refer to the American technology company, headquartered in Cupertino, California.

[0088] It: a pronoun referring to something that is not human.

[0089] Therefore, this sentence can generate input data for the relation classification model:

[0090] Input text: "[CLS] I have a big, round apple in my pocket. It must be delicious. [SEP] I: first-person pronoun. Pocket: A pocket refers to a bag-shaped part sewn on clothes for holding things, such as a pocket. Or a container for holding things, usually made of cloth or leather; a container made of cloth, leather, etc."

[0091] Enter the specified mention code: 0101200000000000000000000000……;

[0092] Input text: "[CLS] I have a big, round apple in my pocket. It must be delicious. [SEP] I: First-person pronoun. Apple: Also known as nai or lingo, it is the fruit of the apple tree. It is usually red, but it depends on the variety. It is rich in minerals and vitamins and is one of the most commonly consumed fruits. Sometimes apple also refers to an American technology company headquartered in Cupertino, California, USA."

[0093] Enter the specified mention code: 0100000000000001200000000000……;

[0094] Input text: "[CLS] I have a big, round apple in my pocket. It must be delicious. [SEP] I: ... it: ... ";

[0095] Enter the specified mention code: 0100000000000000001000000000……;

[0096] Input text: "[CLS] I have a big, round apple in my pocket. It must be delicious. [SEP] Pocket: ... Apple: ...";

[0097] Enter the specified mention code: 0001200000000001200000000000……;

[0098] Input text: "[CLS] I have a big, round apple in my pocket. It must be delicious. [SEP] pocket: ...it: ...";

[0099] Enter the specified mention code: 000120000000000000100000000……;

[0100] Input text: "[CLS]I have a big, round apple in my pocket. It must be delicious. [SEP]Apple: ...it: ...";

[0101] Enter the specified mention code: 000000000000000120100000000……;

[0102] Input the data into the relational classification model, and the calculation process of the model is expressed in the following form: L 0 =Embedding_Lookup(sk)+Segment_Lookup(mids)+Position_Encoding(sk);

[0103] Segment_Lookup is the operation that encodes the input specified mention code into a matrix, sk is the integer index vector after the input sentence is concatenated with the knowledge, and mids is the specified mention code. The remaining symbols are the same as the formula representation of the mention recognition model operation.

[0104] Since the RoBERTa model of the relation classification model is structurally the same as the SpanBERT model of the reference recognition model, the operation formula of the RoBERTa model of the relation classification model is summarized as follows: sk =Transformer_Encode(sk,mids);

[0105] Among them, M sk The encoding matrix output by the model.

[0106] The output encoding vector corresponding to [CLS] is input into the feedforward network layer for classification to obtain the classification result of whether the specified mention has the specified relationship. The formula is as follows:

[0107] Where r^ is the prediction result of whether there is a coreference relationship, MLP is a feedforward neural network, sk [0] is the output code corresponding to [CLS].

[0108] After the relation classification model predicts the relationships between all mention pairs, it can identify sets of mentions that refer to the same object based on the co-reference chains between them. For example, in the sentence "I have a big, round apple in my pocket. It must be delicious," the relation classification model should conclude that "apple" and "it" refer to the same object, forming three independent objects in the sentence along with "I" and "pocket."

[0109] FIG4 is a flowchart of a device for resolving references to external knowledge according to an embodiment. Referring to FIG4 , the device includes the following modules:

[0110] Training data generation module: obtains target text and marks mentions in sentences of the target text as training data for the mention recognition model; the training data includes mention recognition training data and relationship classification training data;

[0111] Building modules: Building a mention recognition model and a relationship classification model;

[0112] Model training module: Use the data obtained in step (1) to train the mention recognition model and the relationship classification model;

[0113] Coreference resolution module: The trained model is used for coreference resolution. That is, the mention recognition model is first used to identify mentions in a sentence. Given any two mentions in a sentence, the knowledge of the specified mentions is retrieved from the knowledge base, concatenated into the sentence, and then input into the relation classification model. The relation classification model predicts whether the input mentions have a coreference relationship.

[0114] Regarding the apparatus in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0115] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0116] Those skilled in the art will readily conceive of other embodiments of the present application after considering the specification and practicing the contents disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present application that follow the general principles of this application and include common knowledge or customary techniques in the art that are not disclosed in this application.

[0117] It will be understood that the present application is not limited to the exact construction that has been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof.

Claims

1. A method for reference resolution with reference to external knowledge, characterized in that: The method comprises the following steps: (1) Generating training data: The training data includes mention recognition training data and relation classification training data; obtaining a target text, marking the mentions in the target text sentences as training data for the mention recognition model; specifying two or three mentions in the text sentences, retrieving the knowledge corresponding to the specified mentions, and marking whether there is a co-reference relationship between the specified mentions and the mentions that have a co-reference relationship as training data for the relation classification model; the training data for the relation classification model includes the cases of specifying two mentions and specifying three mentions; (2) Build a mention recognition model and a relation classification model; the mention recognition model uses the SpanBERT model pre-trained on Chinese corpus; the relation classification model uses the RoBERTa model pre-trained on Chinese corpus; (3) Using the data obtained in step (1) to train the mention recognition model and the relationship classification model; when training the mention recognition model, input a sentence and train the model to predict the label of each character; when training the relationship classification model, when two mentions are specified in the input, if the two mentions refer to the same object, the relationship classification model determines that there is a co-reference relationship and marks both mentions; if the two mentions refer to different objects, the relationship classification model determines that there is no co-reference relationship and does not mark any mention; when three mentions are specified in the input, if the three mentions refer to the same object, the relationship classification model determines that there is a co-reference relationship and marks all the three mentions; if only two refer to the same object, the relationship classification model determines that there is a co-reference relationship and marks the two mentions; if the three mentions refer to different objects, the relationship classification model determines that there is no co-reference relationship and does not mark any mention; (4) Using the trained model for coreference resolution; that is, first using the mention recognition model to identify the mentions in the sentence, specify any two or three mentions in the sentence, retrieve the specified mentions from the knowledge base, concatenate them into the sentence, and then input them into the relation classification model. The relation classification model predicts whether the input mentions have a coreference relationship and the mentions that have a coreference relationship.

2. A method for resolving references to external knowledge according to claim 1, characterized in that: The mention recognition training data in step (1) is used to train the mention recognition model, that is, for an input sentence, 0 is used to represent non-mention, 1 is used to represent the start position of a mention, and 2 is used to represent the non-start position of a mention to annotate the sentence to obtain a mention recognition label sequence.

3. A method for resolving references to external knowledge according to claim 1, characterized in that: When training the recognition model, a sentence is input and the training model predicts the label of each character; The loss function for training the mention recognition model is: Where P is the probability, is the predicted type of the i-th position under the condition of input sentence s wait The type label y at the i-th position i ; Loss_tag is the loss of mention recognition, i is the i-th position, n is the length of the input sentence, is the predicted type value of the i-th position model, y i is the type label at the ith position, and s is the integer index vector of the input sentence.

4. A method for resolving references to external knowledge according to claim 1, characterized in that: The relation classification training data in step (1) is used to train the relation classification model. The relation classification model starts with "[CLS]", followed by the input sentence, followed by "[SEP]", and followed by the knowledge corresponding to the specified mention, keeping the length no more than 512; the specified mention information is represented by the specified mention code, and the specified mention information is represented in the form of a 012 sequence; 0 represents non-specified content, 1 represents the starting position of the specified mention, and 2 represents the non-starting position of the specified mention.

5. A method for resolving references to external knowledge according to claim 1, characterized in that: When training the relation classification model, it is necessary not only to determine whether there is a co-referential relationship in the specified mentions, but also to mark the mentions with co-referential relationship. That is, when training the relation classification model, two or three mentions are specified, and the training model determines whether there is a co-referential relationship in the specified mentions, and marks the mentions with co-referential relationship. The loss function of the training relation classification model is: Where Loss_ref is the loss of the relational classification model, r ^ is the model prediction value of the relation classification, r is the label of the relation classification, sk is the integer index vector corresponding to the concatenated text of the input sentence and knowledge, mids is the input specified mention code, i is the i-th position of the sentence, n is the length of the sentence, is the predicted value of the model at the i-th position in the sentence, t i is the label of the i-th position in the sentence; P is the probability, P(r ^ =r|sk,mids) is the predicted value r of the relational model under the input sk and mids conditions ^ is equal to the probability of label r of the relation classification, is the predicted value of the i-th position under the input sk and mids conditions Equal to the label t at the i-th position i probability.

6. A device for deconstructing references with reference to external knowledge, characterized in that: The device includes the following modules: Training data generation module: obtaining target text, marking the mentions in sentences of the target text as training data for the mention recognition model; the training data includes mention recognition training data and relation classification training data; Building modules: Building a mention recognition model and a relation classification model; the mention recognition model uses the SpanBERT model pre-trained on Chinese corpus; the relation classification model uses the RoBERTa model pre-trained on Chinese corpus; Model training module: Use the obtained data to train the mention recognition model and the relationship classification model; When training a mention recognition model, you input a sentence and train the model to predict the label of each character; When training a relational classification model, if two mentions are specified in the input, if the two mentions refer to the same object, the relational classification model determines that there is a co-reference relationship and marks both mentions; if the two mentions refer to different objects, the relational classification model determines that there is no co-reference relationship and does not mark either. Mentions; when three mentions are specified in the input, if the three mentions refer to the same object, the relation classification model determines that there is a co-reference relationship and marks all three mentions; if only two refer to the same object, the relation classification model determines that there is a co-reference relationship and marks the two mentions; if the three mentions refer to different objects, the relation classification model determines that there is no co-reference relationship and does not mark any mention; Co-reference resolution module: The trained model is used for co-reference resolution; that is, the mention recognition model is first used to identify the mentions in the sentence, and any two or three mentions in the specified sentence are retrieved from the knowledge base. The knowledge of the specified mentions is spliced ​​into the sentence and then input into the relation classification model. The relation classification model predicts whether the input mentions have a co-reference relationship and the mentions that have a co-reference relationship.

Images