Anaphora resolution method and device that references external knowledge

Abstract

The present invention provides a method and apparatus for anaphora resolution that refers to external knowledge. [Solution] The method generates training data and constructs and trains a mention identification model and a relation classification model. First, a sentence is input to train the mention identification model, which marks mentions in the sentence. A sentence specifying two or three mentions is input to connect knowledge corresponding to the mentions, determine whether a coreference relationship exists between the specified mentions, and train the relation classification model to mark mentions where a coreference relationship exists. After the model is trained, it is used for anaphora resolution. The method of the present invention takes into account the semantic information of the entire sentence during the process of anaphora resolution by referring to external knowledge. During the process of training the relation classification model, the model is trained to determine whether a coreference relationship exists between the mentions and mark mentions where a coreference relationship exists. When three mentions are input, this training method allows the model to have a deeper understanding of mentions and coreference relationships, and gives the model stronger anaphora resolution capabilities.

Description

[Technical Field]

[0001] The present invention relates to the technical fields of natural language processing and reference resolution / coreference resolution technology, and more particularly to a method and apparatus for anaphora resolution that refers to external knowledge. [Background technology]

[0002] In linguistics, the use of an abbreviation or alternative name for a previously mentioned word is called anaphora. While anaphora can avoid the awkwardness and overstatement that can result from repeated use of the same word, it can also create problems with anaphoric uncertainty. The process of separating different mentions of the same object into a single set of equivalents is called anaphora resolution. Anaphora resolution is a fundamental research area in the field of NLP and plays an important role in tasks such as reading comprehension, information extraction, and multi-turn dialogue.

[0003] Anaphora resolution has evolved from rule-based to neural network models. Hobbs et al. proposed an anaphora resolution algorithm that uses a breadth-first search algorithm to generate a parsing tree from text analysis, and effectively matches coreference relationships between entity phrases and antecedents based on manual rules. Rule-based anaphora resolution methods achieve the goal of matching anaphors and antecedents by defining and constraining strict rules, but lack scalability. With the expansion of data scale, anaphora resolution methods based on statistical algorithms have emerged. The co-occurrence frequency between antecedents and anaphors is used as an indicator for relationship analysis. Dagan et al. proposed a method to automatically collect statistical information on co-occurrence modes in large corpora, which can resolve anaphoric mentions and syntactic ambiguity. With the rise of machine learning, anaphora resolution methods based on machine learning have emerged. Machine learning methods can be considered a combination of rule-based and statistical methods. McCarthy et al. proposed a mention pair model that transforms the anaphora resolution task into a classification task of determining whether an antecedent and an anaphor match. Luo et al. represent all mentions in a text using a Bayer tree structure and obtain referential chain features using the "ANY" feature representation format. Yang et al. build an entity-mention model using an inductive logic programming algorithm and provide a relational method to organize different entity knowledge and mentions. Iida et al. propose a "tournament" model based on central theory, which compares two candidate antecedents and selects the one with the highest probability to establish a coreference relationship. In recent years, with the development of deep learning, an increasing number of neural network models have been applied to anaphora resolution. In "Improving Coreference Resolution by Learning Entity-Level Distributed Representations," they propose a method to generate cluster-pair vectors and perform anaphora resolution using a clustering method.

[0004] However, anaphora resolution is a challenging task that requires a deep understanding of text and common sense. For example, consider two sentences: "I have a big, round apple in my pocket, and it's about to burst" and "I have a big, round apple in my pocket, and it's sure to be delicious." The coreference of "it" can be either "pocket" or "apple." Anaphora resolution requires understanding the semantic and allusive knowledge of the sentence. Knowledge-aware Pronoun Coreference Resolution proposed an anaphora resolution method that can reference external knowledge, but the model structure employed in this method has weak feature extraction capabilities and does not take into account the semantics of the entire sentence when referencing external knowledge. Subsequent methods for anaphora resolution using text generation methods based on big models cannot perform anaphora resolution by referencing external knowledge. Summary of the Invention [Problem to be solved by the invention]

[0005] In response to the shortcomings of the prior art, the present invention provides an anaphora resolution method that refers to external knowledge. [Means for solving the problem]

[0006] To achieve the above object, the technical solution of the present invention is as follows: A first aspect of the present invention is an anaphora resolution method that refers to external knowledge, comprising the following steps (1) to (4): In the step (1), training data is generated, a target text is obtained, and mentions in the target text sentence are marked as training data for a mention identification model, where the training data includes mention identification training data and relation classification training data. In the step (2), a mention identification model and a relation classification model are constructed. In step (3), the data obtained in step (1) is used to train a mention identification model and a relation classification model. In step (4), the trained model is used for anaphora resolution, i.e., first, the mention identification model is used to identify the mentions in the sentence, then any two mentions in the sentence are specified, the knowledge of the specified mentions is searched from the knowledge base, and the knowledge is spliced ​​into the sentence, and then input into the relationship classification model, which predicts whether the input mentions have a coreference relationship.

[0007] Specifically, in step (1), the mentions in the target text are marked as training data for the mention identification model. Specifically, two or three mentions in the text are specified, knowledge corresponding to the specified mentions is searched, and whether or not a coreference relationship exists between the specified mentions and the mentions that have a coreference relationship are tagged as training data for the relation classification model.

[0008] Specifically, the mention identification training data in step (1) is used to train a mention identification model, i.e., for an input sentence, 0 indicates non-mention, 1 indicates the start position of the mention, and 2 indicates the non-start position of the mention, and the sentence is tagged to obtain a mention identification tag sequence.

[0009] Specifically, when training the mention identification model, a sentence is input and the model is trained to predict the tag of each character. The loss function for training the mention identification model is shown in Equation 3 below:

number

[0010] Specifically, the relationship classification training data in step (1) is used to train a relationship classification model, which starts with "[CLS]", then splices the input sentence, then splices "[SEP]", and then splices the knowledge corresponding to the specified mention so that the length does not exceed 512. The specified mention information is indicated by the specified mention code, and the specified mention information is indicated in the form of a 012 sequence, where 0 indicates non-specified content, 1 indicates the start position of the specified mention, and 2 indicates the non-start position of the specified mention.

[0011] Furthermore, in step (4), the relation classification model predicts whether the input mention has a co-reference relationship, and the relation classification model not only determines whether the specified mention has a co-reference relationship, but also marks the mentions that have a co-reference relationship, and the specific details include the following contents (6.1) and (6.2): In the above (6.1), when two references are specified in the input of the relational classification model, if the two references refer to the same object, the relational classification model determines that there is a co-reference relationship and marks both of the two references. If the two references refer to different objects, the relational classification model determines that there is no co-reference relationship and does not mark any references. In (6.2) above, when three references are specified in the input of the relational classification model, if the three references point to the same object, the relational classification model determines that there is a co-reference relationship and marks all three references. If only two references point to the same object, the relational classification model determines that there is a co-reference relationship and marks the two references. If the three references point to different objects, the relational classification model determines that there is no co-reference relationship and does not mark any references.

[0012] Specifically, when training, the relationship classification model not only determines whether a specified mention has a coreference relationship, but also marks the coreferenced mentions. That is, when training the relationship classification model, two or three mentions are specified, and the model is trained to determine whether the specified mentions have a coreference relationship and mark the coreferenced mentions. The loss function for training the relationship classification model is shown in the following Equation 7:

number

[0013] Furthermore, when training the prediction of the relational classification model, if only two mentions need to be specified, the relational classification model will determine whether the two mentions of the anaphora have a coreference relationship.

[0014] Specifically, the mention identification model in step (2) adopts the SpanBERT model pre-trained on a Chinese corpus, and the relation classification model adopts the RoBERTa model pre-trained on a Chinese corpus.

[0015] In a second aspect of the present invention, there is provided an anaphora resolution device that refers to external knowledge, comprising: a training data generation module, a construction module, a model training module, and an anaphora resolution module; The training data generation module acquires a target text and marks mentions in the target text sentence as training data for a mention identification model, the training data including mention identification training data and relation classification training data. The construction module constructs a mention identification model and a relation classification model. The model training module trains a mention identification model and a relation classification model using the data obtained in step (1). The anaphora resolution module uses the trained model for anaphora resolution: first, it uses a mention identification model to identify mentions in a sentence, then specifies any two mentions in the sentence, searches the knowledge of the specified mentions from a knowledge base, connects them into the sentence, and then inputs them into a relationship classification model, which predicts whether the input mentions have a coreference relationship. [Effects of the Invention]

[0016] The beneficial effects of the present invention are as follows: First, we provide a method for anaphora resolution that references external knowledge. In the process of anaphora resolution with reference to external knowledge, the semantic information of the entire sentence is taken into account, allowing the model to better understand and narrow down the external knowledge based on the semantics of the sentence. Second, the proposed method adds a three-mention situation to the training process of the relation classification model. When three mentions are specified, the model is trained not to determine whether a coreference relationship exists between the mentions, but to mark mentions where a coreference relationship exists. This training method allows the model to have a deeper understanding of mentions and coreference relationships, giving the model stronger anaphora resolution capabilities. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a flowchart of a method according to the present invention. [Figure 2] FIG. 1 is a schematic diagram of input and output of a mention identification model. [Figure 3] FIG. 1 is a schematic diagram of input and output of a relational classification model. [Figure 4] 2 is a flow chart of the device according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, the anaphora resolution method with reference to external knowledge according to the present invention will be described in detail with reference to the accompanying drawings. Unless contradictory, the features of the following examples and embodiments can be combined with each other.

[0019] As shown in FIG. 1, the anaphora resolution method with reference to external knowledge according to the present invention includes the following steps: (1) Generate training data. The text paragraphs are crawled from the network and tagged, and mentions in the sentences are first marked. Then, for each mention, corresponding knowledge is retrieved from the knowledge base to generate training data, which includes mention identification training data and relation classification training data.

[0020] The mention identification training data is used to train the mention identification model. As shown in Figure 2, a sentence is tagged to obtain a mention identification tag sequence by using 0 to indicate non-mention, 1 to indicate the start of a mention, and 2 to indicate the non-start of a mention. For example, take the sentence "I have a big, round apple in my pocket that is about to burst." There are four mentions in the sentence: "I," "pocket," "apple," and "it." The corresponding mention identification tag sequence is "101200000000001201000000."

[0021] The relation classification training data is used to train a relation classification model. The relation classification training data includes double-mention relation classification data and three-mention relation classification data. The double-mention relation classification data specifies two references in a sentence and trains a model to determine whether the two references have a coreference relationship. The three-mention relation classification data specifies three references in a sentence and trains a model to determine whether the three references have a coreference relationship and mark references that have a coreference relationship. If all three references have a coreference relationship, all three references are marked.

[0022] As shown in Figure 3, taking the example of "I have a big, round apple in my pocket, and it's about to burst," samples of both mention relation classification data are as follows: The text to be entered is "[CLS] I have a large, round apple in my pocket, and it's about to burst. [SEP] A pocket is a pouch-like part sewn into clothing to hold things, or a pocket on clothing, or a storage device generally made of cloth or leather, or a device made of cloth, leather, etc. for holding things. An apple, also known as a citrus fruit or an apple, is the fruit of the apple tree, generally red in color but varying depending on the variety. It is rich in minerals and vitamins and is one of the most commonly eaten fruits. Sometimes, apple refers to an American technology company headquartered in Cupertino, California, USA." The specified mention code to enter is "000120000000000120000000000..." The relation classification tag is 0, The mention tag for the coreference relationship existence is 00000000000000000000000000. The text to be entered is "[CLS] I have a large, round apple in my pocket that is about to burst. [SEP] A pocket is a pouch-like part sewn into clothing for storing things, or a pocket in clothing, or a storage device generally made of cloth or leather, or a device made of cloth, leather, etc. for storing things. It is a pronoun and refers to something other than a person." The specified mention code to enter is "000120000000000000100000000..." The relation classification tag is 1, The mention tag for the coreference relationship existence is 001200000000000001000000. The three reference relationship classification data are as follows: The text to be entered is "[CLS] I have a large, round apple in my pocket, and it's about to burst. [SEP] A pocket is a pouch-like part sewn into clothing to hold things, or a pocket on clothing, or a storage device generally made of cloth or leather, or a device made of cloth, leather, etc. for holding things. An apple, also known as a citrus fruit or an apple, is the fruit of the apple tree, generally red in color but varying depending on the variety. It is rich in minerals and vitamins and is one of the most commonly eaten fruits. Sometimes, apple refers to an American technology company headquartered in Cupertino, California, USA." The specified mention code to enter is "010120000000000120000000000..." The relation classification tag is 0, The mention tag for the coreference relationship existence is 00000000000000000000000000. The text to be entered is "[CLS] I have a large, round apple in my pocket, and it's about to burst. [SEP] A pocket is a pouch-like part sewn into clothing to hold things, or a pocket on clothing, or a storage device generally made of cloth or leather, or a device made of cloth, leather, etc. for holding things. An apple, also known as a 'na' or 'ringo', is the fruit of the apple tree, generally red in color but varying depending on the variety. It is rich in minerals and vitamins and is one of the most commonly eaten fruits. Sometimes, the word 'apple' also refers to an American technology company headquartered in Cupertino, California, USA. It is a pronoun, referring to something other than a person." The specified mention code to enter is 000120000000000120100000000... The relation classification tag is 1, The mention tag for the coreference relationship existence is 001200000000000001000000. Here, the input text is constructed by adding [CLS] before the sentence requiring anaphora analysis, followed by [SEP]. The knowledge of the specified mention is added after [SEP], and the specified mention code is entered. The specified mention information is expressed in the form of a 012 sequence, where 0 indicates non-specific content, 1 indicates the start position of the specified mention, and 2 indicates the non-start position of the specified mention. In the above example, the ellipsis after the input anaphora mention code indicates an omitted 0. The relationship classification tag indicates whether the specified mention has a coreference relationship, with 0 indicating no coreference relationship and 1 indicating a coreference relationship. Since three mentions are specified in the input of the three-mention classification data, there are several possibilities for their relationship: none of the three mentions have a coreference relationship, only two of the mentions have a coreference relationship, or all three mentions have a coreference relationship. If none of the three mentions have a coreference relationship, the relationship classification tag is 0; otherwise, it is 1. When the relationship classification tag is 1, it is necessary to mark the coreferenced mentions to distinguish between cases where only two mentions have a coreference relationship and cases where all three mentions have a coreference relationship. To maintain a consistent data format for the relationship classification tasks, coreference mention tags are also used for both mention relationship classification tasks. The coreference mention tag indicates the coreferenced mention in a 012 sequence format. 1 indicates the start position where a coreferenced mention exists, 2 indicates the non-start position where a coreferenced mention exists, and 0 indicates the others.

[0023] (2) Model construction. The construction of the models includes the construction of a mention identification model and a relation classification model.

[0024] The mention identification model is used to tag mentions in sentences. It employs the SpanBERT model, which has been pre-trained on a Chinese corpus. SpanBERT has the same structure as the BERT model, but has been improved in training tasks. Instead of using random marks, training data is generated by shielding consecutive random spans using span masking. A span boundary objective is added to train span boundary representations to predict the entire content of the shielded span, without relying on a single marker representation within it. Due to the improvements in training tasks, SpanBERT is suitable for the tasks of entity identification and mention identification.

[0025] The relation classification model determines whether a coreference relationship exists in the input mentions and marks those that do. The input to the relation classification model specifies the mention sentence and knowledge about the mention. The RoBERTa model, pre-trained on a large corpus, has excellent feature extraction capabilities, so the RoBERTa Chinese pre-trained model is used as the relation classification model.

[0026] (3) Model training. The input sentences of the training data for mention identification are converted into an integer index vector input by SpanBERT, and an output vector corresponding to each position is obtained. Next, the output vector corresponding to each position is input to a linear layer for classification, and a tag for that position is obtained. The loss is obtained by comparing the model's predicted tag sequence with the tagging. The loss is minimized to train the model. The training process of the mention identification model is expressed in the following equations 1 to 3:

number

number

number

[0027] The input sentences of the relation classification training data are converted into integer index vectors, and then input into the model along with the specified mention code. The model is trained to determine whether a coreference relationship exists and mark mentions that do. The training loss function is the sum of the relation classification loss and the mention marking loss. The training process of the relation classification model is expressed by the following equations 4 to 7:

number

number

number

number

[0028] (4) Model predictions. The trained mention identification model and relation classification model are used for anaphora resolution, and the mention identification model is used to identify mentions in a sentence, and then the knowledge corresponding to each mention is searched. All combinations of two mentions are specified, and the corresponding knowledge is connected and input into the relation classification model, and the relation classification model predicts whether the specified mentions have a coreference relationship.

[0029] When the mention identification model makes a prediction, it converts the sentence into an integer index list and inputs it into the model. The model adds the embedding vector corresponding to the integer index and the position code information to obtain a matrix, which is expressed as Equation 8 below:

number

[0030] The calculation for obtaining the location code matrix is ​​performed by constructing a trigonometric function to indicate the location information. Specifically,

number

number

[0031] The resulting matrix is ​​then input to a multi-layer encoder to obtain the code output. The calculation process can be expressed as follows:

number

number

[0032] Here, the encoder operation input passes through multihead attention, skip connection, layer normalization, linear layer, skip connection and layer normalization processes in sequence, and the equation is expressed as follows:

number

number

number

number

[0033] The process of passing the integer index vector in the above sentence through the model to obtain the code matrix is ​​the encoding process of the Transformer, and this calculation process of the model can be summarized in the following formula:

number

number

[0034] After identifying mentions in a sentence using the mention identification model, pairs of identified mentions are paired and the corresponding knowledge is connected and input into the relation classification model, which predicts whether the input mentions have a coreference relationship. Note that the input in the prediction stage only requires two mentions, not three. For example, in the sentence "I have a big, round apple in my pocket, and it must be delicious," the mentions identified by the mention identification model are "me," "pocket," "apple," and "it." Searching for knowledge corresponding to each mention from the knowledge base is as follows: I am the first person pronoun. A pocket is a bag-like part sewn onto clothing for storing things, or a pocket on a garment, or a storage device generally made of cloth or leather, or a storage device made of cloth, leather, etc. Apples, also known as 'na' or 'ringo', are the fruit of the apple tree, generally red in color but varying depending on the variety. Rich in minerals and vitamins, they are one of the most commonly eaten fruits. Sometimes, the word 'apple' also refers to an American technology company headquartered in Cupertino, California. It is a pronoun, referring to something that is not a person. The sentences can therefore generate input data for a relational classification model. The text to be entered is "[CLS] I have a big, round apple in my pocket, and it must be delicious. [SEP] A pocket is a pouch-like part sewn into clothing for storing things, or a pocket on clothing, or a storage device generally made of cloth or leather, or a device for storing things made of cloth, leather, etc." The specified mention code to enter is 01012000000000000000000000000... The input text is: "[CLS] There is a big and round apple in my pocket, and it must be delicious. [SEP] I am a first-person pronoun. An apple, also called nai or ringo in Japanese, is the fruit of an apple tree and generally呈红色 (should be '呈现红色' in Chinese, here it's translated as 'presents a red color' in English), but varies by variety, is rich in minerals and vitamins, and is one of the most commonly eaten fruits by people. Sometimes, 'apple' can also refer to an American technology company headquartered in Cupertino, California, USA. The specified reference code to be input is 010000000000000120000000000……, and The input text is: "[CLS] There is a big and round apple in my pocket, and it must be delicious. [SEP] I am..., it is..." The specified reference code to be input is 010000000000000000100000000……. The input text is: "[CLS] There is a big and round apple in my pocket, and it must be delicious. [SEP] The pocket is..., the apple is..." The specified reference code to be input is 000120000000000120000000000……. The input text is: "[CLS] There is a big and round apple in my pocket, and it must be delicious. [SEP] The pocket is..., it is..." The specified reference code to be input is 000120000000000000100000000……. The input text is: "[CLS] There is a big and round apple in my pocket, and it must be delicious. [SEP] The apple is..., it is..." The specified reference code to be input is 000000000000000120100000000……. Input the data into the relationship classification model, and the calculation process of the model is represented by the following formula:

Number

[0035] Since the RoBERTa model of relational classification is structurally the same as the SpanBERT model of mention identification, the operation of the RoBERTa model of relational classification can be summarized in the following formula:

number

[0036] The output code vector corresponding to [CLS] is input to the feedforward network layer for classification, and the classification result of whether the specified mention has the specified relationship or not is obtained. The formula is expressed as follows:

number

[0037] After predicting the relationships between all pairs of mentions, the relational classification model can obtain a set of mentions that refer to the same object based on the coreference chain between the mentions. For the sentence "I have a big, round apple in my pocket, and it must be delicious," the "apple" and "it" that should be obtained by the relational classification model refer to the same object, and together with "me" and "pocket," they constitute three mutually independent objects in the sentence.

[0038] 4 is a flowchart of an anaphora resolution device that refers to external knowledge according to an embodiment. Referring to FIG. 4, the device includes the following modules: The training data generation module acquires the target text and marks mentions in the target text sentences as training data for the mention identification model, where the training data includes mention identification training data and relation classification training data. The construction module constructs a mention identification model and a relation classification model. The model training module trains the mention identification model and the relation classification model using the data obtained in step (1). The anaphora resolution module uses the trained model for anaphora resolution: first, it uses the mention identification model to identify mentions in the sentence, then specifies any two mentions in the sentence, searches the knowledge of the specified mentions from the knowledge base, splices them into the sentence, and then inputs them into the relationship classification model, which predicts whether the input mentions have a coreference relationship.

[0039] The specific manner in which each module performs an operation in the device in the above embodiment has been described in detail in the embodiment relating to the method, and therefore a detailed description thereof will be omitted here.

[0040] The device embodiments basically correspond to the method embodiments, so please refer to the relevant parts of the description of the method embodiments. The device embodiments described above are merely schematic, and the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, i.e., they may be located in one place or may be arranged in multiple network units. Depending on actual needs, some or all of the modules may be selected to achieve the objectives of the proposed invention. Those skilled in the art can understand and implement the present invention without any creative effort.

[0041] Other embodiments of the present invention will be readily apparent to those skilled in the art after considering the specification and practicing the disclosure. The present invention is intended to cover any modifications, uses, or adaptations of the present invention, which modifications, uses, or adaptations follow the general principles of the present invention and include means known to those skilled in the art or which are not disclosed in the present invention.

[0042] It should be noted that the present invention is not limited to the exact structure described above and shown in the drawings, and various modifications and changes can be made without departing from the scope of the present invention.

Claims

1. An anaphora resolution method that refers to external knowledge, the following steps (1) to (4) being executed by a computer: In the step (1), the computer generates training data including mention identification training data and relationship classification training data, acquires a target text, marks mentions in the target text sentence as training data for a mention identification model, specifies two or three mentions in the text sentence, searches for knowledge corresponding to the specified mentions, and tags whether a coreference relationship exists between the specified mentions and the mentions that have a coreference relationship as training data for a relationship classification model, and the training data for the relationship classification model includes a case where two mentions are specified and a case where three mentions are specified; In step (2), the computer constructs a mention identification model and a relation classification model, the mention identification model adopts a SpanBERT model pre-trained on a Chinese corpus, and the relation classification model adopts a RoBERTa model pre-trained on a Chinese corpus; In step (3), the computer trains a mention identification model and a relation classification model using the data obtained in step (1). When training the mention identification model, a sentence is input and the model is trained to predict a tag for each character. When training the relation classification model, if two mentions are specified in the input, and the two mentions refer to the same object, the relation classification model determines that there is a coreference relationship and marks both of the two mentions; if the two mentions refer to different objects, the relation classification model determines that there is no coreference relationship and does not mark any mentions; if three mentions are specified in the input, and the three mentions refer to the same object, the relation classification model determines that there is a coreference relationship and marks all three mentions; if only two mentions refer to the same object, the relation classification model determines that there is a coreference relationship and marks the two mentions; if the three mentions refer to different objects, the relation classification model determines that there is no coreference relationship and does not mark any mentions. In step (4), the computer uses the trained model for anaphora resolution, i.e., first, identifies the mentions in the sentence using a mention identification model, specifies any two or three mentions in the sentence, searches for knowledge of the specified mentions from a knowledge base, connects them to the sentence, and then inputs them into a relationship classification model, and the relationship classification model predicts whether the input mentions have a coreference relationship and which mentions have a coreference relationship.

2. 2. The method of claim 1, wherein the reference identification training data in step (1) is used to train a reference identification model, i.e., for an input sentence, 0 indicates non-reference, 1 indicates the start position of the reference, and 2 indicates the non-start position of the reference, and the sentence is tagged to obtain a reference identification tag sequence.

3. When training the mention identification model, a sentence is input and the model is trained to predict the tag of each character, and the loss function for training the mention identification model is shown in Equation 3 below: [Equation 3] where P is the probability, 【number】 is the predicted type of the i-th position under the condition that sentence s is input. 【number】 is the type tag y at the i-th position ｉ where Loss_tag is the loss of mention identification, i is the i-th position, and n is the length of the input sentence; 【number】 is the value of the prediction type of the model at the i-th position, and y ｉ 2. The method of claim 1, wherein i is a type tag at the i-th position, and s is an integer index vector of the input sentence.

4. 2. The method for anaphora resolution with reference to external knowledge according to claim 1, wherein the relational classification training data in step (1) is used to train a relational classification model, and the relational classification model starts with "[CLS]", then splices the input sentence, then splices "[SEP]", and then splices knowledge corresponding to the specified mention so that the length does not exceed 512, and indicates the specified mention information with a specified mention code, and indicates the specified mention information in the form of a 012 sequence, where 0 indicates non-specified content, 1 indicates the start position of the specified mention, and 2 indicates the non-start position of the specified mention.

5. When training, the relationship classification model not only determines whether a specified mention has a coreference relationship, but also marks the coreferenced mentions. That is, when training the relationship classification model, two or three mentions are specified, and the model is trained to determine whether a specified mention has a coreference relationship and mark the coreferenced mentions. The loss function for training the relationship classification model is shown in the following Equation 7: [Equation 7] where Loss_ref is the loss of the relation classification model, 【number】 is the model prediction of the relation classification, r is the tag of the relation classification, sk is an integer index vector corresponding to the input sentence and knowledge spliced ​​text, mids is the input specified mention code, i is the i-th position of the sentence, n is the length of the sentence, 【number】 is the model's prediction for the i-th position of the sentence, and t ｉ is the tag at the i-th position of the sentence, P is the probability, 【number】 is the predicted value of the relational model under the condition that sk and mids are input. 【number】 is the probability that r is equal to the tag r of the relation classification, 【number】 is the predicted value of the i-th position under the condition that sk and mids are input. 【number】 is the tag t at the i-th position ｉ 2. The method for resolving anaphora with reference to external knowledge according to claim 1, wherein the probability is equal to:

6. An anaphora resolution device that refers to external knowledge, the anaphora resolution device including: a training data generation module, a construction module, a model training module, and an anaphora resolution module; The training data generation module obtains a target text and marks mentions in the target text sentence as training data for a mention identification model, the training data including mention identification training data and relation classification training data; The construction module constructs a mention identification model and a relationship classification model, wherein the mention identification model adopts a SpanBERT model pre-trained on a Chinese corpus, and the relationship classification model adopts a RoBERTa model pre-trained on a Chinese corpus; The model training module uses the obtained data to train a mention identification model and a relationship classification model. When training the mention identification model, a sentence is input and the model is trained to predict a tag for each character. When training the relationship classification model, if two mentions are specified in the input, and the two mentions refer to the same object, the relationship classification model determines that there is a co-reference relationship and marks both of the two mentions; if the two mentions refer to different objects, the relationship classification model determines that there is no co-reference relationship and does not mark any mentions; if three mentions are specified in the input, and the three mentions refer to the same object, the relationship classification model determines that there is a co-reference relationship and marks all three mentions; if only two mentions refer to the same object, the relationship classification model determines that there is a co-reference relationship and marks the two mentions; if the three mentions refer to different objects, the relationship classification model determines that there is no co-reference relationship and does not mark any mentions. The anaphora resolution module uses a trained model for anaphora resolution, i.e., first identifies the mention in the sentence using a mention identification model, specifies any two or three mentions in the sentence, searches for knowledge of the specified mention from a knowledge base, connects them to the sentence, and then inputs them into a relationship classification model, and the relationship classification model predicts whether the input mention has a coreference relationship and which mentions have a coreference relationship. This is an anaphora resolution device that references external knowledge.

Images