Zero pronoun identification device, zero pronoun identification method, and program

Abstract

To identify zero pronouns in an input sentence using a more simplified mechanism as compared with a conventional art.SOLUTION: A zero pronoun identifying device comprises: a word dividing unit that divides an input sentence; a predicate identifying unit that identifies a predicate from the sentence divided into words by the word dividing unit; and a zero pronoun identifying unit that calculates a span of an item for the predicate using a zero pronoun identifying model which is a model in which an output layer is added to a trained language model, determines whether or not an empty category exists by comparing a score of the span with a score of the empty category for the predicate, and if the empty category exists, determines presence or absence of a zero pronoun by classifying the empty category.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to techniques for identifying zero pronouns. [Background technology]

[0002] In languages such as Japanese, Chinese, and Arabic, it is possible to omit the subject or object if it is clear from the context. These are called zero pronouns or pro (small pro) because by omitting the element, they function as anaphors in the same way as pronouns.

[0003] For example, in the second sentence below in Japanese (JA), "I liked it," the object is omitted. In Japanese, it is clear from the first sentence that the person likes "cake," so it is natural to omit it. However, in English (EN), the object must be expressed with the pronoun "it."

[0004] JA This cake is delicious. I (pro-OBJ) like it.

[0005] EN This cake is delicious. I like (it). Furthermore, in this example, it would be more natural to omit the subject in the second sentence in Japanese and write it as "I liked it," as shown below. However, omitting the subject is not allowed in English.

[0006] JA This cake is delicious. (pro-SBJ) (pro-OBJ) I like it.

[0007] EN This cake is delicious. (I) like (it). Languages such as Japanese that allow the omission of subjects and objects (there are zero pronouns) are called pro-drop languages, while languages such as English that require a subject are called non-pro-drop languages. When translating from a pro-drop language to a non-pro-drop language, the technology to identify zero pronouns in input sentences in the pro-drop language is essential for correctly translating the meaning of the sentence based on the context and situation.

[0008] Zero pronouns are a type of empty category. In linguistics, especially in generative grammar, empty categories refer to the omitted pronoun (zero pronoun) called pro (or small pro), the unstated controlled subject called PRO (or big pro), and the null element (an element without a phonetic form) that expresses traces of movement in WH questions, relative clauses, etc. called T (or trace). Empty categories are sometimes called gaps. [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Linfeng Song, Kun Xu, Yue Zhang, Jianshu Chen, and Dong Yu. Zpr2: Joint zero pronoun recovery and resolution using multi-task learning and bert. In Proceedings of ACL-2020, pp. 5429-5434,2020. [Non-patent document 2] Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. Corefqa: Coreference resolution as query-based span prediction. In Proceedings of ACL-2020, pp. 6953-6963, 2020. Summary of the Invention [Problem to be solved by the invention]

[0010] However, conventional techniques for identifying zero pronouns in an input sentence have the problem of requiring a complex mechanism, such as the need for external tools for syntactic analysis to obtain a syntactic tree.

[0011] The present invention has been made in consideration of the above points, and aims to provide a technique for identifying zero pronouns in an input sentence using a mechanism that is simpler than conventional techniques. [Means for solving the problem]

[0012] According to the disclosed technology, there is provided a word segmentation unit that segments an input sentence; a predicate identification unit that identifies a predicate from the sentence that has been word-divided by the word division unit; A zero pronoun identification model that adds an output layer to a pre-trained language model and a zero pronoun identification device comprising: The zero pronoun identification model receives the preceding descriptor and the sentence as input, The span of the term for the preceding descriptor is determined, and the score of the span is compared with the score of an empty category for the preceding descriptor to determine whether the empty category exists. If the empty category exists, but Zero Pronoun Whether or not judgement and output the judgment result. the score of the span is the product of the probability that the term starts with the word at the start of the span and the probability that the term ends with the word at the end of the span; The score of the empty category is the product of the probability that the start position of the term is a special token in the input sequence to the zero pronoun identification model and the probability that the end position of the term is the special token. A zero pronoun identifier is provided. [Effects of the Invention]

[0013] The disclosed technology provides a technology for identifying zero pronouns in an input sentence using a mechanism that is simpler than conventional techniques. [Brief explanation of the drawings]

[0014] [Figure 1]1 is a configuration diagram of a zero pronoun identification system (zero pronoun identification device) according to an embodiment of the present invention. [Figure 2] FIG. 10 is a diagram illustrating the configuration of a zero pronoun identification model learning unit. [Figure 3] FIG. 10 is a diagram showing the flow of processing when training a zero pronoun identification model. [Figure 4] FIG. 10 is a diagram showing the process flow when identifying zero pronouns. [Figure 5] FIG. 2 is a diagram illustrating a hardware configuration of the apparatus. [Figure 6] This figure shows the number of documents, sentences, and predicates in NPCMJ and OntoNotes 5.0. [Figure 7] Figure 1 shows the number and percentage of zero pronouns in the training data of NPCMJ and OntoNotes. [Figure 8] Figure 1 shows the prediction accuracy of argument span and the identification accuracy of zero pronouns in NPCMJ. [Figure 9] FIG. 10 shows the identification accuracy of zero pronouns by class in NPCMJ. [Figure 10] Figure 1 shows the prediction accuracy of argument span and the identification accuracy of zero pronouns in OntoNotes 5.0. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0016] Furthermore, all of the systems and devices described in this embodiment offer specific improvements over conventional approaches such as those described in Non-Patent Documents 1 and 2, and represent an advancement in the technical field of zero pronoun identification.

[0017] Note that the technologies described in the reference documents cited in the following description are publicly known technologies, but the content of the explanation of the problems with the technologies described in the reference documents is not publicly known technologies. The numbers and names of the reference documents are listed together at the end of the specification. The numbers of the reference documents cited in the following description are indicated as "[1]" etc.

[0018] (Outline of the embodiment) First, an outline of this embodiment will be described. In this embodiment, a mechanism for performing span prediction using a trained language model is used as the basic mechanism for solving the problem. This realizes simple and highly accurate zero pronoun identification.

[0019] In the zero pronoun identification device described below, the span of the argument for a predicate is calculated for each argument type (subject, direct object, indirect object, etc.), and the argument span score is compared with the score for the argument being an empty category to determine whether a specific argument of the predicate is an empty category.Furthermore, it is determined whether the empty category is a zero pronoun, thereby achieving zero pronoun identification with higher accuracy than simple sequential labeling.

[0020] In the following, we will first explain various reference techniques related to zero pronoun identification to facilitate understanding of the technology according to this embodiment, followed by a description of the problem, the configuration of the device according to this embodiment, and its operation.

[0021] (Reference Technology) <Identifying Zero Pronouns as a Labeling Problem> Many of the conventional zero pronoun identification methods treat zero pronoun identification as a tree or sequence labeling problem.

[0022] Xiang et al. [7] and Takeno et al. [5] take the parse tree of a sentence as input, and classify the nodes corresponding to clauses that are maximal projections of predicates according to the presence or absence of zero pronouns (binary) or the type of zero pronoun (multiple-valued). The targets of processing in [7] and [5] are not only zero pronouns but also empty categories that contain traces of movement.

[0023] Song et al. [4] take a sentence as input as a sequence of words and classify all word boundaries according to the presence or absence of a zero pronoun (binary) or the type of zero pronoun (multiple-valued).

[0024] <Question answering using a trained language model> BERT [1] is a language representation model that uses a Transformer encoder to output a word vector for each word in an input sequence, taking into account the surrounding context. In recent years, language representation models are sometimes simply called language models.

[0025] Creating a language model from large-scale language data using a cloze test or a masked language model task, which predicts masked words from the context, is called pre-training, and the created language model is called a pre-trained language model.

[0026] It has been reported that adding an appropriate output layer to a pre-trained language model such as BERT and performing transfer learning (fine-tuning) with the training data of the target task can achieve the highest accuracy in various tasks such as semantic text similarity, natural language inference (textual entailment recognition), question answering, and named entity extraction [1].

[0027] For example, SQuAD-style question answering (QA) is a task in which a text and a question (query) are given, and the answer to the question is in the text, in other words, a substring (span) of the text is the answer [3].

[0028] In SQuAD-style question answering using a pre-trained language model, two sequences, a question and text, are first concatenated using a special symbol to form a single sequence, such as '[CLS] Question [SEP] Text [SEP]', and then given as input to a pre-trained language model. Next, the word vector output by the pre-trained language model for each word in the input sequence is used to predict the probability that that word will be the start and end points of an answer (span) to the question, and the span with the highest probability is extracted from the text as the answer to the question.

[0029] Here, [CLS] is a special token used to create a vector that aggregates information from two input sequences, and [SEP] is a token that represents the separation of the input sequences. In cases where it is necessary to indicate that a question cannot be answered, as in SQuAD v2.0[2], a classifier that determines whether the answer is possible is placed in the output layer of the vector for [CLS].

[0030] <Coreference analysis based on question answering> In recent years, methods have been proposed that apply question answering or span prediction using trained language models to various language processing technologies.

[0031] For a problem relatively similar to zero pronoun identification, Wu et al.[6] proposed a method for realizing coreference analysis using a question-answering framework. In the method disclosed in Reference [6], a sentence containing a mention of a certain entity is treated as a question, with the mention enclosed in the special token <mention>. The problem of extracting a set of mentions that refer to the same entity as the mention in the question is treated as a text sequence classification problem using BIO tagging.

[0032] (Problems and solutions) In the prior art, a complex mechanism was required to realize zero pronoun identification, such as the need for external tools such as syntactic analysis to obtain a syntactic tree.

[0033] Therefore, in this embodiment, Japanese zero pronoun identification is realized using a span prediction framework that uses a trained language model. This makes it possible to realize zero pronoun identification with higher accuracy than methods based on sequence labeling using a simple mechanism based on a trained language model and ground truth data for zero pronoun identification, without requiring external tools such as syntactic analysis to obtain a parse tree as in the past.

[0034] (Description of the Technology According to the Embodiment) <Device configuration example> FIG. 1 shows an example of the overall configuration of a zero pronoun identification system (which may also be called a zero pronoun identification device) according to this embodiment.

[0035] As shown in Figure 1, the zero pronoun identification system includes an input unit 111, a zero pronoun identification model learning unit 110, an output unit 112, a zero pronoun identification training data DB 120, a trained multilingual model DB 130, an input unit 211, a word segmentation unit 210, a predicate identification unit 220, a zero pronoun identification unit 230, an output unit 231, and a zero pronoun identification model DB 240.

[0036] Furthermore, the configuration shown in FIG. 1 may be realized by one device (computer), or may be configured by multiple devices (computers).

[0037] 1, a device 100 may be configured including an input unit 111, a zero pronoun identification model learning unit 110, an output unit 112, and a trained multilingual model DB 130. The device 100 may be called a learning device or a zero pronoun identification device.

[0038] Furthermore, a device 200 may be configured that includes an input unit 211, a word segmentation unit 210, a predicate identification unit 220, a zero pronoun identification unit 230, an output unit 231, and a zero pronoun identification model DB 240. The device 200 may be called an estimation device or a zero pronoun identification device.

[0039] Training (learning) sentences and correct answer data are stored as training data in the zero pronoun identification training data DB 120. The training (learning) sentences may be sentences that have already been subjected to word segmentation and predicate identification, or may be sentences that have not yet been subjected to word segmentation and predicate identification.

[0040] In the learning process described below, an example is shown in which word segmentation and predicate identification are performed during learning as well as during estimation. An example configuration of the zero pronoun identification model training unit 110 in this case is shown in FIG. 2. As shown in FIG. 2, the zero pronoun identification model training unit 110 includes a word segmentation unit 113, a predicate identification unit 114, a zero pronoun identification unit 115, and a parameter update unit 116. The word segmentation unit 113, the predicate identification unit 114, and the zero pronoun identification unit 115 each include the same functions as the word segmentation unit 210, the predicate identification unit 220, and the zero pronoun identification unit 230 used during estimation, respectively.

[0041] <Example of operation> The operation of each unit in the zero pronoun identification system will be outlined with reference to the flowcharts in Figures 3 and 4. The details of each process will be specifically described later.

[0042] First, the operation of training a zero pronoun identification model will be described with reference to the flowchart in Fig. 3. In the following processing, it is assumed that the zero pronoun identification unit 115 reads a trained language model from the trained language model DB 130 and stores a zero pronoun identification model (pre-training model) in a storage unit such as a memory, in which an output layer is added to the trained language model. The processing by the zero pronoun identification unit 115 is performed using the zero pronoun identification model. Alternatively, it may be considered that the zero pronoun identification unit 115 is the zero pronoun identification model.

[0043] In S101 , the input unit 111 reads out sentences that are training data for zero pronoun identification from the zero pronoun identification training data DB 120 , and inputs them to the word segmentation unit 113 .

[0044] In S102, the word dividing unit 113 divides the input sentence into words, and the predicate identifying unit 114 identifies predicates in the sentence.

[0045] The zero pronoun identification unit 115 repeats the processes of S103 to S106 for all argument types of all predicates in the sentence. In S103, the zero pronoun identification unit 115 finds the span that maximizes the score of the argument for the predicate, and in S104 finds the score for the argument of the predicate that is an empty category.

[0046] In S105, the zero pronoun identification unit 115 determines whether the score of the span is equal to or greater than the score of the empty category, and if the determination result is No, the process proceeds to S106, where the empty category is classified to identify the zero pronoun.

[0047] If the determination result in S105 is Yes, the next processing target is processed. When the processing for all argument types of all predicates in the sentence is completed, the process proceeds to S107.

[0048] In S107, the parameter update unit 116 determines whether the learning process has converged, and if the result is No, updates the learning parameters of the zero pronoun identification model (S108). Note that any method may be used to determine whether the learning process has converged. For example, convergence may be determined when the error between the estimation result and the correct data becomes equal to or smaller than a threshold, or when the process has been repeated a predetermined number of times.

[0049] If the determination result in S107 is Yes, the zero pronoun identification model learning unit 110 passes the learning parameters (model parameters) of the current zero pronoun identification model to the output unit 112, and the output unit 112 stores the learning parameters in the zero pronoun identification model DB 240.

[0050] It should be noted that the "model" in this embodiment is a neural network model, and when stored in a storage unit such as a DB, it is stored as data consisting of weighting parameters and the like.

[0051] Next, the operation of identifying (estimating) a zero pronoun from an input sentence will be described with reference to the flowchart in Fig. 4. In this process, it is assumed that the zero pronoun identification unit 230 reads a trained zero pronoun identification model from the zero pronoun identification model DB 240 and stores it in a storage unit such as a memory. The process by the zero pronoun identification unit 230 is performed using the zero pronoun identification model. Alternatively, it may be considered that the zero pronoun identification unit 230 is the zero pronoun identification model.

[0052] In S201, the input unit 211 inputs a sentence to the word segmentation unit 210. In S202, the word segmentation unit 210 segments the input sentence into words, and the predicate identification unit 220 identifies predicates in the sentence from the segmented word group.

[0053] The zero pronoun identification unit 230 repeats the processes of S203 to S206 for all argument types of all predicates in the sentence. In S203, the zero pronoun identification unit 230 finds the span that maximizes the score of the argument for the predicate, and in S204 finds the score for the argument of the predicate that is an empty category.

[0054] In S205, the zero pronoun identification unit 230 determines whether the score of the span is equal to or greater than the score of the empty category, and if the determination result is No, the process proceeds to S206, where the empty category is classified to identify the zero pronoun.

[0055] If the determination result in S205 is Yes, the next processing target is processed. When the processing for all argument types of all predicates in the sentence is completed, the process proceeds to S207.

[0056] In S207, the zero pronoun identification unit 230 passes the zero pronoun to the output unit 231, and the output unit 231 outputs the zero pronoun. Note that outputting the zero pronoun may be in any output form, such as outputting information indicating that there is a zero pronoun for the predicate or outputting the type of zero pronoun.

[0057] The processing operations of the zero pronoun identification system will be described in more detail below.

[0058] (On empty categories and zero pronouns) As mentioned above, in linguistics, especially in generative grammar, an empty category refers to an omitted pronoun (zero pronoun) called pro (or small pro), an unstated controlled subject called PRO (or big pro), and a null element (an element without a phonetic form) that expresses traces of movement in WH questions, relative clauses, etc. called T (or trace). Empty categories are also sometimes called gaps.

[0059] In this embodiment, the type of argument for a predicate, such as a subject, a direct object, or an indirect object, is called an argument type, and is represented by arg.

[0060] For a given predicate, the zero pronoun identification unit 115 / 230 predicts an argument for each argument type required by the predicate as a span of the input sentence, and if no span for that argument type is found, it determines that the argument of that argument type is an empty category.

[0061] When determining whether or not a zero pronoun is present, the zero pronoun identification unit 115 / 230 classifies the empty category into two values: whether or not it is a zero pronoun. In cases where zero pronouns are further subdivided into pro, speaker, hearer, etc., as in the NPCMJ described below, the empty category is classified into multiple values, such as an empty category other than a zero pronoun, pro, speaker, or hearer.

[0062] (Identifying Japanese zero pronouns based on question-answering) In this embodiment, we apply the method for realizing question answering using a trained language model [1] to Japanese zero pronoun identification. That is, we regard zero pronoun identification as a SQuAD-style question-answering in which predicates are questions, sentences are texts, and zero pronouns are answers.

[0063] First, the text to be input is divided into words by preprocessing using morphological analysis software or the like corresponding to the word division unit 113 / 210 and the predicate identification unit 114 / 220, and the predicates in the input sentence are identified.

[0064] When identifying zero pronouns in a sentence, the zero pronoun identification unit 115 / 230 (or the predicate identification unit 114 / 220 that provides input to the zero pronoun identification unit 115 / 230) creates an input sequence of '[CLS] question [SEP] sentence [SEP]'.

[0065] The question for input sentence x is constructed as follows:

[0066] { x qs-C:qs-1 , [S-PRED], x qs:qe , [E-PRED], x qe+1:qe+C} where C is the span x corresponding to the predicate. qs:qe The span x is the size (number of words) of the context window before and after the sentence. qs:qe In the above, qs is the start position of the span and qe is the end position of the span.

[0067] [S-PRED] and [E-PRED] are boundary markers that indicate the beginning and end of a predicate. For example, in the following example sentence 1, excluding the zero pronoun φ, there are five words: "daigaku, e, iki, mashita, ta."

[0068] (Example 1) (φ) I arrived at the university. (pro)-SBJ university at VB AX AXD In example sentence 1, "arrival" is the predicate, and when the context window C=1, x qs-C:qs-1 , is the word just before "arrival" and x qe+1:qe+C is the word immediately following "arrival," so the question is constructed as follows:

[0069] {"to", [S-PRED], "arrived", [E-PRED], "finished"} The following describes the configuration and operation of the zero pronoun identification model, which corresponds to the zero pronoun identification unit 115 / 230. The operation of the zero pronoun identification model described below may be realized entirely by a neural network, or may be realized by combining a neural network with a program other than a neural network.

[0070] (Term extraction and empty category detection using span prediction) In this embodiment, the zero pronoun identification model includes two independent output layers (linear layers) in addition to the trained language model in order to predict the start and end positions of specific types of arguments for predicates that are input to the zero pronoun identification model. Of the two independent output layers, one output layer predicts the start position of an argument, and the other output layer predicts the end position of the argument. The zero pronoun identification model extracts arguments and detects empty categories as follows.

[0071] In input sentence x, span x i:j The score where is the span of term type arg is shown in equation (1). As shown in equation (1), the score arg (i, j) is the term type arg whose starting position is the i-th word x i The probability that the term of term type arg ends at the jth word x j It is defined as the product of the probabilities that

[0072]

number

[0073]

number

[0074]

number

[0075]

number

[0076]

number

[0077] The loss for span prediction of a term related to one predicate during learning (training) is defined as the cross-entropy loss between the correct start position i' and end position j', as shown in equation (6).

[0078]

number

[0079] The probability that the empty category of the term type arg is the class class is defined as shown in the following equation (7). Equation (7) outputs the probability for each class.

[0080]

number

[0081] The loss for classification of empty categories during training is defined as the cross-entropy loss for the correct class label, as shown in equation (8).

[0082]

number

[0083] In other words, the zero pronoun identification model calculates the loss for the span prediction of the argument against the training data (ground truth data). span and the loss for classification of empty categories labelThe weighted sum of the above is used as the objective function (loss function) and training is carried out to optimize (minimize) this.

[0084]

number

[0085] (Example of hardware configuration) The zero pronoun identification system, device 100, and device 200 described above can all be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud. The zero pronoun identification system, device 100, and device 200 are collectively referred to as "devices."

[0086] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0087] Fig. 5 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 5 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected by a bus BS.

[0088] A program for realizing processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0089] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes the functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0090] (Effects of the embodiment) In order to verify the effectiveness of the technology according to this embodiment, an evaluation experiment was carried out, and the details of the experiment will be described below.

[0091] <About the data used in the evaluation experiment> In the evaluation experiments, we used the March 2020 version of the NPCMJ (NINJAL Parsed Corpus of Modern Japanese) created by the National Institute for Japanese Language and Linguistics for Japanese, and OntoNotes 5.0 created by the Linguistic Data Consortium (LDC) in the U.S. for Chinese. Figure 6 shows the number of documents, sentences, and predicates for NPCMJ and OntoNotes 5.0.

[0092] NPCMJ is a corpus of Japanese sentences with phrase structure trees containing empty category information. In this data, arguments of predicates containing zero pronouns are tagged with '-SBJ' (subject), '-OB1' (direct object), and 'OB2' (indirect object). Zero pronouns are also classified as pro, speaker, hearer, etc.

[0093] OntoNotes 5.0 is a corpus annotated with various linguistic information for English, Chinese, and Arabic. In this experiment, we used data annotated with parse trees of phrase structures containing empty categories for Chinese sentences. In this data, arguments for predicates containing zero pronouns are tagged with '-SBJ' (subject), '-OBJ' (object), and '-IO' (indirect object).

[0094] Figure 7 shows the number of zero pronouns in the training data of NPCMJ and OntoNotes, divided into subjects, direct objects, and indirect objects. The figures in parentheses indicate the percentage of zero pronouns appearing in each element. Regarding subjects, there is about a 20% omission rate in both Japanese and English. Direct objects and indirect objects are omitted less frequently than subjects. In particular, in the case of Chinese, the percentage of direct objects and indirect objects omitted is extremely small.

[0095] <About the experiment results> The pre-trained language models used were NICT BERT for Japanese and bert-base-chinese from HuggingFace Transformers for Chinese. Japanese sentences were tokenized using MeCab with the Juman dictionary, and Chinese sentences were tokenized using the BERT Tokenizer. Tokenization corresponds to the word segmentation mentioned above.

[0096] The hyperparameters are batch_size=16, learning_rate=3e-5, training_epoch=4, C=2, α=1.

[0097] As a baseline method, we used sequence classification based on BERT [1]. For a sentence, we solve the sequence labeling problem of BIOES format to identify predicates with specific zero pronouns. We also created different models for zero pronoun identification for different argument types.

[0098] Figure 8 shows the accuracy of term span prediction and zero pronoun identification in NPCMJ. ALL represents the sum of SBJ, OB1, and OB2. Compared to the baseline based on sequence labeling, the technology of the present invention improves zero pronoun identification accuracy by 4 points in F1.

[0099] Figure 9 shows the accuracy of zero pronoun identification by class in NPCMJ. The values in the table represent the values for the sum of SBJ, OB1, and OB2. Even when zero pronouns are subdivided into pro, speaker, and hearer, the technology of the present invention achieves higher zero pronoun identification accuracy than the baseline based on sequence labeling.

[0100] Figure 10 shows the accuracy of term span prediction and zero pronoun identification in OntoNotes 5.0. ALL represents the sum of SBJ, OBJ, and IO. Compared to the baseline based on sequence labeling, our method improves zero pronoun identification accuracy by 9 points in F1.

[0101] (Addendum) The following additional clauses are disclosed in relation to the above-described embodiment. (Additional note 1) Memory and at least one processor coupled to said memory; Including, The processor: Divide the input sentence into words, Identifying predicates from the segmented sentence; Using a zero pronoun identification model, which is a model in which an output layer is added to a trained language model, the span of an argument for a preceding descriptive word is calculated, and the score of the span is compared with the score of an empty category for the preceding descriptive word to determine whether the empty category exists. If the empty category exists, the presence or absence of a zero pronoun is determined by classifying the empty category. Zero pronoun identifier. (Additional note 2) The processor determines that the empty category exists if the score of the empty category is greater than the score of the span. 2. The zero pronoun identification device according to claim 1. (Additional note 3) the score of the span is the product of the probability that the term starts with the word at the start of the span and the probability that the term ends with the word at the end of the span; The score of the empty category is the product of the probability that the start position of the term is a special token in the input sequence to the zero pronoun identification model and the probability that the end position of the term is the special token. 3. The zero pronoun identification device according to claim 1 or 2. (Additional note 4) The processor classifies the empty category into one of a plurality of zero pronoun types and an empty category other than a zero pronoun. 4. A zero pronoun identification device according to any one of claims 1 to 3. (Additional note 5) The processor uses ground truth data to update the parameters of the zero pronoun identification model so as to minimize a weighted sum of a loss related to span prediction and a loss related to empty categories. 5. A zero pronoun identification device according to any one of claims 1 to 4. (Additional note 6) 1. A computer processor-implemented method for zero pronoun identification, comprising: a word segmentation step for segmenting an input sentence; a predicate identification step of identifying a predicate from the sentence divided into words by the word division step; a zero pronoun identification step of determining the span of an argument for a preceding descriptive word using a zero pronoun identification model, which is a model in which an output layer is added to a trained language model, and comparing the score of the span with the score of an empty category for the preceding descriptive word to determine whether the empty category exists; and if the empty category exists, classifying the empty category to determine the presence or absence of a zero pronoun; A zero pronoun identification method comprising: (Additional note 7) A non-transitory storage medium storing a program executable by a computer to perform a zero pronoun identification process, The zero pronoun identification process includes: Divide the input sentence into words, Identifying predicates from the segmented sentence; Using a zero pronoun identification model, which is a model in which an output layer is added to a trained language model, the span of an argument for a preceding descriptive word is calculated, and the score of the span is compared with the score of an empty category for the preceding descriptive word to determine whether the empty category exists. If the empty category exists, the presence or absence of a zero pronoun is determined by classifying the empty category. Non-transitory storage medium. (References) [1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL-2019, pp.4171-4186, 2019. [2] Pranav Rajpurkar, Robin Jia, and Percy Liang. Know what you don't know: Unanswerable questions for squad. In Proceedings of the ACL-2018, pp. 784-789, 2018. [3] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP-2016, pp. 2383-2392, 2016. [4] Linfeng Song, Kun Xu, Yue Zhang, Jianshu Chen, and Dong Yu. Zpr2: Joint zero pronoun recovery and resolution using multi-task learning and bert. In Proceedings of ACL-2020, pp. 5429-5434,2020. [5] Shunsuke Takeno, Masaaki Nagata, and Kazuhide Yamamoto. Empty category detection using path features and distributed case frames. In Proceedings of EMNLP-2015, pp. 1335-1340, 2015. [6] Wei Wu, Fei Wang, Arianna Yuan, Fei Wu, and Jiwei Li. Corefqa: Coreference resolution as query-based span prediction. In Proceedings of ACL-2020, pp. 6953-6963, 2020. [7] Bing Xiang, Xiaoqiang Lue, and Bowen Zhou. Enlisting the ghost: Modeling empty categories for machine translation. In ACL-2013, pp. 822-831, 2013.

[0102] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0103] 110 Zero pronoun identification model learning section 111 Input section 112 Output section 113 Word Segmentation 114 Predicate Identification Unit 115 Zero Pronoun Identification Section 116 Parameter Update Unit 120 Zero Pronoun Identification Training Data DB 130 pre-trained multilingual model database 210 Word Segmentation 211 Input section 220 Predicate Identification Unit 230 Zero Pronoun Identification Section 231 Output section 240 Zero Pronoun Identification Model DB 1000 Drive Device 1001 Recording media 1002 Auxiliary storage 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output Device

Claims

1. a word segmentation unit that segments an input sentence; a predicate identification unit that identifies a predicate from the sentence that has been word-divided by the word division unit; a zero pronoun identification model that is a model in which an output layer is added to a trained language model, The zero pronoun identification model receives an antecedent descriptive word and the sentence as input, calculates an argument span for the antecedent descriptive word, and compares the score of the span with the score of an empty category for the antecedent descriptive word to determine whether the empty category exists; if the empty category exists, determines whether the empty category is a zero pronoun or not, and outputs the determination result; the score of the span is the product of the probability that the term starts with the word at the start of the span and the probability that the term ends with the word at the end of the span; The score of the empty category is the product of the probability that the start position of the term is a special token in the input sequence to the zero pronoun identification model and the probability that the end position of the term is the special token. Zero pronoun identifier.

2. The zero pronoun identification model determines that the empty category exists if the score for the empty category is greater than the score for the span. The zero pronoun identification device of claim 1 .

3. The zero pronoun identification model classifies the empty category into one of a plurality of classes of zero pronouns and an empty category other than zero pronouns.

3. The zero pronoun identification device according to claim 1 or 2.

4. a parameter update unit that updates the parameters of the zero pronoun identification model using ground truth data so as to minimize the weighted sum of the loss related to span prediction and the loss related to the empty category; 4. The zero pronoun identification device according to claim 1, further comprising:

5. A zero pronoun identification method executed by a computer equipped with a zero pronoun identification model, which is a model in which an output layer is added to a trained language model, comprising: a word segmentation step in which the computer segments an input sentence; a predicate identification step in which the computer identifies a predicate from the sentence that has been word-divided by the word division step; the zero pronoun identification model in the computer receives an antecedent descriptive word and the sentence as input, calculates a span of an argument for the antecedent descriptive word, and compares the score of the span with the score of an empty category for the antecedent descriptive word to determine whether the empty category exists; and if the empty category exists, determines whether the empty category is a zero pronoun or not, and outputs a determination result; the score of the span is the product of the probability that the term starts with the word at the start of the span and the probability that the term ends with the word at the end of the span; The score of the empty category is the product of the probability that the start position of the term is a special token in the input sequence to the zero pronoun identification model and the probability that the end position of the term is the special token. Zero pronoun identification method.

6. A program for causing a computer to function as each unit of the zero pronoun identification device according to any one of claims 1 to 4 and the zero pronoun identification model.

Images