Estimation device and learning device

Abstract

This estimation device comprises: an input unit for inputting a first text span and a second text span; and a generation unit having an encoder / decoder model for estimating a connection word and a discourse relationship label between the first text span and the second text span on the basis of the first text span and the second text span, and outputting the connection word and the discourse relationship label.

Description

Estimation device and learning device

[0001] The present invention relates to techniques for inferring discourse relations.

[0002] Generally, when people read a text, they proceed while understanding the semantic connections between a sentence (or clause) and the surrounding sentences (or clauses). Such connections are called discourse relations. In other words, discourse relations show how two sentences (or clauses) are logically / structurally connected.

[0003] Discourse relations may or may not explicitly use conjunctions such as "but." Discourse relations without explicit conjunctions are called implicit discourse relations. In the following, we will focus on implicit discourse relations.

[0004] Implicit discourse relation recognition is a task of estimating the discourse relation between two sentences (or clauses) that do not have an explicit conjunction between them. Prior art for implicit discourse relation recognition includes techniques using classifiers, as disclosed in, for example, Non-Patent Documents 1 and 2.

[0005] Wei Xiang, Bang Wang, Lu Dai, and Yijun Mo. 2022. Encoding and Fusing Semantic Connection and Linguistic Evidence for Implicit Discourse Relation Recognition. In Findings of the Association for Computational Linguistics: ACL 2022, pages 3247-3257, Dublin, Ireland. Association for Computational Linguistics.Wanqiu Long and Bonnie Webber. 2022. Facilitating Contrastive Learning of Discourse Relational Senses by Exploiting the Hierarchy of Sense Relations. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 10704-10716, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.

[0006] As described above, conventional techniques for implicit discourse relation recognition often use classifiers. However, conventional techniques using classifiers have low accuracy in estimating discourse relations that are not present in the training data or that are rarely present in the training data. Furthermore, in discourse relation recognition, it is desirable to estimate both discourse relations and conjunctions.

[0007] The present invention has been made in view of the above points, and has as its object to provide a technique for estimating both discourse relations and conjunctions with high accuracy in implicit discourse relation recognition.

[0008] According to the disclosed technique, there is provided an estimation device including: an input unit that inputs a first text span and a second text span; and a generation unit having an encoder-decoder model that estimates conjunctions and discourse relation labels between the first text span and the second text span based on the first text span and the second text span, and outputs the conjunctions and the discourse relation labels.

[0009] According to the disclosed technology, it is possible to estimate both discourse relations and conjunctions with high accuracy in implicit discourse relation recognition.

[0010] FIG. 1 is a diagram for explaining an example of implicit discourse relation recognition. FIG. 2 is a diagram for explaining an example of implicit discourse relation recognition using a classifier. FIG. 3 is a diagram for explaining an example of implicit discourse relation recognition using a classifier. FIG. 4 is a diagram showing an example of the configuration of an estimation device 100 in an embodiment of the present invention. FIG. 5 is a diagram for explaining an example of the operation of a generation unit 130. FIG. 6 is a diagram for explaining an example of the operation of the generation unit 130. FIG. 7 is a diagram showing an example of the configuration of a learning device 200 in an embodiment of the present invention. FIG. 8 is a diagram for explaining an example of the operation of a learning unit 230. FIG. 9 is a diagram showing experimental results. FIG. 10 is a diagram showing experimental results. FIG. 11 is a diagram showing an example of the hardware configuration of the device.

[0011] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0012] In the following, an example of English is shown as a specific example of inferring discourse relations and conjunctions, but this is just one example. Languages ​​to which the technology according to the present invention can be applied are not limited to English, and the technology according to the present invention can be applied to any language.

[0013] Below, examples of prior art for implicit discourse relation recognition and problems therewith will be described first, and then the technology according to this embodiment will be described.

[0014] (Implicit Discourse Relation Recognition) First, we will explain implicit discourse relation recognition (IDRR), which is the subject of this embodiment. As described above, implicit discourse relation recognition is a task of inferring the discourse relation between two sentences (or clauses) that do not have an explicit conjunction between them.

[0015] A sentence (or clause) targeted in discourse relation recognition may be called an argument, a text span, or a discourse unit. In the following, we will use the terms argument or text span as appropriate.

[0016] The Penn Discourse Tree Bank (PDTB) is a well-known discourse corpus that is often used in discourse relation recognition. The PDTB is a corpus of English newspaper articles in which discourse relation labels are assigned between adjacent text spans.

[0017] More specifically, discourse relation labels in PDTB are expressed in three hierarchical levels. The top level (Level 1) indicates broad categories such as Comparison, the second level (Level 2) specifies types such as Concession, and the third level (Level 3) indicates more detailed relationships between arguments, such as Arg2-as-denier. The current PDTB is more specifically called PDTB3, and in PDTB3 there are four types of Level 1 labels and 22 types of Level 2 labels.

[0018] A period is used to represent each level. For example, labels for three levels are expressed as "Comparison.Concession.Arg2-as-denier." The most appropriate conjunction between arguments is provided as metadata. In this embodiment, discourse relation labels for the first and second levels are estimated, such as "Comparison.Concession."

[0019] An example of implicit discourse relation recognition will be described with reference to Fig. 1. In the example of Fig. 1, Argument1 and Argument2 are in a discourse relation connected by the conjunction "however." However, "however" is not explicitly stated. In this embodiment, an estimation device 100, which will be described later, estimates the discourse relation label "Comparison.Concession" between Argument1 and Argument2 and the conjunction "however."

[0020] (Regarding Implicit Discourse Relation Recognition Using a Classifier) ​​Conventionally, a discourse relation label between two arguments has been estimated using a classifier. Examples are shown in FIGS. 2 and 3. As shown in FIG. 2, Argument 1 and Argument 2 are input to Classifier 1. Classifier 1 estimates and outputs "Comparison.Concession," which is the discourse relation label between Argument 1 and Argument 2. As Classifier 1, for example, a pre-trained large-scale language model such as RoBERTa is used. Note that it is also possible to estimate conjunctions using Classifier 1.

[0021] Figure 3 shows Figure 2 in more detail. As shown in Figure 3, Argument 1 and Argument 2 are tokenized, and the resulting tokens are input to classifier 1 (specifically, the encoder). In classifier 1, each token is converted into an embedding vector in the embedding layer, and the embedding vector is processed by the encoder. Specifically, classifier 1 is trained to estimate discourse relation labels from the embedding vectors of CLS tokens in the final layer, and the trained classifier 1 estimates discourse relation labels. The classification problem using classifier 1 becomes a 4-value classification problem when estimating level 1 labels, and a 22-value classification problem when estimating level 2 labels.

[0022] (About the Issues) As described above, conventional discourse relation recognition techniques use classifiers to estimate discourse relations. However, this technique has low accuracy in estimating discourse relations that are not present in training data or that are rarely present in training data. Furthermore, downstream tasks such as machine translation require processing both discourse relations and conjunctions, so it is desirable to estimate both discourse relation labels and conjunctions in discourse relation recognition.

[0023] The following describes an estimation device 100 and a learning device 200 for solving the above problems. In this embodiment, the estimation device 100 and the learning device 200 are separate devices, but they may be the same device. For example, the estimation device 100 may include the functions of the learning device 200, or the learning device 200 may include the functions of the estimation device 100.

[0024] (Configuration Example of Estimation Apparatus 100) Fig. 4 shows a configuration example of the estimation apparatus 100 according to this embodiment. As shown in Fig. 4, the estimation apparatus 100 includes an input unit 110, a division processing unit 120, a generation unit 130, an output unit 140, and a trained model DB 150.

[0025] The input unit 110 is a target for estimating conjunctions and discourse relation labels. <mask>The division processing unit 120 receives two arguments (Argument 1 and Argument 2) connected by the <mask>, Argument2 are divided into tokens. Note that the string connecting Argument1 and Argument2 is <mask>Using is one example. <mask>A string other than "" may be used as the string connecting Argument1 and Argument2.

[0026] The generation unit 130 receives Argument1 and Argument2. <mask>The generation unit 130 receives a string of tokens connected by the above formula, and outputs "Argument1, conjunction (discourse relation label), Argument2." The conjunction and discourse relation label in "conjunction (discourse relation label)" are the conjunction and discourse relation label estimated between Argument1 and Argument2.

[0027] The output unit 140 outputs "Argument1, conjunction (discourse relation label), Argument2" obtained by the generation unit 130 to the outside.

[0028] In this embodiment, BART (Bidirectional and Auto-Regressive Transformers) is used as the generation unit 130. BART is a model in which a Transformer encoder-decoder model is pre-trained with a large amount of text data. This pre-trained BART is fine-tuned with a task of estimating conjunctions (discourse relation labels), thereby obtaining an encoder-decoder model that functions as the generation unit 130. Note that the generation unit 130 may have an encoder-decoder model, or the generation unit 130 itself may be an encoder-decoder model.

[0029] The trained model DB 150 stores trained parameters of the encoder-decoder model, and the generator 130 reads out these parameters to construct the encoder-decoder model.

[0030] In this embodiment, BART is used as the encoder-decoder model in the generation unit 130, but this is just an example. An encoder-decoder model other than BART may be used in the generation unit 130.

[0031] The encoder-decoder model used in this embodiment is a well-known model, such as the Transformer encoder-decoder model described in the paper "Attention is all you need." Therefore, only an outline of the operation of the encoder-decoder model will be described below.

[0032] (Example of Operation of Generator 130) An example of operation of the generator 130 (encoder-decoder model) during estimation will be described with reference to Figures 5 and 6. Note that in the following description, for convenience of explanation, the description of tokenization may be omitted in some cases.

[0033] In the example in Figure 5, "American Brands just had a different approach," Mr. Wathen says. <mask>"Their approach didn't work" <mask>The sentence connected with is input to the generation unit 130.

[0034] The generation unit 130 generates and outputs "American Brands "just had a different approach," Mr. Wathen says. however (Comparison.Concession) "Their approach didn't work"" from the input sentence. however (Comparison.Concession) is a conjunction (discourse relation label). That is, the generation unit 130 outputs Argument1, a conjunction and discourse relation label, and Argument2. This is because, in the input sentence, <mask>A conjunction and a discourse relation label are inserted.

[0035] In this embodiment, the conjunction and the discourse relation label are output in the form of a conjunction connected to a discourse relation label in parentheses, such as "conjunction (discourse relation label)." This is because the training data of the pre-trained encoder-decoder model contains many expressions such as "a certain expression (its explanation)."

[0036] However, outputting a conjunction and a discourse relation label in the form of "conjunction (discourse relation label)" is just an example, and a conjunction and a discourse relation label may be output in a form other than "conjunction (discourse relation label)."

[0037] 6 is a diagram showing in more detail the operation of the generation unit 130. FIG. 6 shows the encoder 131 and decoder 132 of the encoder-decoder model in the generation unit 130.

[0038] As shown in Fig. 6, the tokens obtained by tokenizing Argument1 and Argument2, the CLS token, and the MASK token are input to the encoder 131. The MASK token is <mask>is the token corresponding to

[0039] In the encoder 131, each token is converted into an embedding vector (Emb), and processing is performed on the embedding vector. The encoder 131 performs calculations using an attention mechanism on the input embedding vector sequence to generate and output a context-aware embedding vector sequence. The output embedding vector sequence is input to the decoder 132.

[0040] Furthermore, a CLS token (or an EOS token) is first input to the decoder 132, and the output tokens from the decoder 132 are sequentially input to the decoder 132. As a result, the decoder 132 generates a token sequence in an autoregressive manner. This operation itself is the same as the operation of a general Transformer encoder-decoder model.

[0041] In FIG. 6, the "LABEL" output from the decoder 132 corresponds to a conjunction and a discourse relation label (specifically, a conjunction (discourse relation label)).

[0042] 7 shows an example of the configuration of the learning device 200 according to this embodiment. As shown in FIG. 7, the learning device 200 includes an input unit 210, a division processing unit 220, a learning unit 230, an output unit 240, and a learning model DB 250.

[0043] The input unit 210 inputs a training dataset as training data. The dataset contains multiple pieces of data for "Argument1, conjunction, discourse relation label, Argument2." The segmentation unit 220 is the same as the segmentation unit 120 in the estimation device 100, and segments the input text into tokens.

[0044] The learning unit 230 includes an encoder-decoder model to be learned, and uses learning data to learn the encoder-decoder model (optimize parameters). Parameters during learning are stored in the learning model DB 250. Note that the encoder-decoder model to be learned may not be a component of the learning unit 230. In other words, the learning unit 230 may include a function for learning the encoder-decoder model to be learned, but may not include the encoder-decoder model to be learned.

[0045] The encoder-decoder model is the encoder-decoder model described in the section on the estimation device 100. That is, the encoder-decoder model is the BART, which is a pre-trained encoder-decoder model. The training unit 230 trains the encoder-decoder model so as to perform the task of estimating conjunctions and discourse relation labels.

[0046] More specifically, the learning unit 230 learns that the encoder-decoder model uses Argument1 and Argument2 as shown in FIG. <mask>The model is trained to generate the correct answer "Argument1, a conjunction and discourse relation label, and Argument2" from a sentence connected by the above. The model training itself can be performed using an existing technique such as backpropagation. In other words, the training unit 230 trains Argument1 and Argument2 by <mask>The parameters of the encoder-decoder model are adjusted so that the output sequence from the encoder-decoder model to which the connected sentences are input matches the correct sequence "Argument1, conjunction and discourse relation label, and Argument2" (so that the error is minimized).

[0047] After the learning is completed, the learned parameters are stored in the learning model DB 250. For example, the output 230 reads the learned parameters from the learning model DB 250 and stores the learned parameters in the trained model DB 150 of the estimation device 100.

[0048] (Example of Operation of Learning Unit 230) An example of operation of the learning unit 230 will be described with reference to Fig. 8. Fig. 8 shows the encoder 131 and decoder 132 in the encoder-decoder model to be learned. As shown in Fig. 8, each token obtained by tokenizing Argument1 and Argument2, a CLS token, and a MASK token are input to the encoder 131.

[0049] In the encoder 131, each token is converted into an embedding vector, and processing is performed on the embedding vector. The encoder 131 performs calculations using an attention mechanism on the input embedding vector sequence to generate and output a sequence of embedding vectors that take context into consideration. The output embedding vector sequence is input to the decoder 132.

[0050] Furthermore, a CLS token (or an EOS token) is first input to the decoder 132, and then a sequence to which tokens of training data (correct answer data) are sequentially added is sequentially input to the decoder 132. The training unit 130 adjusts the parameters of the encoder-decoder model so that the sequence output from the decoder 132 matches the correct answer data. This operation itself is the same as the operation during training of a general Transformer encoder-decoder model. Furthermore, as with general fine tuning, pre-trained parameters are used as the initial values ​​of the encoder-decoder model.

[0051] (Other Examples) In the examples described above, the generation unit 130 of the estimation device 100 generates “Argument1, <mask>, Argument2" and outputs "Argument1, conjunction and discourse relation label, Argument2". <mask>The encoder-decoder model is trained so that it takes "Argument1, conjunction and discourse relation label, Argument2" as input and outputs "Argument1, conjunction and discourse relation label, Argument2".

[0052] However, the above operation is an example. <mask>, Argument2" as input and output "conjunction and discourse relation label." <mask>, Argument2" as input and output "conjunction and discourse relation label."

[0053] In the above case, "conjunction and discourse relation label" may be in the form of "conjunction (discourse relation label)."

[0054] (Experimental Results) Experiments were conducted to confirm the effects of the technology according to this embodiment, and the results are shown in FIGS.

[0055] FIG. 9 shows the accuracy of discourse relation label estimation, etc. FIG. 10 shows the accuracy of conjunction estimation, etc. In FIGS. 9 and 10, "Prior Research 1" indicates a classifier using BERT, as disclosed in Non-Patent Document 1. "Prior Research 2" indicates a classifier using object learning and RoBERTa, as disclosed in Non-Patent Document 2. "Classifier using RoBERTa" indicates the classifier described using FIGS. 2 and 3. "Proposed Method" indicates the technology according to this embodiment (i.e., the estimation device 100).

[0056] Accuracy, Macro-F1, and Weighted-F1 are evaluation indices commonly used in classification problems, with higher values ​​indicating better evaluation. Accuracy (correct answer rate) is the number of correct answers in the prediction results divided by the number of data. Macro-F1 is the F1 value used in multi-class classification and is the average of the F1 values ​​calculated for each class in multi-class classification. The F1 value is the harmonic mean of precision and recall. Weighted-F1 is the average of the F1 values ​​weighted by each label.

[0057] As can be seen from Figures 9 and 10, the "proposed method" is able to simultaneously estimate both conjunctions and discourse relation labels with good accuracy. In particular, as shown in Figure 10, the accuracy of conjunction estimation is significantly improved compared to the conventional technique.

[0058] (Hardware Configuration Example) Any of the devices described in this embodiment (the estimation device 100 and the learning device 200) can be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0059] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0060] Fig. 11 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 11 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected via a bus B. The computer may further include a GPU.

[0061] The program that realizes the processing on the computer is provided by a recording medium 1001, such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0062] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0063] (Summary, Effects, etc. of the Embodiments) As described above, the technology described in the present embodiment makes it possible to estimate both discourse relations and conjunctions with high accuracy by using an encoder-decoder model used in sentence generation, etc. In particular, the accuracy of conjunction estimation can be significantly improved compared to conventional techniques.

[0064] The following additional notes are provided regarding the above-described embodiments.

[0065] <Supplementary Notes> (Supplementary Item 1) An estimation device including a memory and at least one processor connected to the memory, wherein the processor inputs a first text span and a second text span, and estimates a conjunction and a discourse relation label between the first text span and the second text span based on the first text span and the second text span by using an encoder-decoder model, and outputs the conjunction and the discourse relation label. (Supplementary Item 2) The estimation device according to Supplementary Item 1, wherein the processor estimates the conjunction and the discourse relation label between the first text span and the second text span by using the encoder-decoder model, and outputs the first text span, the conjunction, the discourse relation label, and the second text span. (Supplementary Item 3) The estimation device according to Supplementary Item 1 or 2, wherein the processor outputs the conjunction and the discourse relation label in the form of a conjunction (discourse relation label) by using the encoder-decoder model. (Supplementary Item 4) A learning device comprising: a memory; and at least one processor connected to the memory, wherein the processor inputs a first text span and a second text span; and trains the encoder-decoder model to which the first text span and the second text span are input, so that the encoder-decoder model estimates conjunctions and discourse relation labels between the first text span and the second text span, and outputs the conjunctions and the discourse relation labels.

[0066] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims.

[0067] 1 Classifier 100 Estimation device 110 Input unit 120 Segmentation processing unit 130 Generation unit 140 Output unit 150 Trained model DB 200 Learning device 210 Input unit 220 Segmentation processing unit 230 Learning unit 240 Output unit 250 Trained model DB 1000 Drive device 1001 Recording medium 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device< / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask> < / mask>

Claims

1. An estimation device comprising: an input unit for inputting a first text span and a second text span; and a generation unit having an encoder-decoder model for estimating conjunctions and discourse relation labels between the first text span and the second text span based on the first text span and the second text span, and outputting the conjunctions and the discourse relation labels.

2. The estimation device according to claim 1, wherein the encoder-decoder model estimates the conjunction and the discourse relation label between the first text span and the second text span, and outputs the first text span, the conjunction, the discourse relation label, and the second text span.

3. The estimation device according to claim 1 or 2, wherein the encoder-decoder model outputs the conjunction and the discourse relation label in the form of a conjunction (discourse relation label).

4. A learning device comprising: an input unit for inputting a first text span and a second text span; and a learning unit for training the encoder-decoder model to which the first text span and the second text span are input, so that the encoder-decoder model estimates conjunctions and discourse relation labels between the first text span and the second text span, and outputs the conjunctions and the discourse relation labels.

Images