Evaluation device, evaluation method, and program

Abstract

This evaluation device comprises: a substructure acquisition unit that acquires a first substructure that is a substructure of first text and a second substructure that is a substructure of second text; an association unit that obtains an association for each substructure between one or more first substructures and one or more second substructures; and a similarity calculation unit that calculates a similarity between the first text and the second text on the basis of a calculated value that indicates a relationship between the associated first substructure and second substructure.

Description

[Technical Field]

[0001] This invention relates to a technique for calculating the similarity between two texts. [Background technology]

[0002] One prior art for calculating the similarity between two texts is the technique disclosed in Non-Patent Document 1. Non-Patent Document 1 discloses a method for determining the correspondence between tokens in texts based on the cosine similarity of the token embedding vectors of the texts, and then calculating the similarity between the texts based on that correspondence. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] However, prior art such as Non-Patent Document 1 unfairly assigns a low similarity score to expressions that have similar meanings despite having different surface information. In other words, prior art has the problem of not being able to properly evaluate the similarity between texts.

[0005] This invention has been made in view of the above points, and aims to provide a technology for appropriately evaluating the similarity between texts. [Means for solving the problem]

[0006] According to the disclosed technology, for the first text Subtree or path obtained from the syntax tree a first partial structure which is Subtree or path obtained from the syntax tree and a partial structure acquisition unit that acquires a second partial structure which is among one or more first partial structures and one or more second partial structures, Based on the similarity between the first substructure and the second substructure, a pairing unit that determines a pairing for each partial structure, and a similarity calculation unit that calculates the similarity between the first text and the second text based on the Similarity calculated value between the paired first partial structure and second partial structure An evaluation device is provided.

Advantages of the Invention

[0007] According to the disclosed technology, a technology for appropriately evaluating the similarity between texts is provided.

Brief Description of the Drawings

[0008] [Figure 1] It is a configuration diagram of the evaluation device 100. [Figure 2] It is a flowchart for explaining the operation of the evaluation device 100. [Figure 3] It is a diagram for explaining a specific example of partial structure acquisition. [Figure 4] It is a diagram for explaining a specific example of partial structure acquisition. [Figure 5] It is a diagram showing an example of a phrase structure tree with phrase structure labels. [Figure 6] It is a diagram showing an example of a dependency structure tree with edge labels. [Figure 7] It is a diagram showing an example of the hardware configuration of the evaluation device 100. <00001​​​​​​​​​Hereinafter, embodiments of the present invention (this embodiment) will be described with reference to the drawings. The embodiments described below are merely examples, and the embodiments to which the present invention is applied are not limited to the embodiments described below.

[0010] In this specification, references are indicated by numbers such as [1], and the names of the references corresponding to the numbers are listed at the end of the specification.

[0011] The following describes an evaluation method, which is a technology according to this embodiment, for determining the similarity between two texts based on the similarity between the substructures of these texts. However, to facilitate understanding of this technology, the background technology and problems will be explained first.

[0012] In the following explanation of the background technology and challenges, the content disclosed in the references is publicly known, but the explanation of the content disclosed in the references and the explanation of the challenges are not publicly known.

[0013] (Background technology) A representative method for evaluating the similarity between two texts is the BLEU[1] method, which evaluates similarity based on matching surface information. BLEU is the de facto standard for automatic evaluation metrics in machine translation, and it evaluates similarity based on the degree of agreement between word n-grams between the translated text and the reference translation.

[0014] In recent years, many similarity measures based on contextual embedding vectors of words and sentences using large-scale pre-trained models have been proposed. BERTScore[2] proposes a method that determines the correspondence between tokens in texts based on the cosine similarity of the embedding vectors of tokens in texts, and then calculates the similarity between texts based on that correspondence. In this case, the pre-trained model used to calculate the embedding vectors does not undergo fine tuning, and therefore does not require training data.

[0015] On the other hand, BLEURT[3] proposes a method for calculating similarity by adding a linear layer on top of the [CLS] vector of the final layer of BERT[4]. In this case, the parameters of BERT and the linear layer are fine-tuned using manually annotated translation evaluation data, so manually created training data is required.

[0016] Furthermore, methods based on paraphrasing generation using sequence-to-sequence (Seq2Seq) models have also been proposed. Prism[5] proposes a method that uses a paraphrasing generator created as a Seq2Seq model to generate paraphrases from a reference translation to a translated sentence, or vice versa, and uses the score of these paraphrases as an evaluation metric. In this case, the paraphrasing generator is created by training a multilingual translation model, thereby generating paraphrases between the same languages ​​within a framework similar to the zero-shot translation task. Prism also supports 39 languages ​​and has achieved a correlation of the same or higher than all other evaluation metrics in the segment-level evaluation of the WMT 2019 metric shared task.

[0017] (assignment) Methods based on surface information matching have the problem of unfairly assigning low similarity scores to expressions that have similar meanings despite having different surface information (or expressions that have different meanings despite having similar surface information). Methods based on sentence embedding vectors and paraphrase generation have been proposed to address this problem.

[0018] However, these methods do not utilize the structural information inherent in the text. Texts inherently contain structures that reflect the meaning of words, grammatical functions, and the roles of phrases, such as phrase structure, dependency structure, and rhetorical structure. While the structure of a text is thought to be a useful clue in capturing the differences between texts, addressing the problem of unfairly assigning low similarity scores to expressions that have similar meanings despite having different surface appearances, it remains unclear whether structural information is useful in text similarity scales.

[0019] Regarding methods based on sentence embedding vectors, analyses have shown that pre-trained models retain structural information as latent representations [6], but this does not explicitly use structural information in calculating similarity measures.

[0020] (Summary of the embodiment) In this embodiment, the evaluation device 100 uses the structural information of the texts to obtain the substructures of two texts and calculates the similarity between the two texts based on the similarity between the corresponding substructures. More specifically, the evaluation device 100 enumerates the substructures of each of the two texts for which similarity is to be sought, then determines the correspondence between the substructures of the two texts, and finally calculates the similarity between the texts using the vector similarity between the substructures based on the correspondence.

[0021] (Example of device configuration) Figure 1 shows an example of the configuration of the evaluation device 100 in this embodiment. As shown in Figure 1, the evaluation device 100 has an input unit 150, a structure acquisition unit 110, a partial structure acquisition unit 120, a correspondence unit 130, a similarity calculation unit 140, and an output unit 160.

[0022] The operation of the evaluation device 100 will be explained with reference to the flowchart in Figure 2. In S101, text x and text y are input from the input unit 150. Text x and text y each consist of multiple elements. In this embodiment, the elements that make up the text are words, but are not limited to words. "Text" may also be called a "sentence".

[0023] In S102, the structure acquisition unit 110 acquires the structural information of text x and text y. In S103, the partial structure acquisition unit 120 acquires the partial structure from the structural information of text x and the structural information of text y.

[0024] In S104, the correspondence unit 150 calculates the correspondence between the substructure of text x and the substructure of text y.

[0025] In S105, the similarity calculation unit 140 calculates the similarity between text x and text y based on the correspondence obtained in S104. In S106, the output unit 160 outputs the similarity.

[0026] Alternatively, the evaluation device 100 may not include the structure acquisition unit 110 by incorporating the functions of the structure acquisition unit 110 into the partial structure acquisition unit 120. Furthermore, the evaluation device 100 may also not include the structure acquisition unit 110 by calculating the structural information obtained by the structure acquisition unit 110 outside of the evaluation device 100 and inputting the structural information from an external source.

[0027] The following will explain each of the above steps in more detail. Specific examples will be used as appropriate.

[0028] (S101, S102: Input, Structure acquisition) In S101, the input unit 150 outputs the text x=〈x1,···,x i > and text y=〈y1,···,y j > is entered. Each element of text x, which consists of i elements, is a word. Similarly, each element of text y, which consists of j elements, is a word.

[0029] Furthermore, from the input unit 150, the embedding vector of text x <vector x1, ..., vector x i > and the embedding vector of text y <vector y1,..., vector y j > is input. Note that the embedded vector may be calculated from the text x and y by the input unit 150 or the structure acquisition unit 110.

[0030] Note that in the images of mathematical formulas inserted under symbols such as "Equation 1," vectors are shown in bold. However, in the text of the specification, vectors should be written as "vector x." This rule does not apply if it is clear from the context that it is a vector.

[0031] In S102, the structure acquisition unit 110 acquires the structural information of text x and text y by using a syntactic analysis method.

[0032] The structural information to be acquired is not limited to any specific type. Any structural information that allows the text to be represented as a directed graph (including tree structures) may be used.

[0033] Furthermore, the structural information in this embodiment may be any information that includes at least the text divided into predetermined units and information regarding the relationships between those elements. The structural information may also include label information indicating the type of element and the type of relationship between elements, or vector information used during structure extraction (e.g., during phrase structure analysis using a neural network).

[0034] Specifically, structural information such as discourse structure, phrase structure, dependency structure, Universal Dependencies, and AMR (Abstract Meaning Representation) can be used. Universal Dependencies are disclosed, for example, in "Joakim Nivre. Towards a universal grammar for natural language processing. In Computational Linguistics and Intelligent Text Processing, pp. 3-16. Springer, 2015."

[0035] In this embodiment, it is assumed that a phrase structure tree or a dependency structure tree, which is frequently used among structural information and can be obtained with very high accuracy by the structure acquisition unit 110 (for example, an existing structural analyzer), will be acquired as structural information.

[0036] (S103: Obtain partial structure) In S103, for each of the text x and the text y, the partial structure acquisition unit 120 acquires (enumerates) a partial structure to be the target for obtaining the correspondence relationship from the structure information calculated in S102.

[0037] For example, when using a phrase structure tree as the structure information, the partial structure acquisition unit 120 acquires all the subtrees for the phrase structure tree. Each subtree of all the subtrees is one partial structure. When using a dependency structure tree as the structure information, since the subtrees do not include partial structures consisting only of individual words, etc., the partial structure acquisition unit 120 acquires, in addition to the subtrees, each node and the path from each leaf node to each node as partial structures.

[0038] Let the partial structure of text x be s m , and the partial structure s m Define the sequence of word indices included in as idx(s m ). The partial structure acquisition unit 120 calculates the vector s m corresponding to the partial structure s as m as

[0039]

Number

[0040] Here, aggr(s m ) can be any function as long as it is a function for calculating the vector corresponding to s m , and it does not necessarily have to be the average of the word vectors included in the partial structure as in formula (1). The partial structure acquisition unit 120 performs the same calculation for the partial structure t n of text y as in the case of text x.

[0041] <Specific example of partial structure acquisition> Here, we will explain a specific example of obtaining substructures from a given text. Figure 3 shows an example of extracting all subtrees from a phrase structure tree of a text containing words z1 to z6.

[0042] A phrase structure tree is a tree structure (syntactic tree) in which symbols representing the type of phrase in a text, such as noun phrases (NP), verb phrases (VP), prepositional phrases (PP), and sentences (S), are used as internal nodes (branching nodes), and each word in the text is used as a leaf node.

[0043] In the example in Figure 3, the symbols for the type of clause in each internal node are omitted, but for example, z1 and z2 constitute NP, and z1 to z6 constitute S.

[0044] In the example in Figure 3, each subtree is represented by the span (word length) of a word (word subscript). The span can be a single word. For example, (1,1) is a subtree consisting of one word z1. Also, (1,6) is a subtree containing six words z1 to z6.

[0045] Figure 4 shows an example of obtaining a substructure from a dependency structure tree of a text containing words z1 to z6.

[0046] A dependency structure tree is a tree structure (syntax tree) that represents the dependencies between words in a text. Each node represents a word, and an arrow is drawn from the root (parent) to the leaf (child). In the example in Figure 4, each node (z1, etc.) is extracted as a substructure of length (span) 1, the paths from the leaf nodes to each node are extracted as substructures of lengths 2 and 3, and the subtree rooted at z6 is extracted as a substructure of length 6.

[0047] For reference, Figure 5 shows an example of a phrase structure tree with phrase structure labels attached to specific text. Figure 6 shows an example of a dependency structure tree with edge labels attached to the same text as shown in Figure 5.

[0048] The most prominent word in a subtree is called the head. In a dependency structure tree, the head is the word closest to the root, and in a phrase structure tree, it is the word determined by a rule. Labels indicating the role of each subtree in a phrase structure tree, such as S, NP, and VP mentioned earlier, are called phrase structure labels. In a dependency structure tree, the labels attached to the arrows from word to word are called edge labels to the parent node.

[0049] (S104: Correspondence) In S104, the matching unit 130, based on the similarity sim(s,t) between the substructures of the two texts, enumerates the substructures s m ,t n The substructure a(s) of the other text corresponding to each of these m ),a'(t m This process finds the corresponding substructure B in text y for a given substructure A in text x. This process of finding the correspondence (finding the substructure of one text that corresponds to the substructure of the other text) is performed for each substructure of text x and for each substructure of text y.

[0050] The method used by the correspondence unit 130 to determine this correspondence is not limited to a specific method, but can be used, for example, by employing a greedy algorithm, linear programming, or dynamic programming. Below, the greedy algorithm and linear programming will be described as specific examples of methods for determining the correspondence. When using dynamic programming, a sequence of substructures sorted by word length or text occurrence position can be used as input, and the correspondence can be determined using a general dynamic programming method.

[0051] In this embodiment, the similarity sim(s,t) between substructures used to calculate the correspondence relationship may be calculated using any method that yields a value equivalent to the similarity. Specific examples of calculation methods will be described later.

[0052] <Greedy method> First, let's explain the greedy algorithm. The greedy algorithm in this embodiment is based on BERTScore[2]. The correspondence unit 130 finds the correspondence between individual substructures of one text and the substructures of the other text that have the highest similarity score sim(s,t), as shown in equations (2) and (3) below.

[0053]

number

[0054]

number

[0055] (Linear programming) Next, we will explain linear programming. Here, the correspondence unit 130 finds the optimal correspondence between substructures by solving the problem of finding the set that maximizes the sum of the correspondence scores using linear programming. For example, the correspondence unit 130 finds the optimal correspondence by solving the integer programming problem formulated as follows.

[0056]

number

[0057]

number

[0058]

number

[0059] Let's explain the meaning of equation (10) above. Here, let m1 and m2 be the substructures in text x, and let n1 and n2 be the substructures in text y. Equation (10) means that if neither the condition "[0] If m1 is a child of m2 then n1 is a child of n2" nor the condition "If n1 is a child of n2 then m1 is a child of m2" is satisfied, then the correspondence of (m1,n1) and the correspondence of (m2,n2) do not occur simultaneously.

[0060] <Similarity between substructures> Next, we will explain the similarity score sim(s,t). The similarity score sim(s,t) between substructures represents the similarity score between substructures s and t, and is defined, for example, by using the degree of agreement of the surface information of the substructures (word overlap rate) or the similarity of the vectors corresponding to the substructures. For example, using the similarity of the vectors corresponding to the substructures, it can be calculated as follows.

[0061]

number

[0062]

number

[0063] Here, the labels used can include the main word or phrase label of the substructure, or the edge label to the parent node. By filtering the candidates that can be matched in advance in this way, the computational cost of finding the correspondence can be reduced. Furthermore, if filtering is not performed, a score greater than 0 may be assigned to correspondences between uncorresponding substructures, which can become noise in the similarity metric. In contrast, by performing filtering in advance and assigning a score of 0 to substructures that do not clearly correspond, this problem is mitigated and accuracy is improved.

[0064] (S105, S106: Similarity calculation, output) In S105, the similarity calculation unit 140 calculates the similarity between text x and text y based on the correspondence relationship calculated in S104. Here, it is assumed that the correspondence relationship between substructures of each word length l has been obtained.

[0065] The similarity calculation unit 140 first calculates a score F for the correspondence between word lengths l between text x and text y. l P is the average score for the correspondence x. l The average score R for y lTherefore, the calculation is performed as follows. The following calculation is performed for each word length l.

[0066]

number

[0067] Next, the similarity calculation unit 140 calculates the score F for the multiple word lengths obtained above. l Based on this, the final text similarity score F is calculated as follows.

[0068]

number

[0069]

number

[0070] (Example hardware configuration) The evaluation device 100 described in this embodiment can be realized, for example, by having a computer execute a program. This computer may be a physical computer or a virtual machine on the cloud.

[0071] In other words, the device can be realized by using hardware resources such as the CPU and memory built into a computer to execute a program corresponding to the processing performed by the device. The program can be recorded on a computer-readable recording medium (such as portable memory), saved, and distributed. It can also be provided via a network, such as the Internet or email.

[0072] Figure 7 shows an example of the hardware configuration of the computer described above. The computer in Figure 7 has a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, etc., all of which are interconnected by a bus BS. The computer may also be equipped with a GPU.

[0073] The program that enables processing on the computer is provided, for example, on a recording medium 1001 such as a CD-ROM or memory card. When the recording medium 1001 containing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001; it may also be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files and data.

[0074] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when a program startup command is received. The CPU 1004 implements the functions related to the evaluation device 100 according to the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) etc., generated by a program. The input device 1007 consists of a keyboard and mouse, buttons, or a touch panel, etc., and is used to input various operation commands. The output device 1008 outputs the calculation results.

[0075] (Effects of the embodiment) The technology described above, based on the embodiments, makes it possible to appropriately evaluate the similarity between texts.

[0076] To confirm the effectiveness of the technology according to this embodiment (hereinafter referred to as the proposed method), an experiment was conducted to check the correlation between the textual similarity between machine-translated text using the proposed method and the reference translation, and the evaluation by human hands. This experiment will be described below.

[0077] <Settings> The dataset used was from the system-level evaluation task of the Metrics Shared Task[7,8] for WMT20 and WMT21. Because a language capable of parsing multiple types of syntax and determining the main vocabulary was necessary for comparing the structures used, language pairs were limited to those where the target language was English, specifically using a reference translation called ref-A: {cs,de,iu,ja,km,pl,ps,ru}-en for WMT20 and {cs,de,is,ja,ru,zh}-en for WMT21. Baselines were set from the scales used in the task in each year: Prism[5] and BLEURT[3] for WMT20, and Prism and BERTScore[2] for WMT21. Pearson's correlation coefficient was used for evaluation scales, following WMT20, and Williams' Test was used for significance testing between scales.

[0078] The proposed method used SpanBERT[9] as a pre-trained model. Dependent structures were used for structural information, and a RoBERTa-based pipeline from spaCy was used as the dependent structure analyzer.

[0079] A greedy algorithm with word length constraints was used for mapping, and a filtering method was introduced for the similarity scores between substructures, using the main word of each substructure as the label. Weight w l for const The value obtained was used.

[0080] <Experimental Results> The experimental results are shown in Figure 8. Figure 8 shows the correlation coefficients for each language pair in WMT20 / WMT21. Bold indicates the best score among the three measures. † indicates that it is a development dataset.

[0081] As shown in Figure 8, WMT20 achieves higher or comparable accuracy to the baseline for language pairs other than de-en and ps-en, and WMT21 achieves higher accuracy to the baseline for language pairs other than zh-en. While baseline methods such as Prism and BLEURT require large-scale multilingual parallel corpora and manually evaluated data as training data, the proposed method achieves this level of accuracy despite being an unsupervised method using a language model pre-trained on relatively readily available monolingual data.

[0082] This suggests that considering structure in text-to-text similarity measures is effective. On the other hand, compared to BERTScore, which is based on a pre-trained language model using monolingual data, similar to the proposed method, a significant decrease in accuracy is observed in some data. If we consider that BERTScore is based on token-level alignment, while the proposed method extends that alignment to the substructure level, then the constraint of alignment based on structural information may be causing the decrease in accuracy.

[0083] The following additional information is disclosed regarding the embodiments described above.

[0084] <Note> (Additional note 1) Memory and At least one processor connected to the memory, Includes, The aforementioned processor, Obtain the first substructure, which is a substructure of the first text, and the second substructure, which is a substructure of the second text. Find the correspondence between each substructure between one or more first substructures and one or more second substructures. The similarity between the first text and the second text is calculated based on the calculated value representing the relationship between the mapped first substructure and the second substructure. Evaluation device. (Additional note 2) The aforementioned processor seeks to establish a correspondence only between the first and second substructures that have the same word length. The evaluation apparatus described in Appendix 1. (Additional note 3) The processor determines the correspondence between the first and second substructures in such a way that the parent-child relationship between the first and second substructures is not reversed. The evaluation apparatus described in Appendix 1 or 2. (Additional note 4) The aforementioned processor seeks to establish a correspondence only between the first and second substructures that have the same label. An evaluation device as described in any one of the appendices 1 to 3. (Additional note 5) The processor calculates the similarity using the calculated value between the first substructure and the second substructure associated with the first substructure, and the calculated value between the second substructure and the first substructure associated with the second substructure. An evaluation device as described in any one of the appendices 1 through 4. (Additional note 6) The processor calculates the similarity using the calculated value and weights for the word lengths of the substructures. An evaluation device as described in any one of the appendices 1 through 5. (Additional note 7) A computer-based evaluation method, Obtain the first substructure, which is a substructure of the first text, and the second substructure, which is a substructure of the second text. Find the correspondence between each substructure between one or more first substructures and one or more second substructures. The similarity between the first text and the second text is calculated based on the calculated value representing the relationship between the mapped first substructure and the second substructure. Evaluation method. (Additional note 8) A non-temporary storage medium storing a program for causing a computer to function as one of the components in the evaluation device described in any one of the appendices 1 through 6.

[0085] Although this embodiment has been described above, the present invention is not limited to this specific embodiment, and various modifications and changes are possible within the scope of the gist of the invention as described in the claims. [References] [1] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311-318, Philadelphia, Pennsylvania, USA, July 2002. Association for Computational Linguistics. [2] Tianyi Zhang*, Varsha Kishore*, Felix Wu*, Kilian Q. Weinberger, and Yoav Artzi. Bertscore: Evaluating text generation with bert. In International Conference on Learning Representations, 2020. [3] Thibault Sellam, Dipanjan Das, and Ankur Parikh. BLEURT: Learning robust metrics for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. 7881-7892, Online, July 2020. Association for Computational Linguistics. [4] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL-2019, pp. 4171-4186, 2019. [5] Brian Thompson and Matt Post. Automatic machine translation evaluation in many languages via zeroshot paraphrasing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, November 2020. Association for Computational Linguistics. [6] John Hewitt and Christopher D. Manning. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 4129-4138, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics. [7] Nitika Mathur, Johnny Wei, Markus Freitag, Qingsong Ma, and Ondrej Bojar. Results of the WMT20 metrics shared task. In Proceedings of the Fifth Conference on Machine Translation, pp. 688-725, Online, November 2020. Association for Computational Linguistics. [8] Markus Freitag, Ricardo Rei, Nitika Mathur, Chi-kiu Lo, Craig Stewart, George Foster, Alon Lavie, and Ondrej Bojar. Results of the WMT21 metrics shared task: Evaluating metrics with expert-based human evaluations on TED and news domain. In Proceedings of the Sixth Conference on Machine Translation, pp. 733-774, Online, November 2021. Association for Computational Linguistics. [9] Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. SpanBERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, Vol. 8, pp. 64-77, 2020. [Explanation of Symbols]

[0086] 100 Evaluation device 110 Structure acquisition part 120 Substructure acquisition part 130 Correspondence section 140 Similarity calculation part 150 Input section 160 Output section 1000 drive unit 1001 Recording media 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input device 1008 Output device

Claims

1. A substructure acquisition unit that acquires a first substructure, which is a subtree or path obtained from the syntax tree of the first text, and a second substructure, which is a subtree or path obtained from the syntax tree of the second text. A matching unit that determines the correspondence between one or more first substructures and one or more second substructures based on the similarity between the first substructures and the second substructures, A similarity calculation unit calculates the similarity between the first text and the second text based on a calculated value which is the similarity between the mapped first substructure and the second substructure. An evaluation device equipped with the following features.

2. The aforementioned correspondence unit seeks correspondence only between first and second substructures that have the same word length. The evaluation apparatus according to claim 1.

3. The correspondence unit determines the correspondence between the first substructure and the second substructure so that the parent-child relationship between the first substructure and the second substructure is not reversed. The evaluation apparatus according to claim 1.

4. The aforementioned correspondence unit finds correspondences only between the first and second substructures that have the same label. The evaluation apparatus according to claim 1.

5. The similarity calculation unit calculates the similarity between the first text and the second text using the calculated value between the first substructure and the second substructure associated with the first substructure, and the calculated value between the second substructure and the first substructure associated with the second substructure. The evaluation apparatus according to claim 1.

6. The similarity calculation unit calculates the similarity between the first text and the second text using the calculated value and weights for the word lengths of the substructures. The evaluation apparatus according to claim 1.

7. A computer-based evaluation method, Obtain a first substructure, which is a subtree or path obtained from the syntax tree of the first text, and a second substructure, which is a subtree or path obtained from the syntax tree of the second text. Between one or more first substructures and one or more second substructures, a correspondence is determined for each substructure based on the similarity between the first substructure and the second substructure. The similarity between the first text and the second text is calculated based on the calculated similarity between the mapped first substructure and the second substructure. Evaluation method.

8. A program for causing a computer to function as a component in the evaluation apparatus described in any one of claims 1 to 6.

Images