Language processing program, language processing device, and language processing method

Abstract

To provide a language processing program which accurately associates a plurality of texts described in different languages with each other.SOLUTION: A computer executes processing of: extracting a second named entity corresponding to a first named entity which is included in a first text described in a first language from a second text described in a second language; associating the first text with the second text on the basis of a similarity ratio between the first and second named entities and an alignment probability between the first and second named entities; and outputting association information indicating a result obtained by associating the first text with the second text.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to language processing technology. [Background technology]

[0002] A parallel corpus is a corpus that aligns sentences in different languages ​​that are related by translation, while a comparable corpus is a corpus that aligns documents in different languages ​​that relate to the same topic.

[0003] The document pairs in a comparable corpus do not have a strict translation relationship like a parallel corpus. Wikipedia articles written in multiple languages ​​are an example of a comparable corpus.

[0004] Bilingual corpora are important data used in machine translation. However, building a bilingual corpus often requires human translation and checking, which creates a significant workload. For this reason, bilingual corpora are a scarce resource that is difficult to obtain. Furthermore, building a bilingual corpus in a specialized field, such as science and technology, requires specialized knowledge, making bilingual corpora in specialized fields even rarer.

[0005] Furthermore, there are large disparities between languages ​​in terms of existing bilingual corpus resources. While there are relatively many bilingual corpora for major European languages ​​such as English and French, there are few or no bilingual corpora for low-resource languages. Low-resource languages ​​include Basque, Japanese, Arabic, Tamil, Thai, and Indonesian, for which there is little data.

[0006] Regarding bilingual corpora, there are known techniques that use comparable corpora for machine translation of low-resource languages ​​(see, for example, Non-Patent Document 1 and Non-Patent Document 2). There is also known a technique that uses a bidirectional recurrent neural network to extract bilingual sentences (see, for example, Non-Patent Document 3). There is also known a topic estimation device that assigns cross-lingual topics to each document or each word in a non-bilingual corpus that is matched on a document-by-document basis (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2017-151678 [Non-patent literature]

[0008] [Non-Patent Document 1] A. Irvine et al., “Combining Bilingual and Comparable Corpora for Low Resource Machine Translation”, Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 262-270, 2013. [Non-patent document 2] SH Ramesh et al., “Neural Machine Translation for Low Resource Languages ​​using Bilingual Lexicon Induced from Comparable Corpora”, Proceedings of NAACL-HLT 2018: Student Research Workshop, pages 112-119, 2018. [Non-patent document 3] F. Gregoire et al., “Extracting Parallel Sentences with Bidirectional Recurrent Neural Networks to Improve Machine Translation”, Proceedings of the 27th International Conference on Computational Linguistics, pages 1442-1453, 2018. Summary of the Invention [Problem to be solved by the invention]

[0009] When extracting pairs of sentences that have translation relationships based on the semantic similarity between sentences to generate a bilingual corpus from a comparable corpus, the extracted pairs of sentences do not necessarily express the same semantic content.

[0010] This problem is not limited to the case of generating a bilingual corpus from a comparable corpus, but arises whenever comparing various texts written in different languages.

[0011] In one aspect, the present invention aims to accurately align multiple texts written in different languages. [Means for solving the problem]

[0012] In one proposal, the language processing program causes the computer to perform the following processes.

[0013] The computer extracts, from a second text written in a second language, a second named entity corresponding to a first named entity included in a first text written in a first language. The computer aligns the first text with the second text based on the similarity between the first named entity and the second named entity and the alignment probability between the first named entity and the second named entity. The computer outputs alignment information indicating the alignment result between the first text and the second text. [Effects of the Invention]

[0014] According to one aspect, it is possible to accurately associate multiple texts written in different languages. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a functional configuration diagram of a language processing apparatus according to an embodiment. [Figure 2] 10 is a flowchart of a first language process. [Figure 3] FIG. 1 is a functional configuration diagram showing a specific example of a language processing device. [Figure 4A] 10 is a flowchart (part 1) of the second language processing. [Figure 4B] 10 is a flowchart (part 2) of the second language processing. [Figure 5] FIG. 2 is a hardware configuration diagram of an information processing device. DETAILED DESCRIPTION OF THE INVENTION

[0016] Hereinafter, embodiments will be described in detail with reference to the drawings.

[0017] In the techniques of Non-Patent Documents 1 to 3, semantic similarity between sentences is calculated, and pairs of sentences that have a translation relationship are extracted based on the calculated semantic similarity. The semantic similarity is the semantic similarity between sentences, and is different from lexical similarity such as the similarity between words in a translation dictionary. As the semantic similarity, for example, the similarity of context or the similarity of sentence vectors generated from each sentence is used.

[0018] However, even if two sentences have high semantic similarity, it does not necessarily mean that they express the same meaning. Therefore, if a bilingual corpus is generated by matching sentences based solely on semantic similarity, the accuracy of the bilingual corpus may decrease.

[0019] As an example, compare the following English sentence E1 and Japanese sentence J1.

[0020] E1 Biomedical is so easy. J1 Genetics is a breeze.

[0021] In this case, the context similarity between English sentence E1 and Japanese sentence J1 is 0.8, which is a relatively high value. However, the semantic content of English sentence E1 is "Biomedicine is very simple," which is different from the semantic content of Japanese sentence J1.

[0022] As another example, compare the following English sentence E2 with the Japanese sentence J2.

[0023] E2 We report a rare case of ischemic heart disease. J2 Soul Disorder will be releasing a limited edition cassette.

[0024] In this case, the similarity between the sentence vector of English sentence E2 and the sentence vector of Japanese sentence J2 is 0.7, which is a relatively high value. However, the semantic content of English sentence E2 is "We report a rare case of ischemic heart disease," which is different from the semantic content of Japanese sentence J2.

[0025] Therefore, in the embodiment, sentences are compared not only based on the semantic similarity of the sentence pair, but also based on the similarity of the named entities (NE) contained in each sentence. NE represents proper nouns, time expressions, numerical expressions, etc. Proper nouns include organization names (ORGANIZATION), person names (PERSON), place names (LOCATION), and names of proper objects (ARTIFACT). Time expressions include date expressions (DATE) and time expressions (TIME), and numerical expressions include monetary expressions (MONEY) and percentage expressions (PERCENT).

[0026] The more corresponding NEs there are between sentences in different languages, the more likely those sentences are to express the same semantic content. Therefore, by comparing sentences using NE similarity, it is possible to accurately match two sentences that express the same semantic content.

[0027] 1 shows an example of the functional configuration of a language processing device according to an embodiment. The language processing device 101 in FIG. 1 includes an extraction unit 111, an association unit 112, and an output unit 113.

[0028] Fig. 2 is a flowchart showing an example of the first language processing performed by the language processing device 101 of Fig. 1. First, the extraction unit 111 extracts a second NE corresponding to a first NE included in a first text written in a first language from a second text written in a second language (step 201).

[0029] Next, the association unit 112 associates the first text with the second text based on the similarity between the first NE and the second NE and the alignment probability between the first NE and the second NE (step 202).Then, the output unit 113 outputs association information indicating the result of associating the first text with the second text (step 203).

[0030] The language processing device 101 in FIG. 1 can accurately associate a plurality of texts written in different languages.

[0031] Figure 3 shows a specific example of the language processing device 101 of Figure 1. The language processing device 301 of Figure 3 includes a division unit 311, an analysis unit 312, an extraction unit 313, an alignment unit 314, a calculation unit 315, a determination unit 316, an output unit 317, and a storage unit 318. The alignment unit 314 corresponds to the extraction unit 111 of Figure 1, the calculation unit 315 and the determination unit 316 correspond to the association unit 112 of Figure 1, and the output unit 317 corresponds to the output unit 113 of Figure 1.

[0032] The storage unit 318 stores a comparable corpus 321. The comparable corpus 321 includes a plurality of first language documents written in a first language and a plurality of second language documents written in a second language. The first language and the second language are natural languages. Each of the plurality of first language documents is associated with a second language document related to the same topic among the plurality of second language documents. At least one of the first language or the second language may be a low-resource language.

[0033] The dividing unit 311 divides any of the first language documents included in the comparable corpus 321 into sentences to generate a first language sentence set 322-1, and stores the set in the storage unit 318. The dividing unit 311 then divides the second language documents associated with the divided first language documents into sentences to generate a second language sentence set 322-2, and stores the set in the storage unit 318.

[0034] The first language sentence set 322-1 includes M (M is an integer equal to or greater than 1) first language sentences X1(i) (i = 1 to M), and the second language sentence set 322-2 includes N (N is an integer equal to or greater than 1) second language sentences X2(j) (j = 1 to N). The first language sentences X1(i) are examples of first text, and the second language sentences X2(j) are examples of second text.

[0035] The analysis unit 312 performs morphological analysis on each sentence X1(i) in the first language sentence set 322-1 to divide the sentence X1(i) into a plurality of morphemes, generates a first analysis result 323-1, and stores it in the storage unit 318. Then, the analysis unit 312 performs morphological analysis on each sentence X2(j) in the second language sentence set 322-2 to divide the sentence X2(j) into a plurality of morphemes, generates a second analysis result 323-2, and stores it in the storage unit 318. Morphological analysis is an example of natural language processing.

[0036] The first analysis result 323-1 and the second analysis result 323-2 include a plurality of morphemes included in each sentence and information indicating the part of speech of each morpheme, etc. A morpheme may also be a word.

[0037] The extraction unit 313 extracts NEs in the first language from the morphemes included in the first analysis result 323-1 by performing named entity recognition (NER) on the first analysis result 323-1 of each sentence X1(i). NER is an example of natural language processing. Hereinafter, the extracted NEs in the first language will be referred to as NE1(i). NE1(i) is an example of a first NE.

[0038] When the first language sentence set 322-1 is generated from documents in a specific specialized field, the extraction unit 313 may extract NE1(i) by performing NER in that specialized field.

[0039] The alignment unit 314 performs alignment processing on the combination of each sentence X1(i) in the first language sentence set 322-1 and each sentence X2(j) in the second language sentence set 322-2 using the first analysis result 323-1 and the second analysis result 323-2. The alignment processing includes at least one of word alignment and phrase alignment. The alignment processing is an example of natural language processing.

[0040] By performing alignment processing, the alignment unit 314 identifies words or phrases included in sentence X2(j) of each combination that are similar to each of the multiple words or phrases included in sentence X1(i) of the same combination. A phrase includes multiple words.

[0041] Then, alignment unit 314 extracts, from among the words or phrases contained in sentence X2(j), words or phrases similar to NE1(i) contained in sentence X1(i) as NEs in the second language. Hereinafter, the extracted NEs in the second language will be referred to as NE2(j). NE2(j) is an example of a second NE.

[0042] Furthermore, the alignment unit 314 performs an alignment process to determine the alignment probability between each word or phrase included in sentence X1(i) and each word or phrase included in sentence X2(j). The alignment unit 314 then generates extraction results 324 including NE1(i), NE2(j), and alignment information, and stores them in the storage unit 318.

[0043] The alignment information includes the alignment probability between each word included in NE1(i) and each word included in NE2(j), or the alignment probability between NE1(i) and NE2(j). Hereinafter, the alignment probability between each word included in NE1(i) and each word included in NE2(j) may be referred to as the alignment probability of each word pair.

[0044] By performing alignment processing, NE2(j) similar to NE1(i) can be extracted from sentence X2(j) with high accuracy, and alignment information regarding the alignment probability between NE1(i) and NE2(j) can be obtained at the same time.

[0045] The calculation unit 315 uses the first analysis result 323-1, the second analysis result 323-2, and the extraction result 324 to calculate the similarity S1 and the similarity S2 for the combination of each sentence X1(i) in the first language sentence set 322-1 and each sentence X2(j) in the second language sentence set 322-2.

[0046] Similarity S1 represents the similarity between sentence X1(i) and sentence X2(j), and similarity S2 represents the similarity between NE1(i) and NE2(j). Similarity S1 may be, for example, the semantic similarity between sentence X1(i) and sentence X2(j), and similarity S2 may be, for example, the semantic similarity between NE1(i) and NE2(j).

[0047] Next, the calculation unit 315 calculates the alignment probability AP between NE1(i) and NE2(j) using the alignment information included in the extraction result 324. If the alignment information includes the alignment probability of each word pair, the calculation unit 315 calculates the statistical value of the alignment probability of each word pair as the alignment probability AP. The statistical value may be the average, median, maximum, minimum, or the like.

[0048] If the alignment information includes the alignment probability between NE1(i) and NE2(j), the calculation unit 315 uses the alignment probability as the alignment probability AP. By using the alignment information obtained by the alignment process, the alignment probability AP between NE1(i) and NE2(j) can be easily calculated.

[0049] Next, calculation unit 315 calculates the similarity S between sentence X1(i) and sentence X2(j) using similarity S1, similarity S2, and alignment probability AP. Similarity S is used as an evaluation index for evaluating the correspondence between sentence X1(i) and sentence X2(j).

[0050] Based on the result of comparing the similarity S with the threshold T, the determination unit 316 associates sentence X1(i) with sentence X2(j), generates a bilingual corpus 325 including the associated sentences X1(i) and X2(j), and stores the bilingual corpus 325 in the storage unit 318. The output unit 317 outputs the generated bilingual corpus 325. The bilingual corpus 325 is an example of association information indicating the result of associating the first text with the second text.

[0051] For example, the calculation unit 315 generates a sentence vector emb(X1(i)) and a sentence vector emb(X2(j)) from sentence X1(i) and sentence X2(j), and calculates the similarity S1 using the following formula.

[0052] S1=cos(emb(X1(i)),emb(X2(j))) (1)

[0053] cos(emb(X1(i)),emb(X2(j))) represents the cosine similarity between sentence vector emb(X1(i)) and sentence vector emb(X2(j)).

[0054] Furthermore, the calculation unit 315 generates, for example, an NE vector emb(NE1(i)) and an NE vector emb(NE2(j)) from NE1(i) and NE2(j), and calculates the similarity S2 using the following equation.

[0055] S2=cos(emb(NE1(i)),emb(NE2(j))) (2)

[0056] Examples of methods for generating sentence vectors or NE vectors from sentences or NEs include Bag of Words, TF-IDF (Term Frequency-Inverse Document Frequency), and distributed representations. Examples of distributed representations include Word2Vec, Glove (Global Vectors), FastText, and BERT (Bidirectional Encoder Representations from Transformers).

[0057] Next, the calculation unit 315 calculates the similarity S3 and the similarity S using the similarity S1, the similarity S2, and the alignment probability AP according to the following formulas.

[0058] S3=mean(S2*AP) (3) S=(1-p)*S1+p*S3 (4)

[0059] mean(S2*AP) represents the average of S2*AP for one or more NE1(i). However, if NE1(i) is not extracted from sentence X1(i) or if NE2(j) is not extracted from sentence X2(j), mean(S2*AP) = 0. The similarity S3 in formula (3) represents a weighted NE similarity using the alignment probability AP as the weight.

[0060] p is a real number greater than or equal to 0 and less than or equal to 1. For example, a predetermined value is used as p. Equation (4) represents linear interpolation of similarity S1 and similarity S3 using p as an interpolation coefficient. According to equation (4), similarity S is calculated using not only similarity S1 but also similarity S3 calculated from alignment probability AP.

[0061] If the similarity S is greater than the threshold T, the determination unit 316 associates sentence X1(i) and sentence X2(j) and adds them to the bilingual corpus 325; if the similarity S is equal to or less than the threshold T, the determination unit 316 does not add the combination of sentence X1(i) and sentence X2(j) to the bilingual corpus 325.

[0062] By using the alignment probability AP as a weight to calculate the similarity S3, which represents the NE similarity between sentence X1(i) and sentence X2(j), the larger the alignment probability AP, the larger the similarity S3, and as a result, the larger the similarity S. The larger the NE similarity, the more likely sentence X1(i) and sentence X2(j) express the same meaning. Therefore, by evaluating the correspondence between these sentences using the similarity S, it is possible to preferentially select sentence pairs that express the same meaning and add them to the bilingual corpus 325.

[0063] Furthermore, by using an appropriate value as the threshold T for the similarity S, it is possible to exclude from the bilingual corpus 325 sentence pairs that do not express the same meaning despite having a high semantic similarity.

[0064] Here, the language processing performed by the language processing device 301 when the first language is English and the second language is Japanese will be explained using specific examples of sentence X1(i) and sentence X2(j). First, assume that the following English sentence and Japanese sentence are selected as sentence X1(i) and sentence X2(j).

[0065] X1(i) Biomedical is so easy. X2(j) Genetics is a breeze.

[0066] The text X1(i) and the text X2(j) are the same as the above-mentioned English text E1 and Japanese text J1. In this case, from X1(i) and the text X2(j), the following text vectors emb(X1(i)) and text vector emb(X2(j)) are generated.

[0067] emb(X1(i)) = [0.76, 0.3, 0.32,..., 0.4] emb(X2(j)) = [0.35, 0.05, 0.1,..., 0.15]

[0068] Then, from the text vector emb(X1(i)) and the text vector emb(X2(j)), the similarity S1 is calculated by Equation (1), and S1 = 0.8.

[0069] However, since the text X1(i) does not contain NE, NE1(i) and NE2(j) are not extracted. Therefore, S3 = 0. When p = 0.5, the similarity S is calculated as follows by Equation (4).

[0070] S = (1 - 0.5) * 0.8 + 0.5 * 0 = 0.4 (5)

[0071] When T = 0.7, since S < T, the combination of the text X1(i) and the text X2(j) is not added to the parallel corpus 325.

[0072] Next, assume the case where the following English text and Japanese text are selected as the text X1(i) and the text X2(j).

[0073] X1(i) We report a rare case of ischemic heart disease. X2(j) We will sell the cassette of Soul Disorder on a limited basis.

[0074] The text X1(i) and the text X2(j) are the same as the above-mentioned English text E2 and Japanese text J2. In this case, from X1(i) and the text X2(j), the following text vectors emb(X1(i)) and text vector emb(X2(j)) are generated.

[0075] emb(X1(i)) = [0.32, 0.94, 0.82, ..., 0.53] emb(X2(j)) = [0.8, 0.35, 0.73, ..., 0.75]

[0076] And from the sentence vector emb(X1(i)) and the sentence vector emb(X2(j)), the similarity S1 is calculated according to Equation (1), and S1 = 0.7.

[0077] Also, "ischemic heart disease" is extracted as NE1(i) from sentence X1(i), but no NE2(j) is extracted from sentence X2(j). Therefore, S3 = 0. When p = 0.5, the similarity S is calculated as follows according to Equation (4).

[0078] S = (1 - 0.5) * 0.7 + 0.5 * 0 = 0.35 (6)

[0079] When T = 0.7, since S < T, the combination of sentence X1(i) and sentence X2(j) is not added to the parallel corpus 325.

[0080] Thus, when NE1(i) or NE2(j) is not extracted, S3 = 0, so the possibility of S < T becomes high. Therefore, sentence pairs that do not represent the same semantic content despite having a high semantic similarity can be excluded from the parallel corpus 325.

[0081] Next, assume the case where the following English sentence and Japanese sentence are selected as sentence X1(i) and sentence X2(j).

[0082] X1(i) xxx chronic myeloid leukemia xxx BCR-ABL fusion gene. X2(j) Chronic myeloid leukemia##BCR-ABL fusion gene#.

[0083] "xxx" represents an English word, and "#" represents a Japanese character. In this case, the similarity S1 is calculated from the sentence vector emb(X1(i)) and sentence vector emb(X2(j)) using formula (1), and is S1 = 0.9.

[0084] Furthermore, from sentence X1(i), "chronic myeloid leukemia" and "BCR-ABL fusion gene" are extracted as NE1(i). Then, from sentence X2(j), "chronic myeloid leukemia" is extracted as NE2(j) similar to "chronic myeloid leukemia", and "BCR-ABL fusion gene" is extracted as NE2(j) similar to "BCR-ABL fusion gene".

[0085] In this example, the alignment probability AP is the average value of the alignment probability between each word in NE1(i) and each word in NE2(j). If the alignment probability between word W1 in NE1(i) and word W2 in NE2(j) is denoted as P(W1,W2), the alignment probability AP between "chronic myeloid leukemia" and "chronic myeloid leukemia" is calculated using the following formula:

[0086] AP=(P(chronic)+P(myeloid) +P(leukemia) / 3 =(1.0+1.0+1.0) / 3 =1.0 (7)

[0087] In addition, the alignment probability AP between the "BCR-ABL fusion gene" and the "BCR-ABL fusion gene" is calculated using the following formula:

[0088] AP=(P(BCR-ABL,BCR-ABL)+P(fusion, fusion) +P(gene) / 3 =(0.9+1.0+1.0) / 3 =0.97 (8)

[0089] In this example, the similarity S2 between "chronic myeloid leukemia" and "chronic myeloid leukemia" is 1.0, and the similarity S2 between "BCR-ABL fusion gene" and "BCR-ABL fusion gene" is 1.0. Therefore, the similarity S3 is calculated as follows using formula (3):

[0090] S3=(1.0*1.0+1.0*0.97) / 2=0.985 (9)

[0091] When p=0.5, the similarity S is calculated as follows using equation (4):

[0092] S=(1-0.5)*0.9+0.5*0.985=0.94 (10)

[0093] When T=0.7, S>T, so sentences X1(i) and X2(j) are associated with each other and added to the bilingual corpus 325.

[0094] Next, assume that the following English and Japanese sentences are selected as sentence X1(i) and sentence X2(j).

[0095] X1(i) xxx chronic myeloid leukemia xxx BCR-ABL fusion gene. X2(j) CML#####BCR-ABL fusion gene####.

[0096] In this case, the similarity S1 is calculated from the sentence vector emb(X1(i)) and the sentence vector emb(X2(j)) using equation (1), and is S1=0.9.

[0097] Furthermore, "chronic myeloid leukemia" and "BCR-ABL fusion gene" are extracted as NE1(i) from sentence X1(i). Then, from sentence X2(j), "CML" is extracted as NE2(j) similar to "chronic myeloid leukemia", and "BCR-ABL fusion gene" is extracted as NE2(j) similar to "BCR-ABL fusion gene".

[0098] In this example, the alignment probability AP is the average value of the alignment probability between each word in NE1(i) and each word in NE2(j). If the alignment probability between word W1 in NE1(i) and word W2 in NE2(j) is denoted as P(W1,W2), the alignment probability AP between "chronic myeloid leukemia" and "CML" is calculated using the following formula:

[0099] AP=(P(chronic,CML)+P(myeloid,CML) +P(leukemia,CML)) / 3 =(0.3+0.3+0.3) / 3 =0.3 (11)

[0100] Furthermore, the alignment probability AP between the "BCR-ABL fusion gene" and the "BCR-ABL fusion gene" is calculated using formula (8).

[0101] In this example, the similarity S2 between "chronic myeloid leukemia" and "CML" is 1.0, and the similarity S2 between "BCR-ABL fusion gene" and "BCR-ABL fusion gene" is 1.0. Therefore, the similarity S3 is calculated as follows using formula (3):

[0102] S3=(1.0*0.3+1.0*0.97) / 2=0.635 (12)

[0103] When p=0.5, the similarity S is calculated as follows using equation (4):

[0104] S=(1-0.5)*0.9+0.5*0.635=0.77 (13)

[0105] When T=0.7, S>T, so sentences X1(i) and X2(j) are associated with each other and added to the bilingual corpus 325.

[0106] In this way, when NE1(i) and NE2(j) are extracted, S3≠0, and therefore it becomes more likely that S>T. Therefore, it is possible to add sentence pairs that are likely to express the same meaning to the bilingual corpus 325.

[0107] 3, by evaluating the correspondence between sentence pairs based on the similarity S calculated using the alignment probability AP, sentence pairs that do not express the same meaning can be excluded and sentence pairs that are likely to express the same meaning can be preferentially aligned. This allows sentences written in different languages ​​to be aligned with high accuracy, improving the accuracy of the bilingual corpus 325.

[0108] Furthermore, by using the language processing device 301, it is possible to easily generate a parallel corpus 325, which is a scarce resource, from a comparable corpus 321 in a low-resource language or a specific specialized field.

[0109] Furthermore, by training a machine translation model through machine learning using a highly accurate bilingual corpus 325 as training data, it is possible to generate a machine translation model with few mistranslations and high robustness.

[0110] Fig. 4 is a flowchart showing an example of the second language processing performed by the language processing device 301 of Fig. 3. First, the dividing unit 311 divides any first language document included in the comparable corpus 321 into sentences to generate a first language sentence set 322-1 (step 401). Then, the dividing unit 311 divides second language documents associated with the divided first language documents into sentences to generate a second language sentence set 322-2 (step 402).

[0111] Next, the analysis unit 312 performs morphological analysis on each sentence X1(i) in the first language sentence set 322-1 to divide the sentence X1(i) into a plurality of morphemes and generate a first analysis result 323-1 (step 403).Then, the analysis unit 312 performs morphological analysis on each sentence X2(j) in the second language sentence set 322-2 to divide the sentence X2(j) into a plurality of morphemes and generate a second analysis result 323-2 (step 404).

[0112] Next, the extraction unit 313 sets the control variable i to 1 (step 405), and extracts NE1(i) by performing NER on the first analysis result 323-1 of the sentence X1(i) (step 406). Then, the extraction unit 313 sets the control variable j to 1 (step 407).

[0113] Next, the alignment unit 314 extracts NE2(j) by performing alignment processing on the combination of sentence X1(i) in the first language sentence set 322-1 and sentence X2(j) in the second language sentence set 322-2 (step 408). The alignment processing in step 408 also simultaneously obtains alignment information regarding the alignment probability between NE1(i) and NE2(j).

[0114] Next, the alignment unit 314 generates an extraction result 324 including NE1(i), NE2(j), and alignment information (step 409).

[0115] Next, calculation unit 315 uses first analysis result 323-1, second analysis result 323-2, and extraction result 324 to calculate the similarity S between sentence X1(i) and sentence X2(j) according to equations (1) to (4) (step 410).

[0116] Next, the determination unit 316 compares the similarity S with a threshold value T (step 411). If the similarity S is greater than the threshold value T (step 411, YES), the determination unit 316 associates sentence X1(i) with sentence X2(j) and adds them to the bilingual corpus 325 (step 412).

[0117] Next, the extraction unit 313 compares j with N (step 413). If j is smaller than N (step 413, NO), the extraction unit 313 increments j by 1 (step 416), and the language processing device 301 repeats the processes from step 408 onwards.

[0118] If j has reached N (step 413, YES), the extraction unit 313 compares i with M (step 414). If i is smaller than M (step 414, NO), the extraction unit 313 increments i by 1 (step 417), and the language processing device 301 repeats the processes from step 406 onwards.

[0119] If the similarity S is equal to or less than the threshold value T (step 411, NO), the language processing device 301 performs the processes from step 413 onwards. If i has reached M (step 414, YES), the output unit 317 outputs the bilingual corpus 325 (step 415).

[0120] The calculation unit 315 may calculate the similarity S using any of the following calculation formulas instead of formula (4).

[0121] S=w*S1+v*S3 (21) S=(S1*S3)^(1-q) (22) S=(2*S1*S3) / (S1+S3) (23) S=(S1^r)*(S3^(1-r)) (24) S=(1-m)*(1-t)*S1 +(1-m)*t*S3+m*S3 (25) S=(1 / 2)*(S1+S3) +(1 / 2)*(S1-S3)*log((1-s) / (1+s)) (26) S=a*S1^c+b*S3^d (27)

[0122] In equation (21), w and v are predetermined weighting coefficients. Equation (21) represents the weighted addition of the similarity S1 and the similarity S3.

[0123] In equation (22), q is a predetermined scale factor. Equation (22) represents the geometric mean of the similarity S1 and the similarity S3.

[0124] Equation (23) represents the harmonic mean of the similarity S1 and the similarity S3.

[0125] In equation (24), r is a predetermined weight parameter.Equation (24) represents interpolation using the power of the similarity S1 and the power of the similarity S3.

[0126] In equation (25), t is a real number greater than or equal to 0 and less than or equal to 1, and m is a predetermined margin parameter. Equation (25) represents linear interpolation of the similarity S1 and the similarity S3 using the margin parameter.

[0127] In equation (26), s is a predetermined parameter.Equation (26) represents the interpolation of similarity S1 and similarity S3 using a logarithmic function.

[0128] In equation (27), a and b are predetermined coefficients, and c and d are predetermined parameters.Equation (27) represents interpolation of the similarity S1 and the similarity S3 using a polynomial.

[0129] The calculation unit 315 may calculate the similarity S1 and the similarity S2 using any of the following indices or models instead of the cosine similarity of the vectors.

[0130] (1) Jaccard similarity (2) Dot product (3) Dice coefficient (4) Pearson correlation coefficient (5) Spearman correlation coefficient (6) Euclidean distance (7) Squared Euclidean distance (8) Standardized Euclidean distance (9) L2 norm (10) Canberra Distance (11) Chebyshev distance (12) Minkowski distance (13) Mahalanobis distance (14) Jensen-Shannon distance (15) Chi-square distance (16) Levenshtein distance (17) Hamming distance (18) Jaccard / Tanimoto distance (19)LASER(Language-Agnostic Sentence Representations) (20)LaBSE(Language-Agnostic BERT Sentence Embedding)

[0131] The language processing device 301 may perform language processing using a passage consisting of multiple sentences as the first text and the second text instead of a sentence.

[0132] The configurations of the language processing device 101 in Fig. 1 and the language processing device 301 in Fig. 3 are merely examples, and some of the components may be omitted or changed depending on the application or conditions of the language processing device. For example, in the language processing device 301 in Fig. 3, if the first analysis result 323-1 and the second analysis result 323-2 are generated by an external device, the dividing unit 311 and the analyzing unit 312 can be omitted.

[0133] 2, 4A, and 4B are merely examples, and some of the processes may be omitted or changed depending on the configuration or conditions of the language processing device. For example, in the language processing device 301 of Fig. 3, if the first analysis result 323-1 and the second analysis result 323-2 are generated by an external device, the processes of steps 401 to 404 of Fig. 4A can be omitted.

[0134] Equations (1) to (13) and equations (21) to (27) are merely examples, and the language processing device 301 may perform language processing using other calculation formulas.

[0135] Fig. 5 shows an example of the hardware configuration of an information processing device (computer) used as the language processing device 101 in Fig. 1 and the language processing device 301 in Fig. 3. The information processing device in Fig. 5 includes a CPU (Central Processing Unit) 501, a memory 502, an input device 503, an output device 504, an auxiliary storage device 505, a medium drive device 506, and a network connection device 507. These components are hardware and are connected to each other by a bus 508.

[0136] The memory 502 is, for example, a semiconductor memory such as a read-only memory (ROM) or a random access memory (RAM), and stores programs and data used in processing. The memory 502 may operate as the storage unit 318 in FIG.

[0137] 3 by executing a program using the memory 502. The CPU 501 (processor) also operates as the division unit 311, analysis unit 312, extraction unit 313, alignment unit 314, calculation unit 315, and determination unit 316 in FIG. 3 by executing a program using the memory 502.

[0138] The input device 503 is, for example, a keyboard, a pointing device, etc., and is used for inputting instructions or information from a user or operator. The output device 504 is, for example, a display device, a printer, etc., and is used for outputting inquiries or instructions to a user or operator and processing results. The output device 504 may operate as the output unit 113 in FIG. 1 or the output unit 317 in FIG. 3, and the processing results may be the bilingual corpus 325.

[0139] The auxiliary storage device 505 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device, etc. The auxiliary storage device 505 may be a hard disk drive or a solid state drive (SSD). The information processing device stores programs and data in the auxiliary storage device 505 and can use them by loading them into the memory 502. The auxiliary storage device 505 may operate as the storage unit 318 in FIG. 3.

[0140] The medium drive device 506 drives the portable recording medium 509 and accesses the recorded contents thereof. The portable recording medium 509 is a memory device, a flexible disk, an optical disk, a magneto-optical disk, etc. The portable recording medium 509 may be a CD-ROM (Compact Disk Read Only Memory), a DVD (Digital Versatile Disk), a USB (Universal Serial Bus) memory, etc. A user or operator can store programs and data in the portable recording medium 509 and load them into the memory 502 for use.

[0141] In this way, the computer-readable recording medium that stores the program and data used in the processing is a physical (non-transitory) recording medium such as the memory 502, the auxiliary storage device 505, or the portable recording medium 509.

[0142] The network connection device 507 is a communication interface circuit connected to a communication network such as a WAN (Wide Area Network) or a LAN (Local Area Network) and performs data conversion associated with communication. The information processing device receives programs and data from an external device via the network connection device 507 and loads them into the memory 502 for use. The network connection device 507 may operate as the output unit 113 in FIG. 1 or the output unit 317 in FIG. 3.

[0143] 5, some of the components may be omitted or changed depending on the application or conditions of the information processing device. For example, if an interface with a user or operator is not required, the input device 503 and the output device 504 may be omitted. If the portable recording medium 509 or a communication network is not used, the medium drive device 506 or the network connection device 507 may be omitted.

[0144] Although the disclosed embodiments and their advantages have been described in detail, those skilled in the art may make various modifications, additions, and omissions without departing from the scope of the invention as clearly set forth in the claims.

[0145] The following notes are further provided regarding the embodiment described with reference to FIGS. (Appendix 1) extracting, from a second text written in a second language, a second named entity corresponding to a first named entity included in a first text written in a first language; matching the first text with the second text based on a similarity between the first named entity and the second named entity and an alignment probability between the first named entity and the second named entity; outputting association information indicating a result of associating the first text with the second text; A language processing program that causes a computer to execute a process. (Appendix 2) The process of extracting the second named entity from the second text includes: a process of extracting, from among a plurality of words included in the second text, one or a plurality of words similar to the first named entity as the second named entity; A process of calculating an alignment probability between a word included in the first named entity and a word included in the second named entity; Including, The language processing program described in Appendix 1, characterized in that the process of matching the first text and the second text includes a process of calculating a statistical value of the alignment probability between words included in the first named entity and words included in the second named entity as the alignment probability between the first named entity and the second named entity. (Appendix 3) The process of associating the first text with the second text includes: a process of calculating an evaluation index for evaluating the correspondence between the first text and the second text based on the similarity and the alignment probability; a process of associating the first text with the second text based on a result of comparing the evaluation index with a threshold; 2. The language processing program according to claim 1, comprising: (Appendix 4) 4. The language processing program according to claim 1, wherein at least one of the first language and the second language is a low-resource language. (Appendix 5) an extraction unit that extracts, from a second text written in a second language, a second named entity corresponding to a first named entity included in a first text written in a first language; an associating unit that associates the first text with the second text based on a similarity between the first named entity and the second named entity and an alignment probability between the first named entity and the second named entity; an output unit that outputs association information indicating a result of associating the first text with the second text; A language processing device comprising: (Appendix 6) the extraction unit extracts, from among a plurality of words included in the second text, one or more words similar to the first named entity as the second named entity, and calculates an alignment probability between the words included in the first named entity and the words included in the second named entity; The language processing device described in Appendix 5, characterized in that the matching unit calculates a statistical value of the alignment probability between the words included in the first named entity and the words included in the second named entity as the alignment probability between the first named entity and the second named entity. (Appendix 7) The language processing device described in Appendix 5, characterized in that the alignment unit calculates an evaluation index for evaluating the correspondence between the first text and the second text based on the similarity and the alignment probability, and aligns the first text and the second text based on a result of comparing the evaluation index with a threshold. (Appendix 8) 8. The language processing device according to any one of appendices 5 to 7, wherein at least one of the first language and the second language is a low-resource language. (Appendix 9) extracting, from a second text written in a second language, a second named entity corresponding to a first named entity included in a first text written in a first language; matching the first text with the second text based on a similarity between the first named entity and the second named entity and an alignment probability between the first named entity and the second named entity; outputting association information indicating a result of associating the first text with the second text; A language processing method characterized in that the processing is executed by a computer. (Appendix 10) The process of extracting the second named entity from the second text includes: a process of extracting, from among a plurality of words included in the second text, one or a plurality of words similar to the first named entity as the second named entity; A process of calculating an alignment probability between a word included in the first named entity and a word included in the second named entity; Including, The language processing method described in Appendix 9, characterized in that the process of matching the first text and the second text includes a process of calculating a statistical value of the alignment probability between words included in the first named entity and words included in the second named entity as the alignment probability between the first named entity and the second named entity. (Appendix 11) The process of associating the first text with the second text includes: a process of calculating an evaluation index for evaluating the correspondence between the first text and the second text based on the similarity and the alignment probability; a process of associating the first text with the second text based on a result of comparing the evaluation index with a threshold; 10. The language processing method according to claim 9, comprising: (Appendix 12) 12. The language processing method according to any one of claims 9 to 11, wherein at least one of the first language and the second language is a low-resource language. [Explanation of symbols]

[0146] 101, 301 Language processing devices 111, 313 Extraction part 112 Mapping section 113, 317 Output section 311 Split section 312 Analysis Department 314 Alignment Department 315 Calculation Department 316 Judgment section 318 Storage section 321 Comparable Corpus 322-1 First language sentence set 322-2 Second language sentence set 323-1 First analysis result 323-2 Second analysis result 324 Extraction results 325 parallel corpus 501 CPU 502 memory 503 Input Device 504 Output Device 505 Auxiliary storage 506 Media drive unit 507 Network connection device 508 Bus 509 Portable Recording Media

Claims

1. extracting a second named entity corresponding to a first named entity included in a first text written in a first language from a second text written in a second language; matching the first text with the second text based on a similarity between the first named entity and the second named entity and an alignment probability between the first named entity and the second named entity; outputting association information indicating a result of associating the first text with the second text; A language processing program that causes a computer to execute processing.

2. The process of extracting the second named entity from the second text includes: A process of extracting, from among a plurality of words included in the second text, one or a plurality of words similar to the first named entity as the second named entity; determining an alignment probability between a word included in the first named entity and a word included in the second named entity; Including, The language processing program according to claim 1, characterized in that the process of matching the first text with the second text includes a process of calculating a statistical value of an alignment probability between a word contained in the first named entity and a word contained in the second named entity as an alignment probability between the first named entity and the second named entity.

3. The process of associating the first text with the second text includes: calculating an evaluation index for evaluating a correspondence between the first text and the second text based on the similarity and the alignment probability; A process of associating the first text with the second text based on a result of comparing the evaluation index with a threshold value; 2. The language processing program according to claim 1, further comprising:

4. 4. The language processing program according to claim 1, wherein at least one of the first language and the second language is a low-resource language.

5. an extraction unit that extracts a second named entity corresponding to a first named entity included in a first text written in a first language from a second text written in a second language; a matching unit that matches the first text with the second text based on a similarity between the first named entity and the second named entity and an alignment probability between the first named entity and the second named entity; an output unit that outputs association information indicating a result of associating the first text with the second text; A language processing device comprising:

6. extracting a second named entity corresponding to a first named entity included in a first text written in a first language from a second text written in a second language; matching the first text with the second text based on a similarity between the first named entity and the second named entity and an alignment probability between the first named entity and the second named entity; outputting association information indicating a result of associating the first text with the second text; A language processing method, characterized in that the processing is executed by a computer.

Images