Proper noun translation dictionary creation device, and proper noun translation dictionary creation method

Abstract

To provide a technique capable of extracting proper noun translation pairs from a document set.SOLUTION: A proper noun translation dictionary creation device 30 includes: a structuring unit 310 that extracts a combination of a proper noun, a related attribute value that is another proper noun that co-occurs with the proper noun, and a related attribute name that is a label for the related attribute value from a document set; a translation information selection unit 320 that receives selection of the proper nouns to be translated and the associated attribute names used in the translation using the related attribute value corresponding to the related attribute name selected by the translation information selection unit 310; a matching unit 330 that calculates the degree of agreement between proper nouns selected by the translation information selection unit 320 using related attribute values corresponding to the related attribute names selected by the translation information selection unit 320; and a proper noun translation dictionary 340 that holds translation pairs of proper nouns selected by the translation information selection unit based on the degree of agreement calculated by the matching unit 330.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to a proper noun translation dictionary creating device and a proper noun translation dictionary creating method. [Background technology]

[0002] In recent years, with the emergence of highly accurate neural machine translation (NMT), end-to-end translation has become common, where a single model is used to translate input text and output the translated text. NMT uses a bilingual corpus for training. A bilingual corpus is a set of sentences that are translated between multiple languages ​​and correspond to each other in a bilingual format.

[0003] In recent years, NMT has become more accurate, producing grammatically correct and more natural translations. However, it tends to fail to translate named entities that do not appear in the training corpus. Examples of named entities that do not appear in the training corpus include book titles and names of newly developed vehicles and cars, which were created after the NMT training period.

[0004] For sentences containing such named entities, it is possible to use a named entity translation dictionary, i.e., a dictionary that stores translation pairs of named entities between multiple languages, to translate only the named entities using a separate named entity translation dictionary, or to train NMT using the named entity translation dictionary as part of a bilingual corpus.

[0005] However, manually creating a named entity translation dictionary requires time and cost. In response to this problem, Patent Document 1 discloses an example of a technology for automatically extracting translation pairs of proper nouns and the like from documents in two languages ​​that do not have an ideal bilingual relationship.

[0006] The method described in Patent Document 1 uses a document written in a first language and a document written in a second language as input documents, extracts all combinations of named entities in the first language and named entities in the second language, and presents combinations with high likelihood as translation pair candidates. The likelihood is calculated based on statistics such as the translation probability and translation order of named entities, which are calculated by preparing a bilingual corpus with content similar to the document in advance and manually or otherwise matching named entities. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] Japanese Patent Application Laid-Open No. 2006-190107 Summary of the Invention [Problem to be solved by the invention]

[0008] However, the method disclosed in Patent Document 1 cannot extract translation pairs unless a bilingual corpus with similar content exists for the input document. Furthermore, even if the input documents in the first and second languages ​​do not have an ideal bilingual relationship, translation pairs cannot be extracted unless they are written in a similar manner about a common topic.

[0009] For example, if a breaking news article about event A written in a first language and a second language is taken as input, it is expected that there will be a certain degree of correspondence between the proper nouns that appear and that the order in which the proper nouns appear will be similar. However, if the input document pair contains translation-paired proper nouns but covers different topics, such as a document written in the first language being a breaking news article about event A and a document written in the second language being an article that comprehensively explains events B and C related to event A, it will be difficult to extract translation pairs of named entities between the two documents.

[0010] This is because the likelihood that a combination of proper nouns is a translation pair is calculated using statistical values ​​calculated from a bilingual corpus.

[0011] The present invention has been made in consideration of the above points, and has as its object to provide a technique that can extract translation pairs of proper nouns from a document collection. [Means for solving the problem]

[0012] In order to solve the above problem, one representative proper noun translation dictionary creation device of the present invention is a proper noun translation dictionary creation device that extracts translation pairs of proper nouns from a document set, and includes: a structuring unit that extracts from the document set combinations of proper nouns, related attribute values ​​which are other proper nouns that co-occur with the proper nouns, and related attribute names which are labels of the related attribute values; a translation information selection unit that accepts selection of proper nouns to be translated and related attribute names to be used for translation from the proper nouns and related attribute names extracted by the structuring unit; a matching unit that calculates the degree of agreement between the proper nouns selected by the translation information selection unit using the related attribute values ​​corresponding to the related attribute names selected by the translation information selection unit; and a proper noun translation dictionary that holds translation pairs of proper nouns selected by the translation information selection unit based on the degree of agreement calculated by the matching unit. [Effects of the Invention]

[0013] According to the present invention, translation pairs of proper nouns can be extracted from a document collection. [Brief explanation of the drawings]

[0014] [Figure 1] 1 is a schematic configuration diagram showing an example of the configuration of a proper noun translation dictionary creating device according to first to third embodiments. [Figure 2] FIG. 10 is a diagram showing an example of a proper noun translation pair screen displayed on a user terminal. [Figure 3] FIG. 10 is a diagram illustrating an example of the configuration of a structuring unit. [Figure 4] FIG. 10 illustrates an example of the configuration of a translation information selection unit. [Figure 5] FIG. 2 illustrates an example of the configuration of a matching unit. [Figure 6] 10 is a flowchart showing an example of a processing procedure for creating a proper noun translation dictionary. [Figure 7] 10 is a diagram showing an example of proper noun extraction in the structuring unit; FIG. [Figure 8] FIG. 10 is a diagram showing an example of a proper noun selection screen in the translation information selection unit. [Figure 9] FIG. 10 is a diagram showing an example of a related attribute selection screen in the translation information selection unit. [Figure 10] 10A and 10B are diagrams illustrating an example of related attribute value complementation in a matching unit. [Figure 11] FIG. 10 is a diagram illustrating an example of structured information in a matching unit. [Figure 12] FIG. 10 is a diagram illustrating an outline of matching of structured information in a matching unit. [Figure 13] 10 is a table showing an example of a result of calculation of similarity between related attribute values ​​in a matching unit. [Figure 14] 10 is a table showing an example of importance of related attributes stored in a related attribute importance storage unit in a matching unit. [Figure 15] FIG. 10 is a diagram illustrating an example of structured information when repetitive processing occurs in the matching unit. [Figure 16] 10 is a flowchart showing a processing procedure of a repetitive process in a matching unit. [Figure 17] 10 is a flowchart showing a processing procedure for creating a proper noun translation dictionary according to the second embodiment. [Figure 18] FIG. 10 is a diagram showing an example of input and output to a large-scale language model in a structuring unit. [Figure 19] 3 is a diagram showing an example of input and output to a large-scale language model in a matching unit 330. FIG. DETAILED DESCRIPTION OF THE INVENTION

[0015] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[0016] It should be noted that the following description and drawings are examples for explaining the present invention, and that appropriate omissions and simplifications have been made for clarity of explanation. Furthermore, not all of the combinations of features described in the embodiments are necessarily essential to the solutions of the present invention. The present invention is not limited to the embodiments, and all application examples that conform to the concept of the present invention are included in the technical scope of the present invention.

[0017] Furthermore, "translation" in this invention is not limited to identifying pairs of nouns (i.e., translation pairs) that represent the same thing across multiple languages, but also includes identifying nouns with multiple names within the same language. For example, the Japanese satellite "Tsubame" may be written in Japanese documents as "Super Low Attitude Test Satellite," its English translation, or its abbreviation, "SLATS." Registering these four expressions as translation pairs in a dictionary also falls within the scope of what "translation" means in this invention.

[0018] Furthermore, the set of input documents may be composed of one or more documents created according to some standard, regardless of their type or language. In the following embodiment and examples, a translation dictionary creation process for Japanese proper nouns and English proper nouns will be described as an example, but the input documents may be composed of multiple languages, not just Japanese and English, or may be composed of only one language. Furthermore, the proper noun translation dictionary that is finally output is not limited to Japanese-English translation, and a dictionary that supports multiple languages ​​may be output. [Example]

[0019] First, the configuration will be described, but the configuration of this embodiment and that of later-described embodiments 2 and 3 are common in many respects. Therefore, the configuration of embodiment 1 will be described.

[0020] FIG. 1 is a schematic diagram showing an example of the configuration of a proper noun translation dictionary creating device according to first to third embodiments.

[0021] As shown in FIG. 1, a proper noun translation dictionary creation device 30 is connected to a document management device 10 and a user terminal 20 via a network 40 .

[0022] The document management device 10 manages a set of documents to be input to the proper noun translation dictionary creation device 30. Specifically, the document management device 10 has a database that stores a set of documents to be input for creating a proper noun translation dictionary, and further manages documents based on operations by an administrator.

[0023] The user terminal 20 is a device that accepts input and output by a user, and can be realized by an information processing device such as a PC, a tablet, or a smartphone. The user terminal 20 issues instructions to the proper noun translation dictionary creation device 30 in accordance with user operations, and outputs the processing results of the proper noun translation dictionary creation device 30. In other words, the user terminal 20 is provided with the input unit and output unit of the first to third embodiments.

[0024] In FIG. 1, the document management device 10, the user terminal 20, and the proper noun translation dictionary creation device 30 are connected via a network 40, but the document management device 10 and the user terminal 20 may be installed in the same area as the proper noun translation dictionary creation device 30, or may be directly connected to the proper noun translation dictionary creation device 30.

[0025] Finally, we will explain the proper noun translation dictionary creation device 30. The proper noun translation dictionary creation device 30 includes a communication unit 31, a processing unit 32, a memory 33, and a storage unit 34. These are connected via a bus 36.

[0026] First, the communication unit 31 transmits and receives information to and from the document management device 10 and the user terminal 20 via the network 40. For example, the communication unit 31 receives a set of documents to be input to the proper noun translation dictionary creation device 30 from the document management device 10. The communication unit 31 also transmits to the user terminal 20 a selection screen for proper nouns and a selection screen for related attributes in the translation information selection unit 320, which will be described later. The information received by the communication unit 31 is sent to the processing unit 32 and the memory 33.

[0027] The processing unit 32 can be realized by a processor such as a CPU (Central Processing Unit), and executes calculations in accordance with various programs loaded in a memory 33, which will be described later.

[0028] In the memory 33, information required for processing of the proper noun translation dictionary creation program 35 stored in the storage unit 34 is expanded.

[0029] The storage unit 34 has a proper noun translation dictionary creation program 35 and a proper noun translation dictionary 340. The proper noun translation dictionary creation program 35 has a structuring unit 310, a translation information selecting unit 320, and a matching unit 330. These units may be implemented as separate programs, or may be realized as a single program.

[0030] The proper noun translation dictionary 340 is a database that holds translation pairs of proper nouns extracted by the proper noun translation dictionary creation program 35 .

[0031] The "translation pair" referred to here does not necessarily have to be a pair of proper nouns from two languages; as mentioned above, it can be a set of corresponding proper nouns from multiple languages, or it can be a set of corresponding proper nouns that are different expressions within the same language.

[0032] The translation pairs held in the proper noun translation dictionary 340 are displayed on the user terminal 20 in a table format as shown in FIG.

[0033] FIG. 2 is a diagram showing an example of a proper noun translation pair screen displayed on the user terminal 20. As shown in FIG.

[0034] 2, proper noun translation pair screen 21 displays a first proper noun in column (A), a second proper noun corresponding to the first proper noun in column (B), and so on, with proper nouns in columns (A) to (E) that correspond to each other. A user can edit this proper noun translation pair screen 21 via user terminal 20.

[0035] Next, we will explain the structuring unit 310, translation information selection unit 320, and matching unit 330. The specific processing performed by each unit will be explained in detail using the processing flowcharts in each embodiment, so the explanation of each unit using Figure 1 will be brief below.

[0036] FIG. 3 is a diagram showing an example of the configuration of the structuring unit 310. As shown in FIG.

[0037] The structuring unit 310 includes a proper noun extraction unit 311 , a related attribute value extraction unit 312 , and a related attribute aggregation unit 313 .

[0038] The proper noun extraction unit 311 extracts proper nouns from each document in the document set input to the proper noun translation dictionary creation program 35 .

[0039] For each proper noun extracted by the proper noun extraction unit 311, the related attribute value extraction unit 312 extracts other proper nouns that co-occur with that proper noun and their labels as related attribute values ​​and related attribute names for each proper noun.

[0040] Co-occurrence means that two or more words appear together within a certain range of a document, such as in the same sentence, paragraph, or document. A label refers to the type of thing or event that each proper noun represents. For example, a word representing the name of a satellite is labeled "satellite," and a word representing a date is labeled "date."

[0041] The related attribute counting unit 313 counts the number of related attribute values ​​that appear for each related attribute name.

[0042] The proper noun extraction unit 311 may use any method for extracting proper nouns and any method for assigning labels to each proper noun.

[0043] The attribute names and their occurrence frequencies tallied by the related attribute tallying unit 313 are used to display the occurrence frequencies of the related attribute names on a screen for selecting related attribute names to be used in matching translation pairs, allowing the user to select related attribute names to be used in matching translation pairs by referring to the occurrence frequencies of the related attribute names.

[0044] Furthermore, the related attribute values ​​and related attribute names extracted by the related attribute value extraction unit 312 are used as translation information for matching translation pairs of each proper noun.

[0045] FIG. 4 is a diagram showing an example of the configuration of the translation information selection unit 320. As shown in FIG.

[0046] The translation information selection unit 320 includes a proper noun selection unit 321 and a related attribute selection unit 322 .

[0047] The proper noun selection unit 321 selects, from the proper nouns extracted by the structuring unit 310, proper nouns to be used for creating translation pairs.

[0048] The related attribute selection unit 322 selects, for each selected label, a related attribute name to be used for matching translation pairs.

[0049] Regarding the translation information selection unit 320, the selection of proper nouns and related attribute names may be performed by the user, or may be determined by the system in some other way.

[0050] FIG. 5 is a diagram illustrating an example of the configuration of the matching unit 330. As shown in FIG.

[0051] The matching unit includes a related attribute value complementing unit 331, a related attribute value translating unit 332, a related attribute value similarity calculating unit 333, a proper noun agreement calculating unit 334, a dictionary registration determining unit 335, and a related attribute importance storing unit 336.

[0052] The related attribute value complementing unit 331 complements the related attribute values ​​of proper nouns that have the same spelling among the proper nouns selected by the translation information selecting unit 320 .

[0053] The related attribute value translation unit 332 translates the related attribute values ​​of the proper nouns whose related attribute values ​​have been supplemented into one of the languages ​​used in the input document.

[0054] The related attribute value similarity calculation unit 333 calculates the similarity between the related attribute values ​​for each related attribute name possessed by each proper noun for all combinations of proper nouns written in different ways.

[0055] The proper noun similarity calculation unit 334 calculates the similarity between proper nouns using the similarity between each related attribute name calculated by the related attribute value similarity calculation unit 333 and the importance of each related attribute held by the related attribute importance storage unit 336.

[0056] The dictionary registration determination unit 335 determines, based on the degree of coincidence calculated by the proper noun coincidence calculation unit 334, whether the translation pair should be registered in the dictionary.

[0057] 6 is a flowchart showing an example of a processing procedure for creating a proper noun translation dictionary. Hereinafter, the processing procedure in the first embodiment performed by the proper noun translation dictionary creating device 30 will be described with reference to FIGS.

[0058] The proper noun extraction unit 311 extracts proper nouns and their labels from a set of input documents (step S601). For example, the proper noun extraction unit 311 performs morphological analysis on the input documents to divide the documents into words, and assigns to each word a label that corresponds to that word.

[0059] For example, proper nouns that fall under the label "SATELLITE" are assigned "B-SATELLITE" or "I-SATELLITE." Words that do not fall under any label are assigned "OTHER."

[0060] Any named entity extraction method can be used to extract proper nouns. For example, a trained machine learning model that extracts proper nouns corresponding to each label may be used, or a rule-based labeling method based on surrounding words or character strings may be used, or labels may be assigned to words or word combinations predefined in a dictionary. Open source software may also be used.

[0061] FIG. 7 is a diagram showing an example of proper noun extraction in structuring unit 310. As shown in FIG.

[0062] Article 70 is an example of a document written in Japanese announcing a satellite launch.

[0063] From document 72 of article 70, the underlined words "Mitsubishi Heavy Industries," "Japan Aerospace Exploration Agency (JAXA)," "January 24, 2017," "Defense Communications Satellite," "Kirameki 2," "H-IIA," and "Tanegashima Space Center" are extracted as proper nouns.

[0064] Furthermore, the labels for these proper nouns are given as "launch company," "launch company," "date," "satellite type," "satellite," "rocket," and "location," respectively.

[0065] Article 73 is an example of a document written in English announcing a satellite launch.

[0066] From document 75 of article 73, the underlined phrases "Japan," "yesterday," "military communication satellite," "DSN-2," "Tanegashima space port," and "H-IIA" are extracted as proper nouns.

[0067] Furthermore, the labels for these proper nouns are given as "country," "date," "satellite type," "satellite," "location," and "rocket," respectively.

[0068] Although "yesterday" has a date label, the specific date is unknown. Therefore, the specific date is inferred from date 74, which is the date article 73 was written. In this example, date 74 is "2017-01-25" and the extracted proper noun is "yesterday," so it is possible to infer that the date represented by "yesterday" is "2027-01-24."

[0069] 6, the related attribute value extraction unit 312 extracts other proper nouns that co-occur with each proper noun extracted in step S601 and their labels as related attribute values ​​and related attribute names for each proper noun. This step will also be described with reference to FIG.

[0070] An example of processing for extracting related attribute values ​​and related attribute names for "Kirameki No. 2" (word in bold) in document 72 to which the "satellite" label has been assigned will be described.

[0071] If we define co-occurrence as "appearing simultaneously in the same sentence," then from document 72, the related attribute values ​​extracted for "Kirameki 2" are "Mitsubishi Heavy Industries," "Japan Aerospace Exploration Agency (JAXA)," "January 24, 2017," "Defense Communications Satellite," "H-IIA," and "Tanegashima Space Center."

[0072] The related attribute names of these related attribute values ​​are "launch company," "launch company," "date," "satellite type," "rocket," and "location," respectively.

[0073] In document 75, an example of processing for extracting related attribute values ​​and related attribute names for "DSN-2" (word in bold), which is also labeled "satellite," is described below. As with the previous processing, if co-occurrence is defined as "appearing simultaneously in the same sentence," then the sentences that co-occur with "DSN-2" are the first sentence, "Japan yesterday..." and "The satellite...". This is because the first word in the second sentence, "The satellite," references "DSN-2." As such, it is desirable to perform coreference analysis to clarify the reference relationships between words within the document before extracting related attribute values. Coreference analysis can be performed using any method, such as open-source software.

[0074] Therefore, from document 75, "Japan," "yesterday," "military communication satellite," "Tanegashima space port," and "H-IIA" are extracted as related attribute values ​​of "DSN-2."

[0075] The related attribute names of these related attribute values ​​are "country," "date," "satellite type," "satellite," "location," and "rocket," respectively.

[0076] 6, the related attribute counting unit 313 counts the frequency of the related attribute names extracted in step S602 for each label of the proper noun extracted in step S601. In the processing example of step S602 described above, when the frequencies of the related attribute names extracted from documents 72 and 75 are counted, for "satellite," "country" appears once, "date" appears twice, "launch company" appears twice, "satellite type" appears twice, "rocket" appears twice, and "location" appears twice.

[0077] The proper noun selection unit 321 selects proper nouns to be used for creating translation pairs from the proper nouns extracted in step S601 (step S604).

[0078] FIG. 8 is a diagram showing an example of a proper noun selection screen in the translation information selection unit 320. As shown in FIG.

[0079] Table 80 is displayed to the user as a selection screen for proper nouns to be translated. Column 81 is a checkbox that accepts the user's selection. Column 82 is the label of the extracted proper noun. Column 83 is the proper noun extracted for each label.

[0080] The user looks at the proper nouns in column 83 and the labels in column 82 and selects the label of the proper noun for which a translation pair is to be created.

[0081] In table 80, the user selects proper nouns to be translated for each label of the proper noun, but the user may also select whether or not to translate each proper noun individually.

[0082] In step S605 of FIG. 6, the related attribute selection unit 322 selects, for each proper noun selected in step S602, a related attribute name to be used for matching of translation pairs.

[0083] 9 is a diagram showing an example of a related attribute selection screen in the translation information selection unit 320. Screen 90 shows the case where "satellite" and "rocket" are selected in step S604 as shown in Table 80, with Table 91 showing a list of attributes extracted for the satellite and Table 92 showing a list of attributes extracted for the rocket.

[0084] Columns 92 and 95 are checkboxes that accept user selections. Columns 93 and 96 are related attribute names. Columns 94 and 97 are the number of times each related attribute name appears. The number of times each related attribute name appears is displayed based on the frequency of the related attribute names tallied by the related attribute tallying unit 313. Alternatively, a threshold may be set for the number of times each related attribute name appears, and only related attribute names that appear more than the threshold may be displayed on the related attribute selection screen.

[0085] The user refers to the number of occurrences of the related attribute name and selects which related attribute name is to be used as information for matching in matching of translation pairs in the matching unit 330, which will be described later.

[0086] The processing of the proper noun selection unit 321 and the related attribute selection unit 322 may be selected by the user as in this embodiment, or the system may automatically select the proper nouns to be translated and their related attribute names based on, for example, frequency information on proper nouns and frequency information on related attribute names.

[0087] In step S606 of FIG. 6, the related attribute value complementing unit 331 complements the related attribute values ​​of proper nouns that have the same spelling among the proper nouns selected in step S604.

[0088] 10 is a diagram showing an example of related attribute value complementation in the matching unit. Tables 11, 12, and 13 each show the related attribute names and their attribute values ​​selected in step S605 for the satellite "Kirameki 2."

[0089] In this invention, these tables are called the structured information of the satellite "Kirameki 2." Each piece of structured information is extracted from multiple documents that contain descriptions of "Kirameki 2," and the types of attribute names and attribute values ​​extracted vary. In this step, these attribute values ​​are complemented to integrate the structured information of "Kirameki 2" into a single table, namely, Table 14.

[0090] The process of complementing related attribute values ​​will be specifically described below using the example of FIG.

[0091] As for the country, it is only extracted in Table 12, so the country is listed as "Japan" in Table 14 as well.

[0092] Regarding dates, only the year is extracted in Table 12, while the date is also extracted in Tables 11 and 13. In this case, the attribute value that has been extracted to a more specific date is the date in Table 14.

[0093] For the rocket, the same attribute value "H-IIA" is extracted in all tables, so this becomes the rocket's attribute value in Table 14 as is.

[0094] The location is extracted as "Tanegashima" in Table 12, and as "Tanegashima Space Center" in Tables 11 and 13. Of these, the "Tanegashima Space Center" that encompasses either of these is used as the location in Table 14.

[0095] In terms of satellite type, "Defense communications satellite" is extracted only in Table 13, so it will be used as the satellite type in Table 14.

[0096] As different manufacturers were extracted in Table 11 ("Mitsubishi Electric") and Table 12 ("NEC"), both "Mitsubishi Electric" and "NEC" were treated as manufacturers in Table 14.

[0097] 6, the related attribute value translation unit 332 translates into English each related attribute value of the structured information whose attribute values ​​have been supplemented in step S606. Note that while an example of translating Japanese related attribute values ​​into English is described here, it is sufficient to translate either one of the related attribute values ​​so that the related attribute values ​​for which similarity is calculated are in the same language. Furthermore, if the related attribute values ​​for which similarity is calculated are in the same language, no translation is necessary.

[0098] Next, the related attribute value similarity calculation unit 333 calculates the similarity between the related attribute values ​​of the structured information translated into English in step S607 (step S608).

[0099] FIG. 11 is a diagram showing an example of structured information in matching unit 330. As shown in FIG.

[0100] The structured information group 100 shows structured information 101 to 103, whose labels extracted from Japanese documents are made up of related attribute names of proper nouns for satellites and their attribute values.

[0101] The structured information group 200 also shows structured information 201 to 203, whose labels extracted from English documents are made up of related attribute names of proper nouns for satellites and their attribute values.

[0102] The matching unit 330 calculates the similarity between attribute values ​​for all combinations of structured information group 100 and structured information group 200 in Fig. 11. That is, in the example of Fig. 11, the similarity between attribute values ​​is calculated for 3 x 3 = 9 combinations of structured information.

[0103] FIG. 12 is a diagram for explaining an outline of matching of structured information in matching unit 330. As shown in FIG.

[0104] In step S607, the structured information 101 extracted in Japanese is translated into structured information 111 in English.

[0105] 12, a process for matching structured information 111 translated into English with English structured information in the structured information group 200 will be described. Open source software can be used for the translation.

[0106] The similarity between each related attribute value is determined by using two types of similarity: similarity between character strings and similarity between feature vectors, depending on the characteristics of the attribute. Below, we will explain in detail how to calculate the similarity between the six attributes "Country," "Date," "Rocket," "Location," "Satellite Type," and "Manufacturer" that are contained in the structured information 111.

[0107] Since the expressions for "country", "rocket", "location", and "manufacturer" are somewhat standardized, we use the similarity between strings. Pi and str Qi The similarity m i can be calculated using the Levenshtein distance, which is the edit distance between two strings, as shown below (Equation 1).

[0108]

number

[0109] For "type," we use the similarity between feature vectors because there are a wide variety of ways to express it. For example, there are various ways to express the same weather observation satellite, such as "Weather satellite" and "Meteorological satellite." Since calculating the similarity between strings is not suitable for such attributes, we convert these attribute values ​​into semantic feature vectors such as Word2Vec or Glove, and then calculate the similarity between the feature vectors.

[0110] The similarity between feature vectors can be calculated by any calculation method such as cosine similarity. Furthermore, the method of converting related attributes into feature vectors can be any vector conversion method other than the above-mentioned methods.

[0111] It is desirable that the "date" be expressed in a normalized form such as "2017-01-04" when the processing by the proper noun extraction unit 311 is completed, and the similarity is set to 1.0 when the dates match exactly. Here, the date may depend on the country in which the article was written. For example, even for the same launch information, there may be a difference in the date depending on whether it is written in Japan time or Universal Time. To take such cases into consideration, it is desirable to set a predetermined similarity greater than 0, such as setting the similarity to 0.5 when the date difference is one day.

[0112] The similarity between each related attribute value calculated here is preferably normalized to have a value between 0 and 1. In addition, for attributes other than the six mentioned above, any similarity calculation method can be used in accordance with the characteristics of the attribute value of each attribute.

[0113] FIG. 13 is a table showing an example of the results of calculation of the similarity between related attribute values ​​in the matching unit 330.

[0114] Table 130 shows the results of calculating the similarity between the six attribute values ​​of structured information 111 and structured information group 200 (structured information 201, structured information 202, and structured information 203) using the above processing method.

[0115] In step S609 of FIG. 6, the proper noun matching calculation unit 334 calculates the similarity between proper nouns using the similarity between each related attribute name calculated in step S607 and the importance of each related attribute held by the related attribute importance storage unit 336.

[0116] The impact of the similarity of attribute values ​​on the match between proper nouns varies depending on the characteristics of each related attribute. Therefore, the importance of related attributes is calculated and matching is performed with weights assigned to each related attribute.

[0117] FIG. 14 is a table showing an example of the importance of related attributes stored in the related attribute importance storage unit 336 in the matching unit 330.

[0118] Table 140 shows the relationship between the number of different related attribute values ​​of the related attribute and the number of different proper nouns in the target label and the importance.

[0119] The related attribute importance storage unit 336 counts the number of different proper nouns in the target label and the number of different related attribute values ​​for named entities extracted in advance from a set of documents of the same type.

[0120] The importance of related attribute A is calculated by dividing the number of different attribute values ​​of related attribute A by the number of different proper nouns in the target label.

[0121] This formula expresses how much the similarity of the attribute values ​​of each related attribute affects the degree of match between structured information, i.e., between proper nouns. For example, when it comes to satellites, since many satellites are launched from the same country, it is not necessarily the case that they refer to the same satellite even if they are from the same country. However, if the dates are the same, there is a high probability that the satellites match, since not many satellites are launched on the same day.

[0122] In this way, by assigning importance according to the characteristics of each related attribute, it is possible to control the influence of the similarity between related attribute values. Using this importance and the similarity between each related attribute value calculated in step S608, the degree of match T between structured information can be calculated using the following (Equation 2).

[0123]

number

[0124] When applying Table 130 and Table 140 to (Equation 2) to calculate the degrees of agreement between structured information 111 and structured information 201, structured information 202, and structured information 203, the results are 0.952, 2.137, and 0.992, respectively.

[0125] In step S610 of FIG. 6, the dictionary registration determination unit 335 determines whether the translation pair should be registered in the dictionary based on the degree of coincidence calculated in step S609.

[0126] If all the related attribute values ​​match, the degree of match is the sum of the importance values, i.e., 2.5. If the degree of match is 80% or more of the sum, i.e., 2.0 or more, the structured information is deemed to match, and the proper noun pair is registered in the dictionary.

[0127] In the example of FIG. 12, based on the degree of coincidence calculated above, it is determined that the structured information 111 and the structured information 202 match, and the pair “Kirameki No. 2” and “DSN-2” are registered in the proper noun translation dictionary 340 .

[0128] The threshold value can be set to any value. Alternatively, the pair of proper nouns with the highest degree of match may be determined as the translation pair.

[0129] As described above, according to the first embodiment, even when no bilingual corpus exists from a document set, matching of translation pairs of proper nouns is performed using the related attribute values ​​of the proper nouns, and a proper noun translation dictionary can be created. [Example]

[0130] In the second embodiment, an example will be described in which, when the matching unit 330 processes proper nouns with multiple labels, the processes of the related attribute value similarity calculation unit 333 and the proper noun agreement calculation unit 334 in the first embodiment are repeatedly performed.

[0131] FIG. 15 is a diagram showing an example of structured information when repeated processing occurs in the matching unit 330.

[0132] Table 150 and Table 151 show the structured information of the satellite "Yaogan 39" and the structured information of the satellite "Long March 2D", respectively. In this embodiment, it is assumed that "satellite" and "rocket" are selected by the translation information selection unit 320.

[0133] As a characteristic, Table 150 includes "Long March 2D" in rockets, and Table 151 includes "Yaokan 39" in satellites. In this way, there are cases where the proper nouns to be translated are related attribute values ​​of different proper nouns to be translated. In this case, for example, when matching among proper nouns of satellites using labels, there is a possibility that the translation of "Long March 2D" rocket will fail, resulting in a failure to match the translation pair. Conversely, when matching among proper nouns of rockets using labels, there is a possibility that the translation of "Yaokan 39" satellite will fail, resulting in a failure to match the translation pair.

[0134] In order to deal with such cases where translation of attribute values ​​fails, if proper nouns of multiple labels are to be translated at the same time, the matching accuracy is improved by repeating the process.

[0135] FIG. 16 is a flowchart showing the procedure of the repetitive processing in the matching unit 330.

[0136] First, the matching unit 330 performs matching for the satellites (step S161). The processing of the matching unit 330 has been explained in steps S606 to S610 of the first embodiment, so a detailed explanation will be omitted.

[0137] Next, when the number of repetitions is one, or when the number of matchings for the satellite is greater than in the previous loop (YES in step S162), the matching unit 330 performs matching for the rocket (step S163).

[0138] If the process has been repeated two or more times and the number of matches is less than in the previous loop (NO in step S162), the process ends.

[0139] Next, when the number of repetitions is 1, or the number of rocket matches is greater than in the previous loop (YES in step S164), the matching unit 330 returns to the process of matching for the satellite in step S161.

[0140] If the process has been repeated two or more times and the number of matches is less than in the previous loop (NO in step S164), the process ends.

[0141] Next, a specific example where the iterative process is effective will be described below. In this description, it is assumed that the Japanese term "Yaogan 39" and the English term "Yaogan 39" are a translation pair for satellites, and that the English term "Chang Zheng 2D" and the English term "Chang Zheng 2D" are a translation pair for rockets. It is assumed that these translation pairs are identified as translation pairs from structured information by the iterative process of the matching unit 330.

[0142] First, after matching the satellites in step S161, when matching between "Yaogan 39" and "Yaogan 39," which are originally a translation pair, the translation result of "Long March 2D" is "Long March 2D," and because the similarity between the strings and "Chang Zheng 2D" is low, the degree of coincidence does not meet the threshold, and matching is not possible.

[0143] Next, in step S163, matching is performed for rockets, and the translation pair "Long March 2D" and "Chang Zheng 2D" are successfully matched. At this point, the number of repetitions is still one, so the process returns to step S161.

[0144] In the second step S161, the satellites are matched using the result that the rockets "Long March 2D" and "Chang Zheng 2D" are a translation pair. In this matching, the degree of agreement between "Yaogan 39" and "Yaogan 39" is greater than in the first processing, and exceeds the threshold, so they can be registered as a translation pair.

[0145] As described above, according to the second embodiment, if the related attribute value of the proper noun to be translated contains a proper noun of another label to be translated, the matching unit 330 can repeatedly perform the processing to more accurately match translation pairs. [Example]

[0146] Next, in the third embodiment, a large-scale language model is used to perform part of the processing of the structuring unit 310 and the matching unit 330. The large-scale language model here may be, for example, a cloud service such as GPT-3.5 or GPT-4, or an open-source model such as Llama2.

[0147] Fig. 17 is a flowchart showing the processing procedure for creating a proper noun translation dictionary in the second embodiment. The configuration diagram of this embodiment is the same as Fig. 1 described in the first embodiment. Furthermore, the processing performed by the translation information selection unit 320 is also the same, so the description of steps S173 and S174 in the flowchart of Fig. 17 will be omitted. The processing procedure for creating a proper noun translation dictionary in the third embodiment will be described below with reference to the flowchart.

[0148] The structuring unit 310 inputs a prompt (instruction) into the large-scale language model and extracts proper nouns, attribute labels, and related proper nouns (step S171).

[0149] FIG. 18 is a diagram showing an example of input and output to a large-scale language model in the structuring unit 310. As shown in FIG.

[0150] Screen 18 displays prompts 180 for input to the large scale language model and output results 181 from the large scale language model.

[0151] Prompt 180 asks for text in an input document to be given, and for proper nouns and their attribute labels, as well as other related proper nouns (related proper nouns), to be extracted from the text in a table format.

[0152] In the output result 181, a table 182 of extracted proper nouns, attribute labels, and related proper nouns is output.

[0153] Here, for each row of Table 182, the proper noun corresponds to the target proper noun, the attribute label corresponds to the label of that proper noun, and the related proper noun corresponds to the related attribute value. Also, since the related proper nouns are extracted in the proper noun column, their attribute labels correspond to the related attribute labels.

[0154] In this way, the processing of the proper noun extraction unit 311 and the related attribute value extraction unit 312 is replaced with processing using a large-scale language model. This is performed for all documents in the input set, and the number of different related attribute values ​​is tallied (step S172), as in the first embodiment. Next, the translation information selection unit 320 selects proper nouns to be translated (step S173) and related attributes to be used in translation (step S174). As described above, this has already been explained in the first embodiment, so further explanation is omitted.

[0155] Next, the matching unit 330 complements the related attribute values ​​between proper nouns with the same spelling among the proper nouns selected in step S173, as in the first embodiment (step S15).

[0156] Next, the matching unit 330 inputs the prompt into the large-scale language model and calculates the degree of agreement between the structured information (step S176).

[0157] FIG. 19 is a diagram showing an example of input and output to a large-scale language model in the matching unit 330.

[0158] Screen 19 displays prompts 190 for input to the large scale language model and output results 191 from the large scale language model.

[0159] In prompt 190, structured information 1901, 1902 of a plurality of proper nouns with related attribute values ​​completed is given, and a score out of 10 is calculated based on the similarity of each attribute value to determine whether they represent the same satellite.

[0160] In this way, structured information in different languages ​​is directly compared without translating related attribute values, and the degree of agreement is output.

[0161] Finally, based on the degree of coincidence output by the large-scale language model, it is determined whether or not to register the word in the proper noun translation dictionary (step S177). This process is the same as the process in step S610 in the first embodiment, and therefore a description thereof will be omitted.

[0162] As described above, according to the third embodiment, by using a large-scale language model for the processing of the structuring unit 310 and the matching unit 330, it is possible to automate the extraction of proper nouns, the translation between attribute values, and the calculation of the degree of similarity of structured information.

[0163] In this embodiment, the processing of the translation information selection unit 320 is premised on the processing in which selection is made on the user screen, as in embodiment 1. However, by prompting the large-scale language model to narrow down to only the attributes necessary for translation, the large-scale language model can also be used for the processing of the translation information selection unit 320.

[0164] The present invention is not limited to the above-described embodiments and includes various modifications. For example, the above-described embodiments have been described in detail to clearly explain the present invention, and the present invention is not necessarily limited to those including all of the described configurations. Furthermore, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, or to add the configuration of another embodiment to the configuration of one embodiment. Furthermore, it is possible to add, delete, or replace part of the configuration of each embodiment with other configurations. [Explanation of symbols]

[0165] 10... document management device, 20... user terminal, 30... proper noun translation dictionary creation device, 31... communication unit, 32... processing unit, 33... memory, 34... storage unit, 35... proper noun translation dictionary creation program, 310... structuring unit, 320... translation information selection unit, 330... matching unit, 340... proper noun translation dictionary

Claims

1. A proper noun translation dictionary creation device that extracts translation pairs of proper nouns from a document collection, comprising: a structuring unit that extracts from the document set a combination of a proper noun, a related attribute value that is another proper noun that co-occurs with the proper noun, and a related attribute name that is a label of the related attribute value; a translation information selection unit that accepts selection of proper nouns to be translated and related attribute names to be used for translation from the proper nouns and related attribute names extracted by the structuring unit; a matching unit that calculates a degree of agreement between the proper nouns selected by the translation information selection unit using related attribute values ​​corresponding to the related attribute names selected by the translation information selection unit; a proper noun translation dictionary that holds translation pairs of proper nouns selected by the translation information selection unit based on the degree of coincidence calculated by the matching unit; A proper noun translation dictionary creation device comprising:

2. 2. The proper noun translation dictionary creation device according to claim 1, The matching unit A proper noun translation dictionary creation device that calculates a degree of agreement between proper nouns selected by the translation information selection unit using a degree of similarity between related attribute values ​​corresponding to related attribute names selected by the translation information selection unit.

3. 3. The proper noun translation dictionary creation device according to claim 2, The matching unit A proper noun translation dictionary creation device that calculates the degree of agreement between proper nouns selected by the translation information selection unit using the similarity of character strings between related attribute values ​​corresponding to the related attribute names selected by the translation information selection unit.

4. 3. The proper noun translation dictionary creation device according to claim 2, The matching unit A proper noun translation dictionary creation device that converts related attribute values ​​corresponding to related attribute names selected by the translation information selection unit into feature vectors and calculates the degree of agreement between the proper nouns selected by the translation information selection unit using the similarity between the feature vectors.

5. 3. The proper noun translation dictionary creation device according to claim 2, The matching unit calculating the importance of the related attribute name based on the number of different proper nouns relative to the number of different related attribute names; A proper noun translation dictionary creation device that calculates the degree of agreement between the proper nouns using the similarity between the related attribute values ​​and the importance of the related attribute names.

6. 3. The proper noun translation dictionary creation device according to claim 2, the structuring unit further includes a related attribute counting unit that counts the frequency of the related attribute name for each label of the proper noun; The translation information selection unit displays the frequency of the related attribute names collected by the related attribute collection unit.

7. 3. The proper noun translation dictionary creation device according to claim 2, The matching unit further includes a related attribute value complementing unit that complements the related attribute values ​​between proper nouns with the same spelling among the proper nouns selected by the translation information selecting unit.

8. 3. The proper noun translation dictionary creation device according to claim 2, The matching unit calculating a degree of agreement between proper nouns that are related attribute values ​​of the proper nouns to be translated; Translating the related attribute value of the proper noun to be translated using the calculated degree of coincidence; The proper noun translation dictionary creation device calculates the degree of agreement between the proper nouns to be translated using the degree of similarity between the translated related attribute values.

9. 3. The proper noun translation dictionary creation device according to claim 2, The structured portion is A proper noun translation dictionary creation device that extracts combinations of the proper nouns, the related attribute values, and the related attribute names from the document set using a large-scale language model.

10. 3. The proper noun translation dictionary creation device according to claim 2, The matching unit A proper noun translation dictionary creation device that uses a large-scale language model to input related attribute values ​​corresponding to related attribute names selected by the translation information selection unit and calculates the degree of agreement between the proper nouns selected by the translation information selection unit.

11. A proper noun translation dictionary creation method for a proper noun translation dictionary creation device that extracts translation pairs of proper nouns from a document collection, comprising: extracting from the document set a combination of a proper noun, a related attribute value which is another proper noun that co-occurs with the proper noun, and a related attribute name which is a label for the related attribute value; Accepting a selection of a proper noun to be translated and a related attribute name to be used for translation from the extracted proper nouns and related attribute names; calculating a degree of agreement between the selected proper nouns using related attribute values ​​corresponding to the selected related attribute names; A proper noun translation dictionary creating method for storing translation pairs of the selected proper nouns based on the calculated degree of coincidence.

Images