Information processing device and information processing method

Abstract

To reduce the burden in making up a dictionary for conversion into easy Japanese.SOLUTION: An information processing apparatus 100 of the present invention comprises: a collecting section 1 that collects predetermined sentence data; a text mining section 2 that morpheme-analyzes collected sentence data, dependency-analyzes collected sentence data on the basis of a result of the morpheme analysis, creates a graph based on a dependency structure identified by dependency analysis, and generates a cluster by clustering the graph created; a summarizing section 3 that primarily summarizes sentence data including a first feature word belonging to a first cluster, one of clusters generated, from the collected sentence data and extracts a second feature word appearing a predetermined value or greater of the appearing number of times from the primarily-summarized sentence data and secondarily summarizes sentence data including the extracted second feature word; a creating section 4 that creates a conversion dictionary using the secondarily-summarized sentence data; and a learning section 5 that makes a natural-language processing model learn a created conversion dictionary.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing apparatus and an information processing method.

Background Art

[0002] In recent years, there have been many demands for clearly explaining sentences written in Japanese. Specifically, there are demands such as converting ordinary Japanese sentences into easy-to-understand Japanese for foreigners and children, converting Kansai dialect into standard language, and translating sentences with many special medical terms used. To meet these demands, there has been active development of technologies for converting difficult Japanese into easy (understandable) Japanese.

[0003] For example, Patent Document 1 discloses a sentence simplification system that provides a document written in plain Japanese more easily than before. This sentence simplification system displays each first option in order to select an easy-to-understand method for making a difficult-to-understand phrase easy to understand using an easy phrase for each difficult-to-understand phrase, displays each second option of a plurality of second methods in order to select a pre-related processing method related to the phrase that appears immediately before the difficult-to-understand phrase, and displays each third option of a plurality of third methods in order to select a post-related processing method related to the phrase that appears immediately after the difficult-to-understand phrase. For each difficult-to-understand phrase, it memorizes the selected items and the specified easy phrases, etc. Then, it simplifies the original sentence written in Japanese based on the selected items.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The invention described in Patent Document 1 can be described as an invention for converting to simplified Japanese, which involves analyzing text using program logic and performing the conversion process within that program. On the other hand, there are also known inventions for converting to simplified Japanese that use machine learning with a natural language processing model. Since sentences with the same meaning can be written in various ways, converting to simplified Japanese using a natural language processing model requires learning a vast number of conversion patterns (conversion dictionaries). However, there is a problem in that creating the dictionaries to be learned requires a great deal of time and effort.

[0006] In view of these circumstances, the present invention aims to reduce the burden of creating dictionaries for converting Japanese into simplified Japanese. [Means for solving the problem]

[0007] The present invention solves the aforementioned problems, A collection unit that collects specified text data, A text mining unit that performs morphological analysis on the collected text data, performs dependency analysis on the collected text data based on the results of the morphological analysis, creates a graph based on the dependency structure identified by the dependency analysis, and generates clusters by clustering the created graph, Belonging to the first cluster, which is one of the clusters generated as described above. Using the first characteristic word, An aggregation unit that first aggregates text data containing a first characteristic word from the collected text data, extracts a second characteristic word that appears more than a predetermined number of times from the first aggregated text data, and second aggregates text data containing the extracted second characteristic word. A creation unit that creates a conversion dictionary using the secondary aggregated text data, The information processing device includes a learning unit that trains a natural language processing model with the conversion dictionary created above.

[0008] Furthermore, the present invention is Information processing device, The steps include: collecting specified text data, The steps include: performing morphological analysis on the collected text data; performing dependency analysis on the collected text data based on the results of the morphological analysis; creating a graph based on the dependency structure identified by the dependency analysis; and generating clusters by clustering the created graph. Belonging to the first cluster, which is one of the clusters generated as described above. Using the first characteristic word, The process involves first aggregating text data containing a first characteristic word from the collected text data, extracting a second characteristic word that appears more than a predetermined number of times from the first aggregated text data, and second aggregating text data containing the extracted second characteristic word. The steps include creating a conversion dictionary using the secondary aggregated text data, This information processing method comprises the steps of: training a natural language processing model with the conversion dictionary created above. [Effects of the Invention]

[0009] According to the present invention, the burden of creating dictionaries for converting to simplified Japanese can be reduced. [Brief explanation of the drawing]

[0010] [Figure 1] This is a functional configuration diagram of the information processing device in this embodiment. [Figure 2] This is a flowchart of the preparation process in this embodiment. [Figure 3] This is a flowchart of the conversion process in this embodiment. [Modes for carrying out the invention]

[0011] [composition] Figure 1 is a functional configuration diagram of the information processing device in this embodiment. The information processing device 100 is a computer that converts complex Japanese text into simplified Japanese. The information processing device 100 includes hardware such as an input unit, an output unit, a control unit, and a storage unit. For example, if the control unit is composed of a CPU (Central Processing Unit), the information processing by the computer including the control unit is realized by program execution processing by the CPU. The storage unit included in the computer stores various programs to realize the functions of the computer according to the CPU's instructions. This realizes cooperation between software and hardware. The programs can be provided by recording them on a recording medium or via a network. The output unit may also include the function of a display unit that displays on a screen.

[0012] The information processing device 100 comprises a collection unit 1, a text mining unit 2, an aggregation unit 3, a creation unit 4, a learning unit 5, a forced replacement unit 6, and a conversion unit 7. The information processing device 100 also stores a document DB 11, a conversion dictionary DB 12, a forced replacement dictionary DB 13, and a natural language processing model 14. "DB" stands for Database.

[0013] Collection unit 1 collects predetermined text data. This predetermined text data is valuable to users of the service that converts text to simplified Japanese, and is not limited to, for example, text data about the user or text data about the field to which the user belongs. Collection unit 1 can collect such text data from, for example, websites in related fields or books in related fields, according to the search criteria entered by the user.

[0014] The text mining unit 2 performs text mining on predetermined text data. The text mining performed by the text mining unit 2 is disclosed in Japanese Patent Publication No. 2019-117484 (hereinafter referred to as "Reference 1") and is well known, so a detailed explanation will be omitted.

[0015] Based on the results of text mining, the aggregation unit 3 aggregates, among the collected text data, the text data that is beneficial to the user. The creation unit 4 creates a conversion dictionary using the aggregated text data. The conversion dictionary is a dictionary that summarizes the conversion patterns when converting text data. The learning unit 5 causes the created conversion dictionary to be machine-learned by the natural language processing model 14. The forced replacement unit 6 replaces the word specified by the user with another word desired by the user. The replacement by the forced replacement unit 6 takes precedence over the conversion based on the natural language processing model 14. Also, the forced replacement unit 6 creates the forced replacement dictionary described later. The conversion unit 7 inputs the target text data into the natural language processing model 14 and converts it.

[0016] The text DB 11 is a database that stores the text data collected by the collection unit 1 in a predetermined format. The conversion dictionary DB 12 is a database that stores the conversion dictionary that summarizes (stores in association) the set of the text data aggregated by the aggregation unit 3 and the text data after conversion of the aggregated text data. The forced replacement dictionary DB 13 is a database that stores the forced replacement dictionary that summarizes the set of the word specified by the user and the word replaced by the forced replacement unit 6 for the word. The natural language processing model 14 is a machine learning model for enabling a computer to understand words that humans use daily and for processing natural language, which is the language of humans, by a machine. The natural language processing model 14 can adopt well-known ones, and detailed descriptions are omitted.

[0017] (Details of the processing of the collection unit 1) The collection unit 1 can divide the description content into text units by providing delimiters such as full stops, line breaks, parentheses, etc. for the description content of the homepage or book, and collect text data. Also, the collection unit 1 can assign a unique ID (identifier, Identifier) to each collected text data. As a result, the collection unit 1 can create text list data that summarizes the set of the ID and the text data.

[0018] (Details of the processing in Text Mining Unit 2) Using the invention described in Reference 1, the text mining unit 2 performs text mining on all the text data collected by the collection unit 1, that is, all the text data in the text list data. Specifically, the text mining unit 2 performs morphological analysis on each of the collected text data. The text mining unit 2 also performs dependency analysis on each of the collected text data based on the results of the morphological analysis. The text mining unit 2 also creates a graph for each text data based on the dependency structure identified by the dependency analysis. The text mining unit 2 then merges the graphs created for each text data to create a single graph. The text mining unit 2 also performs clustering of the merged graph. As a result, one or more clusters are generated for the merged graph. The text mining unit 2 can also extract one or more feature words from each of the generated clusters. Feature words are words that appear with a term frequency (TF) of a predetermined value or more. The predetermined value may be determined for each cluster, for example, or it may be determined to be the same value uniformly for all clusters.

[0019] The text mining unit 2 can set a category for each generated cluster. The category for each cluster symbolizes one of several sub-concepts indicated by the content of the collected text data, and can be set by the operator of the information processing device 100. The operator may, for example, set the category by inferring it from the characteristic words included in each cluster, or by inferring it from words other than the characteristic words included in each cluster, or by inferring it from words other than the words included in each cluster.

[0020] (Details of the processing in aggregation unit 3) The aggregation unit 3 uses a characteristic word (first characteristic word) belonging to one of the clusters (first cluster) generated by text mining to aggregate text data containing that characteristic word from all text data collected by the collection unit 1 (primary aggregation). The primary aggregated text data can be said to be text data that focuses on a specific categorized concept and is limited to a more specific field. The aggregation unit 3 can perform primary aggregation as many times as there are categories, i.e., clusters.

[0021] Furthermore, the aggregation unit 3 can identify words contained in the primary aggregated text data and aggregate the frequency of occurrence of each identified word, as well as the (multiple) IDs assigned to the text data containing the identified word, for each word. The aggregation unit 3 can create word list data that summarizes pairs of words, their frequency of occurrence, and the IDs of the text data containing that word. Word list data can be prepared for each category.

[0022] Furthermore, the aggregation unit 3 refers to the word list data, extracts characteristic words (second characteristic words) that satisfy predetermined conditions from the primary aggregated text data, and aggregates the text data containing the extracted characteristic words (secondary aggregation). The predetermined conditions can be, for example, "appears with a number of occurrences equal to or greater than a predetermined value," but are not limited to this. Also, the above "predetermined value" can be set to any numerical value. When the condition "appears with a number of occurrences equal to or greater than a predetermined value" is used, the aggregation unit 3 can select text data containing frequently used characteristic words from the primary aggregated text data through secondary aggregation.

[0023] (Details of the processing in creation section 4) First, the creation unit 4 compares the secondary aggregated text data with the word list data created by the aggregation unit 3 and extracts text data with matching IDs. Here, the word list data used for matching can be, for example, a word list data of words belonging to one or more categories arbitrarily selected by the operator after prior verification of their validity. The creation unit 4 can also exclude text data with an extremely small number of characters or an extremely large number of characters from the extracted text data.

[0024] Next, the creation unit 4 outputs sentence data containing the target word, using the ID as the key, in order of the most frequently occurring word, up to a predetermined number (e.g., 10 per word) from the sentence data extracted in the previous step. Here, the creation unit 4 outputs sentence data up to the limit, according to a predetermined upper limit on the number of output items. Next, the creation unit 4 uses the output sentence data as the pre-conversion sentence data and creates a pre-conversion sentence list data by summarizing the pre-conversion sentence data.

[0025] Meanwhile, the creation unit 4 extracts words from the word list data created by the aggregation unit 3 that have an occurrence frequency of a predetermined value or more. At this time, the categories related to the word list data from which words are to be extracted can be, for example, one or more types of categories arbitrarily selected by the operator after confirming their validity in advance, but are not limited to these. The extracted words are set as the pre-conversion words. The operator sets the words obtained by converting the pre-conversion words into simplified Japanese (by manual means) as the post-conversion words and inputs them into the information processing device 100. Next, the creation unit 4 creates a word dictionary that combines the pairs of pre-conversion words and post-conversion words. At this time, it is preferable to predetermine the word conversion rules to prevent fluctuations in the conversion.

[0026] Next, the creation unit 4 creates a conversion dictionary using the pre-conversion text list data and the word dictionary created in the previous step. Specifically, if a word contained in the text data in the pre-conversion text list data is a pre-conversion word in the word dictionary, the creation unit 4 converts that word to a post-conversion word. As a result, post-conversion text data containing the post-conversion word is created for each of the pre-conversion text data. The creation unit 4 then creates a conversion dictionary by combining the pairs of pre-conversion text data and post-conversion text data.

[0027] The creation unit 4 allows the operator to manually modify parts of the pre- and post-conversion sentence data (which may be parts of the pre- or post-conversion sentence data) that are not converted words. Multiple types of such modifications can be performed. These modifications are useful because they provide multiple versions of sentences with the same meaning but different wording, resulting in Japanese sentences that are not only easy to understand but also easy to read (natural-sounding) for a machine-based word conversion. Modifying parts of the sentence other than the converted words improves the conversion dictionary and thus the accuracy of the conversion. Note that modifying parts of the sentence other than the converted words is optional.

[0028] [process] The processes performed by the information processing device 100 of this embodiment will now be described. Figure 2 is a flowchart of the preparation process in this embodiment. Figure 3 is a flowchart of the conversion process in this embodiment.

[0029] (Preparation process) The preparation process proceeds as follows: First, the collection unit 1 collects predetermined text data that is valuable to the user according to the search conditions provided by the user (Step A1). Next, the collection unit 1 performs text processing to assign IDs to the collected text data (Step A2). Text processing creates a text list data. Next, the text mining unit 2 performs text mining on all the text data in the text list data (Step A3). Next, the aggregation unit 3 performs primary aggregation of the text data based on the results of the text mining (Step A4). Next, the aggregation unit 3 performs secondary aggregation of the text data based on the results of the primary aggregation (Step A5). Next, the creation unit 4 creates a conversion dictionary using the secondary aggregated text data (Step A6). Next, the learning unit 5 trains the created conversion dictionary on the natural language processing model 14 (Step A7). Finally, the forced replacement unit 6 creates a forced replacement dictionary that combines pairs of words specified by the user and words that the forced replacement unit 6 replaces with those words (Step A8).

[0030] This completes the preparation process. Note that the creation of the forced replacement dictionary in step A8 can be performed at any time, not just after step A7.

[0031] (Conversion process) The conversion process proceeds as follows. The conversion process begins when the user inputs the text data to be converted into the information processing device 100. First, the forced replacement unit 6 performs forced replacement on the input text data (step B1). Based on the forced replacement dictionary stored in the forced replacement dictionary DB 13, the forced replacement unit 6 extracts a word (first word) that the user wants to convert from the input text data and converts it to another word (second word). Next, the conversion unit 7 inputs the forcibly replaced text data into the natural language processing model 14 and converts it (step B2). Finally, the forced replacement unit 6 performs forced replacement on the converted text data (step B3). Based on the forced replacement dictionary, the forced replacement unit 6 extracts a word (first word) that the user wants to convert from the converted text data and converts it to another word (second word). As a result, the information processing device 100 can output text data written in simplified Japanese.

[0032] This completes the conversion process. Note that the forced replacement in steps B1 and B3 is optional and can be omitted. Furthermore, by performing forced replacement (step B1) before the conversion by the natural language processing model 14 (step B2), it is possible to reliably reflect the user's desired custom word replacements in the converted text data.

[0033] According to this embodiment, the words to be converted can be limited to words categorized into a specific field through primary aggregation, and further limited to words that meet conditions such as high frequency of occurrence through secondary aggregation. Therefore, the number of words to be registered in the conversion dictionary can be reduced, and the number of conversion patterns to be learned for conversion to simplified Japanese can be reduced. On the other hand, the limited words are those desired by the user and can provide useful text data for the user. Therefore, the burden of creating a dictionary for conversion to simplified Japanese can be reduced. Furthermore, since the system can independently force-replace user-specified words with desired words within the text data to be converted, it can convert the text data to match the user's preferences, taking precedence over conversions performed by the natural language processing model 14.

[0034] [Specific example] (1) The conversion unit 7 can convert the sentence "It was a difficult mission, but I managed to complete it" to "It was a difficult job, but I completed it." The difficult words "difficult" and "mission" are converted to the simpler words "difficult" and "job," respectively. Also, the adverb "somehow" can be removed as its absence does not fundamentally change the meaning, thus converting the sentence to a simpler one. (2) The conversion unit 7 can convert the sentence "Please be sure to bring your seal" to the sentence "Please be sure to bring your seal." The difficult words "seal" and "bring" are converted to the simpler words "seal" and "bring," respectively. (3) The conversion unit 7 can convert the sentence "We decline inquiries regarding the holding of the regular general meeting" to "We decline inquiries related to holding the regular general meeting." The difficult words "hold" and "decline" are converted to the simpler words "hold" and "decline," respectively. In addition, by maintaining the conversion dictionary created by the creation unit 4, the parts of the sentence other than the converted words can be made easier to read (changes in the wording before and after when converting nouns to verbs, changes from polite language to standard language). (4): The conversion unit 7 can convert the sentence "When coming to the venue, please use public transportation." into the sentence "When coming to the venue, please use the bus or train." Difficult words such as "come," "public transportation," and "use" are converted into simpler words such as "come," "bus or train," and "use," respectively.

[0035] [Differentiation] (a) While this invention describes the case of converting difficult Japanese into easy Japanese, it can also be applied to other languages. Furthermore, this invention can also be applied to translation into different types of languages.

[0036] (b) It is also possible to realize technologies that appropriately combine the various technologies described in this embodiment. (c): The software described in this embodiment can be implemented as hardware, and the hardware can be implemented as software. (d) In addition, hardware, software, flowcharts, etc. can be modified as appropriate without departing from the spirit of the present invention. [Explanation of Symbols]

[0037] 100 Information Processing Devices 1. Collection Department 2. Text Mining Section 3. Aggregation Section 4. Creation Section 5. Learning Department 6. Forced replacement section 7 Conversion Unit 11 Text DB 12 Conversion Dictionary DB 13 Forced Replacement Dictionary Database 14 Natural Language Processing Models

Claims

1. A collection unit that collects specified text data, A text mining unit that performs morphological analysis on the collected text data, performs dependency analysis on the collected text data based on the results of the morphological analysis, creates a graph based on the dependency structure identified by the dependency analysis, and generates clusters by clustering the created graph, An aggregation unit that, using a first characteristic word belonging to a first cluster, which is one of the generated clusters, performs primary aggregation of text data containing the first characteristic word from the collected text data, extracts a second characteristic word that appears more than a predetermined number of times from the primary aggregated text data, and performs secondary aggregation of text data containing the extracted second characteristic word, A creation unit that creates a conversion dictionary using the secondary aggregated text data, An information processing device comprising: a learning unit that trains a natural language processing model on the conversion dictionary created above.

2. The information processing apparatus according to claim 1, further comprising a conversion unit that inputs target text data into the natural language processing model and converts it.

3. The information processing apparatus according to claim 2, further comprising a forced replacement unit that replaces the first word with a pre-specified second word if the target text data or the converted target text data contains the first word.

4. Information processing device, The steps include: collecting specified text data, The steps include: performing morphological analysis on the collected text data; performing dependency analysis on the collected text data based on the results of the morphological analysis; creating a graph based on the dependency structure identified by the dependency analysis; and generating clusters by clustering the created graph. The steps include: first aggregating text data containing the first characteristic word from the collected text data using the first characteristic word belonging to the first cluster, which is one of the generated clusters; second aggregating text data containing the extracted second characteristic word from the primary aggregated text data, and second aggregating the text data containing the extracted second characteristic word; The steps include creating a conversion dictionary using the secondary aggregated text data, An information processing method comprising the step of training a natural language processing model with the conversion dictionary created above.

Images